Making quick-look flare spectrograms and movies: Difference between revisions

From EOVSA Wiki
Jump to navigation Jump to search
(Created page with "This page documents instructions for EOVSA Scientists-on-Duty (SoD) to create quicklook flare spectrograms and movies as part of their daily routines. == Producing EOVSA quick-look flare spectrograms == === Step 1: Checking Possible flares === Verify the possible flares on the daily EOVSA Solar Dynamic Spectrogram, for example: http://ovsa.njit.edu/browser/?suntoday_date=2024-05-07 none|thumb|center|500px| In this example, we see...")
 
 
(5 intermediate revisions by the same user not shown)
Line 1: Line 1:
This page documents instructions for EOVSA Scientists-on-Duty (SoD) to create quicklook flare spectrograms and movies as part of their daily routines.  
This page documents instructions for EOVSA Scientists-on-Duty (SoD) to create quicklook flare spectrograms and movies as part of their daily routines.  
== Prerequisites ==
Login into the pipeline machine with your account:
<pre style="font-family:courier">ssh -X <your_user_name>@pipeline</pre>
If your default shell is not bash, enter bash by
<pre style="font-family:courier">bash</pre>
=== Configuring Access to the Interim Database (IDB) ===
To process and calibrate EOVSA raw "Interim" Database (IDB) data, access to the SQL database containing the calibration data is required. Perform the following steps to configure access:
Obtain Database Credentials: Contact Bin Chen to request the <username>, <account_name>, and <password> for database access.
Create a ".netrc" File: Create a ".netrc" file in your home directory ("$HOME") with the following contents, replacing "<username>," "<account_name>," and "<password>" with the actual database credentials:
<pre>machine eovsa-db0.cgb0fabhwkos.us-west-2.rds.amazonaws.com
          login <username>
          account <account_name>
          password <password></pre>
Secure the ".netrc" File: To ensure that the file is only accessible by you, set its permissions to only allow owner read/write:
<pre>chmod 600 ~/.netrc</pre>
=== Set up the Python Environment ===
If the following is not already in your ~/.bashrc file, do the following
<pre style="font-family:courier">alias loadpyenv3.8='source /home/user/.setenv_pyenv38'
export EOVSADBJSON=/common/python/current/EOVSADB.json </pre>
Load the Python 3.8 environment
<pre style="font-family:courier">loadpyenv3.8
ipython --pylab</pre>


== Producing EOVSA quick-look flare spectrograms ==
== Producing EOVSA quick-look flare spectrograms ==
Line 25: Line 63:


[[File:FLM20240507.png|none|thumb|center|500px|]]
[[File:FLM20240507.png|none|thumb|center|500px|]]
=== Step 2: Set up the environment === 


Login into the pipeline machine with your account:
=== Step 2: Analyze the flare ===
 
<pre style="font-family:courier">ssh -X [your_user_name]@pipeline</pre>
 
If your default shell is not bash, enter bash by
<pre style="font-family:courier">bash</pre>
 
If the following is not already in your ~/.bashrc file, do the following
<pre style="font-family:courier">alias loadpyenv3.8='source /home/user/.setenv_pyenv38'
export EOVSADBJSON=/common/python/current/EOVSADB.json </pre>


Enter a working directory. We have limited space under /home, so it is better to go to your directory under /data1/.
Enter a working directory. We have limited space under /home, so it is better to go to your directory under /data1/.
Line 43: Line 70:
<pre style="font-family:courier">cd /data1/<your_user_name>/</pre>
<pre style="font-family:courier">cd /data1/<your_user_name>/</pre>


Load the Python 3.8 environment
Obtain IDB files by providing a time range that encloses the flare. This step will also perform various calibration steps (absolute flux, attenuator gain, and feed rotation), so it may take a while (a few minutes per file). During the course, it will also ask you to confirm the files to be processed. Each IDB file is supposed to be 10-minute long. The naming convention is "IDByyyymmddhhmmdd," with the time indicating the start time of the file.
 
<pre style="font-family:courier">loadpyenv3.8
ipython --pylab</pre>


In Python, enter the following:
In Python, enter the following:
Line 53: Line 77:
from eovsapy.util import Time </pre>  
from eovsapy.util import Time </pre>  


=== Step 3: Analyze the flare === 
<pre style="font-family:courier">files = fs.calIDB(Time(['2024-05-07 16:20','2024-05-07 16:35'])) </pre>
 
Obtain IDB files by providing a time range that encloses the flare. This step will also perform various calibration steps (absolute flux, attenuator gain, and feed rotation), so it may take a long while (~hour depending on your flare duration). During the course, it will also ask you to confirm the files to be processed. Each IDB file is supposed to be 10-minute long. The naming convention is "IDByyyymmddhhmmdd," with the time indicating the start time of the file.


<pre style="font-family:courier">The timerange corresponds to these files (will take about 8 minutes to process)
<pre style="font-family:courier">The timerange corresponds to these files (will take about 8 minutes to process)
Line 61: Line 83:
/data1/eovsa/fits/IDB/20240507/IDB20240507163024
/data1/eovsa/fits/IDB/20240507/IDB20240507163024
Do you want to continue? (say no if you want to adjust timerange) [y/n]?y </pre>
Do you want to continue? (say no if you want to adjust timerange) [y/n]?y </pre>
<pre style="font-family:courier">files = fs.calIDB(Time(['2024-05-07 16:20','2024-05-07 16:35'])) </pre>


The previous step, if successful, will produce a list of IDB files under the current directory
The previous step, if successful, will produce a list of IDB files under the current directory
Line 69: Line 90:
Inspect the spectrograms and see if all the antennas look okay.  
Inspect the spectrograms and see if all the antennas look okay.  


<pre style="font-family:courier">out, spec = fs.inspect(files) </pre>
<pre style="font-family:courier">out, spec_tp, spec_xp, time_axis, freq_axis = fs.inspect(files) </pre>


But, if one or more antennas are known to be working, enter the list of the good ones. In this example, we have antennas 7 and 10 that were being serviced at the time. So let us exclude them using the "ant_str" parameter. Note the antenna names go from 1 to 13.
Two figures will be produced. One two-panel figure shows the median total-power spectrogram across all antennas and a cross-power spectrogram across a set of baselines defined by "uvrange" (default to 45 to 300 m).  


<pre style="font-family:courier"> out, spec = fs.inspect(files, ant_str='ant1-6 ant8-9 ant11-13') </pre>


[[File:Out_spec.png|none|thumb|center|500px|]]
Another shows total-power spectrograms for all 16 antennas. (Note the antenna names go from 1 to 16. Ant 14 is the 27-m for calibrations, and Ants 15 and 16 are not yet connected to the system, but will be available soon.)


To better see the flare, you can change the spec vmax as:
[[File:Specs_quicklook.jpg|none|thumb|center|500px|]]
[[File:Tpspec_all.jpg|none|thumb|center|500px|]]


<pre style="font-family:courier"> imshow(spec,vmax=30,vmin=-1) </pre>
To fine-tune the color normalization, you can directly call the plt_quicklook_specs() function by providing spec_tp, spec_xp, and out:


[[File:Out_spec2.png|none|thumb|center|500px|]]
<pre style="font-family:courier"> fs.plt_quicklook_specs(spec_tp, spec_xp, out=out, vmin_xp=1., vmax_xp=100., vmin_tp=10., vmax_tp=600.) </pre>


Use the figure above to choose the background interval (bgidx), the maximum intensity and the frequencies.  
Check the [https://ovsa.njit.edu/wiki/index.php/Expanded_Owens_Valley_Solar_Array#EOVSA_Observing_Log EOVSA Observing Log] for antennas that were not working. We should also check the total-power spectrograms to see if there are any anomalies. In this example, except for Ant 10, all others look okay. However, the observing log says Ant 7 has issues with the Y polarization. Let us exclude both of them using the "ant_str" parameter. Also, adjust the normalization for the spectrograms using "vmin_tp" and "vmax_tp" for total power and "vmin_xp" and "vmax_xp" for cross power.


The tpk (Time of the peak) determines the name of the resulting files (.png and .fits), and also determines the flare_id.
<pre style="font-family:courier"> out, spec_tp, spec_xp = fs.inspect(files, ant_str='ant1-6 ant8-9 ant11-13', vmin_xp=1., vmax_xp=100., vmin_tp=10., vmax_tp=600.) </pre>


It's better to keep the formate of tpk as tpk='yyyy-mm-dd hh:mm:00' and add the flare time on wiki in the format of "hh:mm":  
Use the figure above to 1) choose the background interval (bgidx), 2) identify the flare peak time, 3) select frequencies for plotting. The box at the top-right of the plot shows the time index/time string, frequency index/frequency string, flux for total power and cross power at the location of your cursor. Move your cursor around and write the information down. The parameter "tpk" flare peak time determines the name of the resulting files (.png and .fits) and the flare_id. In this example, I found the peak time is very close to 2024-05-07T16:29:01. See the snapshot below.
 
[[File:Specs_quicklook_adj.jpg|none|thumb|center|500px|]]


<pre style="font-family:courier"> f, ax0, ax1 = fs.make_plot(out,bgidx=[200,210],vmin=0.1, vmax=110, lcfreqs=[120,190,270,350],ant_str='ant1-6 ant8-9 ant11-13', tpk='2024-05-07 16:30:00') </pre>
<pre style="font-family:courier"> f, ax0, ax1 = fs.make_plot(out,bgidx=[200,210],vmin=0.1, vmax=110, lcfreqs=[120,190,270,350],ant_str='ant1-6 ant8-9 ant11-13', tpk='2024-05-07 16:30:00') </pre>

Latest revision as of 13:02, 13 October 2024

This page documents instructions for EOVSA Scientists-on-Duty (SoD) to create quicklook flare spectrograms and movies as part of their daily routines.

Prerequisites

Login into the pipeline machine with your account: 

ssh -X <your_user_name>@pipeline

If your default shell is not bash, enter bash by 

bash

Configuring Access to the Interim Database (IDB)

To process and calibrate EOVSA raw "Interim" Database (IDB) data, access to the SQL database containing the calibration data is required. Perform the following steps to configure access:

Obtain Database Credentials: Contact Bin Chen to request the <username>, <account_name>, and <password> for database access.

Create a ".netrc" File: Create a ".netrc" file in your home directory ("$HOME") with the following contents, replacing "<username>," "<account_name>," and "<password>" with the actual database credentials: 

machine eovsa-db0.cgb0fabhwkos.us-west-2.rds.amazonaws.com
           login <username>
           account <account_name>
           password <password>

Secure the ".netrc" File: To ensure that the file is only accessible by you, set its permissions to only allow owner read/write: 

chmod 600 ~/.netrc

Set up the Python Environment

If the following is not already in your ~/.bashrc file, do the following 

alias loadpyenv3.8='source /home/user/.setenv_pyenv38'
export EOVSADBJSON=/common/python/current/EOVSADB.json

Load the Python 3.8 environment 

loadpyenv3.8
ipython --pylab


Producing EOVSA quick-look flare spectrograms

Step 1: Checking Possible flares

Verify the possible flares on the daily EOVSA Solar Dynamic Spectrogram, for example:

http://ovsa.njit.edu/browser/?suntoday_date=2024-05-07

Daily spec 20240507.png

In this example, we see a possible flare that happened around 16:30 UT, which appears as a bright vertical stripe in the daily cross-power dynamic spectrum.

For better visualization and flare time precision, check the higher-resolution dynamic spectra at:

http://ovsa.njit.edu/flaremon/. "XSPYYYYMMDDHHMMSS.png" are the file names you are looking for.

XSP20240507153014.png

Since 2024-May-05, real-time flare detection figures and list are also available at:

http://ovsa.njit.edu/flaremon/FLM20240507.png http://ovsa.njit.edu/flaremon/flarelist/flarelist_2024-05-07.txt

FLM20240507.png

Step 2: Analyze the flare

Enter a working directory. We have limited space under /home, so it is better to go to your directory under /data1/. 

cd /data1/<your_user_name>/

Obtain IDB files by providing a time range that encloses the flare. This step will also perform various calibration steps (absolute flux, attenuator gain, and feed rotation), so it may take a while (a few minutes per file). During the course, it will also ask you to confirm the files to be processed. Each IDB file is supposed to be 10-minute long. The naming convention is "IDByyyymmddhhmmdd," with the time indicating the start time of the file.

In Python, enter the following: 

from eovsapy import flare_spec as fs 
from eovsapy.util import Time 
files = fs.calIDB(Time(['2024-05-07 16:20','2024-05-07 16:35'])) 
The timerange corresponds to these files (will take about 8 minutes to process)
/data1/eovsa/fits/IDB/20240507/IDB20240507162024
/data1/eovsa/fits/IDB/20240507/IDB20240507163024
Do you want to continue? (say no if you want to adjust timerange) [y/n]?y

The previous step, if successful, will produce a list of IDB files under the current directory 

In [21]: print(files)
['./IDB20240507162024', './IDB20240507163024']

Inspect the spectrograms and see if all the antennas look okay. 

out, spec_tp, spec_xp, time_axis, freq_axis = fs.inspect(files)

Two figures will be produced. One two-panel figure shows the median total-power spectrogram across all antennas and a cross-power spectrogram across a set of baselines defined by "uvrange" (default to 45 to 300 m).


Another shows total-power spectrograms for all 16 antennas. (Note the antenna names go from 1 to 16. Ant 14 is the 27-m for calibrations, and Ants 15 and 16 are not yet connected to the system, but will be available soon.)

Specs quicklook.jpg
Tpspec all.jpg

To fine-tune the color normalization, you can directly call the plt_quicklook_specs() function by providing spec_tp, spec_xp, and out: 

 fs.plt_quicklook_specs(spec_tp, spec_xp, out=out, vmin_xp=1., vmax_xp=100., vmin_tp=10., vmax_tp=600.)

Check the EOVSA Observing Log for antennas that were not working. We should also check the total-power spectrograms to see if there are any anomalies. In this example, except for Ant 10, all others look okay. However, the observing log says Ant 7 has issues with the Y polarization. Let us exclude both of them using the "ant_str" parameter. Also, adjust the normalization for the spectrograms using "vmin_tp" and "vmax_tp" for total power and "vmin_xp" and "vmax_xp" for cross power. 

 out, spec_tp, spec_xp = fs.inspect(files, ant_str='ant1-6 ant8-9 ant11-13', vmin_xp=1., vmax_xp=100., vmin_tp=10., vmax_tp=600.)

Use the figure above to 1) choose the background interval (bgidx), 2) identify the flare peak time, 3) select frequencies for plotting. The box at the top-right of the plot shows the time index/time string, frequency index/frequency string, flux for total power and cross power at the location of your cursor. Move your cursor around and write the information down. The parameter "tpk" flare peak time determines the name of the resulting files (.png and .fits) and the flare_id. In this example, I found the peak time is very close to 2024-05-07T16:29:01. See the snapshot below.

Specs quicklook adj.jpg 


 f, ax0, ax1 = fs.make_plot(out,bgidx=[200,210],vmin=0.1, vmax=110, lcfreqs=[120,190,270,350],ant_str='ant1-6 ant8-9 ant11-13', tpk='2024-05-07 16:30:00')
Eovsa.spec.flare id 20240507163000.png

A second background interval can be defined right after the first one, e.g., bg2idx=[1000,1010]

In this example, the output files are eovsa.spec.flare_id_20240507163000.fits and eovsa.spec.flare_id_20240507163000.fits.

At the end, copy the .fits file to /common/webplots/events/2024: 

 cp eovsa.spec.flare_id_20240507163000.fits /common/webplots/events/2024

and include the flare in the wiki Flare List:

http://ovsa.njit.edu/wiki/index.php/2024

The flare position have been copied from the https://solarmonitor.org/

Retrieved from "http://ovsa.njit.edu//wiki/index.php?title=Making_quick-look_flare_spectrograms_and_movies&oldid=10657"