EOVSA Data Analysis Tutorial 2022: Difference between revisions

From EOVSA Wiki
Jump to navigation Jump to search
 
(187 intermediate revisions by 2 users not shown)
Line 1: Line 1:
=Introduction to EOVSA and Datasets=
=1. Introduction to EOVSA and Datasets=
[[file:Eovsa1.png|thumb|right|500px|EOVSA]]
[[file:Eovsa1.png|center|500px|EOVSA]]
EOVSA (Expanded Owens Valley Solar Array) is a solar-dedicated radio interferometer operated by the New Jersey Institute of Technology. EOVSA observes the full disk of the Sun at all times when the Sun is >10 degrees above the local horizon, which is season dependent and ranges from 7-12 hours duration centered on 20 UT. Like any radio interferometer, the fundamental measurement for imaging is the correlated amplitude and phase between each pair of antennas, which is called a “complex visibility.” EOVSA’s 13 antennas form 78 such visibilities at any frequency and instant of time, i.e. 78 measurements of the spatial Fourier transform of the solar brightness distribution. EOVSA records these visibilities at 451 science frequency channels each second, in four polarization products (XX,YY,XY,YX), as well as additional total flux measurements from each individual antenna. These data are then processed through a pipeline processing system (the block diagram in the Figure 1 shows the data flow in the pipeline). One of the outputs of the pipeline is a visibility database in a widely used open-standard format called a CASA measurement set (or “ms”; CASA is the Common Astronomy Software Applications package used by many modern interferometer arrays).
EOVSA (Expanded Owens Valley Solar Array) is a solar-dedicated radio interferometer operated by the New Jersey Institute of Technology. EOVSA observes the full disk of the Sun at all times when the Sun is >10 degrees above the local horizon, which is season dependent and ranges from 7-12 hours duration centered on 20 UT. Like any radio interferometer, the fundamental measurement for imaging is the correlated amplitude and phase between each pair of antennas, which is called a “complex visibility.” EOVSA’s 13 antennas form 78 such visibilities at any frequency and instant of time, i.e. 78 measurements of the spatial Fourier transform of the solar brightness distribution. EOVSA records these visibilities at 451 science frequency channels each second, in four polarization products (XX, YY, XY, YX), as well as additional total flux measurements from each individual antenna. These data are then processed through a pipeline processing system whose data flow is shown in the block diagram (Figure 1). One of the outputs of the pipeline is a visibility database in a widely used open-standard format called a CASA measurement set (or “ms”; CASA is the Common Astronomy Software Applications package used by many modern interferometer arrays).


We deliver the radio interferometry data on the following three levels:
EOVSA delivers the radio interferometry data on the following three levels:
[[file:EOVSA_data_levels.PNG|thumb|right|500px|Pipeline block diagram]]
[[file:EOVSA_data_levels.PNG|thumb|right|500px|Figure 1: Pipeline block diagram]]
Level 0 - Raw visibility data from the instrument. This includes observations of cosmic sources for phase calibration, and gain and pointing observations required for total power calibration.
* '''Level 0''' - Raw visibility data - This includes observations of cosmic sources for phase calibration, and gain and pointing observations required for total power calibration.
Level 1 - Calibrated visibility data. After applying calibration and other preliminary processing to level 0 data, we create the calibrated visibility data in CASA ms format (second column in Figure 1). These visibility data have all of the required content to produce Level 2 images and spectrogram data in standard FITS format. The following tutorial will guide you through how to make EOVSA images and spectrogram from visibility data. We provide a set of standard ms’s for each day (red solid boxes in Figure 1) for users who wish to start with visibility data. You can retrieve EOVSA 1-min averaged visibility data in CASA ms format from this page. Full-resolution EOVSA visilibity data in CASA ms format will be provide per request. Please contact the EOVSA team if you wish to have Level 1 visibility data for a specific event.
* '''Level 1''' - Calibrated visibility data - After applying calibration and other preliminary processing to level 0 data, we create the calibrated visibility data in CASA ms format (second column in Figure 1). These visibility data have all of the required content to produce Level 2 images and spectrogram data in standard FITS format. The following tutorial will guide you through how to make EOVSA images and spectrogram from visibility data. We provide a set of standard ms’s for each day (pink solid boxes in Figure 1) for users who wish to start with visibility data. You can retrieve EOVSA 1-min averaged visibility data in CASA ms format from this [http://www.ovsa.njit.edu/fits/UDBms_slfcaled/ page]. Full-resolution EOVSA visibility data in CASA ms format will be provided per request. Please contact the *EOVSA team if you wish to have Level 1 visibility data for a specific event.
Level 2 - Images and spectrogram data in standard FITS format. Most users, however, will prefer to work with spectrogram (frequency-time) and image data, which are also outputs of the pipeline system shown in Figure 1 (orange boxes). Spectrograms are provided as standard FITS tables containing the frequency list, list of times, and data in both total power and a sum of amplitudes over intermediate-length baselines (cross power). Likewise, image data products are in FITS format with standard keywords and are converted into the Helioprojective Cartesian coordinate system compatible with the World Coordinate System (WCS) convention, along with correct registration for the spatial, spectral, and temporal coordinates. Both the spectrogram and image data products are calibrated properly and have physical radio intensity units (sfu for spectrograms and brightness temperature for radio images).
* '''Level 2''' - Images and spectrogram data in standard FITS format - Most users, however, will prefer to work with spectrogram (frequency-time) and image data, which are also outputs of the pipeline system shown in Figure 1 (orange boxes). Spectrograms are provided as standard FITS tables containing the frequency list, list of times, and data in both total power and a sum of amplitudes over intermediate-length baselines (cross power). Likewise, image data products are in FITS format with standard keywords and are converted into the Helioprojective Cartesian coordinate system compatible with the World Coordinate System (WCS) convention, along with correct registration for the spatial, spectral, and temporal coordinates. Both the spectrogram and image data products are calibrated and have physical radio intensity units (sfu for spectrograms and brightness temperature for radio images).


=Browsing and Obtaining EOVSA data=
=2. Browsing and Obtaining EOVSA data=
[[file:EOVSA_Browser.PNG|thumb|right|500px|EOVSA Browser]]
[[file:EOVSA_Browser.PNG|thumb|right|500px|Figure 2: EOVSA Browser]]
[[file:fig-browser.png|thumb|right|500px|Browsing EOVSA data in RHESSI Browser]]
[[file:RHESSI_browser.png|thumb|right|500px|Figure 3: RHESSI Browser]]


Currently the most convenient way for browsing EOVSA data is through the [http://sprg.ssl.berkeley.edu/~tohban/browser/ RHESSI Browser]. First, check the "EOVSA Radio Data" box on the data selection area (top-left corner). Then select year/month/date to view the overall EOVSA dynamic spectrum. Note if a time is selected at early UTC hours (e.g., 0-3 UT), it will show the EOVSA dynamic spectrum from the previous day. Also note that EOVSA data were not commissioned for spectroscopic imaging prior to April 2017. An example of the overview EOVSA dynamic spectrum for 2017 Aug 21 is shown in the figure on the right.
====EOVSA Browser====
The most convenient way for browsing '''Level 2''' EOVSA data is through the [http://ovsa.njit.edu/browser EOVSA Browser]. The overview EOVSA dynamic spectra on the top-left panel are from the median of the uncalibrated cross-power visibilities at a few short baselines, which are not (but a good proxy of) the total-power dynamic spectra. The effects of spatial information blended in the cross-power visibilities can be clearly seen as the "U"-shaped features throughout the day, which are due to the movement of the Sun across the sky that effectively changes the length and orientation of the baselines. Flare emission can be seen in the dynamic spectra, which usually appears as vertical bright features across many frequency bands. The pipeline quicklook full-disk images are also displayed for the given frequencies along with multi-wavelength full-disk maps such as SDO AIA, HMI, BBSO H-alpha by hovering on the buttons on the bottom of each map..  More information can be found on this [http://ovsa.njit.edu/data-browsing.html page]. The figure on the right shows an example of the overview EOVSA dynamic spectrum for 2021 May 07.


The overview EOVSA dynamic spectra are from the median of the <b>uncalibrated</b> cross-power visibilities at a few short baselines, which are not (but a good proxy of) the total-power dynamic spectra. The effects of spatial information blended in the cross-power visibilities can be clearly seen as the "U"-shaped features throughout the day, which are due to the movement of the Sun across the sky that effectively changes the length and orientation of the baselines. Flare times can be easily seen in the EOVSA dynamic spectra, which usually appear as vertical bright features across many frequency bands. More information can be found on [http://ovsa.njit.edu/data-browsing.html this page].
====RHESSI Browser====
EOVSA data can also be browsed through [http://sprg.ssl.berkeley.edu/~tohban/browser/ RHESSI Browser]. Check the "EOVSA Radio Data" box on the data selection area (top-left corner). Then select year/month/date to view the overall EOVSA dynamic spectrum. Note if a time is selected at early UTC hours (e.g., 0-3 UT), it will show the EOVSA dynamic spectrum from the previous day. Also, note that EOVSA data were not commissioned for spectroscopic imaging prior to April 2017.  


We are currently working on a pipeline to create quicklook full-disk images for implementation at the RHESSI Browser in a way similar as the RHESSI quicklook images. Please stay tuned.
===Level 0 (raw visibility) Data===
Once you identify the flare time, you can find the full-resolution (1-s cadence) uncalibrated visibility files (in Miriad format) at [http://www.ovsa.njit.edu/fits/IDB/ this link]. Each data file is usually 10 minutes in duration. The name convention is YYYYMMDD (folder name) /IDBYYYYMMDDHHMMSS (file name), where the time in the file name indicates the start time of the visibility data.  However, to be useful for imaging, these files must be first calibrated and then converted to a CASA measurement set.  At present, this can only be done with software at the OVRO site although we are working on software to allow this processing to be done elsewhere. For now, please request this processing to be done for you by emailing Dale Gary, Bin Chen, Sijie Yu, or others you may know in the solar radio group.


Once you identify the flare time, you can find the full-resolution (1-s cadence) uncalibrated visibility files (in Miriad format) at [http://www.ovsa.njit.edu/fits/IDB/ this link]. Each data file is usually 10 minutes in duration. Name convention is YYYYMMDD/IDBYYYYMMDDHHMMSS, where the time in the file name indicates the start time of the visibility data.
===Level 1 (calibrated visibility) Data===
The images you see in the browser are generated by an automated pipeline that also produces calibrated CASA ms datasets, but at 1-min integration.  If a 1-min cadence is sufficient for your purposes, you may wish to download the calibrated data and perform your own imaging.  The 1-min cadence calibrated data are at [http://www.ovsa.njit.edu/fits/UDBms_slfcaled/ this link]. You should be aware that a model disk has been subtracted from the visibilities in these files, which is what you want if you are interested in imaging active regions or flares. It is possible to add the model back, but this is not available (yet) in standard software, so please contact someone in the NJIT solar radio group for help with that.


=Software requirements and installation=  
===Other useful links===
EOVSA visibility data processing: SunCASA A Python wrapper around CASA for synthesis imaging and visualizing solar spectral imaging data. CASA is one of the leading software tools for "supporting the data post-processing needs of the next generation of radio astronomical telescopes such as ALMA and VLA", an international effort led by the National Radio Astronomy Observatory. The current version of CASA uses Python (3.6) interface. More information about CASA can be found on NRAO's CASA website.
NOTE: CASA is available ONLY on UNIX-BASED PLATFORMS, and therefore, so is SunCASA.


EOVSA image data processing: GSFIT A IDL-widget(GUI)-based spectral fitting package called gsfit, which provides a user-friendly display of EOVSA image cubes and an interface to fast fitting codes (via platform-dependent shared-object libraries).
For this tutorial, we will demonstrate using SunCASA to create EOVSA image and spectrogram from the visibility data observed during a M-class flare on 2021 May 7. There are two approaches in accessing the SunCASA package:


[convenient] Through this notebook, together with Google Colaboratory (colab) which hosts this notebook on free virtual environment that requires no setup and runs entirely in the cloud. If you are into this option, skip the following Installation section and go directly to the Tour Section.
* [https://join.slack.com/t/eovsatutorial-2021/shared_invite/zt-sob838f4-zvs_qH3M4yTbWGlPIiw6og EOVSA User Forum]
* [http://www.ovsa.njit.edu/wiki/index.php/Recent_Flare_List_(2021) EOVSA Flare list]
* [http://ovsa.njit.edu/browser EOVSA browser]
* [http://www.ovsa.njit.edu/wiki/index.php/Expanded_Owens_Valley_Solar_Array EOVSA wiki]
* [http://www.ovsa.njit.edu/fits/UDBms_slfcaled/ EOVSA 1-min averaged CASA ms database]
* [https://github.com/suncasa/suncasa-src SunCASA on Github] and [https://pypi.org/project/suncasa/ PyPI]


=Download Sample EOVSA Data for the Tutorial=  
=3. Software requirements and installation=  
=Information on the EOVSA Visibility Data (CASA Measurement Set)=
EOVSA data processing and analysis are developed over two packages:
With the pip installation, the CASA modules may be used in a standard Pythonic manner. For example, CASA tasks can be invoked using import, while CASA tools are Python classes that first need to be instantiated to create usable objects. A useful first task to run is listobs, which provides a summary of the measurement set.
* '''[https://github.com/suncasa/suncasa SunCASA]''' A Python wrapper around [https://casa.nrao.edu/ CASA (the Common Astronomy Software Applications package)] for synthesis imaging and visualizing solar spectral imaging data. CASA is one of the leading software tools for "supporting the data post-processing needs of the next generation of radio astronomical telescopes such as ALMA and VLA", an international effort led by the [https://public.nrao.edu/ National Radio Astronomy Observatory]. The current version of CASA uses Python (3.6) interface. More information about CASA can be found on [https://casa.nrao.edu/ NRAO's CASA website ]. Note, CASA is available ONLY on UNIX-BASED PLATFORMS (and therefore, so is SunCASA).  


=Cross-Power Dynamic Spectrum=
* '''[https://github.com/Gelu-Nita/GSFIT GSFIT]''' A IDL-widget (GUI)-based spectral fitting package called GSFIT, which provides a user-friendly display of EOVSA image cubes and an interface to fast-fitting codes (via platform-dependent shared-object libraries).
 
For this tutorial, we will demonstrate using SunCASA to create EOVSA image and spectrogram from the visibility data observed for an M class flare on 2021 May 07. There are two approaches to accessing the SunCASA package:
 
# Through [https://colab.research.google.com/ Google Colaboratory (colab)] which hosts a free virtual environment that requires no setup and runs entirely in the cloud. For example, the [https://colab.research.google.com/drive/19NQb6Emb9HvKX4QHq9ZYCP3RM6nT7sDL#scrollTo=cLdDVptBGG-X EOVSA workspace] of this tutorial explains the analysis setup.
# Install on your own machine, and run the notebook as a regular Jupyter Notebook. See [http://ovsa.njit.edu//wiki/index.php/EOVSA_Data_Analysis_Tutorial_2022#Installation_of_SunCASA_(optional) SunCASA Installation] section below, also [http://www.ovsa.njit.edu/wiki/index.php/SunCASA_Installation this page] and [http://www.ovsa.njit.edu/wiki/index.php/GSFIT_Installation GSFIT Installation] for instructions.
 
=4. Download Sample EOVSA Data for the Tutorial=
 
For this tutorial, we use an M4 class flare on 07 May 2021, around 19:00 UT occurred near the solar east limb as viewed from Earth, and was well observed by Solar Orbiter (including the STIX instrument). For other recent EOVSA events, check here.
Here, we provide a calibrated and self-calibrated EOVSA visibility dataset in CASA ms format (IDB20210507_1840-1920XXYY.cal.10s.slfcaled.ms) which is ready for science analyses purposes,
 
=5. Information on the EOVSA Visibility Data (CASA Measurement Set)=
 
With the pip installation, the CASA modules may be used in a standard Python manner. For example, CASA tasks can be invoked using import, while CASA tools are Python classes that first need to be instantiated to create usable objects. A useful first task to run is [https://casa.nrao.edu/docs/taskref/listobs-task.html listobs], which provides a summary of the measurement set.
 
<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
## in SunCASA
from casatasks import listobs
import os
msfile = 'IDB20210507_1840-1920XXYY.cal.10s.slfcaled.ms'
rc = listobs(vis=msfile, listfile = msfile + '.listobs', overwrite=True)
 
print(os.popen("cat " + msfile + ".listobs").read())
</pre>
[[file:Listobs_20210507.png|thumb|center|1400px|Figure 4: Output of listobs task]]
 
=6. Self-calibration (optional)=
Self-calibration is often needed in bringing out details of the flare at multiple frequencies. An example script, run in SunCASA, for doing self-calibration of the 2017 Aug 21 flare at ~20:20 UT can be found at [http://www.ovsa.njit.edu/wiki/index.php/Tohban_EOVSA_Imaging_Tutorial_A-Z here]. The EOVSA workspace discussed along with this tutorial anyway uses the self-calibrated dataset for analysis.
 
=7. Cross-Power Dynamic Spectrum=
The first module we introduce is [https://github.com/suncasa/suncasa/blob/main/suncasa/utils/dspec.py''dspec'']. This module allows you to generate a cross-power dynamic spectrum from an MS file, and visualize it. You can select a subset of data by specifying a [https://casa.nrao.edu/Release3.3.0/docs/UserMan/UserMansu112.html time range], [https://casaguides.nrao.edu/index.php/Selecting_Spectral_Windows_and_Channels spectral windows/channels],  [https://casaguides.nrao.edu/index.php/Antenna/Baseline_Selection_Syntax_with_or_without_Autocorrelations antenna baseline], or [https://casa.nrao.edu/Release3.3.0/docs/UserMan/UserMansu113.html uvrange]. The selection syntax follows the ''CASA'' convention. More information of CASA selection syntax may be found in the above links or the [https://casa.nrao.edu/casadocs/casa-5.4.0/data-selection/data-selection-in-a-measurementset Measurement Selection Syntax].
The first module we introduce is [https://github.com/suncasa/suncasa/blob/main/suncasa/utils/dspec.py''dspec'']. This module allows you to generate a cross-power dynamic spectrum from an MS file, and visualize it. You can select a subset of data by specifying a [https://casa.nrao.edu/Release3.3.0/docs/UserMan/UserMansu112.html time range], [https://casaguides.nrao.edu/index.php/Selecting_Spectral_Windows_and_Channels spectral windows/channels],  [https://casaguides.nrao.edu/index.php/Antenna/Baseline_Selection_Syntax_with_or_without_Autocorrelations antenna baseline], or [https://casa.nrao.edu/Release3.3.0/docs/UserMan/UserMansu113.html uvrange]. The selection syntax follows the ''CASA'' convention. More information of CASA selection syntax may be found in the above links or the [https://casa.nrao.edu/casadocs/casa-5.4.0/data-selection/data-selection-in-a-measurementset Measurement Selection Syntax].
[[file:fig-dspec.png|thumb|right|300px|Figure 1: EOVSA cross power dynamic spectrum at stokes XX and YY]]
 
[[file:Dynspec_20210507.png|thumb|right|300px|Figure 4: EOVSA cross power dynamic spectrum at stokes XX polarization]]
 
<pre style="background-color: #FCEBD9;">
<pre style="background-color: #FCEBD9;">
from suncasa.utils import dspec as ds
## in SunCASA
import matplotlib.pyplot as plt
from suncasa import dspec as ds
          ## define the visbility data file
import time
msfile = 'IDB20220330_concat_slfcaled__backsub.ms'
           ## define the output filename of the dynamic spectrum  
           ## define the output filename of the dynamic spectrum  
specfile = msfile + '.dspec.npz'
specfile = msfile + '.dspec.npz'
           ## select relatively short baselines within a length (here I use 0.15~0.5km),
           ## The example below shows the cross-power spectrogram from a baseline selected using the parameter "bl".
           ## and take a median cross all of them (with the domedian parameter)
           ## bl = '4&9' means selecting a baseline from Antenna ID 4 (Antenna Name "eo05") correlating with Antenna ID 9
           ## alternatively, you can use the "bl" parameter to select individual baseline(s)
          ## (Antenna Name "eo10") - refer listobs output.
uvrange = '0.15~0.5km'
           ## you can also use the "bl" parameter to select multiple baseline(s), i.e., bl='0&2;4&9;8&11'.
          ## this step generates a dynamic spectrum and saves it to specfile
            
dspec=ds.get_dspec(vis=msfile, specfile=specfile, uvrange=uvrange, domedian=True)
           ## Hover over "Dspec" in the next line to see additional arguments such as frequency range ("spw"), and time range ("timeran")  
           ## dspec is a Python dictionary that contains the resulting dynamic spectrum.
           ## or use help(ds.Dspec)
           ## A copy is saved in "specfile" as a numpy npz file.
           ## this step generates a dynamic spectrum and saves it to specfile as a numpy .npz file
          ## Other optional parameters are available for more selection criteria
d = ds.Dspec(msfile, bl='4&9', specfile=specfile)
          ## such as frequency range ("spw"), and time range ("timeran")
d.plot(vmin=None, vmax=None, pol='XX')
           ## Use "ds.get_dspec?" to see more options
print(d.data.shape) # The shape gives the dimensions of polarization, baselines, frequencies, time       
dspec['spec'].shape
</pre>
           ## (1, 1, 451, 1798) One polarization, 451 frequencies and 1798 times
dspec.keys()
          ## ['tim', 'pol', 'uvrange', 'bl', 'timeran', 'freq', 'spec', 'spw']


          ## The following command displays the resulting cross-power dynamic spectrum
Alternative methods of using the "dspec" task are discussed in the EOVSA workspace.
          ## Use ds.plt_dspec? to check more plotting options
 
ds.plt_dspec(specfile)  #alternatively the next step can also be used
NOTE: If you are using your own machine, the plotting should be in interactive mode. In that mode, hovering your mouse over the dynamic spectrum allows you to read the time, frequency, and flux information under the cursor at the bottom of the window.
ds.plt_dspec(dspec)
 
         
=8. Quick-Look Imaging=
</pre>
We use CASA's CLEAN algorithm for EOVSA imaging. As the default coordinate system is equatorial, solar coordinate transformation and image registration need to be done to place the image into the usual Helioprojective Cartesian Coordinates (X- and Y-axes are Solar X and Solar Y in arcsecond unit, respectively). We have bundled a number of these steps into a module named  [https://github.com/suncasa/suncasa/blob/master/utils/qlookplot.py ''qlookplot''], allowing users to generate an observing summary plot showing the cross power dynamic spectrum, GOES light curves and EOVSA quick-look images.
Now you should have a popup window showing the dynamic spectrum. Hover your mouse over the dynamic spectrum, you can read the time and frequency information at the bottom of the window.
 
Now, let us start with making a summary plot of the EOVSA image at the spectral windows 2 to 5 (2.4–3.7 GHz).
 
[[file:EOVSAimg_20210507.png|thumb|right|400px|Figure 5: EOVSA full-Sun single-band quicklook image]]


=Quick-Look Imaging=
<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
Imaging EOVSA data involves image cleaning, as well as solar coordinate transformation and image registration. We bundled a number of these steps ino a module named [https://github.com/suncasa/suncasa/blob/master/utils/qlookplot.py ''qlookplot''], allowing users to generate an observing summary plot showing the cross power dynamic spectrum, GOES light curves and EOVSA quick-look images. Now let us start with making a summary plot of EOVSA image at the spectral window 5 (5.4 GHz).
[[file:fig-qlookplot0.png|thumb|right|300px|EOVSA full-Sun single-band quicklook image]]
<pre style="background-color: #FCEBD9">
## in SunCASA
## in SunCASA
from suncasa.utils import qlookplot as ql
from suncasa.utils import qlookplot as ql
msfile = 'IDB20170821201800-202300.4s.slfcaled.ms'
          ## (Optional) Supply the npz file of the dynamic spectrum from previous step.  
## (Optional) Supply the npz file of the dynamic spectrum from previous step.  
          ## If not provided, the program will generate a new one from the visibility.
## If not provided, the program will generate a new one from the visibility.
         
specfile = msfile + '.dspec.npz'
timerange = '19:02:00~19:02:10' ## set the time interval          
## set the time interval
spw = '2~5'                     ## select frequency range        
timerange = '20:21:10~20:21:18'
stokes = 'XX'                   ## select stokes XX          
## select frequency range from 5 GHz to 6 GHz
plotaia = False                ## Initially, turn off AIA image plotting, default is True
spw = '5~6GHz'
## select stokes XX
stokes = 'XX' 
## turn off AIA image plotting, default is True
plotaia = False


ql.qlookplot(vis=msfile, specfile=specfile, timerange=timerange, spw=spw, \
ql.qlookplot(vis=msfile, specfile=specfile, timerange=timerange, spw=spw, \
     stokes=stokes, plotaia=plotaia)
     stokes=stokes, plotaia=plotaia, restoringbeam=['60arcsec'], robust=0.5)
</pre>
</pre>


The empty panels are there as we only selected one polarization for imaging/spectroscopy. Feel free to explore by adjusting the above parameters, e.g. use a different time range ("timerange" parameter) and/or frequency range ("spw" parameter).  
The empty panels are there as only one polarization is selected for imaging/spectroscopy.  
 
Feel free while working in [https://colab.research.google.com/drive/19NQb6Emb9HvKX4QHq9ZYCP3RM6nT7sDL#scrollTo=cLdDVptBGG-X EOVSA Workspace] to explore by adjusting the above parameters, e.g. use a different time range ("timerange" parameter) and/or frequency range ("spw" parameter).  
 
====Quick-look Imaging with AIA data====
 
With [https://github.com/suncasa/suncasa/blob/master/utils/qlookplot.py ''qlookplot''], it is easy to engage solar data from SDO/AIA in the summary plot. Setting ''plotaia=True'' in the [https://github.com/suncasa/suncasa/blob/master/utils/qlookplot.py ''qlookplot''] command will download SDO/AIA data at the given time to current directory and add it to the summary plot.
 
[[file:EOVSA_AIA_20210507.png|thumb|right|400px|Figure 6: EOVSA full-Sun single-band quicklook image with SDO/AIA as background]]


With [https://github.com/suncasa/suncasa/blob/master/utils/qlookplot.py ''qlookplot''], it is easy to engage solar data from SDO/AIA in the summary plot.  Setting ''plotaia=True'' in the [https://github.com/suncasa/suncasa/blob/master/utils/qlookplot.py ''qlookplot''] command will download SDO/AIA data at the given time to current directory and add it to the summary plot.
[[file:fig-qlookplot1.png|thumb|right|300px|EOVSA full-Sun single-band quicklook image with SDO/AIA as background]]
<pre style="background-color: #FCEBD9">
<pre style="background-color: #FCEBD9">
## in SunCASA
## in SunCASA
outfits = 'EOVSA_20210507T190205.000000.outim.image.fits'
ql.qlookplot(vis=msfile, specfile=specfile, timerange=timerange, spw=spw, \
ql.qlookplot(vis=msfile, specfile=specfile, timerange=timerange, spw=spw, \
     stokes=stokes, plotaia=True)
     stokes=stokes, plotaia=True)
</pre>
</pre>
The resulting radio image is a 4-D datacube (in solar X-pos, Y-pos, frequency, and polarization), which is, by default, saved as a fits file ''msfile + '.outim.image.fits''' under your working directory. The name of the output fits file can be specified using the "outfits" parameter. In this example, we combine all selected frequencies (specified in keyword "spw") into one image (i.e., multi-frequency synthesis). Therefore the third axis only has one plane. The fourth axis contains polarization. In this example, however, only the first one is populated ("XX"). Here is the relevant information in the header of the resulting FITS file.
 
<pre>                      
The resulting radio image is a 4-D datacube (in solar X-pos, Y-pos, frequency, and polarization), which is, by default, saved as a fits file Instrument_yymmddTHHMMS.ffffff.outim.image.fits under your working directory. An alternate name for the output fits file can be specified using the outfits parameter. In this example, we combine all selected frequencies (specified in keyword spw) into one image (i.e., multi-frequency synthesis). Therefore the third axis only has one plane. The fourth axis contains polarization. In this example, the fourth axis only has one plane as well (XX). Here is the relevant information (first few lines) in the header of the resulting FITS file.
NAXIS  =                    4                                                
 
NAXIS1  =                  512/ Nx
<pre>
NAXIS2  =                  512/ Ny                                            
In [1]: from astropy.io import fits
NAXIS3  =                    1/ number of frequency                                         
In [2]: hdu = fits.open('EOVSA_20210507T190230.000000.outim.image.fits')[0]
NAXIS4  =                    2/ number of polarization
In [3]: hdu.header
Out[3]:
SIMPLE  =                    T / conforms to FITS standard
BITPIX  =                  -64 / array data type
NAXIS  =                    4 / number of array dimensions
NAXIS1  =                  512 / Nx
NAXIS2  =                  512 / Ny
NAXIS3  =                    1 / number of frequencies
NAXIS4  =                    1 / number of polarizations
</pre>
</pre>
By default, [https://github.com/suncasa/suncasa/blob/master/utils/qlookplot.py ''qlookplot''] produces a full sun radio image (512x512 with a pixel size of 5"). If you know where the radio source is (e.g., from the previous full-Sun imaging), you can make a partial solar image around the source by specifying the image center ("xycen"), pixel scale ("cell"), and image field of view ("fov"). Here we show an example that images a 8-s interval around 20:21:14 UT using multi-frequency synthesis in 12-14 GHz and a smaller restoring beam. The microwave source is show to bifurcate into two components, which correspond pretty well with the double flare ribbons in SDO/AIA.  
 
[[file:fig-qlookplot3.png|thumb|right|300px|EOVSA multi-frequency synthesis quicklook image]]
By default, qlookplot produces a full sun radio image (512x512 with a pixel size of 5"). If you know where the radio source is (e.g., from the previous full-Sun imaging), you can make a partial solar image around the source by specifying the image center (xycen), pixel scale (cell), and image field of view (fov). Here we show an example that images for each spectral window in this data set (from spw 0 to 49) at the same time interval around 19:02 UT. At high frequencies, the microwave source concentrates near the flare site, corresponding pretty well with the flare kernel in SDO/AIA. At low frequencies, the microwave source extends to higher altitudes in the corona. Again, feel free to change some of these parameters to explore what they do.
 
====Quick-look Imaging with AIA data and all spw====
Next, we will make images for every single spectral window in this data set (from spw 0 to 47).  
 
[[file:EOVSA_AIA_allspw_20210507.png|thumb|right|400px|Figure 7: EOVSA multi-band quicklook image]]
 
<pre style="background-color: #FCEBD9">
<pre style="background-color: #FCEBD9">
## in SunCASA
## in SunCASA
xycen = [375, 45] ## image center for clean in solar X-Y in arcsec
from suncasa.utils.mstools import time2filename
cell=['2.0arcsec'] ## pixel scale
timerange = '19:02:00~19:02:10'  ## set the time interval
imsize=[128]   ## number of pixels in X and Y. If only one value is provided, NX = NY
spw = ['{}'.format(l) for l in range(48)].    ## select (almost) all spectral windows from spw id #0 to #47
fov = [100,100] ## field of view of the zoomed-in panels in unit of arcsec
outfits = time2filename(visibility_data,timerange=timerange)+'.outim.image.allbd.fits'
ql.qlookplot(vis=msfile, specfile=specfile, timerange='20:21:10~20:21:18', \
 
              spw='12GHz~14GHz', stokes='XX', restoringbeam=['6arcsec'],\
stokes = 'XX'.            ## select stokes XX
              imsize=imsize,cell=cell,xycen=xycen,fov=fov,calpha=1.0)
xycen = [-900, 280]       ## image center for clean in solar X-Y in arcsec
</pre>
cell=['2.0arcsec']       ## pixel scale
imsize=[128]             ## number of pixels in X and Y. If only one value is provided, NX = NY
fov = [300,300]           ## field of view of the zoomed-in panels in unit of arcsec
 
plotaia = True            ## turn off AIA image plotting, default is True
aiawave = 1600            ## AIA passband in Å. The options are [171,131,304,335,211,193,94,1600,1700]
acmap = 'gray_r'          ## Choose the coloar map for AIA images. If not provided, the program will use default AIA colormap.


Next, we will make images for every single spectral window in this data set (from spw 1 to 30, spw 0 is in the visibility data, but is not calibrated for this event).
ql.qlookplot(vis=visibility_data, specfile=specfile, timerange=timerange,  
[[file:fig-qlookplot_mbds.png|thumb|right|300px|EOVSA multi-band quicklook image]]
            spw=spw, stokes=stokes, plotaia=plotaia, aiawave=aiawave,
<pre style="background-color: #FCEBD9">
            restoringbeam=['60arcsec'], robust = 0.5, acmap=acmap,
## in SunCASA
            imsize=imsize,cell=cell,xycen=xycen,fov=fov,  
xycen = [375, 45]  ## image center for clean in solar X-Y in arcsec
            outfits=outfits,overwrite=False)
cell=['2.0arcsec'] ## pixel size
imsize=[128]  ## x and y image size in pixels.
fov = [100,100]  ## field of view of the zoomed-in panels in unit of arcsec
spw = ['{}'.format(s) for s in range(1,31)]
clevels = [0.5, 1.0]  ## contour levels to fill in between.
calpha=0.35  ## now tune down the alpha
restoringbeam=['6arcsec']
ql.qlookplot(vis=msfile, specfile=specfile, timerange=timerange, spw=spw, stokes=stokes, \
            restoringbeam=restoringbeam,imsize=imsize,cell=cell, \
            xycen=xycen,fov=fov,clevels=clevels,calpha=calpha)
</pre>
The output FITS file as a 4-d cube is saved to ''msfile + '.outim.image.fits''' under your working directory. The 30 spectral windows used for spectral imaging are combined in the output FITS file. So NAXIS3 (frequency axis) is 30.
<pre>                       
NAXIS  =                    4 / number of array dimensions                   
NAXIS1  =                  128                                                 
NAXIS2  =                  128                                                 
NAXIS3  =                  30                                                 
NAXIS4  =                    2 
</pre>
</pre>


===Quick-Look Imaging series===
=9. Downloading fits files to your local system=
<pre style="background-color: #FCEBD9;">
This section is for interested users who wish to generate FITS files with full control of all parameters being used for synthesis imaging. Please follow the [https://colab.research.google.com/drive/19NQb6Emb9HvKX4QHq9ZYCP3RM6nT7sDL#scrollTo=cLdDVptBGG-X EOVSA Workspace] for an example on downloading image fits files.
# In SunCASA
msfile = 'IDB20170821201800-202300.4s.slfcaled.ms'
specfile = msfile + '.dspec.npz'
## set the time interval
timerange = '2017/08/21/20:21:10~2017/08/21/20:21:18'
## Bin width for time averaging
twidth = 1
## frequency range
spw = ['0~3']
## image center for clean in solar X-Y in arcsec
xycen = [375, 45]
## number of pixels in X and Y. If only one value is provided, NX = NY
imsize = [128]
## field of view of the zoomed-in panels in unit of arcsec
fov = [100., 100.]
## pixel scale
cell = ['2.0arcsec']
## select stokes XX
stokes = 'XX'
## for EOVSA data, set usemsphacenter to False
usemsphacenter = False
## set True if make a movie
mkmovie = True
## set True if generate compressed fits
docompress = True
#### ---- Control knobs for AIA plotting ---- ####
## set True if plot AIA images as the background
plotaia = True
## provide the path to the directory where the AIA fits files are located. Otherwise, set it to be ''
aiadir = 'Directory-where-AIA-fits-files-are-located'
## AIA passband. The options are [171,131,304,335,211,193,94,1600,1700]
aiawave = 171
## numbers of CPU threads for computing
ncpu = 2


=10. Spectral Fitting with GSFIT (optional)=
The .fits images obtained in the [https://colab.research.google.com/drive/19NQb6Emb9HvKX4QHq9ZYCP3RM6nT7sDL#scrollTo=cLdDVptBGG-X EOVSA Workspace] can be read as an input to the GSFIT package. The screenshot of that is shown in Figure 9. Refer [http://ovsa.njit.edu/wiki/index.php/GSFIT_Help this page] for a detailed description of further GSFIT analysis.


ql.qlookplot(vis=msfile, specfile=specfile, timerange=timerange, spw=spw,
[[file:GSFIT_20220507.PNG|thumb|right|400px|Figure 9: Image and the corresponding spectrum on GSFIT GUI.]]
          xycen=xycen, imsize=imsize, fov=fov, cell=cell, usemsphacenter=usemsphacenter,
          plotaia=plotaia, aiadir=aiadir, aiawave=aiawave,
          mkmovie=mkmovie, twidth=twidth, ncpu=ncpu, docompress=docompress, stokes=stokes)


</pre>
=11. Installation of SunCASA (optional)=
If you want to install SunCASA on your local machine follow this section. If not, run SunCASA directly on the Colab Workspace.


=Downloading fits files to your local system with Batch-Mode Imaging=
PIP wheels for SunCASA and its CASA dependencies are available as a Python 3 module from [https://pypi.org/ PyPI]. This allows simple installation across most of the UNIX-BASED PLATFORMS (Linux & macOS) and import into standard *Python 3.6* environments. The RHEL 6/7  are the officially supported platforms for the casa modules. We have also tested the SunCASA/CASA packages under other Linux-based platforms such as Ubuntu 18, Scientific Linux 6/7, CentOS 7, as well as macOS Big Sur to some extent. YMMV for other versions of Linux or macOS. We do not recommend the use of Conda until CASA's compatibility with Conda is better understood.  
This section is for interested users who wish to generate FITS files with full control on all parameters being used for synthesis imaging. We provide one example SunCASA script for generating 30-band spectral imaging maps, and another for iterating over time to produce a time series of these maps.
====Producing a 30-band map cube for a given time====
An example script can be found at [https://github.com/binchensun/eovsa-tutorial/blob/master/rhessi18/imaging_example.py this Github link]. If you are on the AWS server Virgo, it is under /common/data/eovsa_tutorial/imaging_example.py. First, download or copy the script to your own working directory and cd to your directory.
<pre style="background-color:#FCEBD9;">
# in SunCASA
CASA <##>: cd your_working_directory
CASA <##>: !cp /common/data/eovsa_tutorial/imaging_example.py ./
</pre>


[[file:fig-specimg.png|thumb|right|300px|Example multi-frequency images at a single time integration]]
====Requirements====
The following prerequisites must be present on the host machine before installing SunCASA:


Second (optional), change inputs in the following block in your copy of the "imaging_example.py" script and save the changes. This block has definitions for time range, image center and FOV, antennas used, cell size, number of pixels, etc.
* '''Python 3.6''' (3.7 may also work, its compatibility with CASA is not fully understood yet)
<pre>
* '''For Linux:''' libgfortran3 ([https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/7/html/system_administrators_guide/ch-yum yum] or [https://help.ubuntu.com/community/AptGet/Howto) apt-get] install)
################### USER INPUT GOES IN THIS BLOK ################
* '''For MacOS:''' gcc ([https://brew.sh/) brew install], [https://www.xquartz.org/ XQuartz] and [https://apps.apple.com/us/app/xcode/id497799835?mt=12 Xcode])
vis = 'IDB20170821201800-202300.4s.slfcaled.ms'  # input visibility data
trange = '2017/08/21/20:21:00~2017/08/21/20:21:30' # time range for imaging
xycen = [380., 50.] # center of the output map (in solar X and Y, Unit: arcsec)
xran = [340., 420.]  # plot range in solar X. Unit: arcsec
yran = [10., 90.] # plot range in solar Y. Unit: arcsec
antennas = '' # Use all 13 EOVSA 2-m antennas by default
npix = 256 # number of pixels in the image
cell = '1arcsec' # pixel scale in arcsec
pol = 'XX' # polarization to image, use XX for now
pbcor = True # correct for primary beam response?
outdir = './images' # Specify where to save the output fits files
outimgpre = 'EO' # Something to add to the output image name
</pre>


Then, run the script in SunCASA via
====Installating SunCASA====
<pre style="background-color:#FCEBD9;">
Installation instructions are as follows (from a UNIX terminal window). First create a Python virtual environment named suncasaenv under the $HOME directory:
CASA <##>: execfile('imaging_example.py')
</pre>


The output is a combined 30-band FITS file saved under "outdir". The naming conversion is outimgpre + YYYYMMDDTHHMMSS_ALLBD.fits. In this example, it is "./images/EO20170821T202115.000_ALLBD.fits". Here is the detailed information of the axes of the output FITS file.
<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
<pre>                        
$ cd $HOME
NAXIS  =                    4 / number of array dimensions                   
$ python3.6 -m venv suncasaenv
NAXIS1  =                  128                                                 
NAXIS2  =                  128                                                 
NAXIS3  =                  30                                                 
NAXIS4  =                    2 
</pre>
</pre>


From this point, you can use your favorite language to read the fits files and plot them. The last block of the example script uses SunPy.map to generate the plot shown on the right. For SSWIDL users, please refer to [[#Plotting Images in SSWIDL|Section 3.3.3.3]].
'''NOTE:'''  We strongly recommend using a Python virtual environment to prevent breaking any packages within a pre-existing Python environment.


====Producing a time series of 30-band maps (a 4-d cube)====
Then use [https://pypi.org/project/suncasa/ pip] to install SunCASA within the newly-created virtual environment:
An example script can be found at [https://github.com/binchensun/eovsa-tutorial/blob/master/rhessi18/imaging_timeseries_example.py this Github link]. If you are on the AWS server Virgo, it is under /common/data/eovsa_tutorial/imaging_timeseries_example.py. The script enables parallel-processing (based on a customized task "ptclean3" -- do not ask me why we have "3" in the task name). To invoke more CPU processes for parallel-processing, change "nthreads" in the preambles from 1 to the number of threads you wish to use. 


<pre style="background-color:#FCEBD9;">
<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
# in SunCASA
$ source suncasaenv/bin/activate
CASA <##>: cd your_working_directory
(suncasaenv) $ pip install --upgrade pip
CASA <##>: !cp /common/data/eovsa_tutorial/imaging_timeseries_example.py ./
(suncasaenv) $ pip install suncasa
</pre>
</pre>
Similar as the previous script, change inputs in the following block of your copy of the "imaging_timeseries_example.py" script and save the changes.


[[file:fig-specmovie.png|thumb|right|300px|Example multi-frequency images at one time frame in the [https://web.njit.edu/~sjyu/download/eovsa-tutorial/movie.html output javascript movie]]]
'''NOTE:'''  If this does not work, it could be due to unsuccessful installation of some dependencies. Running these commands should address this.
<pre>
################### USER INPUT GOES IN THIS BLOK ####################
vis = 'IDB20170821201800-202300.4s.slfcaled.ms' # input visibility data
specfile = vis + '.dspec.npz' ## input dynamic spectrum
nthreads = 1  # Number of processing threads to use
overwrite = True  # whether to overwrite the existed fits files.
trange = ''  # time range for imaging, default to all times in the data
twidth = 1  # make one image out of every 2 time integrations
xycen = [380., 50.]  # center of the output map in solar X and Y. Unit: arcsec
xran = [340., 420.]  # plot range in solar X. Unit: arcsec
yran = [10., 90.]  # plot range in solar Y. Unit: arcsec
antennas = ''  # default is to use all 13 EOVSA 2-m antennas.
npix = 128  # number of pixels in the image
cell = '2arcsec'  # pixel scale in arcsec
pol = 'XX'  # polarization to image, use XX for now
pbcor = True  # correct for primary beam response?
grid_spacing = 5. * u.deg  # grid spacing in degrees
outdir = './image_series/'  # Specify where to save the output fits files
imresfile = 'imres.npz'  # File to write summary of the imaging results
outimgpre = 'EO'  # Something to add to the image name
</pre>
 
Then, run the script in SunCASA by
<pre style="background-color:#FCEBD9;">
CASA <##>: execfile('imaging_timeseries_example.py')
</pre>


The output fits images and a summary file imresfile are saved under "outdir" (in this example "./image_timeseries"). The naming convention of output fits images is the same as [[#Producing a 30-band map at a given time|the previous section]]. The summary yields the time, frequency, and path to every image.
<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
<pre style="background-color:#FCEBD9;">
(suncasaenv) $ pip install casatasks
CASA <##>: imres = np.load(outdir + imresfile)['imres'].item()
(suncasaenv) $ pip install casatools
CASA <##>: imres.keys()
(suncasaenv) $ pip install casadata
['FreqGHz', 'ImageName', 'Succeeded', 'BeginTime', 'EndTime']
(suncasaenv) $ pip install PyQt5
CASA <##>: ls image_series/*.fits | head -5
(suncasaenv) $ pip install sunpy[all]
image_series/EO20170821T201800.500_ALLBD.fits
(suncasaenv) $ pip install suncasa
image_series/EO20170821T201804.500_ALLBD.fits
image_series/EO20170821T201808.500_ALLBD.fits
image_series/EO20170821T201812.500_ALLBD.fits
image_series/EO20170821T201816.500_ALLBD.fits
</pre>
</pre>
The last block of the example script uses SunPy.map to generate a series of plots and wraps them as a [https://web.njit.edu/~sjyu/download/eovsa-tutorial/movie.html javascript movie].


====Plotting Images in SSWIDL====
To exit the python venv, type "deactivate" from the terminal. However, the rest of this tutorial '''assumes the venv is active''' (to reactivate, type source $HOME/suncasaenv/bin/activate)
All the output files are in standard FITS format in the Helioprojective Cartesian coordinate system (that most spacecraft solar image data adopt; FITS header CTYPE is HPLN-TAN and HPLT-TAN). They are fully compatible with [https://hesperia.gsfc.nasa.gov/rhessidatacenter/complementary_data/maps/ the SSWIDL map suite] which deals with FITS files. We have prepared SSWIDL routines to convert the single time or time-series FITS files to an array of SSWIDL map structure. The scripts are available in the [https://github.com/binchensun/eovsa-tutorial/tree/master/rhessi18 Github repository of our tutorial]. For those working on Virgo, local copies are placed under /common/data/eovsa_tutorial/. Two scripts are relevant to this tutorial:
* [https://github.com/binchensun/eovsa-tutorial/blob/master/rhessi18/casa_readfits.pro casa_readfits.pro]: read an array of multi-frequency FITS files into FITS header (index) and data.
* [https://github.com/binchensun/eovsa-tutorial/blob/master/rhessi18/casa_fits2map.pro casa_fits2map.pro]: convert an array of multi-frequency FITS files into an array of SSWIDL map structure (adapted from P. T. Gallagher's hsi_fits2map.pro)


First, start SSWIDL from command line:
====Updating a previous installation of SunCASA====
<pre style="background-color:#FCEBD9;">
You can update suncasa to its latest version by running:
sswidl
<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
(suncasaenv) $ pip install --upgrade suncasa
</pre>
</pre>


Find and convert the fits files:
====Sanity check====
<pre  style="background-color:#FCEBD9;">
With the pip installation, SunCASA, as well as CASA, may be used in a standard Pythonic manner. For example, SunCASA modules and CASA tasks can be invoked using “import”, while CASA tools first need to be instantiated:
; cd to your path that contains casa_readfits.pro and casa_fits2map.pro. If on Virgo, just add the path
IDL> add_path, '/common/data/eovsa_tutorial/'
IDL> fitsdir = '/common/data/eovsa_tutorial/image_series/'
IDL> fitsfiles = file_search(fitsdir+'*_ALLBD.fits')
; The resulting fitfiles contains all multi-frequency FITS files under "fitsdir"
; The following command converts the fits files to an array of map structures.
; "casa_fits2map.pro" also work well on a single FITS file.  
; (Optional) keyword "calcrms" is to calculate rms and dynamic range (SNR) of the maps in a user-defined "empty" region
;        of the map specified by "rmsxran" and "rmsyran"
;;; Here we load 10 time frames as a demonstration
IDL> casa_fits2map, fitsfiles[45:54], eomaps [, /calcrms, rmsxran=[260., 320.], rmsyran=[-70., 0.]]
</pre>


The resulting maps have a shape of [# of frequencies, # of polarizations, # of times]
<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
<pre style="background-color:#FCEBD9;">
(suncasaenv) $ ipython
IDL> help,eomaps
Python 3.6.9 (default, Jun 16 2021, 22:21:26)
EOMAPS          STRUCT    = -> <Anonymous> Array[30, 1, 10]
Type 'copyright', 'credits' or 'license' for more information
</pre>
IPython 7.16.1 -- An enhanced Interactive Python. Type '?' for help.
They have compatible keywords recognized by, e.g., plot_map.pro. Example of the output map keywords:
In [1]: import suncasa
<pre  style="background-color:#FCEBD9;">
In [2]: help(suncasa)
IDL> help,eomaps,/str
In [3]: import casatasks
** Structure <36009f8>, 24 tags, length=131336, data length=131332, refs=1:
In [4]: help(casatasks)
  DATA            DOUBLE    Array[128, 128]
  XC              DOUBLE          378.99175
  YC              DOUBLE          49.001472
  DX              DOUBLE          2.0000000
  DY              DOUBLE          2.0000000
  TIME            STRING    '21-Aug-2017 20:20:59.500'
  ID              STRING    'EOVSA XX 3.413GHz'
  DUR            DOUBLE          3.9999997
  XUNITS          STRING    'arcsec'
  YUNITS          STRING    'arcsec'
  ROLL_ANGLE      DOUBLE          0.0000000
  ROLL_CENTER    DOUBLE    Array[2]
  FREQ            DOUBLE          3.4125694
  FREQUNIT        STRING    'GHz'
  STOKES          STRING    'XX'
  DATAUNIT        STRING    'K'
  DATATYPE        STRING    'Brightness Temperature'
  BMAJ            DOUBLE        0.0097377778
  BMIN            DOUBLE        0.0097377778
  BPA            DOUBLE          0.0000000
  RSUN            DOUBLE          948.03989
  L0              FLOAT          0.00000
  B0              DOUBLE          6.9299225
  COMMENT        STRING    'Converted by CASA_FITS2MAP.PRO'
</pre>
</pre>


[[file:fig-specimg_idl.png|thumb|right|300px|Example multi-frequency EOVSA images at one time plotted in SSWIDL]]
An example for plotting all images at a selected time:
<pre  style="background-color:#FCEBD9;">
; choose a time pixel
IDL> plttime = anytim('2017-08-21T20:21:15')
IDL> dt = min(abs(anytim(eomaps[0,0,*].time)-plttime),tidx)
; use maps at this time, first polarization, and all bands
IDL> eomap = reform(eomaps[*,0,tidx])
IDL> window,0,xs=1000,ys=800
IDL> !p.multi=[0,6,5]
IDL> loadct, 3
IDL> for i=0, 29 do plot_map,eomap[i],grid=5.,$
        tit=string(eomap[i].freq,format='(f5.2)')+' GHz', $
        chars=2.0,xran=[340.,420.],yran=[10.,90.]
</pre>


=Self-calibration (optional)=
The use of python3 venv is a simple built-in method of containerizing the pip install such that multiple versions of SunCASA can be kept on a single machine in different environments. In addition, SunCASA is built and tested using standard (python 3.6) libraries which can be replicated with a fresh venv, keeping the libraries needed for SunCASA isolated from other libraries which may already be installed on your machine.
Self-calibration is often needed in bringing out details of the flare at multiple frequencies. An example script, run in SunCASA, for doing self-calibration of the 2017 Aug 21 flare at ~20:20 UT can be found at [http://www.ovsa.njit.edu/wiki/index.php/Tohban_EOVSA_Imaging_Tutorial_A-Z this tutorial].

Latest revision as of 21:56, 7 July 2022

1. Introduction to EOVSA and Datasets

EOVSA

EOVSA (Expanded Owens Valley Solar Array) is a solar-dedicated radio interferometer operated by the New Jersey Institute of Technology. EOVSA observes the full disk of the Sun at all times when the Sun is >10 degrees above the local horizon, which is season dependent and ranges from 7-12 hours duration centered on 20 UT. Like any radio interferometer, the fundamental measurement for imaging is the correlated amplitude and phase between each pair of antennas, which is called a “complex visibility.” EOVSA’s 13 antennas form 78 such visibilities at any frequency and instant of time, i.e. 78 measurements of the spatial Fourier transform of the solar brightness distribution. EOVSA records these visibilities at 451 science frequency channels each second, in four polarization products (XX, YY, XY, YX), as well as additional total flux measurements from each individual antenna. These data are then processed through a pipeline processing system whose data flow is shown in the block diagram (Figure 1). One of the outputs of the pipeline is a visibility database in a widely used open-standard format called a CASA measurement set (or “ms”; CASA is the Common Astronomy Software Applications package used by many modern interferometer arrays).

EOVSA delivers the radio interferometry data on the following three levels:

Figure 1: Pipeline block diagram
  • Level 0 - Raw visibility data - This includes observations of cosmic sources for phase calibration, and gain and pointing observations required for total power calibration.
  • Level 1 - Calibrated visibility data - After applying calibration and other preliminary processing to level 0 data, we create the calibrated visibility data in CASA ms format (second column in Figure 1). These visibility data have all of the required content to produce Level 2 images and spectrogram data in standard FITS format. The following tutorial will guide you through how to make EOVSA images and spectrogram from visibility data. We provide a set of standard ms’s for each day (pink solid boxes in Figure 1) for users who wish to start with visibility data. You can retrieve EOVSA 1-min averaged visibility data in CASA ms format from this page. Full-resolution EOVSA visibility data in CASA ms format will be provided per request. Please contact the *EOVSA team if you wish to have Level 1 visibility data for a specific event.
  • Level 2 - Images and spectrogram data in standard FITS format - Most users, however, will prefer to work with spectrogram (frequency-time) and image data, which are also outputs of the pipeline system shown in Figure 1 (orange boxes). Spectrograms are provided as standard FITS tables containing the frequency list, list of times, and data in both total power and a sum of amplitudes over intermediate-length baselines (cross power). Likewise, image data products are in FITS format with standard keywords and are converted into the Helioprojective Cartesian coordinate system compatible with the World Coordinate System (WCS) convention, along with correct registration for the spatial, spectral, and temporal coordinates. Both the spectrogram and image data products are calibrated and have physical radio intensity units (sfu for spectrograms and brightness temperature for radio images).

2. Browsing and Obtaining EOVSA data

Figure 2: EOVSA Browser
Figure 3: RHESSI Browser

EOVSA Browser

The most convenient way for browsing Level 2 EOVSA data is through the EOVSA Browser. The overview EOVSA dynamic spectra on the top-left panel are from the median of the uncalibrated cross-power visibilities at a few short baselines, which are not (but a good proxy of) the total-power dynamic spectra. The effects of spatial information blended in the cross-power visibilities can be clearly seen as the "U"-shaped features throughout the day, which are due to the movement of the Sun across the sky that effectively changes the length and orientation of the baselines. Flare emission can be seen in the dynamic spectra, which usually appears as vertical bright features across many frequency bands. The pipeline quicklook full-disk images are also displayed for the given frequencies along with multi-wavelength full-disk maps such as SDO AIA, HMI, BBSO H-alpha by hovering on the buttons on the bottom of each map.. More information can be found on this page. The figure on the right shows an example of the overview EOVSA dynamic spectrum for 2021 May 07.

RHESSI Browser

EOVSA data can also be browsed through RHESSI Browser. Check the "EOVSA Radio Data" box on the data selection area (top-left corner). Then select year/month/date to view the overall EOVSA dynamic spectrum. Note if a time is selected at early UTC hours (e.g., 0-3 UT), it will show the EOVSA dynamic spectrum from the previous day. Also, note that EOVSA data were not commissioned for spectroscopic imaging prior to April 2017.

Level 0 (raw visibility) Data

Once you identify the flare time, you can find the full-resolution (1-s cadence) uncalibrated visibility files (in Miriad format) at this link. Each data file is usually 10 minutes in duration. The name convention is YYYYMMDD (folder name) /IDBYYYYMMDDHHMMSS (file name), where the time in the file name indicates the start time of the visibility data. However, to be useful for imaging, these files must be first calibrated and then converted to a CASA measurement set. At present, this can only be done with software at the OVRO site although we are working on software to allow this processing to be done elsewhere. For now, please request this processing to be done for you by emailing Dale Gary, Bin Chen, Sijie Yu, or others you may know in the solar radio group.

Level 1 (calibrated visibility) Data

The images you see in the browser are generated by an automated pipeline that also produces calibrated CASA ms datasets, but at 1-min integration. If a 1-min cadence is sufficient for your purposes, you may wish to download the calibrated data and perform your own imaging. The 1-min cadence calibrated data are at this link. You should be aware that a model disk has been subtracted from the visibilities in these files, which is what you want if you are interested in imaging active regions or flares. It is possible to add the model back, but this is not available (yet) in standard software, so please contact someone in the NJIT solar radio group for help with that.

Other useful links

3. Software requirements and installation

EOVSA data processing and analysis are developed over two packages:

  • SunCASA A Python wrapper around CASA (the Common Astronomy Software Applications package) for synthesis imaging and visualizing solar spectral imaging data. CASA is one of the leading software tools for "supporting the data post-processing needs of the next generation of radio astronomical telescopes such as ALMA and VLA", an international effort led by the National Radio Astronomy Observatory. The current version of CASA uses Python (3.6) interface. More information about CASA can be found on NRAO's CASA website . Note, CASA is available ONLY on UNIX-BASED PLATFORMS (and therefore, so is SunCASA).
  • GSFIT A IDL-widget (GUI)-based spectral fitting package called GSFIT, which provides a user-friendly display of EOVSA image cubes and an interface to fast-fitting codes (via platform-dependent shared-object libraries).

For this tutorial, we will demonstrate using SunCASA to create EOVSA image and spectrogram from the visibility data observed for an M class flare on 2021 May 07. There are two approaches to accessing the SunCASA package:

  1. Through Google Colaboratory (colab) which hosts a free virtual environment that requires no setup and runs entirely in the cloud. For example, the EOVSA workspace of this tutorial explains the analysis setup.
  2. Install on your own machine, and run the notebook as a regular Jupyter Notebook. See SunCASA Installation section below, also this page and GSFIT Installation for instructions.

4. Download Sample EOVSA Data for the Tutorial

For this tutorial, we use an M4 class flare on 07 May 2021, around 19:00 UT occurred near the solar east limb as viewed from Earth, and was well observed by Solar Orbiter (including the STIX instrument). For other recent EOVSA events, check here. Here, we provide a calibrated and self-calibrated EOVSA visibility dataset in CASA ms format (IDB20210507_1840-1920XXYY.cal.10s.slfcaled.ms) which is ready for science analyses purposes,

5. Information on the EOVSA Visibility Data (CASA Measurement Set)

With the pip installation, the CASA modules may be used in a standard Python manner. For example, CASA tasks can be invoked using import, while CASA tools are Python classes that first need to be instantiated to create usable objects. A useful first task to run is listobs, which provides a summary of the measurement set. 

## in SunCASA
from casatasks import listobs
import os
msfile = 'IDB20210507_1840-1920XXYY.cal.10s.slfcaled.ms'
rc = listobs(vis=msfile, listfile = msfile + '.listobs', overwrite=True)

print(os.popen("cat " + msfile + ".listobs").read())
Figure 4: Output of listobs task

6. Self-calibration (optional)

Self-calibration is often needed in bringing out details of the flare at multiple frequencies. An example script, run in SunCASA, for doing self-calibration of the 2017 Aug 21 flare at ~20:20 UT can be found at here. The EOVSA workspace discussed along with this tutorial anyway uses the self-calibrated dataset for analysis.

7. Cross-Power Dynamic Spectrum

The first module we introduce is dspec. This module allows you to generate a cross-power dynamic spectrum from an MS file, and visualize it. You can select a subset of data by specifying a time range, spectral windows/channels, antenna baseline, or uvrange. The selection syntax follows the CASA convention. More information of CASA selection syntax may be found in the above links or the Measurement Selection Syntax.

Figure 4: EOVSA cross power dynamic spectrum at stokes XX polarization
## in SunCASA
from suncasa import dspec as ds
import time
           ## define the output filename of the dynamic spectrum 
specfile = msfile + '.dspec.npz'
           ## The example below shows the cross-power spectrogram from a baseline selected using the parameter "bl".
           ## bl = '4&9' means selecting a baseline from Antenna ID 4 (Antenna Name "eo05") correlating with Antenna ID 9 
           ## (Antenna Name "eo10") - refer listobs output.
           ## you can also use the "bl" parameter to select multiple baseline(s), i.e., bl='0&2;4&9;8&11'.
           
           ## Hover over "Dspec" in the next line to see additional arguments such as frequency range ("spw"), and time range ("timeran") 
           ## or use help(ds.Dspec)
           ## this step generates a dynamic spectrum and saves it to specfile as a numpy .npz file
d = ds.Dspec(msfile, bl='4&9', specfile=specfile) 
d.plot(vmin=None, vmax=None, pol='XX')
print(d.data.shape) # The shape gives the dimensions of polarization, baselines, frequencies, time

Alternative methods of using the "dspec" task are discussed in the EOVSA workspace.

NOTE: If you are using your own machine, the plotting should be in interactive mode. In that mode, hovering your mouse over the dynamic spectrum allows you to read the time, frequency, and flux information under the cursor at the bottom of the window.

8. Quick-Look Imaging

We use CASA's CLEAN algorithm for EOVSA imaging. As the default coordinate system is equatorial, solar coordinate transformation and image registration need to be done to place the image into the usual Helioprojective Cartesian Coordinates (X- and Y-axes are Solar X and Solar Y in arcsecond unit, respectively). We have bundled a number of these steps into a module named qlookplot, allowing users to generate an observing summary plot showing the cross power dynamic spectrum, GOES light curves and EOVSA quick-look images.

Now, let us start with making a summary plot of the EOVSA image at the spectral windows 2 to 5 (2.4–3.7 GHz).

Figure 5: EOVSA full-Sun single-band quicklook image
## in SunCASA
from suncasa.utils import qlookplot as ql
           ## (Optional) Supply the npz file of the dynamic spectrum from previous step. 
           ## If not provided, the program will generate a new one from the visibility.
           
timerange = '19:02:00~19:02:10' ## set the time interval           
spw = '2~5'                     ## select frequency range          
stokes = 'XX'                   ## select stokes XX           
plotaia = False                 ## Initially, turn off AIA image plotting, default is True

ql.qlookplot(vis=msfile, specfile=specfile, timerange=timerange, spw=spw, \
    stokes=stokes, plotaia=plotaia, restoringbeam=['60arcsec'], robust=0.5)

The empty panels are there as only one polarization is selected for imaging/spectroscopy.

Feel free while working in EOVSA Workspace to explore by adjusting the above parameters, e.g. use a different time range ("timerange" parameter) and/or frequency range ("spw" parameter).

Quick-look Imaging with AIA data

With qlookplot, it is easy to engage solar data from SDO/AIA in the summary plot. Setting plotaia=True in the qlookplot command will download SDO/AIA data at the given time to current directory and add it to the summary plot.

Figure 6: EOVSA full-Sun single-band quicklook image with SDO/AIA as background
## in SunCASA
outfits = 'EOVSA_20210507T190205.000000.outim.image.fits'
ql.qlookplot(vis=msfile, specfile=specfile, timerange=timerange, spw=spw, \
    stokes=stokes, plotaia=True)

The resulting radio image is a 4-D datacube (in solar X-pos, Y-pos, frequency, and polarization), which is, by default, saved as a fits file Instrument_yymmddTHHMMS.ffffff.outim.image.fits under your working directory. An alternate name for the output fits file can be specified using the outfits parameter. In this example, we combine all selected frequencies (specified in keyword spw) into one image (i.e., multi-frequency synthesis). Therefore the third axis only has one plane. The fourth axis contains polarization. In this example, the fourth axis only has one plane as well (XX). Here is the relevant information (first few lines) in the header of the resulting FITS file. 

In [1]: from astropy.io import fits
In [2]: hdu = fits.open('EOVSA_20210507T190230.000000.outim.image.fits')[0]
In [3]: hdu.header
Out[3]:
SIMPLE  =                    T / conforms to FITS standard
BITPIX  =                  -64 / array data type
NAXIS   =                    4 / number of array dimensions
NAXIS1  =                  512 / Nx
NAXIS2  =                  512 / Ny
NAXIS3  =                    1 / number of frequencies
NAXIS4  =                    1 / number of polarizations

By default, qlookplot produces a full sun radio image (512x512 with a pixel size of 5"). If you know where the radio source is (e.g., from the previous full-Sun imaging), you can make a partial solar image around the source by specifying the image center (xycen), pixel scale (cell), and image field of view (fov). Here we show an example that images for each spectral window in this data set (from spw 0 to 49) at the same time interval around 19:02 UT. At high frequencies, the microwave source concentrates near the flare site, corresponding pretty well with the flare kernel in SDO/AIA. At low frequencies, the microwave source extends to higher altitudes in the corona. Again, feel free to change some of these parameters to explore what they do.

Quick-look Imaging with AIA data and all spw

Next, we will make images for every single spectral window in this data set (from spw 0 to 47).

Figure 7: EOVSA multi-band quicklook image
## in SunCASA
from suncasa.utils.mstools import time2filename
timerange = '19:02:00~19:02:10'   ## set the time interval
spw = ['{}'.format(l) for l in range(48)].     ## select (almost) all spectral windows from spw id #0 to #47
outfits = time2filename(visibility_data,timerange=timerange)+'.outim.image.allbd.fits'

stokes = 'XX'.            ## select stokes XX
xycen = [-900, 280]       ## image center for clean in solar X-Y in arcsec
cell=['2.0arcsec']        ## pixel scale
imsize=[128]              ## number of pixels in X and Y. If only one value is provided, NX = NY
fov = [300,300]           ## field of view of the zoomed-in panels in unit of arcsec

plotaia = True            ## turn off AIA image plotting, default is True
aiawave = 1600            ## AIA passband in Å. The options are [171,131,304,335,211,193,94,1600,1700]
acmap = 'gray_r'          ## Choose the coloar map for AIA images. If not provided, the program will use default AIA colormap.

ql.qlookplot(vis=visibility_data, specfile=specfile, timerange=timerange, 
             spw=spw, stokes=stokes, plotaia=plotaia, aiawave=aiawave, 
             restoringbeam=['60arcsec'], robust = 0.5, acmap=acmap,
             imsize=imsize,cell=cell,xycen=xycen,fov=fov, 
             outfits=outfits,overwrite=False)

9. Downloading fits files to your local system

This section is for interested users who wish to generate FITS files with full control of all parameters being used for synthesis imaging. Please follow the EOVSA Workspace for an example on downloading image fits files.

10. Spectral Fitting with GSFIT (optional)

The .fits images obtained in the EOVSA Workspace can be read as an input to the GSFIT package. The screenshot of that is shown in Figure 9. Refer this page for a detailed description of further GSFIT analysis.

Figure 9: Image and the corresponding spectrum on GSFIT GUI.

11. Installation of SunCASA (optional)

If you want to install SunCASA on your local machine follow this section. If not, run SunCASA directly on the Colab Workspace.

PIP wheels for SunCASA and its CASA dependencies are available as a Python 3 module from PyPI. This allows simple installation across most of the UNIX-BASED PLATFORMS (Linux & macOS) and import into standard *Python 3.6* environments. The RHEL 6/7 are the officially supported platforms for the casa modules. We have also tested the SunCASA/CASA packages under other Linux-based platforms such as Ubuntu 18, Scientific Linux 6/7, CentOS 7, as well as macOS Big Sur to some extent. YMMV for other versions of Linux or macOS. We do not recommend the use of Conda until CASA's compatibility with Conda is better understood.

Requirements

The following prerequisites must be present on the host machine before installing SunCASA:

  • Python 3.6 (3.7 may also work, its compatibility with CASA is not fully understood yet)
  • For Linux: libgfortran3 (yum or apt-get install)
  • For MacOS: gcc (brew install, XQuartz and Xcode)

Installating SunCASA

Installation instructions are as follows (from a UNIX terminal window). First create a Python virtual environment named suncasaenv under the $HOME directory: 

$ cd $HOME
$ python3.6 -m venv suncasaenv

NOTE: We strongly recommend using a Python virtual environment to prevent breaking any packages within a pre-existing Python environment.

Then use pip to install SunCASA within the newly-created virtual environment: 

$ source suncasaenv/bin/activate
(suncasaenv) $ pip install --upgrade pip
(suncasaenv) $ pip install suncasa

NOTE: If this does not work, it could be due to unsuccessful installation of some dependencies. Running these commands should address this. 

(suncasaenv) $ pip install casatasks
(suncasaenv) $ pip install casatools
(suncasaenv) $ pip install casadata
(suncasaenv) $ pip install PyQt5
(suncasaenv) $ pip install sunpy[all]
(suncasaenv) $ pip install suncasa

To exit the python venv, type "deactivate" from the terminal. However, the rest of this tutorial assumes the venv is active (to reactivate, type source $HOME/suncasaenv/bin/activate)

Updating a previous installation of SunCASA

You can update suncasa to its latest version by running: 

(suncasaenv) $ pip install --upgrade suncasa

Sanity check

With the pip installation, SunCASA, as well as CASA, may be used in a standard Pythonic manner. For example, SunCASA modules and CASA tasks can be invoked using “import”, while CASA tools first need to be instantiated: 

(suncasaenv) $ ipython
Python 3.6.9 (default, Jun 16 2021, 22:21:26)
Type 'copyright', 'credits' or 'license' for more information
IPython 7.16.1 -- An enhanced Interactive Python. Type '?' for help.
In [1]: import suncasa
In [2]: help(suncasa)
In [3]: import casatasks
In [4]: help(casatasks)


The use of python3 venv is a simple built-in method of containerizing the pip install such that multiple versions of SunCASA can be kept on a single machine in different environments. In addition, SunCASA is built and tested using standard (python 3.6) libraries which can be replicated with a fresh venv, keeping the libraries needed for SunCASA isolated from other libraries which may already be installed on your machine.

Retrieved from "http://ovsa.njit.edu//wiki/index.php?title=EOVSA_Data_Analysis_Tutorial_2022&oldid=6567"