EOVSA Data Products: Difference between revisions

From EOVSA Wiki
Jump to navigation Jump to search
No edit summary
 
(222 intermediate revisions by 4 users not shown)
Line 1: Line 1:
Welcome to the EOVSA data products documentation. EOVSA observes the full disk of the Sun at all times when the Sun is >10 degrees above the local horizon, which is season dependent and ranges from 7-12 hours duration centered on 20 UT. During January, coverage will be 16-24 UT daily, 7 days/week. EOVSA will provide microwave spectra and imaging at up to 451 frequencies between 1 and 18 GHz, which is well suited to the temperatures, densities, and magnetic field strengths of the lower corona and upper chromosphere. We deliver the radio interferometry data in level 0-2. Our working definition is that Level 0 is the raw visibility data from the instrument, whereas Level 1 corresponds to the calibrated visibility data, which have all of the required content to produce Level 2 images and spectrogram data in standard FITS format.
=Introduction=
EOVSA observes the full disk of the Sun at all times when the Sun is >10 degrees above the local horizon (season dependent and ranges from 7-12 hours duration centered on 20 UT). EOVSA records data at 451 science frequency channels each second, in four polarization products, as well as additional total flux measurements from each individual antenna. Figure 1 summarizes the different levels of data we produce. The later sections will give a more detailed description and usage examples.
[[File:pipeline_flowchart.jpg|center|600px|EOVSA pipeline block diagram/flow chart ]]


[[File:pipeline_flowchart.jpg|center|600px]]
=Level 0 - Raw visibility data from the instrument=


Full disk images at 6 selected frequency bands centered at 1.4, 3.0, 4.5, 6.8, 10.2, and 13.9 GHz will be provided once per day, calibrated in brightness temperature. Total power full-Sun-integrated spectrograms calibrated in solar flux units will be provided at 451 frequencies and 1 s time resolution. Similar cross-power spectrograms will also be provided, which have much better sensitivity to weak bursts. EOVSA will also provide quick look images and spatially-resolved microwave spectra of any bursts that are identified in the cross-power spectra. Such spatially-resolved spectra can be analyzed to provide quantitative diagnostics of the magnetic field, plasma, and particle energy parameters in the burst source.
As outlined in Figure 1, EOVSA creates raw data products in the left-hand column (labeled Level 0). This includes observations of cosmic sources for phase calibration, and gain and pointing observations required for total power calibration.


==Raw "Interim" Database (IDB) visibility data==
Full-resolution raw "Interim" Database (IDB) visibility data. They are stored in Miriad format, and hence may not be that useful for most people. Be patient after clicking the link--this is a very long list of directories, one for each available date. Recent data (latest few months) can be retrieved from the following page:


https://www.ovsa.njit.edu/fits/IDB/


For older data, visit the UC/Berkeley hosting page:


=All-day TP spectrum=
https://research.ssl.berkeley.edu/data/eovsa/IDB/
The all-day TP spectrum provides a daily dynamic spectrum of the sun-integrated flux in the broad frequency range 1-18 GHz at up to 451 science frequency channels with a cadence of 1 s.
[[File:TPSP.jpeg|center|600px]]


=All-day XP spectrum=
==Raw 1-min-averaged visibility data==
The all-day XP spectrum provides a daily dynamic spectrum of mean flux from baselines with "intermediate" lengths in the broad frequency range 1-18 GHz at up to 451 science frequency channels with a cadence of 1 s.  
This is the same as for the IDB data, except with 1-minute time integration applied. This is typically not useful for flares, but is perfectly fine for imaging active regions and full Sun. These data can be retrieved from the following page:
[[File:XPSP.jpeg|600px| center]]


=Synoptic 7-band Images=
http://www.ovsa.njit.edu/fits/UDB/
The synoptic 7-band images provide full-disk images at 7 standard frequency ranges integrated over an entire day.[[File:synoptic_image.jpg| center |400px]]


=All-day 1-min avg. visibility=
=Level 0.5 - Calibrated visibility data=
After applying calibration and other preliminary processing to the raw (level 0) data, we create the CASA ms’s in the second column in Figure 1 (labeled "level 0.5"). These visibility data are in the Fourier domain of the true images in the plane of the sky and are not immediately ready for spectral imaging analysis yet. However, they have all of the required content to produce images and spectrogram data in standard FITS format (level 1.0).  We provide a set of standard ms’s for each day (red boxes in Figure 1), for use by researchers who know how to deal with visibility data. These data are more suitable for experienced users to exploit the full potential of EOVSA data, such as spatially resolved spectral analysis. Processing these data requires CASA or sunCASA (https://github.com/suncasa/suncasa-src). Please refer to our tutorial at [[EOVSA_Data_Analysis_Tutorial]].
 
==Calibrated full-resolution visibility data for flare events==
Calibrated and self-calibrated visibility data for flare events (purple boxes in Figure 1) will typically be available within 7 days after they are taken. They will be released at our flare list site soon: https://ovsa.njit.edu/flarelist
 
==Self-calibrated 1-min-averaged visibility data==
EOVSA 1-min averaged visibility data in CASA ms format can be retrieved from the following page:
 
http://www.ovsa.njit.edu/fits/UDBms_slfcaled
 
=Level 1.0 - Images and spectrogram data in standard FITS format =
 
Level 1.0 data are for users who prefer to work with spectrogram (frequency-time) and image data directly, which are also outputs of the pipeline system shown in Figure 1 (orange boxes). They are perfectly suitable to be used as context data for comparison with other multi-wavelength observations but are not (yet) intended for quantitative spatially resolved spectral analysis.
 
Spectrograms are provided as standard FITS tables containing the frequency list, list of times, and data in both total power (TP) and a sum of amplitudes over intermediate-length baselines (cross power or XP). Likewise, image data products are in FITS format with standard keywords and are converted into the Helioprojective Cartesian coordinate system compatible with the World Coordinate System (WCS) convention, along with correct registration for the spatial, spectral, and temporal coordinates. Both the spectrogram and image data products are calibrated and have physical radio intensity units (sfu for spectrograms and brightness temperature for radio images).
 
We provide the following level 1 data products:
* Synoptic products:
** '''All-day spectrograms''': Full-day total-power (TP) and cross-power (XP) spectrograms (i.e., no spatial resolution) at full spectral and time resolution in FITS format. One file per day.
** '''All-day synoptic images''': Full disk images at 7 selected frequency bands centered at 1.4, 3.0, 4.5, 6.8, 10.2, 13.9 and 17.0 GHz are produced once per day utilizing the earth-rotation synthesis, calibrated in brightness temperature. This is because EOVSA has a limited number of baselines and we need a long integration to fill up the uv domain in order to make full-disk images.
* Event-based products:
** '''Flare spectrograms''': These are full time and frequency resolution spectrograms produced from the median of calibrated cross-power visibilities in FITS format, cropped to cover the flare duration. Preflare background is also subtracted. Compared to total-power spectrograms, these spectrograms have the advantage of revealing details of the flare evolution by "filtering out" the large-scale, continuous background from the visibilities. Note that for certain flares that have a large source size, the flux can be lower than its true values (as a fraction of the flux will be "resolved out").
** '''Pipeline-produced spectral images''': We also have a semi-automated flare imaging pipeline to produce calibrated (and self-calibrated) images at 12-s cadence at up to 10 frequency bands. They are saved in standard FITS format and have been registered into Helioprojective coordinates. They can be read by SSWIDL or astropy/sunpy. These data have already been calibrated to physical units and are usually good to be compared with context data. But please be cautious when using them for quantitative spectral analysis.
 
{| class="wikitable"
|+ Summary of EOVSA Level 1 Data Products
|-
! scope="col"| Category
! scope="col"| Data Product
! scope="col"| Naming Convention
! scope="col"| Download Link
|-
! rowspan="2" | Synoptic Spectrograms
| All-day TP Spectrograms
| EOVSA_TPall_yyyymmdd.fts
!rowspan="9" | https://ovsa.njit.edu/browser
|-
| All-day XP Spectrograms
| EOVSA_XPall_yyyymmdd.fts
|-
! rowspan="7" | Synoptic Images
|-
| Synoptic 1.4 GHz images
| eovsa_yyyymmdd.spw00-01.tb.disk.fits
|-
| Synoptic 3.0 GHz images
| eovsa_yyyymmdd.spw02-05.tb.disk.fits
|-
| Synoptic 4.5 GHz images
| eovsa_yyyymmdd.spw06-10.tb.disk.fits
|-
| Synoptic 6.8 GHz images
| eovsa_yyyymmdd.spw11-20.tb.disk.fits
|-
| Synoptic 10.2 GHz images
| eovsa_yyyymmdd.spw21-30.tb.disk.fits
|-
| Synoptic 13.9 GHz images
| eovsa_yyyymmdd.spw31-43.tb.disk.fits
|-
| Synoptic 17.0 GHz images
| eovsa_yyyymmdd.spw44-49.tb.disk.fits
|-
! rowspan="1" | Flare Spectrograms
| Full-resolution cross-power Spectrogram
| eovsa.spec.flare_id_YYYYMMDDHHMMSS.fits
!rowspan="2" | https://ovsa.njit.edu/flarelist
|-
! rowspan="1" | Flare Spectral Images
| Pipeline-produced spectral images
| eovsa.lev1_mbd_12s.YYYY-MM-DDTHHMMSSZ.image.fits
|-
|}
 
==Browsing and Downloading level 1 data==
[[File:eovsa_browser.jpg|right|thumb|EOVSA Browser]]
[[file:EOVSA_flarelist.jpg|right|thumb|EOVSA Flare List]]
===Synoptic level 1 data===
EOVSA Level 1 synoptic data products can be retrieved with the following steps:
* Go to [http://ovsa.njit.edu/browser/ EOVSA browser] page.
* Browse to the date of interest.
* Click "synoptic fits" button next to the calendar tool.
* Select the data product based on the names listed in the table above.
 
 
===Flare level 1 data===
EOVSA flare list with spectrograms and spectral images can be queried and downloaded at https://ovsa.njit.edu/flarelist. Users can use the top box to select a time range of interest and query our flare list. The results are displayed in the dropdown box. An interactive plot of the flare light curves will be shown at the bottom of the page once an event is highlighted (by clicking on the flare ID). Quicklook plots and FITS files of the spectrograms and flare movies can be accessed by clicking the icons in each flare record.
 
==Reading and Using level 1 data==
===Introduction===
All our level 1 data products are in FITS format. All the images have standard, WCS-compatible coordinates. Users can use their favorite method to read these files. In the following, we provide minimal examples to read them with Astropy and Sunpy.
 
===Event-Based Data Products===
* An example of how to read and plot the flare spectrograms and images in Python (with [https://www.astropy.org/ Astropy] and [https://sunpy.org/ SunPy]) can be accessed at [https://colab.research.google.com/drive/1Y3ONWCxLPYvWda5_LqFNxafJtwZDNJBD?usp=sharing#scrollTo=ueiMoHbdxfo- this Google Colab Jupyter notebook].
* We are working on an example with SSWIDL and will post it soon.
 
===Synoptic Data Products===
* An example of how to read and plot the flare spectrograms and images in Python (with [https://www.astropy.org/ Astropy], [https://sunpy.org/ SunPy], and optionally, [https://docs.sunpy.org/projects/radiospectra/en/latest/ radiospectra]) can be accessed at [https://colab.research.google.com/drive/11vYHSA6YmaJm0Fl-lHXtNuV3Y6WEcif_#scrollTo=-qLxlKn90lSL this Google Colab Jupyter notebook].
 
 
Additional examples for IDL users. For spectrograms:
<pre style="background-color: #FCEBD9;">
function dspec,filename,doplot=doplot
  ; Read EOVSA Dynamic Spectrum FITS file <filename> and return a spectrogram object.
  ; Optionally show an overview plot if doplot switch is set
  ;
  ; Usage:
  ;    s = dspec(<filename>)            ; Returns spectrogram object
  ;    s = dspec(<filename>,/doplot)    ; Plots spectrum and returns spectrogram object
  ;
  ; To access the data in the spectrogram object, use
  ;    spec = s.get(/spectrogram)    (Array of amplitudes in SFU, of size ntimes, nfreq)
  ;    fghz = s.get(/spectrum_axis)  (Array of frequencies in GHz, of size nfreq)
  ;    ut = s.get(/time_axis)        (Array of UT times in anytim format, of size ntimes)
 
  default,doplot,0
  spec = mrdfits(filename,0)
  freq = mrdfits(filename,1)
  time = mrdfits(filename,2)
  fghz = freq.sfreq
  ut = anytim(time)
  s = spectrogram(spec,ut,fghz)
  if doplot then begin
    window,/free,xsiz=1024,ysiz=600
    ; Find min and max of data from 5% to 95% of sorted array (eliminates outliers)
    sarr = sort(spec)
    dlim = minmax(spec[sarr[n_elements(sarr)*0.05:n_elements(sarr)*0.95]])
    ; Set drange with margin factor of 2 on low end and 5 on high end
    s.set,drange=dlim*[0.5,5]
    loadct,3
    s.plot,/log,/xsty,/ysty,ytitle='Frequency [GHz]',charsize=1.5
  endif
return,s
end
</pre>
[[File:IDL_TPSP.png|center|500px]]
 
 
For synoptic images:
<pre style="background-color: #FCEBD9;">
read_sdo,'eovsa_20191225.spw11-20.tb.disk.fits',header,data,/UNCOMP_DELETE
index2map,header,data,eomap
plot_map,eomap
</pre>
 
[[File:eovsa_20191225_image_sswidl.jpg| center |250px]]
 
=Requesting EOVSA Data or Analysis Assistance=
The pipeline-processed synoptic and flare event data (level 1.0) are science-ready and usually sufficient for many purposes. However, the list may not be complete, and the pipeline-processed data do not have the full-time and frequency resolution necessary for certain in-depth quantitative spectral analyses. If you are interested in (1) working on events that are not currently included in our level 1.0 database and/or (2) needing assistance from an EOVSA team member in detailed or quantitative analysis that requires EOVSA data processing beyond that offered by the level 1.0 products, please use the following Data Request Form to submit such requests.
 
[https://forms.gle/xPo7G3fwGwQhEmdLA EOVSA data request form]
 
We will normally respond to your request within 2-3 working days. Note that for (1), standard acknowledgments and reference citations would be sufficient. For (2), the EOVSA team member who helped with the data analysis/interpretation needs to be included as a co-author for publications that utilize the relevant EOVSA data. Please refer to https://ovsa.njit.edu//wiki/index.php/EOVSA_Data_Policy for details of our data policy.

Latest revision as of 18:44, 30 October 2024

Introduction

EOVSA observes the full disk of the Sun at all times when the Sun is >10 degrees above the local horizon (season dependent and ranges from 7-12 hours duration centered on 20 UT). EOVSA records data at 451 science frequency channels each second, in four polarization products, as well as additional total flux measurements from each individual antenna. Figure 1 summarizes the different levels of data we produce. The later sections will give a more detailed description and usage examples.

EOVSA pipeline block diagram/flow chart

Level 0 - Raw visibility data from the instrument

As outlined in Figure 1, EOVSA creates raw data products in the left-hand column (labeled Level 0). This includes observations of cosmic sources for phase calibration, and gain and pointing observations required for total power calibration.

Raw "Interim" Database (IDB) visibility data

Full-resolution raw "Interim" Database (IDB) visibility data. They are stored in Miriad format, and hence may not be that useful for most people. Be patient after clicking the link--this is a very long list of directories, one for each available date. Recent data (latest few months) can be retrieved from the following page:

https://www.ovsa.njit.edu/fits/IDB/

For older data, visit the UC/Berkeley hosting page:

https://research.ssl.berkeley.edu/data/eovsa/IDB/

Raw 1-min-averaged visibility data

This is the same as for the IDB data, except with 1-minute time integration applied. This is typically not useful for flares, but is perfectly fine for imaging active regions and full Sun. These data can be retrieved from the following page:

http://www.ovsa.njit.edu/fits/UDB/

Level 0.5 - Calibrated visibility data

After applying calibration and other preliminary processing to the raw (level 0) data, we create the CASA ms’s in the second column in Figure 1 (labeled "level 0.5"). These visibility data are in the Fourier domain of the true images in the plane of the sky and are not immediately ready for spectral imaging analysis yet. However, they have all of the required content to produce images and spectrogram data in standard FITS format (level 1.0). We provide a set of standard ms’s for each day (red boxes in Figure 1), for use by researchers who know how to deal with visibility data. These data are more suitable for experienced users to exploit the full potential of EOVSA data, such as spatially resolved spectral analysis. Processing these data requires CASA or sunCASA (https://github.com/suncasa/suncasa-src). Please refer to our tutorial at EOVSA_Data_Analysis_Tutorial.

Calibrated full-resolution visibility data for flare events

Calibrated and self-calibrated visibility data for flare events (purple boxes in Figure 1) will typically be available within 7 days after they are taken. They will be released at our flare list site soon: https://ovsa.njit.edu/flarelist

Self-calibrated 1-min-averaged visibility data

EOVSA 1-min averaged visibility data in CASA ms format can be retrieved from the following page:

http://www.ovsa.njit.edu/fits/UDBms_slfcaled

Level 1.0 - Images and spectrogram data in standard FITS format

Level 1.0 data are for users who prefer to work with spectrogram (frequency-time) and image data directly, which are also outputs of the pipeline system shown in Figure 1 (orange boxes). They are perfectly suitable to be used as context data for comparison with other multi-wavelength observations but are not (yet) intended for quantitative spatially resolved spectral analysis.

Spectrograms are provided as standard FITS tables containing the frequency list, list of times, and data in both total power (TP) and a sum of amplitudes over intermediate-length baselines (cross power or XP). Likewise, image data products are in FITS format with standard keywords and are converted into the Helioprojective Cartesian coordinate system compatible with the World Coordinate System (WCS) convention, along with correct registration for the spatial, spectral, and temporal coordinates. Both the spectrogram and image data products are calibrated and have physical radio intensity units (sfu for spectrograms and brightness temperature for radio images).

We provide the following level 1 data products:

  • Synoptic products:
    • All-day spectrograms: Full-day total-power (TP) and cross-power (XP) spectrograms (i.e., no spatial resolution) at full spectral and time resolution in FITS format. One file per day.
    • All-day synoptic images: Full disk images at 7 selected frequency bands centered at 1.4, 3.0, 4.5, 6.8, 10.2, 13.9 and 17.0 GHz are produced once per day utilizing the earth-rotation synthesis, calibrated in brightness temperature. This is because EOVSA has a limited number of baselines and we need a long integration to fill up the uv domain in order to make full-disk images.
  • Event-based products:
    • Flare spectrograms: These are full time and frequency resolution spectrograms produced from the median of calibrated cross-power visibilities in FITS format, cropped to cover the flare duration. Preflare background is also subtracted. Compared to total-power spectrograms, these spectrograms have the advantage of revealing details of the flare evolution by "filtering out" the large-scale, continuous background from the visibilities. Note that for certain flares that have a large source size, the flux can be lower than its true values (as a fraction of the flux will be "resolved out").
    • Pipeline-produced spectral images: We also have a semi-automated flare imaging pipeline to produce calibrated (and self-calibrated) images at 12-s cadence at up to 10 frequency bands. They are saved in standard FITS format and have been registered into Helioprojective coordinates. They can be read by SSWIDL or astropy/sunpy. These data have already been calibrated to physical units and are usually good to be compared with context data. But please be cautious when using them for quantitative spectral analysis.
Summary of EOVSA Level 1 Data Products
Category Data Product Naming Convention Download Link
Synoptic Spectrograms All-day TP Spectrograms EOVSA_TPall_yyyymmdd.fts https://ovsa.njit.edu/browser
All-day XP Spectrograms EOVSA_XPall_yyyymmdd.fts
Synoptic Images
Synoptic 1.4 GHz images eovsa_yyyymmdd.spw00-01.tb.disk.fits
Synoptic 3.0 GHz images eovsa_yyyymmdd.spw02-05.tb.disk.fits
Synoptic 4.5 GHz images eovsa_yyyymmdd.spw06-10.tb.disk.fits
Synoptic 6.8 GHz images eovsa_yyyymmdd.spw11-20.tb.disk.fits
Synoptic 10.2 GHz images eovsa_yyyymmdd.spw21-30.tb.disk.fits
Synoptic 13.9 GHz images eovsa_yyyymmdd.spw31-43.tb.disk.fits
Synoptic 17.0 GHz images eovsa_yyyymmdd.spw44-49.tb.disk.fits
Flare Spectrograms Full-resolution cross-power Spectrogram eovsa.spec.flare_id_YYYYMMDDHHMMSS.fits https://ovsa.njit.edu/flarelist
Flare Spectral Images Pipeline-produced spectral images eovsa.lev1_mbd_12s.YYYY-MM-DDTHHMMSSZ.image.fits

Browsing and Downloading level 1 data

EOVSA Browser
EOVSA Flare List

Synoptic level 1 data

EOVSA Level 1 synoptic data products can be retrieved with the following steps:

  • Go to EOVSA browser page.
  • Browse to the date of interest.
  • Click "synoptic fits" button next to the calendar tool.
  • Select the data product based on the names listed in the table above.


Flare level 1 data

EOVSA flare list with spectrograms and spectral images can be queried and downloaded at https://ovsa.njit.edu/flarelist. Users can use the top box to select a time range of interest and query our flare list. The results are displayed in the dropdown box. An interactive plot of the flare light curves will be shown at the bottom of the page once an event is highlighted (by clicking on the flare ID). Quicklook plots and FITS files of the spectrograms and flare movies can be accessed by clicking the icons in each flare record.

Reading and Using level 1 data

Introduction

All our level 1 data products are in FITS format. All the images have standard, WCS-compatible coordinates. Users can use their favorite method to read these files. In the following, we provide minimal examples to read them with Astropy and Sunpy.

Event-Based Data Products

Synoptic Data Products


Additional examples for IDL users. For spectrograms: 

function dspec,filename,doplot=doplot
  ; Read EOVSA Dynamic Spectrum FITS file <filename> and return a spectrogram object.
  ; Optionally show an overview plot if doplot switch is set
  ; 
  ; Usage:
  ;    s = dspec(<filename>)            ; Returns spectrogram object
  ;    s = dspec(<filename>,/doplot)    ; Plots spectrum and returns spectrogram object
  ;
  ; To access the data in the spectrogram object, use
  ;    spec = s.get(/spectrogram)    (Array of amplitudes in SFU, of size ntimes, nfreq)
  ;    fghz = s.get(/spectrum_axis)  (Array of frequencies in GHz, of size nfreq)
  ;    ut = s.get(/time_axis)        (Array of UT times in anytim format, of size ntimes)
  
  default,doplot,0
  spec = mrdfits(filename,0)
  freq = mrdfits(filename,1)
  time = mrdfits(filename,2)
  fghz = freq.sfreq
  ut = anytim(time)
  s = spectrogram(spec,ut,fghz)
  if doplot then begin
    window,/free,xsiz=1024,ysiz=600
    ; Find min and max of data from 5% to 95% of sorted array (eliminates outliers)
    sarr = sort(spec)
    dlim = minmax(spec[sarr[n_elements(sarr)*0.05:n_elements(sarr)*0.95]])
    ; Set drange with margin factor of 2 on low end and 5 on high end
    s.set,drange=dlim*[0.5,5]
    loadct,3
    s.plot,/log,/xsty,/ysty,ytitle='Frequency [GHz]',charsize=1.5
  endif
return,s
end
IDL TPSP.png


For synoptic images: 

read_sdo,'eovsa_20191225.spw11-20.tb.disk.fits',header,data,/UNCOMP_DELETE
index2map,header,data,eomap
plot_map,eomap
Eovsa 20191225 image sswidl.jpg

Requesting EOVSA Data or Analysis Assistance

The pipeline-processed synoptic and flare event data (level 1.0) are science-ready and usually sufficient for many purposes. However, the list may not be complete, and the pipeline-processed data do not have the full-time and frequency resolution necessary for certain in-depth quantitative spectral analyses. If you are interested in (1) working on events that are not currently included in our level 1.0 database and/or (2) needing assistance from an EOVSA team member in detailed or quantitative analysis that requires EOVSA data processing beyond that offered by the level 1.0 products, please use the following Data Request Form to submit such requests.

EOVSA data request form

We will normally respond to your request within 2-3 working days. Note that for (1), standard acknowledgments and reference citations would be sufficient. For (2), the EOVSA team member who helped with the data analysis/interpretation needs to be included as a co-author for publications that utilize the relevant EOVSA data. Please refer to https://ovsa.njit.edu//wiki/index.php/EOVSA_Data_Policy for details of our data policy.

Retrieved from "http://ovsa.njit.edu//wiki/index.php?title=EOVSA_Data_Products&oldid=10818"