EOVSA Imaging Workshop

Software and sample data

Here are some basic information on CASA and the software we are developing:

• Obtaining CASA; CASA Guides
• suncasa A CASA/python-based package being developed for imaging and visualizing spectral imaging data. A useful (but potentially buggy) routine (helioimage2fits.py) is available for registering EOVSA/VLA (and ALMA) CASA images.

Software setup

For local people, you can just ssh onto pipeline (@Owens Valley) or our local server baozi.hpcnet.campus.njit.edu and launch CASA. Everything has been set up already. The following instructions are for people who want to (or have to) set up their own environment.

• Download latest version of CASA. I'm using version 4.7.0, but latest version should work.

Also CASA by default does not have EOVSA observatory information. You can manully add EOVSA observatory information in CASA

import os
obsdict = {'MJD': 57447.0, 'Name': 'EOVSA', 'Type': 'WGS84', 'Long': -118.287, 'Lat': 37.2332, 'Height': 1207.13, 'X': 0.0, 'Y': 0.0, 'Z': 0.0, 'Source': 'Dale Gary'}
obstable = os.getenv("CASAPATH").split(' ')[0]+"/data/geodetic/Observatories"
tb.open(obstable,nomodify=False)
nrows = tb.nrows()
tb.addrows(1)
for k in obsdict.keys():
	tb.putcell(k,nrows,obsdict[k])
tb.close()

or download this, unzip (as Observatories), and place the directory under YOUR_CASA_INSTALLATION_PATH/data/geodetic/ import os print YOUR_CASA_INSTALLATION_PATH can be found by using CASA

os.getenv("CASAPATH").split(' ')[0]

• Download the suncasa package from the github page, and put it under YOURPATH as suncasa
• Incorporate suncasa into CASA:

Create ~/.casa/init.py -- this is a convenient way to put in any python codes you want to run every time when CASA is initialized (you can also do this manually, of course, after CASA is launched)

import sys
sys.path.append('YOURPATH')

• helioimage2fits calls astropy and sunpy within CASA. However, earlier versions of CASA (<5.0) does not have Astropy, and no version has sunpy. So we have to install Sunpy, and possibly also Astropy (depending on your CASA version)
• Install SunPy and/or Astropy into CASA. If you do not have Sunpy or Astropy in your python installation, the most convenient way to do so is through Anaconda (which would be your python version). If you don't have Anaconda, refer to this installation guide. With Anaconda in place, do the following to install Sunpy and Astropy

conda config --add channels conda-forge
conda install sunpy
conda install astropy

Next, in order to call Sunpy and Astropy within CASA, we also need to add their paths to CASA's search path. This is done by adding your installed anaconda "site-packages" to ~/.casa/init.py. On a Mac, the default path is "~/anaconda/lib/python2.7/site-packages"

sys.path.append('FULL PATH TO YOUR CONDA site-packages') 

Note: "~" for home directory does not work for the line above. You need to make replace it with the full path, or use the Python function os.path.expanduser() to find it.

Another way to install SunPy or AstroPy is as following -- this may fail if the CASA version does not match the most updated sunpy version, unfortunately.

from setuptools.command import easy_install
easy_install.main(['--user', 'pip'])
import pip
pip.main(['install', 'astropy'])
pip.main(['install', 'sunpy''])

• Okay, if nothing goes wrong, you've done the job! Now try and see if you can call suncasa.utils.helioimage2fits in CASA as follows

from suncasa.utils import helioimage2fits as hf

If there is no complain, you have successfully installed CASA, Sunpy, Astropy, and suncasa!

Data description and data access

To get started, we have made some calibrated flare and active region data in CASA measurement set format, available here. Here are some things to consider when working with the data:

• Antennas 1-8 and 12 have a wide sky coverage, while antennas 9-11 and 13 are the older equatorial mounts that can only cover +/- 55 degrees from the meridian. All antennas were tracking for the flare, but for portions of the all-day scan (roughly before 1600 UT or after 2400 UT) only 9 antennas will be tracking. Data for non-tracking antennas is NOT flagged, so if you try to image these periods, you'll need to manually omit ants 9-11 and 13.
• Because of the 2.5 GHz high-pass filters, data are taken only for bands 4-34. These will be labeled spectral window (spw) 0-31 in the measurement sets. However, band 4 (spw 0) is currently not calibrated, so only the 31 spectral windows 1-31 (bands 5-34, or frequencies 3-18 GHz) are valid data. You'll need to omit spw 0 when making images.
• In principle, circular polarization should be valid and meaningful, but no polarization calibration has been done (yet) so no non-ideal behavior (leakage terms, etc.) have been accounted for.

Flare Data

Folder Flare_20170910 contains two measurement sets at full 1-s time resolution, each with 10-min duration:

1. IDB20170910154625.corrected.ms (171 MB) [1] containing the first 10 minutes of the flare (15:46:25-15:56:25 UT), calibrated but no self-calibration
2. IDB20170910154625.corrected.ms.xx.selfcaled (66 MB) [2] self-calibrated data for polarization XX
3. IDB20170910155625.corrected.ms (232 MB) [3] containing the second 10 minutes of the flare (15:56:25-16:06:25 UT)
4. IDB20170910155625.corrected.ms.xx.selfcaled (77 MB) [4] self-calibrated data for polarization XX

All Day Quiet Sun/Active Region Data

Folder AR_20170710 contains 8 calibrated measurements sets at 60-s time resolution.

1. The seven UDB20170710*.corrected.ms are files for each scan separately, ranging in size from 7 - 150 MB depending on the length of the scan.
2. UDB20170710_allday.ms (268 MB) [5] is the same data concatenated into a single file. However, each scan has a separate source ID.

The AR data are also calibrated but not self-calibrated. In fact, we need to create a strategy for self-calibration of such data. You can get an overview of the AR coverage on that day by looking at the plot at http://ovsa.njit.edu/flaremon/daily/2017/XSP20170710.png. The separate scan files above are labeled with the time of the start of the scan, and each scan covers the time of continuous solar data between each calibration scan.

Make your first EOVSA flare image

Download this CASA script. It does basic clean on the full Sun using this slfcaled measurement set data near the flare peak (1600 UT) with 10-s integration, as well as image registration (into standard FITS file with helioprojective coordinates), and plotting using sunpy.

Fig. 1: Example EOVSA image for 2017 Sep 10 flare at around 1600 UT.

In CASA, cd to your data directory and do the following

execfile('imaging_example_20170910T1600.py')

You are supposed to see a plot similar to the one on the right. I have tested the script on both our servers on campus and Owens Valley. If not, something went wrong, most likely due to compatibility issues...

Possible Topics

We think that the best approach to the imaging workshop is not for everyone to work on the same topic, but rather to consider (and perhaps prioritize) different topics and then put together small teams of people who might be interested in investigating them. Here is a list of some possible topics, but in no prioritized order. Many topics will be somewhat interconnected, but we should avoid time-wasting situations where people working on one topic have to wait until another, uncompleted topic is done.

Calibration

Most calibrations needed for basic imaging are well in hand, so additional efforts are not really needed for an imaging workshop. Here is a list of (semi-)automatic calibrations we are doing so far. All can be improved, but such improvements will not make any major qualitative difference in the images.

1. Gain corrections: To account for changes of the attenuation settings (esp. during flares). See this page for details.
2. Correcting for polarization mixing due to feed rotation: See this page for details.
3. Antenna-based amplitude calibration: made by referencing total power and auto-correlation amplitudes to RSTN/Penticton total solar flux density measurements. See this page for details.
4. Reference phase calibration: antenna-based, band-averaged phases derived from "reference" observations of strong celestial radio sources every night. See this page for details]
5. Daily phase calibration: antenna-based multi-band delays resulting from the phase difference as a function of frequency between those derived from phase calibrators (usually observed 3 times during the day) and the reference calibrator. See this page for details.

There are, however, two other calibrations still being developed:

1. Polarization calibration: To calibrate absolute degree of circular polarization. We believe that the information already exists in available data, but the procedures need to be worked out. Additional observations are possible during this week, if needed.
2. Per-channel bandpass (or single-band-delay) calibration within individual bands: To calibrate the phase/amplitude variations from one science channel to another. Procedures need to be worked out.
3. Real-time spectral-kurtosis flagging of RFI, and other forms of RFI excision.

Pipeline Imaging

We have developed the basis for pipeline imaging of data on a daily basis, examples of the 2017 Jul 10 data can be seen here. These 10-min averages at a few bands might be okay, but lots of improvement is needed. Mainly, these improvements should be in the form of better imaging strategies (which are the subject of some of the topics below), and in better data-quality checks. In addition to these 10-min average images, we also envision making shorter (10-s?) images at a few bands during flares. Some of the RHESSI experience might come in handy here. Improving and finalizing the pipeline imaging is a high priority for this meeting!

Flare Imaging

We have succeeded in making flare multi-frequency movies, which are extremely interesting and revealing, but are quite time-consuming and human-intensive. We need strategies to reduce the number of parameters for different imaging cases, and perhaps to automate or aid in clean-box choices. When converting data-cubes to spatially-resolved spectra, issues of total power calibration come into play.

Self-calibration Strategies

(Bin) Currently I am using a sliding window of five bands (band_to_selfcal +- 2 bands) to create a multi-frequency synthesized image as the input model for doing self-calibration at the band in question. Any alternative strategies? How about channel-by-channel selfcal?

Imaging Strategies

Clean? MEM? Optimum way to do spectral imaging? Here is an example made for the Sep 10 flare (Bin).

Quiet Sun and Active Region Imaging

We desperately need help in this area. We have made simple AR images that are quite promising, but full-disk images are still quite bad. Stephen can probably do wonders, and hopefully teach us to do so also. We should explore the use of additional data (say full-disk images from GAVRT) and how that might be used. How do we handle the issue of solar rotation, which can move an active region by multiple arcminutes over a 12-h day?

Self-calibration Strategies

We can probably do some self-calibration based on bright AR, but can we self-calibration on the solar disk? Gordon would like to explore the self-cal corrections vs. time, but it requires someone succeeding in doing an all-day self-calibration in the first place.

Imaging Strategies

Given solar rotation, and the limitations of EOVSA uv coverage, what are the best imaging strategies for full-disk and AR imaging?

Software Development

Currently we are using Python for analyzing calibration data and CASA for script-based imaging.

1. What is the best approach to make an interactive imaging tool to serve the community (for both experts and non-experts)?
2. How about the tool to visualize/analyze/make use of the resulted 4-D dynamic spectral imaging data?

Notes

Imaging Priority order

1. flare selfcal Tim
2. active region rotation Stephen
3. solar disk model Stephen
4. pipeline heuristics Group (Thurs. morning)

Three types of users to keep in mind

• want packaged data without further processing (catalog data)

choices made for user, such as standard set of bands, (png and fits images and fits dynamic spectrum of flares)

• parameters of their choosing, but without knowing intricacies of instrument (gui)
• Black-belt user who goes back to raw data, but not often doing their own calibration

1. pipeline quicklook Bin, Sijie, Sa"m

Two levels of quicklook at standard frequencies

• Full-disk 10-min, limited resolution
• Zoomed-in 1-min, full resolution (requires locating in space and time)

1. pipeline parallelization optimization
2. visibility weighting
3. user interface Jim, Sa"m

Action Item

Create FITS file of daily spectra. (Jim $$$)

Calibration Priority order

1. polarization calibration Sam
2. total power calibration
3. relative vs spectrum Gordon
4. gain calibration Gordon
5. RFI excision Gelu, Gordon, Jim