EOVSA Data Analysis Tutorial

Software

We have developed two packages for EOVSA data processing and analysis:

• SunCASA A wrapper around CASA for imaging and visualizing spectral imaging data of the Sun. More information about CASA can be found on NRAO's CASA website . Note, (Sun)CASA is ONLY AVAILABLE on UNIX-BASED PLATFORMS (sorry Windows users).
• GSFIT A IDL-widget(GUI)-based spectral fitting package called gsfit, which provides a user-friendly display of EOVSA image cubes and an interface to fast fitting codes (via platform-dependent shared-object libraries).

There are two approaches in accessing our software packages. One is through our Amazon AWS server (recommended for participants of the EOVSA tutorial at the RHESSI 18 Workshop). Another is to install them on your own machine. We discuss the first approach in Section 1.1, and the second in Section 1.2.1 (SunCASA) and 1.2.2 (GSFIT).

Connecting to AWS server

We use an Amazon AWS Lightsail server for testing purposes. The server has 2 CPUs, 8 GB RAM, and 160 GB SSD storage. It runs CentOS 7 (1901-01) Linux. Please limit your usage to LIGHTWEIGHT DATA PROCESSING ONLY.

NOTE: THE ACCOUNT IS ONLY INTENDED FOR THE EOVSA TUTORIAL, BUT *NOT* FOR CARRYING OUT ACTUAL DATA REDUCTION

• Obtain SSH Key from Bin Chen
• Put it under a secure location on your own machine.
• Follow remaining directions depending on your client machine.

Linux / Mac

Recommend to use "~/.ssh" (create if it does not exist by "mkdir ~/.ssh").

• Edit the permission of ~/.ssh and the key (here I use ~/.ssh as the directory to place your key)

chmod 700 ~/.ssh
chmod 400 ~/.ssh/guest-virgo.pem

• Log on to test AWS server (password-less)

ssh -X -i ~/.ssh/guest-virgo.pem guest@virgo.arcs.az.njit.edu

Windows (MobaXterm)

Recommend to use "Documents\MobaXterm\home\.ssh", which should exist if you have already installed the free MobaXterm[1].

• Create new session, click SSH, enter virgo.arcs.az.njit.edu for Remote host, guest for username.
• On advanced SSH settings tab, click Use private key, navigate to and select file guest_key.pem
• Close setup window and click the new sessions icon, which will log you in.

Once the connection is setup, you will have access to SunCASA and GSFIT (included in the sswIDL installation): Enter SunCASA

[guest@ip-172-26-5-203 ~]$ suncasa

Enter sswIDL

[guest@ip-172-26-5-203 ~]$ sswidl

Installation on Your Own Machine

SunCASA

MacOS

We have packed a standard disk image (dmg) for installation on MacOS. It has been tested to work under Mojave (macOS v10.14). YMMV for earlier versions of macOS. Installation steps below:

• Download SunCASA disk image (SunCASA-0.7.4_Pre-release.OSX.dmg).
• If you do not have Java SE Development Kit (JDK) installed on your Mac, please download from the official site and install it before SunCASA installation. The latest version (JDK 12) was tested to work properly. Earlier versions may also work but YMMV.
• Open the disk image file (if your browser does not do so automatically).
• Drag the SunCASA application to the Applications folder of your hard disk.
• Eject the SunCASA disk image.
• Double-click the SunCASA application to run it for the first time. If the OS does not allow you to install apps from non-Apple sources, please Change the settings in "System Preferences-> Security & Privacy -> General" and "Allow applications downloaded from: Mac App store and identified developers". If the OS still reports that the app is damaged, please turn off the OS Gatekeeper for now by running command sudo spctl --master-disable in Terminal. Then double-click the SunCASA application again.
• Initialize SunCASA. To do so, run !install_suncasa from a SunCASA prompt. This step also allow you to create symbolic links to the SunCASA version and its executables (Administrator privileges are required), which will allow you to run suncasa, casaviewer, casaplotms, etc. from any terminal command line.
• Important: Make sure to turn the OS Gatekeeper back on after the installation by running sudo spctl --master-enable in Terminal.
• Optional: To update the data repository, run !update-data from the SunCASA prompt.
• Restart SunCASA by exiting the current SunCASA prompt, and run suncasa in Terminal or double-click the SunCASA application icon.
• Now SunCASA has been up and running on your Mac.

Linux

• To install SunCASA for Linux, we have packaged up a binary distribution of SunCASA which is available as a downloadable tar file. We have tested the package under Scientific Linux 6 (derived from RHEL 6) and CentOS 7 (equivalent to RHEL 7). They may work under some other Linux distributions (e.g. Ubuntu), but YMMV.
• The following prerequisite packages that need to be installed to be able to install SunCASA. Use command "yum install yourpackagename" in Terminal to install.
  • xauth
  • libXft
  • libXi
  • libXrandr
  • libXfixes
  • libXcursor
  • libXinerama
  • libGL
  • libXpm

• Download the packages
  • Package for RedHat6 or equivalent
  • Package for RedHat7 or equivalent
• Untar the package

tar -xzvf SunCASA-release-##version##.tar.gz

You do not have to have root or sudo permission to install or run SunCASA. The package is self-contained in the (untarred) directory "SunCASA-release-##version##". You can move/delete the SunCASA directory as you wish. But to use the executables, do the following:

• All executables, including suncasa are in the SunCASA-release-##version##/bin directory. Include these executables to your path for once (examples below are in bash)

cd SunCASA-release-##version##/bin
PATH=`pwd`:$PATH 

• Then you can initialize SunCASA with the command in bash

install_suncasa

• Optional: To update the data repository, run !update-data from the SunCASA prompt.
• Now SunCASA has been successfully installed on your machine. Open a new Terminal and run suncasa.

GSFIT

We have developed a IDL-widget(GUI)-based spectral fitting package called gsfit, which provides a user-friendly display of EOVSA image cubes and an interface to fast fitting codes (via platform-dependent shared-object libraries). Fits to individual spectra can be done quickly for manual investigation, while parallel/multi-core batch processing of selected blocks of data can also be performed using a command prompt application called gsfitcp. A helper routine called gsfitview allows further display and investigation of the fitting results.

GSFIT SSW_UPGRADE Instalation

The release version of the IDL GSFIT package is intended to be distributed through the SSW IDL repository [2]. Although installed as a stand-alone package, the GSFIT code relies on a series of IDL support routines that are part of the gx_simulator package. Therefore, in addition to installing the gsfit package , the gx_simulator package must be also installed. This installation can be performed by issuing an upgrade command, i.e.

IDL> ssw_upgrade,/gsfit,/gx_simulator,/gen,/spawn,/loud,/passive_ftp 

GSFIT Manual SWW Installation and Setup

However, if as of today the GSFIT package has not been yet released through SSW, you may perform a manual install by copying the directory structure located at https://github.com/Gelu-Nita/GSFIT to your local machine SSW directory, $ssw/packages/gsfit/' If such manual installation is performed, the $ssw/gen/setup/setup.ssw_env script must be edited by adding or altering the following lines:

setenv SSW_GSFIT $SSW_PACKAGES/gsfit$

setenv SSW_PACKAGES_INSTR "packages/gsfit [...]"

setenv SSW_PACKAGES_ALL "$SSW_GSFIT [...]"

where [...] denote the definitions already existent in the original setup.ssw_env script.

GSFIT SSW Instrument Setup

Please note that, regardless the method chosen to install any of these two packages, to ensure proper functionality, one should make sure that sswidl.bat (on Windows platforms) or cshrc (on Unix or Mac platforms) scripts are properly updated such as to include gx_simulator and gsfit in the SSW_INSTR path declaration, i.e.

set SSW_INSTR=gx_simulator, gsfit [...] 

where [..] denotes any other SSW packages already installed.

GSFIT Manual Stand-Alone Installation

Although not reccomended, one may in principle choose to install the gsfit and gx_simulator packages in standalone directories that are not part of the SSW repositories. However if this (not recommended) option is used, the gsfit and gx_simulator paths should be explicitly added to the IDL !path global variable, i.e.

and explicitly add this path to your IDL path structure.

IDL> !path='..//gsfit/idl:'+!path
IDL> !path='..//gx_simulator/idl:'+!path

where ../ should be replaced with the explicit paths to the respective local machine repositories.

GSFIT Linux / Mac OS Requirements

On Mac and Unix platforms, the gfortran compiler must be also installed to allow automatic compilation of the source code located in the ..//gsfit/unix directory, in order to generate a series of shared libraries, when gsfit is launched for the first time. WARNING: For this action to be successfully completed, the user must have writing rights to the ..//gsfit/unix directory.

GSFIT Windows OS Requirements

The GSFIT package is distributed along with a set of dynamic link libraries located in the ..//gsfit/win subfolder, which have been compiled assuming a WIN64 architecture. For Win32, please contact us to inquire about alternative options.

Spectral Fitting with GSFIT

GSFIT GUI Application

The GSFIT interactive GUI application may be launched as follows

IDL> gsfit [,nthreads]

where nthreads is an optional argument that indicates the number of parallel asynchronous threads to be used when performing the fit tasks. By default, GSFT launches with only one thread, but the user may interactively add or delete threads as needed at the run-time up to the number of CPUs available on the system.

GSFIT GUI Organization and Functionality

The GSFIT GUI is organized in three main sections:

• GSFIT Top Main Menu Toolbar, which may be used to perform most of the actions.
• GSFIT Left Panel, which displays a microwave map corresponding to the observing frequency and time instance selected via two horizontal scroll bars.
• GSFIT Middle Panel, which displays the microwave spectrum corresponding to the selected pixel in the right panel map.
• GSFIT Right Panel, which is contain three main elements:
  • GSFIT List Scrollbar, which allows the user to locate spatially and temporarily the fit already performed and inspect the fit results
  • GSFIT Settings page, which allows the user to setup a series of the spectral fit options and perform some actions via a local Toolbar menu.
  • GSFIT Task Queue page, which displays a list of completed, active, and pending fitting tasks.

GSFIT Top Main Menu Toolbar

This toolbar contain a series of action buttons described below.

Upload EOVSA Map Cube

EOVSA map cubes are IDL sav files containg a [ntimes x nfrequencies] brightness temperature (Tb) map structure. Follow this link to learn how such EOVSA map cubes are produced. When uploaded, GSFIT converts these Tb maps to Solar Flux (total intensity or circular polarization) maps.

Select Color Table

Use this button to select a color table for displaying the microwave maps and spectra. GSFIT uses as default the IDL color table #39 (Rainbow+White)

Select Mouse Mode

Use this set of mutual exclusive radio buttons to switch between four modes of mouse tool operation.

• FOV/Pixel Selection Mode: Use this mouse mode to zoom in and out the field of view (FOV) of the microwave maps displayed on the GSFIT Left Panel, or to select the pixel for which the microwave spectrum is displayed in theGSFIT Middle Panel.
  • To zoom in, press, hold, move, and release the left mouse button to select the desired rectangular FOV (rubber-band selection process). To revert to the full FOV, use a single left-button click.
  • Use the right button to select a map pixel on the GSFIT Left Panel. The currently selected pixel is indicated by a set og horizontal and vertical dotted lines. Alternatively, the pixel may be selected using the Cursor X and Cursor Y numerical controls located below the map plot area.
• Rectangular ROI Selection Mode: Use this mouse mode to define a rectangular region of interest (ROI) through a left button rubber-band selection process. The selected ROI is preserved and displayed as long it is not replaced by another ROI selection, or deleted by a single left button click.
• Polygonal ROI Selection Mode: Use this mouse mode to define the vertices of polygonal ROI through a series of sequential left button clicks. To finalize the process, use a right button click to define the last vertex, which will be automatically connected with the first one. The selected ROI is preserved and displayed as long it is not replaced by another ROI selection, or deleted by a single left button click.
• Free-hand ROI Selection Mode: Use this mouse mode to define an arbitrarily shaped ROI contour through a continues movement of the mouse cursor while the left button is pressed. To finalize the process, release the left button to define the last vertex of the countour, which will be automatically connected with the first one. The selected ROI is preserved and displayed as long it is not replaced by another ROI selection, or deleted by a single left button click.

Save ROI

Use this button to save to an IDL file the current ROI selection as a set of two arrays, xroi and yroi, containing its vertex heliocentric coordinates.

Restore ROI

Use this button to import from an IDL file a set of xroi and yroi ROI vertex coordinates previously saved by GSFIT, or generated by alternative means.

Select Fit Frequency Range

Use this set off drop lists controls to select the frequency range to be used for spectral fitting. By default, the full frequency range is selected when a map data cube is uploaded, but the user may choose to restrict this default frequency range for various reasons, such as data quality, or apparent source nonuniformity.

Fit Selected Pixel

Use this button to fit the microwave spectrum displayed on the GSFIT Middle Panel using the selected frequency range and the settings displayed on the GSFIT Settings panel.

Fit Range

Use this button to perform one or more of the following actions:

• Define a rectangular ROI starting at the current cursor position, if no ROI has been previously defined
• Select a time range and time resolution for producing a sequence of fits for each ROI pixel
• Ignore the current ROI and Time Range selections and choose instead to fit the entire map cube
• Use or change the current FIT Frequency Range selection
• Enqueue and start processing the fitting of all ROI pixels over the user-defined time interval (default)
• Enqueue and Wait for future processing of the fit task of all ROI pixels over the user-defined time interval
• Generate and save the fit task for future batch processing by the GSFIT Command Prompt (GSFITCP) application.

Depending on whether or not a ROI is already defined, the user is exposed to a different dialog box to define a fitting task.

• 

  Fit Task Dialog Box if no ROI has been already defined

• 

  Fit Task Dialog Box if a ROI has been already defined

To enqueue the fit tasks for immediate or future processing by GSFIT, the following rules are followed:

• All ROI spectra from a given time frame are processed simultaneously by being as evenly as possible assigned to all active asynchronous parallel threads

• The individual time frames are enqueued and processed sequentially

An example about how the tasks are scheduled is provided in the GSFIT Task Queue section.

NOTE: If a fit task is saved for future batch processing, the scheduling is performed by the GSFIT Command Prompt (GSFITCP) application using the number of active parallel threads available to it.

Process Pending Tasks

Use this button to start processing all pending tasks in the GSFIT Queue

Abort/Delete Tasks

Use this button to select one of the options from the Delete Options dialog box.

GSFIT Left Panel

• 

  A: Selected EOVSA map displayed with no Integrated Flux Threshold applied

• 

  B: Selected EOVSA map displayed with 1 sfu Integrated Flux Threshold applied

GSFIT Middle Panel

• 
• 
• 

GSFIT Right Panel

GSFIT Task Queue

This page displays the current status of the already processed, active, or pending fit tasks. The GSFIT top menu toolbar provides several options for controlling the GSFIT Task Queue.

Here we provide an example of how the tasks are enqueued and processed in two diffrent circumstances:

If a 4x4 ROI; 4 time frames is enqueued while only 1 parallel thread is active, only 16 pixels from the same time frame are processed at any given time. If, instead, 4 active threads are used, batches of 4 pixels are processed by 4 asynchronous threads at any given time. Note that, in the case of 4 active threads, the time needed to fully process a given time frame is dictated by the longest time needed to process one of the 4 pixel batched of that frame, which is about half the time needed to process all 16 pixels in a single thread.

• 

  A: GSFIT Queue active processing: 4x4 ROI; 4 time frames; 1 parallel thread

• 

  B: GSFIT Fit Queue active processing: 4x4 ROI; 4 time frames; 4 parallel threads

• 

  C: GSFIT Fit Queue finalized task: 4x4 ROI; 4 time frames; 4 parallel threads