EOVSA Wiki - User contributions [en]

Radio Data from Around the World

2023-10-18T10:07:12Z

Schhabra: /* Radio Data Access */

== Radio Data Access ==

{| class="wikitable" style="text-align: center; width: 1300px; height: 200px;"
|-
| '''Instrument''' || style="width: 400pt;" | '''Frequency Range/Central Frequencies''' ||style="width: 700pt;" |'''Cadence''' || '''Observing Time (UT)'''|| style="width: 250pt;" |'''Dates of Operation (start/end)''' || '''FOV''' || style="width: 130pt;" | '''Imaging FOV / Resolution''' || style="width: 250pt;" |'''Spectroscopy Frequency Resolution''' || style="width: 250pt;" | '''Useful Links''' || '''Points of Contact for Data '''
|-

| ALMA || 92-96, 104-108, 190-194, 202-206, 229-233, 245-249, 338.6-342.6, 350.6-345.6 GHz || 1 s || 13 - 21 || 2016 - ongoing || Sun, sub active region scales || 6.54-66″/0.62-11.6″ || 2 GHz || [https://ui.adsabs.harvard.edu/abs/2022FrASS...9.7368B/abstract Harvard] || Stephen White (stephen.white.24@spaceforce.mil) Tim Bastian (tbastian@nrao.edu)
|-



| EOVSA || 1 - 18 GHz || Sample time: 20 ms: Full Sweep: 1 s || ~14 - 24 || 2017 - ongoing || Full Sun ||57″/<math>\nu_{GHz}</math> x 51″/<math>\nu_{GHz}</math> || 0.500 GHz || [http://ovsa.njit.edu//wiki/index.php You Are Here] [http://ovsa.njit.edu/browser/?suntoday_date=2023-08-06 EOVSA Browser] || Dale Gary (dgary@njit.edu) Bin Chen (bin.chen@njit.edu)
|-

| F10.7|| 2.8 GHz || 3 times daily || 17 (18), 20, 23 (22) || 1947 - ongoing || Full Sun, unresolved || N/A || N/A || [https://www.spaceweather.gc.ca/forecast-prevision/solar-solaire/solarflux/sx-en.php Space Weather Monitor] || Sam Schonfeld (schonfsj@gmail.com)
|-



| LOFAR || LBA: 10 - 80 MHz, HBA: 120 - 240 MHz || 10ms || 8.5-16 || Time Allocation by Proposal|| [https://drive.google.com/file/d/1YaBnSdzuEvxWee-wUAUZcUzxiaO1524J/view?usp=sharing FOV Info] || [https://drive.google.com/file/d/1YaBnSdzuEvxWee-wUAUZcUzxiaO1524J/view?usp=sharing FOV Info] || 10 kHz || [https://docs.google.com/document/d/103S1ANjTPXDIWnOAlhEAbQy2oLm5ZOcqV8x1eR0hvxs/edit LOFAR Info Doc] || Pietro Zucca (zucca@astron.nl) 
Anshu Kumari (anshusingh628@gmail.com)
|-

| LWA-NM || 10-88 MHz || < 1 s || 15 - 23 || April 2011 - ongoing|| Full-sky || All-sky, ~2 - 15° (zenith) || 7.8 - 612 kHz || [https://leo.phys.unm.edu/~lwa/specs.html LWA Spec] || Stephen White (stephen.white.24@spaceforce.mil)
|-



| MWA || 80 - 300 MHz || 0.5 s || 01 - 08 || 2014 - ongoing || 30° || depends on configuration; ~1 - 2′/200 MHz || 10/40 kHz || [https://asvo.mwatelescope.org/ MWA]|| Surajit Mondal (surajit.mondal@njit.edu)
|-

| NoRP || 1, 2, 3.75, 9.4, 17, 35, 80 GHz || 0.1 s || 23 - 7 || ~1979 - ongoing (3.75 GHz since 1951) || Full Sun, unresolved || N/A || N/A || [https://solar.nro.nao.ac.jp/norp/index.html NRO] || Stephen White (stephen.white.24@spaceforce.mil)
|-

| NoRH || 17, 34 GHz || 1 s, daily || 22.5 - 6.5 || 7/1/1992 - 3/31/2020|| Full Sun || 10″, 5″ || N/A || [https://hinode.isee.nagoya-u.ac.jp/ICCON/ Hinode Site] || Stephen White (stephen.white.24@spaceforce.mil), Nat Gopalswamy (nat.gopalswamy@nasa.gov)
|-

| OVRO-LWA || 15-88 MHz || 1 ms (dynamic spectrum); 100 ms (fast visibilities); 10 s (regular mode) || 14 - 24 || under commissioning (some data already available) || Full Sky || 5′ at 80 MHz || 24 kHz (dynamic spectrum and regular mode), 100 kHz (fast visibilities) || [link to come] || Dale Gary (dgary@njit.edu) Bin Chen (bin.chen@njit.edu)
|-

| PSP/FIELDS || LFR: 10 kHz - 2.4 MHz HFR:1.6 kHz - 19.2 MHz || 3.5–56 s (higher during [https://sppgway.jhuapl.edu/encounters encounters]) || 0 - 24 || 2018/10/02 - Present (during PSP Perihelia) || N/A || N/A || Non-uniform, Δν/ν ~ 4.5% || [https://sprg.ssl.berkeley.edu/data/psp/data/sci/fields/ FIELDS data] [https://fields.ssl.berkeley.edu/data/ FIELDS SOC] [https://cdaweb.gsfc.nasa.gov/ CDAW for FIELDS] [https://parker.gsfc.nasa.gov/crocs.html Coordinated Radiodiagnostics]|| Marc Pulupa (pulupa@berkeley.edu) Sam Badman (samuel.badman@cfa.harvard.edu)
|-



| RSTN || 245, 410, 610, 1415, 4995, 8800, 15400 MHz || 1 s || 0 - 24 || 1980 - ongoing|| Full Sun, unresolved || N/A || N/A || [https://www.ngdc.noaa.gov/stp/space-weather/solar-data/solar-features/solar-radio/rstn-1-second/ NOAA], [https://www.sws.bom.gov.au/World_Data_Centre/2/8/9 SWS] || Stephen White (stephen.white.24@spaceforce.mil)
|-



| VLA || 1-18 GHz; 0.3 GHz (P-Band) || 50 ms - 5 s, depending on observing frequency || 15 - 23 || from 1990s; based on proposals || 42′ at 1 GHz || 1.3 - 46″ @ 1.3 GHz depending on configuration|| Depends on configuration || [https://data.nrao.edu/portal/#/ NRAO Archive] [http://www.ovsa.njit.edu/wiki/index.php/VLA_Data_Survey Data Survey]|| Tim Bastian (tbastian@nrao.edu)
|-
| NRH || 150, 173, 228, 270, 298, 327, 327.5, 408, 432, 445 MHz || 125 ms || 8 - 15 || Ongoing || Full Sun || 6′ @ 150 MHz, 2′ @ 450 MHz || || [https://secchirh.obspm.fr/ NRH] || Ludwig Klein (ludwig.klein@obspm.fr) Abdallah Hamini (abdallah.hamini@obspm.fr)
|-

| E-Callisto || Varies: Multiple Instruments Throughout World || Varies || Varies || Ongoing || Full Sun - Unresolved || Varies || Varies || [https://www.e-callisto.org/Data/data.html E-Callisto] || N/A
|-

|}

== Other Useful Resources ==

# The [https://lesia.obspm.fr/cesra/sites.html CESRA-related data centers] is another place where you can find a list of different Radio instruments.
# The plots below show the Observation times of different instruments:

[[file:Solar_ghz.png|thumb|center|600px|'''Figure 1:''' Radio Instruments (Solar Dedicated and Proposal based observing between 1 - 40 GHz) (Image Credit: Anshu Kumari)]]

[[file:Solarplus_mhz.png|thumb|center|600px|'''Figure 2:''' Radio Instruments (Solar Dedicated and Proposal based observing below 1 GHz) (Image Credit: Anshu Kumari)]]

[[file:Facilities.png|thumb|center|1000px|'''Figure 3:''' Radio Instruments (Solar Dedicated and Proposal based observing between 10 MHz - 10 GHz) (Image Credit: Bin Chen)]]

Owens Valley Solar Arrays

2023-08-14T18:40:49Z

Schhabra: /* Radio Data from Around The Heliosphere */

[[File:Eovsa1.png|border|text-top|800px]]

<big>[http://ovsa.njit.edu/ EOVSA] (Expanded Owens Valley Solar Array) is a solar-dedicated radio interferometer operated by the New Jersey Institute of Technology and serving as a '''National Science Foundation Geospace Facility'''. [[File:NSF.jpg|70px]]
<pre>Operation of EOVSA is supported by the National Science Foundation under Grant No. AGS-2130832. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the National Science Foundation. </pre>
This wiki serves as the site for EOVSA documentation. </big>

== EOVSA Documentation ==

* <big>General</big>
** [[Downconversion and Frequency Tuning]]
** [[Dealing with Radio Frequency Interference]]
** [[Switching between 200 MHz and 300 MHz Correlator]]
** [[Observing in "Fast" Mode]]

* <big>Computer-Network</big>
** [[Computing Systems]]
** [[Network]]

* <big>Control System</big>
** [[27-m Antenna Commands]]
** [[Schedule Commands]]
** [[Control Commands]]
** [[Attenuation and Level Control]]

* <big>Hardware</big>
** [[Hardware Overview]]
** [[2.1-m Antennas]]
** [[27-m Antennas]]

* <big>System Software</big>
** [[Calibration Database]]
** [[Stateframe Database]]
** [[Database Maintenance]]
** [[Create CASA measurement sets]]

* <big>Calibration</big>
**[[Calibration Overview]]
**[[Pointing Calibration]]
**[[Total Power Calibration]]
**[[System Gain Calibration]]
**[[Antenna Position]] (Baseline Calibration)
**[[Reference Gain Calibration]]
**[[Daily Gain Calibration]]
**[[Delay Calibration]]
**[[Bandpass Calibration]]
**[[Polarization Calibration]]
**[[Calibrator Survey]]
**[[Practical Calibration Tutorial]]

* <big>[[Starburst]]</big>

== Using EOVSA Data ==
* <big>[[EOVSA Data products]]</big>
* <big>Analysis Software</big>
** [https://github.com/suncasa/suncasa SunCASA] A wrapper around [https://casa.nrao.edu/ CASA (the Common Astronomy Software Applications package)] for synthesis imaging and visualizing solar spectral imaging data. CASA is one of the leading software tool for "supporting the data post-processing needs of the next generation of radio astronomical telescopes such as ALMA and VLA", an international effort led by the [https://public.nrao.edu/ National Radio Astronomy Observatory]. The current version of CASA uses Python (2.7) interface. More information about CASA can be found on [https://casa.nrao.edu/ NRAO's CASA website ]. Note, CASA is available ONLY on UNIX-BASED PLATFORMS (and therefore, so is SunCASA).
** [https://github.com/Gelu-Nita/GSFIT GSFIT] A IDL-widget(GUI)-based spectral fitting package called gsfit, which provides a user-friendly display of EOVSA image cubes and an interface to fast fitting codes (via platform-dependent shared-object libraries).
** [[Spectrogram Software]]
** [[Mapping Software]]
* <big>Data Analysis Guides</big>
** <big>[[EOVSA Data Analysis Tutorial 2022]]</big> and <big>[https://colab.research.google.com/drive/19NQb6Emb9HvKX4QHq9ZYCP3RM6nT7sDL#scrollTo=cLdDVptBGG-X EOVSA Workspace]</big> at [https://sphere.boulder.swri.edu/ SPHERE 2022 Workshop]
** <big>[https://colab.research.google.com/drive/1lSLLxgOG6b8kgu9Sk6kSKvrViyubnXG6?usp=sharing#scrollTo=xbXyyLmCFCGL EOVSA Data Analysis Tutorial at RHESSI 19 Workshop]</big>
** <big>[[EOVSA Data Analysis Tutorial]]</big> at [http://rhessi18.umn.edu/ RHESSI XVIII Workshop]
** [[Self-Calibrating Flare Data]] Example script and guides for self-calibrating EOVSA flare data (to be completed)


**[[IDB flare pipeline]] Tutorial to run the flare pipeline for quicklook images



* <big>EOVSA Modeling Guide</big>
**[[GX Simulator]]

* Other helpful links
** [https://casaguides.nrao.edu CASA Guides]
** [http://www.lmsal.com/solarsoft/ SolarSoft IDL]
** [http://www.atnf.csiro.au/computing/software/miriad/userguide/userhtml.html Miriad Guides]
** [https://sites.google.com/site/fgscodes/ Fast Gyrosynchrotron Codes (Alexey Kuznetsov's website)]
** [[Basic GitHub Tutorial]]


* <big>[[Full Disk Simulations]]</big>
* <big>[[All-Day Synthesis Issues]]</big>
* <big>[[Analyzing Pre-2017 Data]]</big>
* <big>[[Fixing Pipeline Problems pre-2021-Feb-07]]</big>

== System Software ==

* LabVIEW software
* Python code [https://github.com/dgary50/eovsa Github repository]
* [[Python3 Code Installation]]

== Observing Log ==
[[2016 November]]; [[2016 December| December]]

[[2017 January]]; [[2017 February | February]]; [[2017 March | March]]; [[2017 April | April]]; [[2017 May | May]]; [[2017 June | June]];
[[2017 July | July]]; [[2017 August | August]]; [[2017 September | September]]; [[2017 October | October]]; [[2017 November | November]]; [[2017 December | December]]

[[2018 January]]; [[2018 February | February]]; [[2018 March | March]]; [[2018 April | April]]; [[2018 May | May]]; [[2018 June | June]];
[[2018 July | July]]; [[2018 August | August]]; [[2018 September | September]]; [[2018 October | October]]; [[2018 November | November]]; [[2018 December | December]]

[[2019 January]]; [[2019 February | February]]; [[2019 March | March]]; [[2019 April | April]]; [[2019 May | May]]; [[2019 June | June]];
[[2019 July | July]]; [[2019 August | August]]; [[2019 September | September]]; [[2019 October | October]]; [[2019 November | November]]; [[2019 December | December]]

[[2020 January]]; [[2020 February | February]]; [[2020 March | March]]; [[2020 April | April]]; [[2020 May | May]]; [[2020 June | June]];
[[2020 July | July]]; [[2020 August | August]]; [[2020 September | September]]; [[2020 October | October]]; [[2020 November | November]]; [[2020 December | December]]

[[2021 January]]; [[2021 February | February]]; [[2021 March | March]]; [[2021 April | April]]; [[2021 May | May]]; [[2021 June | June]];
[[2021 July | July]]; [[2021 August | August]]; [[2021 September | September]]; [[2021 October | October]]; [[2021 November | November]]; [[2021 December | December]]

[[2022 SQL Outage]]

[[2023 January]]; [[2023 February | February]]; [[2023 March | March]]; [[2023 April | April]]; [[2023 May | May]]; [[2023 June | June]];
[[2023 July | July]]; [[2023 August | August]]; [[2023 September | September]]; [[2023 October | October]]; [[2023 November | November]]; [[2023 December | December]]

== Tohbans ==

[[Trouble Shooting Guide]]

[[Tohban Records]]

[[Owen's Notes]]

[[Caius' Notes]]

[[Tohban EOVSA Imaging Tutorial A-Z]]

[[Tohban Guide to Self Calibration and Imaging for EOVSA]]

[[Guide to Upgrade SolarSoft(SSW)]]

== EOVSA Flare List ==

See [https://docs.google.com/spreadsheets/d/1P8jHuDRF93dMflU6RMQcsJqVepD9vFkPkofV8Imj4xA/edit?usp=sharing this link] for a list of EOVSA flares as a Google Spreadsheet.

[[Recent Flare List (2021-)]]

[http://ovsa.njit.edu/jay/rd_db.php An older link] is available at the EOVSA website.

== EOVSA Publications ==
Here is a (partial) list of publications that utilize EOVSA data. See also the collection of EOVSA publications at [https://ui.adsabs.harvard.edu/public-libraries/eQ7HfPkySqydu-B8BCt6QQ this NASA/ADS Library].
; 2023
: [https://ui.adsabs.harvard.edu/abs/2023arXiv230107840M/abstract Mondal, S., Chen, B. & Yu, S. (2023) ApJ, submitted] ''Multifrequency microwave imaging of weak transients from the quiet solar corona''
; 2022
: [https://ui.adsabs.harvard.edu/abs/2022FrASS...940945L/abstract Lörinčík et al (2022) Frontiers, 9, 1] ''Rapid variations of Si IV spectra in a flare observed by interface region imaging spectrograph at a sub-second cadence''
: [https://ui.adsabs.harvard.edu/abs/2022NatCo..13.7680K/abstract Kou et al. (2022) Nature Communications 13, 7680] ''Microwave imaging of quasi-periodic pulsations at flare current sheet''
: [https://ui.adsabs.harvard.edu/abs/2022Natur.606..674F/abstract Fleishman et al. (2022) Nature 606, 674] ''Solar flare accelerates nearly all electrons in a large coronal volume''
: [https://ui.adsabs.harvard.edu/abs/2022ApJ...932...92L/abstract Li, X., et al., (2022) ApJ, 932, 92] ''Modeling Electron Acceleration and Transport in the Early Impulsive Phase of the 2017 September 10th Solar Flare''
: [https://ui.adsabs.harvard.edu/abs/2022ApJ...930..154L/abstract Liu, N., et al., (2022), ApJ, 930, 154] ''Multi-instrument Comparative Study of Temperature, Number Density, and Emission Measure during the Precursor Phase of a Solar Flare''
: [https://ui.adsabs.harvard.edu/abs/2022arXiv220503518Z/abstract Zhang et al. (2022), ApJ, 932, 53] ''Implications for additional plasma heating driving the extreme-ultraviolet late phase of a solar flare with microwave imaging spectroscopy''
: [https://ui.adsabs.harvard.edu/abs/2022A%26A...657A..51L/abstract Lopez et al. (2021), A&A, 657, A51] ''A solar flare driven by thermal conduction observed in mid-infrared''
; 2021
: [https://ui.adsabs.harvard.edu/abs/2021ApJ...923..213W/abstract Wei et al. (2021), ApJ, 923, 213] ''Coronal Magnetic Field Measurements along a Partially Erupting Filament in a Solar Flare''
: [https://ui.adsabs.harvard.edu/abs/2021ApJ...919...44S/abstract Shaik & Gary (2021), ApJ, 919, 44] ''Implications of Flat Optically Thick Microwave Spectra in Solar Flares for Source Size and Morphology''
: [https://ui.adsabs.harvard.edu/abs/2021ApJ...915...12K/abstract Kocharov et al. (2021), ApJ, 915, 12] ''Multiple Sources of Solar High-energy Protons''
: [https://ui.adsabs.harvard.edu/abs/2021ApJ...908L..55C/abstract Chen et al. (2021), ApJL, 908, L55] ''Energetic Electron Distribution of the Coronal Acceleration Region: First results from Joint Microwave and Hard X-ray Imaging Spectroscopy''
: [https://ui.adsabs.harvard.edu/abs/2021ApJ...906..132C/abstract Chhabra et al. (2021), ApJ, 906, 132] ''Imaging Spectroscopy of CME-Associated Solar Radio Bursts''
; 2020
: [https://ui.adsabs.harvard.edu/abs/2020ApJ...905..165R/abstract Reeves et al. (2020), ApJ, 905, 165] ''Hot Plasma Flows and Oscillations in the Loop-top Region During the September 10 2017 X8.2 Solar Flare''
: [https://ui.adsabs.harvard.edu/abs/2020ApJ...900...17Y/abstract Yu et al. (2020), ApJ, 900, 17] ''Magnetic Reconnection During the Post Impulsive Phase of the X8.2 Solar Flare: Bi-Directional Outflows as a Cause of Microwave and X-ray Bursts''
: [https://ui.adsabs.harvard.edu/abs/2020NatAs...4.1140C/abstract Chen et al. (2020b), Nature Astronomy, 4, 1140] ''Measurement of magnetic field and relativistic electrons along a solar flare current sheet''
: [https://ui.adsabs.harvard.edu/abs/2020ApJ...895L..50C/abstract Chen et al. (2020a), ApJL, 895, 50] ''Microwave Spectral Imaging of an Erupting Magnetic Flux Rope: Implications for the Standard Solar Flare Model in Three Dimensions''
: [https://ui.adsabs.harvard.edu/abs/2020FrASS...7...22K/abstract Kuroda et al. (2020), Frontiers, 7, 22] ''Evolution of Flare-accelerated Electrons Quantified by Spatially Resolved Analysis''
: [https://ui.adsabs.harvard.edu/abs/2020ApJ...891L..34G/abstract Glesener et al. (2020), ApJL, 891, 34] ''Accelerated Electrons Observed Down to <7 keV in a NuSTAR Solar Microflare''
: [https://ui.adsabs.harvard.edu/abs/2020ApJ...889...72K/abstract Karlicky at al. (2020), ApJ, 889, 72] ''Drifting Pulsation Structure at the Very Beginning of the 2017 September 10 Limb Flare''
: [https://ui.adsabs.harvard.edu/abs/2020Sci...367..278F/abstract Fleishman et al. (2020), Science, 367, 278] ''Decay of the coronal magnetic field can release sufficient energy to power a solar flare''
: [https://ui.adsabs.harvard.edu/abs/2020AAS...23538501G/abstract Gary et al. (2020), BAAS 52, 385.01] [https://aas235-aas.ipostersessions.com/default.aspx?s=97-69-9E-4B-34-19-68-53-1B-C6-21-0C-16-1C-5C-82&guestview=true Direct link to AAS iPoster] ''A new view of the solar atmosphere: daily full-disk multifrequency radio images from EOVSA''
; 2018
: [https://ui.adsabs.harvard.edu/#abs/2018ApJ...864...63P/abstract Polito et al. (2018), ApJ, 864, 63] ''Broad Non-Gaussian Fe XXIV Line Profiles in the Impulsive Phase of the 2017 September 10 X8.3-class Flare Observed by Hinode/EIS''
: [https://ui.adsabs.harvard.edu/#abs/2018ApJ...863...83G/abstract Gary et al. (2018), ApJ, 863, 83] ''Microwave and Hard X-Ray Observations of the 2017 September 10 Solar Limb Flare''
: [https://ui.adsabs.harvard.edu/#abs/2018ApJ...852...32K/abstract Kuroda et al. (2018), ApJ, 852, 32] ''Three-dimensional Forward-fit Modeling of the Hard X-ray and the Microwave Emissions of the 2015 June 22 M6.5 flare''
; 2017
: [https://ui.adsabs.harvard.edu/abs/2017NatAs...1E..85W/abstract Wang et al. (2017), Nature Astronomy, 1, 85] ''High-resolution observations of flare precursors in the low solar atmosphere''
; 2016
: [https://ui.adsabs.harvard.edu/abs/2016JAI.....541009N/abstract Nita et al. (2016), J. Astron. Instr., 5, 1641009-7366] ''EOVSA Implementation of a Spectral Kurtosis Correlator for Transient Detection and Classification''

== VLA Flare List and Publications ==
See [http://www.ovsa.njit.edu/wiki/index.php/VLA_Data_Survey#List_of_Jansky_VLA_Solar_Observations this link] for a list of flare observations made by the [https://science.nrao.edu/facilities/vla/ Karl G. Jansky Very Large Array] (VLA). Below is a partial list of publications that utilize VLA solar data (see also [https://ui.adsabs.harvard.edu/public-libraries/ZwbjpLo9RS-viufWEoQ95Q this NASA/ADS Library]).
* [https://ui.adsabs.harvard.edu/abs/2022ApJ...940..137L/abstract Luo et al. (2022), ApJ, 940, 137] ''Multiple Regions of Nonthermal Quasiperiodic Pulsations during the Impulsive Phase of a Solar Flare''
* [https://ui.adsabs.harvard.edu/abs/2021ApJ...922..134B/abstract Battaglia et al. (2021), ApJ, 922, 134] ''Multiple Electron Acceleration Instances during a Series of Solar Microflares Observed Simultaneously at X-Rays and Microwaves''
* [https://ui.adsabs.harvard.edu/abs/2021ApJ...911....4L/abstract Luo et al. (2021), ApJ, 911, 4] ''Radio Spectral Imaging of an M8.4 Eruptive Solar Flare: Possible Evidence of a Termination Shock''
* [https://ui.adsabs.harvard.edu/abs/2021ApJ...910...40Z/abstract Zhang et al. (2021), ApJ, 910, 40] ''Multiwavelength Observations of the Formation and Eruption of a Complex Filament''
* [https://ui.adsabs.harvard.edu/abs/2020ApJ...904...94S/abstract Sharma et al. (2020), ApJ, 904, 94] ''Radio and X-Ray Observations of Short-lived Episodes of Electron Acceleration in a Solar Microflare''
* [https://ui.adsabs.harvard.edu/abs/2019ApJ...884...63C/abstract Chen et al. (2019), ApJ, 884, 63] ''Radio Spectroscopic Imaging of a Solar Flare Termination Shock: Split-band Feature as Evidence for Shock Compression''
* [https://ui.adsabs.harvard.edu/abs/2019ApJ...872...71Y/abstract Yu & Chen (2019), ApJ, 872, 71] ''Possible Detection of Subsecond-period Propagating Magnetohydrodynamics Waves in Post-reconnection Magnetic Loops during a Two-ribbon Solar Flare''
* [https://ui.adsabs.harvard.edu/abs/2018ApJ...866...62C/abstract Chen et al. (2018), ApJ, 866, 62] ''Magnetic Reconnection Null Points as the Origin of Semirelativistic Electron Beams in a Solar Jet
''
* [https://ui.adsabs.harvard.edu/abs/2017ApJ...848...77W/abstract Wang et al. (2016), ApJ, 848, 77] ''Dynamic Spectral Imaging of Decimetric Fiber Bursts in an Eruptive Solar Flare''
* [https://ui.adsabs.harvard.edu/abs/2015Sci...350.1238C/abstract Chen et al. (2015), Science, 350, 1238] ''Particle acceleration by a solar flare termination shock''
* [https://ui.adsabs.harvard.edu/abs/2014ApJ...794..149C/abstract Chen et al. (2014), ApJ, 794, 149] ''Direct Evidence of an Eruptive, Filament-hosting Magnetic Flux Rope Leading to a Fast Solar Coronal Mass Ejection''
* [https://ui.adsabs.harvard.edu/abs/2013ApJ...763L..21C/abstract Chen et al. (2013), ApJL, 763, 21] ''Tracing Electron Beams in the Sun's Corona with Radio Dynamic Imaging Spectroscopy''

==Radio Data from Around The Heliosphere==
* [http://ovsa.njit.edu//wiki/index.php/Radio_Data_from_Around_the_World#Radio_Data_Access '' Radio Data '']

Owens Valley Solar Arrays

2023-08-14T18:40:17Z

Schhabra: /* Radio Data from Around The Heliosphere */

Owens Valley Solar Arrays

2023-08-14T18:39:10Z

Schhabra:

Radio Data from Around the World

2023-08-09T03:55:24Z

Schhabra: /* Radio Data Access */

== Radio Data Access ==

{| class="wikitable" style="text-align: center; width: 1300px; height: 200px;"
|-
| '''Instrument''' || style="width: 400pt;" | '''Frequency Range/Central Frequencies''' ||style="width: 700pt;" |'''Cadence''' || '''Observing Time (UT)'''|| style="width: 250pt;" |'''Dates of Operation (start/end)''' || '''FOV''' || style="width: 130pt;" | '''Imaging FOV / Resolution''' || style="width: 250pt;" |'''Spectroscopy Frequency Resolution''' || style="width: 250pt;" | '''Useful Links''' || '''Points of Contact for Data '''
|-

| ALMA || 92-96, 104-108, 190-194, 202-206, 229-233, 245-249, 338.6-342.6, 350.6-345.6 GHz || 1 s || 13 - 21 || 2016 - ongoing || Sun, sub active region scales || 6.54-66″/0.62-11.6″ || 2 GHz || [https://ui.adsabs.harvard.edu/abs/2022FrASS...9.7368B/abstract Harvard] || Stephen White (stephen.white.24@spaceforce.mil) Tim Bastian (tbastian@nrao.edu)
|-



| EOVSA || 1 - 18 GHz || Sample time: 20 ms: Full Sweep: 1 s || ~14 - 24 || 2017 - ongoing || Full Sun ||57″/<math>\nu_{GHz}</math> x 51″/<math>\nu_{GHz}</math> || 0.500 GHz || [http://ovsa.njit.edu//wiki/index.php You Are Here] [http://ovsa.njit.edu/browser/?suntoday_date=2023-08-06 EOVSA Browser] || Dale Gary (dgary@njit.edu) Bin Chen (bin.chen@njit.edu)
|-

| F10.7|| 2.8 GHz || 3 times daily || 17 (18), 20, 23 (22) || 1947 - ongoing || Full Sun, unresolved || N/A || N/A || [https://www.spaceweather.gc.ca/forecast-prevision/solar-solaire/solarflux/sx-en.php Space Weather Monitor] || Sam Schonfeld (schonfsj@gmail.com)
|-



| LOFAR || LBA: 10 - 80 MHz, HBA: 120 - 240 MHz || 10ms || 8.5-16 || Time Allocation by Proposal|| [https://drive.google.com/file/d/1YaBnSdzuEvxWee-wUAUZcUzxiaO1524J/view?usp=sharing FOV Info] || [https://drive.google.com/file/d/1YaBnSdzuEvxWee-wUAUZcUzxiaO1524J/view?usp=sharing FOV Info] || 10 kHz || [https://docs.google.com/document/d/103S1ANjTPXDIWnOAlhEAbQy2oLm5ZOcqV8x1eR0hvxs/edit LOFAR Info Doc] || Pietro Zucca (zucca@astron.nl) 
Anshu Kumari (anshusingh628@gmail.com)
|-

| LWA-NM || 10-88 MHz || < 1 s || 15 - 23 || April 2011 - ongoing|| Full-sky || All-sky, ~2 - 15° (zenith) || 7.8 - 612 kHz || [https://leo.phys.unm.edu/~lwa/specs.html LWA Spec] || Stephen White (stephen.white.24@spaceforce.mil)
|-



| MWA || 80 - 300 MHz || 0.5 s || 01 - 08 || 2014 - ongoing || 30° || depends on configuration; ~1 - 2′/200 MHz || 10/40 kHz || [https://asvo.mwatelescope.org/ MWA]|| Surajit Mondal (surajit.mondal@njit.edu)
|-

| NoRP || 1, 2, 3.75, 9.4, 17, 35, 80 GHz || 0.1 s || 23 - 7 || ~1979 - ongoing (3.75 GHz since 1951) || Full Sun, unresolved || N/A || N/A || [https://solar.nro.nao.ac.jp/norp/index.html NRO] || Stephen White (stephen.white.24@spaceforce.mil)
|-

| NoRH || 17, 34 GHz || 1 s, daily || 22.5 - 6.5 || 7/1/1992 - 3/31/2020|| Full Sun || 10″, 5″ || N/A || [https://hinode.isee.nagoya-u.ac.jp/ICCON/ Hinode Site] || Stephen White (stephen.white.24@spaceforce.mil), Nat Gopalswamy (nat.gopalswamy@nasa.gov)
|-

| OVRO-LWA || 15-88 MHz || 1 ms (dynamic spectrum); 100 ms (fast visibilities); 10 s (regular mode) || 14 - 24 || under commissioning (some data already available) || Full Sky || 5′ at 80 MHz || 24 kHz (dynamic spectrum and regular mode), 100 kHz (fast visibilities) || [link to come] || Dale Gary (dgary@njit.edu) Bin Chen (bin.chen@njit.edu)
|-

| PSP/FIELDS || LFR: 10 kHz - 2.4 MHz HFR:1.6 kHz - 19.2 MHz || 7 s || 0 - 24 || 2018/10/02 - Present during PSP Perihelia || N/A || N/A || Non-uniform, Δν/ν ~ 4.5% || [https://sprg.ssl.berkeley.edu/data/psp/data/sci/fields/ FIELDS data] [https://sprg.ssl.berkeley.edu/data/psp/data/sci/fields/ Data Release Notes] || Marc Pulupa (pulupa@berkeley.edu) Sam Badman (samuel.badman@cfa.harvard.edu)
|-



| RSTN || 245, 410, 610, 1415, 4995, 8800, 15400 MHz || 1 s || 0 - 24 || 1980 - ongoing|| Full Sun, unresolved || N/A || N/A || [https://www.ngdc.noaa.gov/stp/space-weather/solar-data/solar-features/solar-radio/rstn-1-second/ NOAA], [https://www.sws.bom.gov.au/World_Data_Centre/2/8/9 SWS] || Stephen White (stephen.white.24@spaceforce.mil)
|-



| VLA || 1-18 GHz; 0.3 GHz (P-Band) || 50 ms - 5 s, depending on observing frequency || 15 - 23 || from 1990s; based on proposals || 42′ at 1 GHz || 1.3 - 46″ @ 1.3 GHz depending on configuration|| Depends on configuration || [https://data.nrao.edu/portal/#/ NRAO Archive] [http://www.ovsa.njit.edu/wiki/index.php/VLA_Data_Survey Data Survey]|| Tim Bastian (tbastian@nrao.edu)
|-
| NRH || 150, 173, 228, 270, 298, 327, 327.5, 408, 432, 445 MHz || 125 ms || 8 - 15 || Ongoing || Full Sun || 6′ @ 150 MHz, 2′ @ 450 MHz || || [https://secchirh.obspm.fr/ NRH] || L. Klein (ludwig.klein@obspm.fr) and A. Hamini (abdallah.hamini@obspm.fr)
|-

| E-Callisto || Varies: Multiple Instruments Throughout World || Varies || Varies || Ongoing || Full Sun - Unresolved || Varies || Varies || [https://www.e-callisto.org/Data/data.html E-Callisto] || N/A
|-

|}

== Other Useful Resources ==

# The [https://lesia.obspm.fr/cesra/sites.html CESRA-related data centers] is another place where you can find a list of different Radio instruments.
# The plots below show the Observation times of different instruments:

[[file:Solar_ghz.png|thumb|center|600px|'''Figure 1:''' Radio Instruments (Solar Dedicated and Proposal based observing between 1 - 40 GHz) (Image Credit: Anshu Kumari)]]

[[file:Solarplus_mhz.png|thumb|center|600px|'''Figure 2:''' Radio Instruments (Solar Dedicated and Proposal based observing below 1 GHz) (Image Credit: Anshu Kumari)]]

[[file:Facilities.png|thumb|center|1000px|'''Figure 3:''' Radio Instruments (Solar Dedicated and Proposal based observing between 10 MHz - 10 GHz) (Image Credit: Bin Chen)]]

Radio Data from Around the World

2023-08-09T03:36:35Z

Schhabra: /* Radio Data Access */

== Radio Data Access ==

{| class="wikitable" style="text-align: center; width: 1300px; height: 200px;"
|-
| '''Instrument''' || style="width: 400pt;" | '''Frequency Range/Central Frequencies''' ||style="width: 700pt;" |'''Cadence''' || '''Observing Time (UT)'''|| style="width: 250pt;" |'''Dates of Operation (start/end)''' || '''FOV''' || style="width: 130pt;" | '''Imaging FOV / Resolution''' || style="width: 250pt;" |'''Spectroscopy Frequency Resolution''' || style="width: 250pt;" | '''Useful Links''' || '''Points of Contact for Data '''
|-

| ALMA || 92-96, 104-108, 190-194, 202-206, 229-233, 245-249, 338.6-342.6, 350.6-345.6 GHz || 1s || 13 - 21 || 2016 - ongoing || Sun, sub active region scales || 6.54-66″ / 0.62-11.6″ || 2 GHz || [https://ui.adsabs.harvard.edu/abs/2022FrASS...9.7368B/abstract Harvard] || Stephen White (stephen.white.24@spaceforce.mil) Tim Bastian (tbastian@nrao.edu)
|-



| EOVSA || 1-18 GHz || Sample time: 20ms: Full Sweep: 1s || ~14-24 || 2017 - ongoing || Full Sun ||57″/<math>\nu_{GHz}</math> x 51″/<math>\nu_{GHz}</math> || 0.500 GHz || [http://ovsa.njit.edu//wiki/index.php You Are Here] [http://ovsa.njit.edu/browser/?suntoday_date=2023-08-06 EOVSA Browser] || Dale Gary (dgary@njit.edu) Bin Chen (bin.chen@njit.edu)
|-

| F10.7|| 2.8 GHz || 3 times daily || 17 (18), 20, 23 (22) || 1947 - ongoing || Full Sun, unresolved || N/A || N/A || [https://www.spaceweather.gc.ca/forecast-prevision/solar-solaire/solarflux/sx-en.php Space Weather Monitor] || Sam Schonfeld (schonfsj@gmail.com)
|-



| LOFAR || LBA: 10 - 80 MHz, HBA: 120 - 240 MHz || 10ms || 8.5-16 || Time Allocation by Proposal|| [https://drive.google.com/file/d/1YaBnSdzuEvxWee-wUAUZcUzxiaO1524J/view?usp=sharing FOV Info] || [https://drive.google.com/file/d/1YaBnSdzuEvxWee-wUAUZcUzxiaO1524J/view?usp=sharing FOV Info] || 10 kHz || [https://docs.google.com/document/d/103S1ANjTPXDIWnOAlhEAbQy2oLm5ZOcqV8x1eR0hvxs/edit LOFAR Info Doc] || Pietro Zucca (zucca@astron.nl) 
Anshu Kumari (anshusingh628@gmail.com)
|-

| LWA-NM || 10-88 MHz || < 1s || 15 - 23 || April 2011 - ongoing|| Full-sky || All-sky, ~2-15° (zenith) || 7.8 - 612 kHz || [https://leo.phys.unm.edu/~lwa/specs.html LWA Spec] || Stephen White (stephen.white.24@spaceforce.mil)
|-



| MWA || 80-300MHz || 0.5s || 01:00-08:00 || 2014 - ongoing || 30deg || depends on configuration; ~1-2′/200MHz || 10/40 kHz || [https://asvo.mwatelescope.org/ MWA]|| Surajit Mondal (surajit.mondal@njit.edu)
|-

| NoRP || 1, 2, 3.75, 9.4, 17, 35, 80 GHz || 0.1s || 23 - 7 || ~1979 - ongoing (3.75 GHz since 1951) || Full Sun, unresolved || N/A || N/A || [https://solar.nro.nao.ac.jp/norp/index.html NRO] || Stephen White (stephen.white.24@spaceforce.mil)
|-

| NoRH || 17, 34 GHz || 1s, daily || 22.5 - 6.5 || 7/1/1992 - 3/31/2020|| Full Sun || 10″, 5″ || N/A || [https://hinode.isee.nagoya-u.ac.jp/ICCON/ Hinode Site] || Stephen White (stephen.white.24@spaceforce.mil), Nat Gopalswamy (nat.gopalswamy@nasa.gov)
|-

| OVRO-LWA || 15-88 MHz || 1ms (dynamic spectrum); 100ms (fast visibilities); 10s (regular mode) || 14-24:00 || under commissioning (some data already available) || Full Sky || 5′ at 80MHz || 24kHz (dynamic spectrum and regular mode), 100 kHz (fast visibilities) || [link to come] || Dale Gary (dgary@njit.edu) Bin Chen (bin.chen@njit.edu)
|-

| PSP/FIELDS || LFR: 10 kHz-2.4 MHz HFR:1.6 kHz-19.2 MHz || 7s || 0-24 || 2018/10/02 - Present during PSP Perihelia || N/A || N/A || Non-uniform, Δν/ν ~ 4.5% || [https://sprg.ssl.berkeley.edu/data/psp/data/sci/fields/ FIELDS data] [https://sprg.ssl.berkeley.edu/data/psp/data/sci/fields/ Data Release Notes] || Marc Pulupa (pulupa@berkeley.edu) Sam Badman (samuel.badman@cfa.harvard.edu)
|-



| RSTN || 245, 410, 610, 1415, 4995, 8800, 15400 MHz || 1s || 0 - 24 || 1980 - ongoing|| Full Sun, unresolved || N/A || N/A || [https://www.ngdc.noaa.gov/stp/space-weather/solar-data/solar-features/solar-radio/rstn-1-second/ NOAA], [https://www.sws.bom.gov.au/World_Data_Centre/2/8/9 SWS] || Stephen White (stephen.white.24@spaceforce.mil)
|-



| VLA || 1-18 GHz; 0.3 GHz (P-Band) || 50 ms - 5 s, depending on observing frequency || 15 - 23 || from 1990s; based on proposals || 42′ at 1 GHz || 1.3-46 ″ @ 1.3GHz depending on configuration|| Depends on configuration || [https://data.nrao.edu/portal/#/ NRAO Archive] [http://www.ovsa.njit.edu/wiki/index.php/VLA_Data_Survey Data Survey]|| Tim Bastian (tbastian@nrao.edu)
|-
| NRH || 150, 173, 228, 270, 298, 327, 327.5, 408, 432, 445 MHz || 125 ms || 8 - 15 || Ongoing || Full Sun || 6′ @ 150 MHz, 2′ @ 450 MHz || || [https://secchirh.obspm.fr/ NRH] || L. Klein (ludwig.klein@obspm.fr) and A. Hamini (abdallah.hamini@obspm.fr)
|-

| E-Callisto || Varies: Multiple Instruments Throughout World || Varies || Varies || Ongoing || Full Sun - Unresolved || Varies || Varies || [https://www.e-callisto.org/Data/data.html E-Callisto] || N/A
|-

|}

== Other Useful Resources ==

# The [https://lesia.obspm.fr/cesra/sites.html CESRA-related data centers] is another place where you can find a list of different Radio instruments.
# The plots below show the Observation times of different instruments:

[[file:Solar_ghz.png|thumb|center|600px|'''Figure 1:''' Radio Instruments (Solar Dedicated and Proposal based observing between 1 - 40 GHz) (Image Credit: Anshu Kumari)]]

[[file:Solarplus_mhz.png|thumb|center|600px|'''Figure 2:''' Radio Instruments (Solar Dedicated and Proposal based observing below 1 GHz) (Image Credit: Anshu Kumari)]]

[[file:Facilities.png|thumb|center|1000px|'''Figure 3:''' Radio Instruments (Solar Dedicated and Proposal based observing between 10 MHz - 10 GHz) (Image Credit: Bin Chen)]]

Radio Data from Around the World

2023-08-09T03:32:13Z

Schhabra: /* Radio Data Access */

== Radio Data Access ==

{| class="wikitable" style="text-align: center; width: 1300px; height: 200px;"
|-
| '''Instrument''' || style="width: 400pt;" | '''Frequency Range/Central Frequencies''' ||style="width: 700pt;" |'''Cadence''' || '''Observing Time (UT)'''|| style="width: 250pt;" |'''Dates of Operation (start/end)''' || '''FOV''' || style="width: 130pt;" | '''Imaging FOV / Resolution''' || style="width: 250pt;" |'''Spectroscopy Frequency Resolution''' || style="width: 250pt;" | '''Useful Links''' || '''Points of Contact for Data '''
|-

| ALMA || 92-96, 104-108, 190-194, 202-206, 229-233, 245-249, 338.6-342.6, 350.6-345.6 GHz || 1s || 13 - 21 || 2016 - ongoing || Sun, sub active region scales || 6.54-66″ / 0.62-11.6″ || 2 GHz || [https://ui.adsabs.harvard.edu/abs/2022FrASS...9.7368B/abstract Harvard] || Stephen White (stephen.white.24@spaceforce.mil) Tim Bastian (tbastian@nrao.edu)
|-



| EOVSA || 1-18 GHz || Sample time: 20ms: Full Sweep: 1s || ~14-24 || 2017 - ongoing || Full Sun ||57″/<math>\nu_{GHz}</math> x 51″/<math>\nu_{GHz}</math> || 0.500 GHz || [http://ovsa.njit.edu//wiki/index.php You Are Here] [http://ovsa.njit.edu/browser/?suntoday_date=2023-08-06 EOVSA Browser] || Dale Gary (dgary@njit.edu) Bin Chen (bin.chen@njit.edu)
|-

| F10.7|| 2.8 GHz || 3 times daily || 17 (18), 20, 23 (22) || 1947 - ongoing || Full Sun, unresolved || N/A || N/A || [https://www.spaceweather.gc.ca/forecast-prevision/solar-solaire/solarflux/sx-en.php Space Weather Monitor] || Sam Schonfeld (schonfsj@gmail.com)
|-



| LOFAR || LBA: 10 - 80 MHz, HBA: 120 - 240 MHz || 10ms || 8.5-16 || Time Allocation by Proposal|| [https://drive.google.com/file/d/1YaBnSdzuEvxWee-wUAUZcUzxiaO1524J/view?usp=sharing FOV Info] || [https://drive.google.com/file/d/1YaBnSdzuEvxWee-wUAUZcUzxiaO1524J/view?usp=sharing FOV Info] || 10 kHz || [https://docs.google.com/document/d/103S1ANjTPXDIWnOAlhEAbQy2oLm5ZOcqV8x1eR0hvxs/edit LOFAR Info Doc] || Pietro Zucca (zucca@astron.nl) 
Anshu Kumari (anshusingh628@gmail.com)
|-

| LWA-NM || 10-88 MHz || < 1s || 15 - 23 || April 2011 - ongoing|| Full-sky || All-sky, ~2-15° (zenith) || 7.8 - 612 kHz || [https://leo.phys.unm.edu/~lwa/specs.html LWA Spec] || Stephen White (stephen.white.24@spaceforce.mil)
|-



| MWA || 80-300MHz || 0.5s || 01:00-08:00 || 2014 - ongoing || 30deg || depends on configuration; ~1-2′/200MHz || 10/40 kHz || [https://asvo.mwatelescope.org/ MWA]|| Surajit Mondal (surajit.mondal@njit.edu)
|-

| NoRP || 1, 2, 3.75, 9.4, 17, 35, 80 GHz || 0.1s || 23 - 7 || ~1979 - ongoing (3.75 GHz since 1951) || Full Sun, unresolved || N/A || N/A || [https://solar.nro.nao.ac.jp/norp/index.html NRO] || Stephen White (stephen.white.24@spaceforce.mil)
|-

| NoRH || 17, 34 GHz || 1s, daily || 22.5 - 6.5 || 7/1/1992 - 3/31/2020|| Full Sun || 10″, 5″ || N/A || [https://hinode.isee.nagoya-u.ac.jp/ICCON/ Hinode Site] || Stephen White (stephen.white.24@spaceforce.mil), Nat Gopalswamy (nat.gopalswamy@nasa.gov)
|-

| OVRO-LWA || 15-88 MHz || 1ms (dynamic spectrum); 100ms (fast visibilities); 10s (regular mode) || 14-24:00 || under commissioning (some data already available) || Full Sky || 5′ at 80MHz || 24kHz (dynamic spectrum and regular mode), 100 kHz (fast visibilities) || [link to come] || Dale Gary (dgary@njit.edu) Bin Chen (bin.chen@njit.edu)
|-

| PSP/FIELDS || LFR: 10 kHz-2.4 MHz HFR:1.6 kHz-19.2 MHz || 7s || 0-24 || 2018/10/02 - Present during PSP Perihelia || N/A || N/A || Non-uniform, Δν/ν ~ 4.5% || [https://sprg.ssl.berkeley.edu/data/psp/data/sci/fields/ FIELDS data] [https://sprg.ssl.berkeley.edu/data/psp/data/sci/fields/ Data Release Notes] || Marc Pulupa (pulupa@berkeley.edu) Sam Badman (samuel.badman@cfa.harvard.edu)
|-



| RSTN || 245, 410, 610, 1415, 4995, 8800, 15400 MHz || 1s || 0 - 24 || 1980 - ongoing|| Full Sun, unresolved || N/A || N/A || [https://www.ngdc.noaa.gov/stp/space-weather/solar-data/solar-features/solar-radio/rstn-1-second/ NOAA], [https://www.sws.bom.gov.au/World_Data_Centre/2/8/9 SWS] || Stephen White (stephen.white.24@spaceforce.mil)
|-



| VLA || 1-18 GHz; 0.3 GHz (P-Band) || 50 ms - 5 s, depending on observing frequency || 15 - 23 || from 1990s; based on proposals || 42′ at 1 GHz || 1.3-46 ″ @ 1.3GHz depending on configuration|| Depends on configuration || [https://data.nrao.edu/portal/#/ NRAO Archive] [http://www.ovsa.njit.edu/wiki/index.php/VLA_Data_Survey Data Survey]|| Tim Bastian (tbastian@nrao.edu)
|-
| NRH || 150, 173, 228, 270, 298, 327, 327.5, 408, 432, 445 MHz || 125 ms || 8 - 15 || Ongoing || Full Sun || 6′ @ 150 MHz, 2′ @ 450 MHz || || [https://secchirh.obspm.fr/ NRH] || L. Klein (ludwig.klein@obspm.fr) and A. Hamini (abdallah.hamini@obspm.fr)
|-

| E-Callisto || Varies: Multiple Instruments Throughout World || Varies || Varies || Ongoing || Full Sun - Unresolved || Varies || Varies || [https://www.e-callisto.org/Data/data.html E-Callisto] || N/A
|-

|}

== Other Useful Resources ==

# The [https://lesia.obspm.fr/cesra/sites.html CESRA-related data centers] is another place where you can find a list of different Radio instruments.
# The plots below show the Observation times of different instruments:

[[file:Solar_ghz.png|thumb|center|600px|'''Figure 1:''' Radio Instruments (Solar Dedicated and Proposal based observing between 1 - 40 GHz) (Image Credit: Anshu Kumari)]]

[[file:Solarplus_mhz.png|thumb|center|600px|'''Figure 2:''' Radio Instruments (Solar Dedicated and Proposal based observing below 1 GHz) (Image Credit: Anshu Kumari)]]

[[file:Facilities.png|thumb|center|1000px|'''Figure 3:''' Radio Instruments (Solar Dedicated and Proposal based observing between 10 MHz - 10 GHz) (Image Credit: Bin Chen)]]

Radio Data from Around the World

2023-08-09T03:30:55Z

Schhabra: /* Radio Data Access */

== Radio Data Access ==

{| class="wikitable" style="text-align: center; width: 1300px; height: 200px;"
|-
| '''Instrument''' || style="width: 400pt;" | '''Frequency Range/Central Frequencies''' ||style="width: 700pt;" |'''Cadence''' || '''Observing Time (UT)'''|| style="width: 250pt;" |'''Dates of Operation (start/end)''' || '''FOV''' || style="width: 130pt;" | '''Imaging FOV / Resolution''' || style="width: 250pt;" |'''Spectroscopy Frequency Resolution''' || style="width: 250pt;" | '''Useful Links''' || '''Points of Contact for Data '''
|-

| ALMA || 92-96, 104-108, 190-194, 202-206, 229-233, 245-249, 338.6-342.6, 350.6-345.6 GHz || 1s || 13 - 21 || 2016 - ongoing || Sun, sub active region scales || 6.54-66″ / 0.62-11.6″ || 2 GHz || [https://ui.adsabs.harvard.edu/abs/2022FrASS...9.7368B/abstract Harvard] || Stephen White (stephen.white.24@spaceforce.mil) Tim Bastian (tbastian@nrao.edu)
|-



| EOVSA || 1-18 GHz || Sample time: 20ms: Full Sweep: 1s || ~14-24 || 2017 - ongoing || Full Sun ||57″/<math>\nu_{GHz}</math> x 51″/<math>\nu_{GHz}</math> || 0.500 GHz || [http://ovsa.njit.edu//wiki/index.php You Are Here] [http://ovsa.njit.edu/browser/?suntoday_date=2023-08-06 EOVSA Browser] || Dale Gary (dgary@njit.edu) Bin Chen (bin.chen@njit.edu)
|-

| F10.7|| 2.8 GHz || 3 times daily || 17 (18), 20, 23 (22) || 1947 - ongoing || Full Sun, unresolved || N/A || N/A || [https://www.spaceweather.gc.ca/forecast-prevision/solar-solaire/solarflux/sx-en.php Space Weather Monitor] || Sam Schonfeld (schonfsj@gmail.com)
|-



| LOFAR || LBA: 10 - 80 MHz, HBA: 120 - 240 MHz || 10ms || 8.5-16 || Time Allocation by Proposal|| [https://drive.google.com/file/d/1YaBnSdzuEvxWee-wUAUZcUzxiaO1524J/view?usp=sharing FOV Info] || [https://drive.google.com/file/d/1YaBnSdzuEvxWee-wUAUZcUzxiaO1524J/view?usp=sharing FOV Info] || 10 kHz || [https://docs.google.com/document/d/103S1ANjTPXDIWnOAlhEAbQy2oLm5ZOcqV8x1eR0hvxs/edit LOFAR Info Doc] || Anshu Kumari (anshusingh628@gmail.com)
|-

| LWA-NM || 10-88 MHz || < 1s || 15 - 23 || April 2011 - ongoing|| Full-sky || All-sky, ~2-15° (zenith) || 7.8 - 612 kHz || [https://leo.phys.unm.edu/~lwa/specs.html LWA Spec] || Stephen White (stephen.white.24@spaceforce.mil)
|-



| MWA || 80-300MHz || 0.5s || 01:00-08:00 || 2014 - ongoing || 30deg || depends on configuration; ~1-2′/200MHz || 10/40 kHz || [https://asvo.mwatelescope.org/ MWA]|| Surajit Mondal (surajit.mondal@njit.edu)
|-

| NoRP || 1, 2, 3.75, 9.4, 17, 35, 80 GHz || 0.1s || 23 - 7 || ~1979 - ongoing (3.75 GHz since 1951) || Full Sun, unresolved || N/A || N/A || [https://solar.nro.nao.ac.jp/norp/index.html NRO] || Stephen White (stephen.white.24@spaceforce.mil)
|-

| NoRH || 17, 34 GHz || 1s, daily || 22.5 - 6.5 || 7/1/1992 - 3/31/2020|| Full Sun || 10″, 5″ || N/A || [https://hinode.isee.nagoya-u.ac.jp/ICCON/ Hinode Site] || Stephen White (stephen.white.24@spaceforce.mil), Nat Gopalswamy (nat.gopalswamy@nasa.gov)
|-

| OVRO-LWA || 15-88 MHz || 1ms (dynamic spectrum); 100ms (fast visibilities); 10s (regular mode) || 14-24:00 || under commissioning (some data already available) || Full Sky || 5′ at 80MHz || 24kHz (dynamic spectrum and regular mode), 100 kHz (fast visibilities) || [link to come] || Dale Gary (dgary@njit.edu) Bin Chen (bin.chen@njit.edu)
|-

| PSP/FIELDS || LFR: 10 kHz-2.4 MHz HFR:1.6 kHz-19.2 MHz || 7s || 0-24 || 2018/10/02 - Present during PSP Perihelia || N/A || N/A || Non-uniform, Δν/ν ~ 4.5% || [https://sprg.ssl.berkeley.edu/data/psp/data/sci/fields/ FIELDS data] [https://sprg.ssl.berkeley.edu/data/psp/data/sci/fields/ Data Release Notes] || Marc Pulupa (pulupa@berkeley.edu) Sam Badman (samuel.badman@cfa.harvard.edu)
|-



| RSTN || 245, 410, 610, 1415, 4995, 8800, 15400 MHz || 1s || 0 - 24 || 1980 - ongoing|| Full Sun, unresolved || N/A || N/A || [https://www.ngdc.noaa.gov/stp/space-weather/solar-data/solar-features/solar-radio/rstn-1-second/ NOAA], [https://www.sws.bom.gov.au/World_Data_Centre/2/8/9 SWS] || Stephen White (stephen.white.24@spaceforce.mil)
|-



| VLA || 1-18 GHz; 0.3 GHz (P-Band) || 50 ms - 5 s, depending on observing frequency || 15 - 23 || from 1990s; based on proposals || 42′ at 1 GHz || 1.3-46 ″ @ 1.3GHz depending on configuration|| Depends on configuration || [https://data.nrao.edu/portal/#/ NRAO Archive] [http://www.ovsa.njit.edu/wiki/index.php/VLA_Data_Survey Data Survey]|| Tim Bastian (tbastian@nrao.edu)
|-
| NRH || 150, 173, 228, 270, 298, 327, 327.5, 408, 432, 445 MHz || 125 ms || 8 - 15 || Ongoing || Full Sun || 6′ @ 150 MHz, 2′ @ 450 MHz || || [https://secchirh.obspm.fr/ NRH] || L. Klein (ludwig.klein@obspm.fr) and A. Hamini (abdallah.hamini@obspm.fr)
|-

| E-Callisto || Varies: Multiple Instruments Throughout World || Varies || Varies || Ongoing || Full Sun - Unresolved || Varies || Varies || [https://www.e-callisto.org/Data/data.html E-Callisto] || N/A
|-

|}

== Other Useful Resources ==

# The [https://lesia.obspm.fr/cesra/sites.html CESRA-related data centers] is another place where you can find a list of different Radio instruments.
# The plots below show the Observation times of different instruments:

[[file:Solar_ghz.png|thumb|center|600px|'''Figure 1:''' Radio Instruments (Solar Dedicated and Proposal based observing between 1 - 40 GHz) (Image Credit: Anshu Kumari)]]

[[file:Solarplus_mhz.png|thumb|center|600px|'''Figure 2:''' Radio Instruments (Solar Dedicated and Proposal based observing below 1 GHz) (Image Credit: Anshu Kumari)]]

[[file:Facilities.png|thumb|center|1000px|'''Figure 3:''' Radio Instruments (Solar Dedicated and Proposal based observing between 10 MHz - 10 GHz) (Image Credit: Bin Chen)]]

Radio Data from Around the World

2023-08-08T16:49:55Z

Schhabra: /* Radio Data Access */

== Radio Data Access ==

{| class="wikitable" style="text-align: center; width: 1300px; height: 200px;"
|-
| '''Instrument''' || style="width: 400pt;" | '''Frequency Range/Central Frequencies''' ||style="width: 700pt;" |'''Cadence''' || '''Observing Time (UT)'''|| style="width: 250pt;" |'''Dates of Operation (start/end)''' || '''FOV''' || style="width: 130pt;" | '''Imaging FOV / Resolution''' || style="width: 250pt;" |'''Spectroscopy Frequency Resolution''' || style="width: 250pt;" | '''Useful Links''' || '''Points of Contact for Data '''
|-

| ALMA || 92-96, 104-108, 190-194, 202-206, 229-233, 245-249, 338.6-342.6, 350.6-345.6 GHz || 1s || 13 - 21 || 2016 - ongoing || Sun, sub active region scales || 6.54-66″ / 0.62-11.6″ || 2 GHz || [https://ui.adsabs.harvard.edu/abs/2022FrASS...9.7368B/abstract Harvard] || Stephen White (stephen.white.24@spaceforce.mil) Tim Bastian (tbastian@nrao.edu)
|-



| EOVSA || 1-18 GHz || Sample time: 20ms: Full Sweep: 1s || ~14-24 || 2017 - ongoing || Point ||57″/<math>\nu_{GHz}</math> x 51″/<math>\nu_{GHz}</math> || 0.500 GHz || [http://ovsa.njit.edu//wiki/index.php You Are Here] [http://ovsa.njit.edu/browser/?suntoday_date=2023-08-06 EOVSA Browser] || Dale Gary (dgary@njit.edu) Bin Chen (bin.chen@njit.edu)
|-

| F10.7|| 2.8 GHz || 3 times daily || 17 (18), 20, 23 (22) || 1947 - ongoing || Full Sun, unresolved || N/A || N/A || [https://www.spaceweather.gc.ca/forecast-prevision/solar-solaire/solarflux/sx-en.php Space Weather Monitor] || Sam Schonfeld (schonfsj@gmail.com)
|-



| LOFAR || LBA: 10 - 80 MHz, HBA: 120 - 240 MHz || 10ms || 8.5-16 || Time Allocation by Proposal|| [https://drive.google.com/file/d/1YaBnSdzuEvxWee-wUAUZcUzxiaO1524J/view?usp=sharing FOV Info] || [https://drive.google.com/file/d/1YaBnSdzuEvxWee-wUAUZcUzxiaO1524J/view?usp=sharing FOV Info] || 10 kHz || [https://docs.google.com/document/d/103S1ANjTPXDIWnOAlhEAbQy2oLm5ZOcqV8x1eR0hvxs/edit LOFAR Info Doc] || Anshu Kumari (anshusingh628@gmail.com)
|-

| LWA-NM || 10-88 MHz || < 1s || 15 - 23 || April 2011 - ongoing|| Full-sky || All-sky, ~2-15° (zenith) || 7.8 - 612 kHz || [https://leo.phys.unm.edu/~lwa/specs.html LWA Spec] || Stephen White (stephen.white.24@spaceforce.mil)
|-



| MWA || 80-300MHz || 0.5s || 01:00-08:00 || 2014 - ongoing || 30deg || depends on configuration; ~1-2′/200MHz || 10/40 kHz || [https://asvo.mwatelescope.org/ MWA]|| Surajit Mondal (surajit.mondal@njit.edu)
|-

| NoRP || 1, 2, 3.75, 9.4, 17, 35, 80 GHz || 0.1s || 23 - 7 || ~1979 - ongoing (3.75 GHz since 1951) || Full Sun, unresolved || N/A || N/A || [https://solar.nro.nao.ac.jp/norp/index.html NRO] || Stephen White (stephen.white.24@spaceforce.mil)
|-

| NoRH || 17, 34 GHz || 1s, daily || 22.5 - 6.5 || 7/1/1992 - 3/31/2020|| Full Sun || 10″, 5″ || N/A || [https://hinode.isee.nagoya-u.ac.jp/ICCON/ Hinode Site] || Stephen White (stephen.white.24@spaceforce.mil), Nat Gopalswamy (nat.gopalswamy@nasa.gov)
|-

| OVRO-LWA || 15-88 MHz || 1ms (dynamic spectrum); 100ms (fast visibilities); 10s (regular mode) || 14-24:00 || under commissioning (some data already available) || Full Sky || 5′ at 80MHz || 24kHz (dynamic spectrum and regular mode), 100 kHz (fast visibilities) || [link to come] || Dale Gary (dgary@njit.edu) Bin Chen (bin.chen@njit.edu)
|-

| PSP/FIELDS || LFR: 10 kHz-2.4 MHz HFR:1.6 kHz-19.2 MHz || 7s || 0-24 || 2018/10/02 - Present during PSP Perihelia || N/A || N/A || Non-uniform, Δν/ν ~ 4.5% || [https://sprg.ssl.berkeley.edu/data/psp/data/sci/fields/ FIELDS data] [https://sprg.ssl.berkeley.edu/data/psp/data/sci/fields/ Data Release Notes] || Marc Pulupa (pulupa@berkeley.edu) Sam Badman (samuel.badman@cfa.harvard.edu)
|-



| RSTN || 245, 410, 610, 1415, 4995, 8800, 15400 MHz || 1s || 0 - 24 || 1980 - ongoing|| Full Sun, unresolved || N/A || N/A || [https://www.ngdc.noaa.gov/stp/space-weather/solar-data/solar-features/solar-radio/rstn-1-second/ NOAA], [https://www.sws.bom.gov.au/World_Data_Centre/2/8/9 SWS] || Stephen White (stephen.white.24@spaceforce.mil)
|-



| VLA || 1-18 GHz; 0.3 GHz (P-Band) || 50 ms - 5 s, depending on observing frequency || 15 - 23 || from 1990s; based on proposals || 42′ at 1 GHz || 1.3-46 ″ @ 1.3GHz depending on configuration|| Depends on configuration || [https://data.nrao.edu/portal/#/ NRAO Archive] [http://www.ovsa.njit.edu/wiki/index.php/VLA_Data_Survey Data Survey]|| Tim Bastian (tbastian@nrao.edu)
|-
| NRH || 150, 173, 228, 270, 298, 327, 327.5, 408, 432, 445 MHz || 125 ms || 8 - 15 || Ongoing || Full Sun || 6′ @ 150 MHz, 2′ @ 450 MHz || || [https://secchirh.obspm.fr/ NRH] || L. Klein (ludwig.klein@obspm.fr) and A. Hamini (abdallah.hamini@obspm.fr)
|-

| E-Callisto || Varies: Multiple Instruments Throughout World || Varies || Varies || Ongoing || Full Sun - Unresolved || Varies || Varies || [https://www.e-callisto.org/Data/data.html E-Callisto] || N/A
|-

|}

== Other Useful Resources ==

# The [https://lesia.obspm.fr/cesra/sites.html CESRA-related data centers] is another place where you can find a list of different Radio instruments.
# The plots below show the Observation times of different instruments:

[[file:Solar_ghz.png|thumb|center|600px|'''Figure 1:''' Radio Instruments (Solar Dedicated and Proposal based observing between 1 - 40 GHz) (Image Credit: Anshu Kumari)]]

[[file:Solarplus_mhz.png|thumb|center|600px|'''Figure 2:''' Radio Instruments (Solar Dedicated and Proposal based observing below 1 GHz) (Image Credit: Anshu Kumari)]]

[[file:Facilities.png|thumb|center|1000px|'''Figure 3:''' Radio Instruments (Solar Dedicated and Proposal based observing between 10 MHz - 10 GHz) (Image Credit: Bin Chen)]]

Radio Data from Around the World

2023-08-08T16:44:28Z

Schhabra: /* Other Useful Resources */

== Radio Data Access ==

{| class="wikitable" style="text-align: center; width: 1300px; height: 200px;"
|-
| '''Instrument''' || style="width: 400pt;" | '''Frequency Range/Central Frequencies''' ||style="width: 700pt;" |'''Cadence''' || '''Observing Time (UT)'''|| style="width: 250pt;" |'''Dates of Operation (start/end)''' || '''FOV''' || style="width: 130pt;" | '''Imaging FOV / Resolution''' || style="width: 250pt;" |'''Spectroscopy Frequency Resolution''' || style="width: 250pt;" | '''Useful Links''' || '''Points of Contact for Data '''
|-

| ALMA || 92-96, 104-108, 190-194, 202-206, 229-233, 245-249, 338.6-342.6, 350.6-345.6 GHz || 1s || 13 - 21 || 2016 - ongoing || Sun, sub active region scales || 6.54-66″ / 0.62-11.6″ || 2 GHz || [https://ui.adsabs.harvard.edu/abs/2022FrASS...9.7368B/abstract Harvard] || Stephen White (stephen.white.24@spaceforce.mil) Tim Bastian (tbastian@nrao.edu)
|-



| EOVSA || 1-18 GHz || Sample time: 20ms: Full Sweep: 1s || ~14-24 || 2017 - ongoing || Point ||57″/<math>\nu_{GHz}</math> x 51″/<math>\nu_{GHz}</math> || 0.500 GHz || [http://ovsa.njit.edu//wiki/index.php You Are Here] || Dale Gary (dgary@njit.edu) Bin Chen (bin.chen@njit.edu)
|-

| F10.7|| 2.8 GHz || 3 times daily || 17 (18), 20, 23 (22) || 1947 - ongoing || Full Sun, unresolved || N/A || N/A || [https://www.spaceweather.gc.ca/forecast-prevision/solar-solaire/solarflux/sx-en.php Space Weather Monitor] || Sam Schonfeld (schonfsj@gmail.com)
|-



| LOFAR || LBA: 10 - 80 MHz, HBA: 120 - 240 MHz || 10ms || 8.5-16 || Time Allocation by Proposal|| [https://drive.google.com/file/d/1YaBnSdzuEvxWee-wUAUZcUzxiaO1524J/view?usp=sharing FOV Info] || [https://drive.google.com/file/d/1YaBnSdzuEvxWee-wUAUZcUzxiaO1524J/view?usp=sharing FOV Info] || 10 kHz || [https://docs.google.com/document/d/103S1ANjTPXDIWnOAlhEAbQy2oLm5ZOcqV8x1eR0hvxs/edit LOFAR Info Doc] || Anshu Kumari (anshusingh628@gmail.com)
|-

| LWA-NM || 10-88 MHz || < 1s || 15 - 23 || April 2011 - ongoing|| Full-sky || All-sky, ~2-15° (zenith) || 7.8 - 612 kHz || [https://leo.phys.unm.edu/~lwa/specs.html LWA Spec] || Stephen White (stephen.white.24@spaceforce.mil)
|-



| MWA || 80-300MHz || 0.5s || 01:00-08:00 || 2014 - ongoing || 30deg || depends on configuration; ~1-2′/200MHz || 10/40 kHz || [https://asvo.mwatelescope.org/ MWA]|| Surajit Mondal (surajit.mondal@njit.edu)
|-

| NoRP || 1, 2, 3.75, 9.4, 17, 35, 80 GHz || 0.1s || 23 - 7 || ~1979 - ongoing (3.75 GHz since 1951) || Full Sun, unresolved || N/A || N/A || [https://solar.nro.nao.ac.jp/norp/index.html NRO] || Stephen White (stephen.white.24@spaceforce.mil)
|-

| NoRH || 17, 34 GHz || 1s, daily || 22.5 - 6.5 || 7/1/1992 - 3/31/2020|| Full Sun || 10″, 5″ || N/A || [https://hinode.isee.nagoya-u.ac.jp/ICCON/ Hinode Site] || Stephen White (stephen.white.24@spaceforce.mil), Nat Gopalswamy (nat.gopalswamy@nasa.gov)
|-

| OVRO-LWA || 15-88 MHz || 1ms (dynamic spectrum); 100ms (fast visibilities); 10s (regular mode) || 14-24:00 || under commissioning (some data already available) || Full Sky || 5′ at 80MHz || 24kHz (dynamic spectrum and regular mode), 100 kHz (fast visibilities) || [link to come] || Dale Gary (dgary@njit.edu) Bin Chen (bin.chen@njit.edu)
|-

| PSP/FIELDS || LFR: 10 kHz-2.4 MHz HFR:1.6 kHz-19.2 MHz || 7s || 0-24 || 2018/10/02 - Present during PSP Perihelia || N/A || N/A || Non-uniform, Δν/ν ~ 4.5% || [https://sprg.ssl.berkeley.edu/data/psp/data/sci/fields/ FIELDS data] [https://sprg.ssl.berkeley.edu/data/psp/data/sci/fields/ Data Release Notes] || Marc Pulupa (pulupa@berkeley.edu) Sam Badman (samuel.badman@cfa.harvard.edu)
|-



| RSTN || 245, 410, 610, 1415, 4995, 8800, 15400 MHz || 1s || 0 - 24 || 1980 - ongoing|| Full Sun, unresolved || N/A || N/A || [https://www.ngdc.noaa.gov/stp/space-weather/solar-data/solar-features/solar-radio/rstn-1-second/ NOAA], [https://www.sws.bom.gov.au/World_Data_Centre/2/8/9 SWS] || Stephen White (stephen.white.24@spaceforce.mil)
|-



| VLA || 1-18 GHz; 0.3 GHz (P-Band) || 50 ms - 5 s, depending on observing frequency || 15 - 23 || from 1990s; based on proposals || 42′ at 1 GHz || 1.3-46 ″ @ 1.3GHz depending on configuration|| Depends on configuration || [https://data.nrao.edu/portal/#/ NRAO Archive] || Tim Bastian (tbastian@nrao.edu)
|-
| NRH || 150, 173, 228, 270, 298, 327, 327.5, 408, 432, 445 MHz || 125 ms || 8 - 15 || Ongoing || Full Sun || 6′ @ 150 MHz, 2′ @ 450 MHz || || [https://secchirh.obspm.fr/ NRH] || L. Klein (ludwig.klein@obspm.fr) and A. Hamini (abdallah.hamini@obspm.fr)
|-

| E-Callisto || Varies: Multiple Instruments Throughout World || Varies || Varies || Ongoing || Full Sun - Unresolved || Varies || Varies || [https://www.e-callisto.org/Data/data.html E-Callisto] || N/A
|-

|}

== Other Useful Resources ==

# The [https://lesia.obspm.fr/cesra/sites.html CESRA-related data centers] is another place where you can find a list of different Radio instruments.
# The plots below show the Observation times of different instruments:

[[file:Solar_ghz.png|thumb|center|600px|'''Figure 1:''' Radio Instruments (Solar Dedicated and Proposal based observing between 1 - 40 GHz) (Image Credit: Anshu Kumari)]]

[[file:Solarplus_mhz.png|thumb|center|600px|'''Figure 2:''' Radio Instruments (Solar Dedicated and Proposal based observing below 1 GHz) (Image Credit: Anshu Kumari)]]

[[file:Facilities.png|thumb|center|1000px|'''Figure 3:''' Radio Instruments (Solar Dedicated and Proposal based observing between 10 MHz - 10 GHz) (Image Credit: Bin Chen)]]

Radio Data from Around the World

2023-08-08T16:44:04Z

Schhabra: /* Other Useful Resources */

== Radio Data Access ==

{| class="wikitable" style="text-align: center; width: 1300px; height: 200px;"
|-
| '''Instrument''' || style="width: 400pt;" | '''Frequency Range/Central Frequencies''' ||style="width: 700pt;" |'''Cadence''' || '''Observing Time (UT)'''|| style="width: 250pt;" |'''Dates of Operation (start/end)''' || '''FOV''' || style="width: 130pt;" | '''Imaging FOV / Resolution''' || style="width: 250pt;" |'''Spectroscopy Frequency Resolution''' || style="width: 250pt;" | '''Useful Links''' || '''Points of Contact for Data '''
|-

| ALMA || 92-96, 104-108, 190-194, 202-206, 229-233, 245-249, 338.6-342.6, 350.6-345.6 GHz || 1s || 13 - 21 || 2016 - ongoing || Sun, sub active region scales || 6.54-66″ / 0.62-11.6″ || 2 GHz || [https://ui.adsabs.harvard.edu/abs/2022FrASS...9.7368B/abstract Harvard] || Stephen White (stephen.white.24@spaceforce.mil) Tim Bastian (tbastian@nrao.edu)
|-



| EOVSA || 1-18 GHz || Sample time: 20ms: Full Sweep: 1s || ~14-24 || 2017 - ongoing || Point ||57″/<math>\nu_{GHz}</math> x 51″/<math>\nu_{GHz}</math> || 0.500 GHz || [http://ovsa.njit.edu//wiki/index.php You Are Here] || Dale Gary (dgary@njit.edu) Bin Chen (bin.chen@njit.edu)
|-

| F10.7|| 2.8 GHz || 3 times daily || 17 (18), 20, 23 (22) || 1947 - ongoing || Full Sun, unresolved || N/A || N/A || [https://www.spaceweather.gc.ca/forecast-prevision/solar-solaire/solarflux/sx-en.php Space Weather Monitor] || Sam Schonfeld (schonfsj@gmail.com)
|-



| LOFAR || LBA: 10 - 80 MHz, HBA: 120 - 240 MHz || 10ms || 8.5-16 || Time Allocation by Proposal|| [https://drive.google.com/file/d/1YaBnSdzuEvxWee-wUAUZcUzxiaO1524J/view?usp=sharing FOV Info] || [https://drive.google.com/file/d/1YaBnSdzuEvxWee-wUAUZcUzxiaO1524J/view?usp=sharing FOV Info] || 10 kHz || [https://docs.google.com/document/d/103S1ANjTPXDIWnOAlhEAbQy2oLm5ZOcqV8x1eR0hvxs/edit LOFAR Info Doc] || Anshu Kumari (anshusingh628@gmail.com)
|-

| LWA-NM || 10-88 MHz || < 1s || 15 - 23 || April 2011 - ongoing|| Full-sky || All-sky, ~2-15° (zenith) || 7.8 - 612 kHz || [https://leo.phys.unm.edu/~lwa/specs.html LWA Spec] || Stephen White (stephen.white.24@spaceforce.mil)
|-



| MWA || 80-300MHz || 0.5s || 01:00-08:00 || 2014 - ongoing || 30deg || depends on configuration; ~1-2′/200MHz || 10/40 kHz || [https://asvo.mwatelescope.org/ MWA]|| Surajit Mondal (surajit.mondal@njit.edu)
|-

| NoRP || 1, 2, 3.75, 9.4, 17, 35, 80 GHz || 0.1s || 23 - 7 || ~1979 - ongoing (3.75 GHz since 1951) || Full Sun, unresolved || N/A || N/A || [https://solar.nro.nao.ac.jp/norp/index.html NRO] || Stephen White (stephen.white.24@spaceforce.mil)
|-

| NoRH || 17, 34 GHz || 1s, daily || 22.5 - 6.5 || 7/1/1992 - 3/31/2020|| Full Sun || 10″, 5″ || N/A || [https://hinode.isee.nagoya-u.ac.jp/ICCON/ Hinode Site] || Stephen White (stephen.white.24@spaceforce.mil), Nat Gopalswamy (nat.gopalswamy@nasa.gov)
|-

| OVRO-LWA || 15-88 MHz || 1ms (dynamic spectrum); 100ms (fast visibilities); 10s (regular mode) || 14-24:00 || under commissioning (some data already available) || Full Sky || 5′ at 80MHz || 24kHz (dynamic spectrum and regular mode), 100 kHz (fast visibilities) || [link to come] || Dale Gary (dgary@njit.edu) Bin Chen (bin.chen@njit.edu)
|-

| PSP/FIELDS || LFR: 10 kHz-2.4 MHz HFR:1.6 kHz-19.2 MHz || 7s || 0-24 || 2018/10/02 - Present during PSP Perihelia || N/A || N/A || Non-uniform, Δν/ν ~ 4.5% || [https://sprg.ssl.berkeley.edu/data/psp/data/sci/fields/ FIELDS data] [https://sprg.ssl.berkeley.edu/data/psp/data/sci/fields/ Data Release Notes] || Marc Pulupa (pulupa@berkeley.edu) Sam Badman (samuel.badman@cfa.harvard.edu)
|-



| RSTN || 245, 410, 610, 1415, 4995, 8800, 15400 MHz || 1s || 0 - 24 || 1980 - ongoing|| Full Sun, unresolved || N/A || N/A || [https://www.ngdc.noaa.gov/stp/space-weather/solar-data/solar-features/solar-radio/rstn-1-second/ NOAA], [https://www.sws.bom.gov.au/World_Data_Centre/2/8/9 SWS] || Stephen White (stephen.white.24@spaceforce.mil)
|-



| VLA || 1-18 GHz; 0.3 GHz (P-Band) || 50 ms - 5 s, depending on observing frequency || 15 - 23 || from 1990s; based on proposals || 42′ at 1 GHz || 1.3-46 ″ @ 1.3GHz depending on configuration|| Depends on configuration || [https://data.nrao.edu/portal/#/ NRAO Archive] || Tim Bastian (tbastian@nrao.edu)
|-
| NRH || 150, 173, 228, 270, 298, 327, 327.5, 408, 432, 445 MHz || 125 ms || 8 - 15 || Ongoing || Full Sun || 6′ @ 150 MHz, 2′ @ 450 MHz || || [https://secchirh.obspm.fr/ NRH] || L. Klein (ludwig.klein@obspm.fr) and A. Hamini (abdallah.hamini@obspm.fr)
|-

| E-Callisto || Varies: Multiple Instruments Throughout World || Varies || Varies || Ongoing || Full Sun - Unresolved || Varies || Varies || [https://www.e-callisto.org/Data/data.html E-Callisto] || N/A
|-

|}

== Other Useful Resources ==

# The [https://lesia.obspm.fr/cesra/sites.html CESRA-related data centers] is another place where you can find a list of different Radio instruments.
# The plots below show the Observation times of different instruments:

[[file:Solar_ghz.png|thumb|center|600px|'''Figure 1:''' Radio Instruments (Solar Dedicated and Proposal based observing between 1 - 40 GHz) (Image Credit: Anshu Kumari)]]

[[file:Solarplus_mhz.png|thumb|center|600px|'''Figure 2:''' Radio Instruments (Solar Dedicated and Proposal based observing below 1 GHz) (Image Credit: Anshu Kumari)]]

[[file:Facilities.png|thumb|center|600px|'''Figure 3:''' Radio Instruments (Solar Dedicated and Proposal based observing between 10 MHz - 10 GHz) (Image Credit: Bin Chen)]]

Radio Data from Around the World

2023-08-08T16:42:33Z

Schhabra: /* Other Useful Resources */

== Radio Data Access ==

{| class="wikitable" style="text-align: center; width: 1300px; height: 200px;"
|-
| '''Instrument''' || style="width: 400pt;" | '''Frequency Range/Central Frequencies''' ||style="width: 700pt;" |'''Cadence''' || '''Observing Time (UT)'''|| style="width: 250pt;" |'''Dates of Operation (start/end)''' || '''FOV''' || style="width: 130pt;" | '''Imaging FOV / Resolution''' || style="width: 250pt;" |'''Spectroscopy Frequency Resolution''' || style="width: 250pt;" | '''Useful Links''' || '''Points of Contact for Data '''
|-

| ALMA || 92-96, 104-108, 190-194, 202-206, 229-233, 245-249, 338.6-342.6, 350.6-345.6 GHz || 1s || 13 - 21 || 2016 - ongoing || Sun, sub active region scales || 6.54-66″ / 0.62-11.6″ || 2 GHz || [https://ui.adsabs.harvard.edu/abs/2022FrASS...9.7368B/abstract Harvard] || Stephen White (stephen.white.24@spaceforce.mil) Tim Bastian (tbastian@nrao.edu)
|-



| EOVSA || 1-18 GHz || Sample time: 20ms: Full Sweep: 1s || ~14-24 || 2017 - ongoing || Point ||57″/<math>\nu_{GHz}</math> x 51″/<math>\nu_{GHz}</math> || 0.500 GHz || [http://ovsa.njit.edu//wiki/index.php You Are Here] || Dale Gary (dgary@njit.edu) Bin Chen (bin.chen@njit.edu)
|-

| F10.7|| 2.8 GHz || 3 times daily || 17 (18), 20, 23 (22) || 1947 - ongoing || Full Sun, unresolved || N/A || N/A || [https://www.spaceweather.gc.ca/forecast-prevision/solar-solaire/solarflux/sx-en.php Space Weather Monitor] || Sam Schonfeld (schonfsj@gmail.com)
|-



| LOFAR || LBA: 10 - 80 MHz, HBA: 120 - 240 MHz || 10ms || 8.5-16 || Time Allocation by Proposal|| [https://drive.google.com/file/d/1YaBnSdzuEvxWee-wUAUZcUzxiaO1524J/view?usp=sharing FOV Info] || [https://drive.google.com/file/d/1YaBnSdzuEvxWee-wUAUZcUzxiaO1524J/view?usp=sharing FOV Info] || 10 kHz || [https://docs.google.com/document/d/103S1ANjTPXDIWnOAlhEAbQy2oLm5ZOcqV8x1eR0hvxs/edit LOFAR Info Doc] || Anshu Kumari (anshusingh628@gmail.com)
|-

| LWA-NM || 10-88 MHz || < 1s || 15 - 23 || April 2011 - ongoing|| Full-sky || All-sky, ~2-15° (zenith) || 7.8 - 612 kHz || [https://leo.phys.unm.edu/~lwa/specs.html LWA Spec] || Stephen White (stephen.white.24@spaceforce.mil)
|-



| MWA || 80-300MHz || 0.5s || 01:00-08:00 || 2014 - ongoing || 30deg || depends on configuration; ~1-2′/200MHz || 10/40 kHz || [https://asvo.mwatelescope.org/ MWA]|| Surajit Mondal (surajit.mondal@njit.edu)
|-

| NoRP || 1, 2, 3.75, 9.4, 17, 35, 80 GHz || 0.1s || 23 - 7 || ~1979 - ongoing (3.75 GHz since 1951) || Full Sun, unresolved || N/A || N/A || [https://solar.nro.nao.ac.jp/norp/index.html NRO] || Stephen White (stephen.white.24@spaceforce.mil)
|-

| NoRH || 17, 34 GHz || 1s, daily || 22.5 - 6.5 || 7/1/1992 - 3/31/2020|| Full Sun || 10″, 5″ || N/A || [https://hinode.isee.nagoya-u.ac.jp/ICCON/ Hinode Site] || Stephen White (stephen.white.24@spaceforce.mil), Nat Gopalswamy (nat.gopalswamy@nasa.gov)
|-

| OVRO-LWA || 15-88 MHz || 1ms (dynamic spectrum); 100ms (fast visibilities); 10s (regular mode) || 14-24:00 || under commissioning (some data already available) || Full Sky || 5′ at 80MHz || 24kHz (dynamic spectrum and regular mode), 100 kHz (fast visibilities) || [link to come] || Dale Gary (dgary@njit.edu) Bin Chen (bin.chen@njit.edu)
|-

| PSP/FIELDS || LFR: 10 kHz-2.4 MHz HFR:1.6 kHz-19.2 MHz || 7s || 0-24 || 2018/10/02 - Present during PSP Perihelia || N/A || N/A || Non-uniform, Δν/ν ~ 4.5% || [https://sprg.ssl.berkeley.edu/data/psp/data/sci/fields/ FIELDS data] [https://sprg.ssl.berkeley.edu/data/psp/data/sci/fields/ Data Release Notes] || Marc Pulupa (pulupa@berkeley.edu) Sam Badman (samuel.badman@cfa.harvard.edu)
|-



| RSTN || 245, 410, 610, 1415, 4995, 8800, 15400 MHz || 1s || 0 - 24 || 1980 - ongoing|| Full Sun, unresolved || N/A || N/A || [https://www.ngdc.noaa.gov/stp/space-weather/solar-data/solar-features/solar-radio/rstn-1-second/ NOAA], [https://www.sws.bom.gov.au/World_Data_Centre/2/8/9 SWS] || Stephen White (stephen.white.24@spaceforce.mil)
|-



| VLA || 1-18 GHz; 0.3 GHz (P-Band) || 50 ms - 5 s, depending on observing frequency || 15 - 23 || from 1990s; based on proposals || 42′ at 1 GHz || 1.3-46 ″ @ 1.3GHz depending on configuration|| Depends on configuration || [https://data.nrao.edu/portal/#/ NRAO Archive] || Tim Bastian (tbastian@nrao.edu)
|-
| NRH || 150, 173, 228, 270, 298, 327, 327.5, 408, 432, 445 MHz || 125 ms || 8 - 15 || Ongoing || Full Sun || 6′ @ 150 MHz, 2′ @ 450 MHz || || [https://secchirh.obspm.fr/ NRH] || L. Klein (ludwig.klein@obspm.fr) and A. Hamini (abdallah.hamini@obspm.fr)
|-

| E-Callisto || Varies: Multiple Instruments Throughout World || Varies || Varies || Ongoing || Full Sun - Unresolved || Varies || Varies || [https://www.e-callisto.org/Data/data.html E-Callisto] || N/A
|-

|}

== Other Useful Resources ==

# The [https://lesia.obspm.fr/cesra/sites.html CESRA-related data centers] is another place where you can find a list of different Radio instruments.
# The plots below show the Observation times of different instruments:

[[file:Solar_ghz.png|thumb|center|600px|'''Figure 1:''' Radio Instruments (Solar Dedicated and Proposal based observing between 1 - 40 GHz) (Plot Credit: Anshu Kumari)]]

[[file:Solarplus_mhz.png|thumb|center|600px|'''Figure 2:''' Radio Instruments (Solar Dedicated and Proposal based observing below 1 GHz) (Plot Credit: Anshu Kumari)]]

File:Solarplus mhz.png

2023-08-08T16:41:08Z

Schhabra:

Radio Data from Around the World

2023-08-08T16:37:41Z

Schhabra: /* Other Useful Resources */

== Radio Data Access ==

{| class="wikitable" style="text-align: center; width: 1300px; height: 200px;"
|-
| '''Instrument''' || style="width: 400pt;" | '''Frequency Range/Central Frequencies''' ||style="width: 700pt;" |'''Cadence''' || '''Observing Time (UT)'''|| style="width: 250pt;" |'''Dates of Operation (start/end)''' || '''FOV''' || style="width: 130pt;" | '''Imaging FOV / Resolution''' || style="width: 250pt;" |'''Spectroscopy Frequency Resolution''' || style="width: 250pt;" | '''Useful Links''' || '''Points of Contact for Data '''
|-

| ALMA || 92-96, 104-108, 190-194, 202-206, 229-233, 245-249, 338.6-342.6, 350.6-345.6 GHz || 1s || 13 - 21 || 2016 - ongoing || Sun, sub active region scales || 6.54-66″ / 0.62-11.6″ || 2 GHz || [https://ui.adsabs.harvard.edu/abs/2022FrASS...9.7368B/abstract Harvard] || Stephen White (stephen.white.24@spaceforce.mil) Tim Bastian (tbastian@nrao.edu)
|-



| EOVSA || 1-18 GHz || Sample time: 20ms: Full Sweep: 1s || ~14-24 || 2017 - ongoing || Point ||57″/<math>\nu_{GHz}</math> x 51″/<math>\nu_{GHz}</math> || 0.500 GHz || [http://ovsa.njit.edu//wiki/index.php You Are Here] || Dale Gary (dgary@njit.edu) Bin Chen (bin.chen@njit.edu)
|-

| F10.7|| 2.8 GHz || 3 times daily || 17 (18), 20, 23 (22) || 1947 - ongoing || Full Sun, unresolved || N/A || N/A || [https://www.spaceweather.gc.ca/forecast-prevision/solar-solaire/solarflux/sx-en.php Space Weather Monitor] || Sam Schonfeld (schonfsj@gmail.com)
|-



| LOFAR || LBA: 10 - 80 MHz, HBA: 120 - 240 MHz || 10ms || 8.5-16 || Time Allocation by Proposal|| [https://drive.google.com/file/d/1YaBnSdzuEvxWee-wUAUZcUzxiaO1524J/view?usp=sharing FOV Info] || [https://drive.google.com/file/d/1YaBnSdzuEvxWee-wUAUZcUzxiaO1524J/view?usp=sharing FOV Info] || 10 kHz || [https://docs.google.com/document/d/103S1ANjTPXDIWnOAlhEAbQy2oLm5ZOcqV8x1eR0hvxs/edit LOFAR Info Doc] || Anshu Kumari (anshusingh628@gmail.com)
|-

| LWA-NM || 10-88 MHz || < 1s || 15 - 23 || April 2011 - ongoing|| Full-sky || All-sky, ~2-15° (zenith) || 7.8 - 612 kHz || [https://leo.phys.unm.edu/~lwa/specs.html LWA Spec] || Stephen White (stephen.white.24@spaceforce.mil)
|-



| MWA || 80-300MHz || 0.5s || 01:00-08:00 || 2014 - ongoing || 30deg || depends on configuration; ~1-2′/200MHz || 10/40 kHz || [https://asvo.mwatelescope.org/ MWA]|| Surajit Mondal (surajit.mondal@njit.edu)
|-

| NoRP || 1, 2, 3.75, 9.4, 17, 35, 80 GHz || 0.1s || 23 - 7 || ~1979 - ongoing (3.75 GHz since 1951) || Full Sun, unresolved || N/A || N/A || [https://solar.nro.nao.ac.jp/norp/index.html NRO] || Stephen White (stephen.white.24@spaceforce.mil)
|-

| NoRH || 17, 34 GHz || 1s, daily || 22.5 - 6.5 || 7/1/1992 - 3/31/2020|| Full Sun || 10″, 5″ || N/A || [https://hinode.isee.nagoya-u.ac.jp/ICCON/ Hinode Site] || Stephen White (stephen.white.24@spaceforce.mil), Nat Gopalswamy (nat.gopalswamy@nasa.gov)
|-

| OVRO-LWA || 15-88 MHz || 1ms (dynamic spectrum); 100ms (fast visibilities); 10s (regular mode) || 14-24:00 || under commissioning (some data already available) || Full Sky || 5′ at 80MHz || 24kHz (dynamic spectrum and regular mode), 100 kHz (fast visibilities) || [link to come] || Dale Gary (dgary@njit.edu) Bin Chen (bin.chen@njit.edu)
|-

| PSP/FIELDS || LFR: 10 kHz-2.4 MHz HFR:1.6 kHz-19.2 MHz || 7s || 0-24 || 2018/10/02 - Present during PSP Perihelia || N/A || N/A || Non-uniform, Δν/ν ~ 4.5% || [https://sprg.ssl.berkeley.edu/data/psp/data/sci/fields/ FIELDS data] [https://sprg.ssl.berkeley.edu/data/psp/data/sci/fields/ Data Release Notes] || Marc Pulupa (pulupa@berkeley.edu) Sam Badman (samuel.badman@cfa.harvard.edu)
|-



| RSTN || 245, 410, 610, 1415, 4995, 8800, 15400 MHz || 1s || 0 - 24 || 1980 - ongoing|| Full Sun, unresolved || N/A || N/A || [https://www.ngdc.noaa.gov/stp/space-weather/solar-data/solar-features/solar-radio/rstn-1-second/ NOAA], [https://www.sws.bom.gov.au/World_Data_Centre/2/8/9 SWS] || Stephen White (stephen.white.24@spaceforce.mil)
|-



| VLA || 1-18 GHz; 0.3 GHz (P-Band) || 50 ms - 5 s, depending on observing frequency || 15 - 23 || from 1990s; based on proposals || 42′ at 1 GHz || 1.3-46 ″ @ 1.3GHz depending on configuration|| Depends on configuration || [https://data.nrao.edu/portal/#/ NRAO Archive] || Tim Bastian (tbastian@nrao.edu)
|-
| NRH || 150, 173, 228, 270, 298, 327, 327.5, 408, 432, 445 MHz || 125 ms || 8 - 15 || Ongoing || Full Sun || 6′ @ 150 MHz, 2′ @ 450 MHz || || [https://secchirh.obspm.fr/ NRH] || L. Klein (ludwig.klein@obspm.fr) and A. Hamini (abdallah.hamini@obspm.fr)
|-

| E-Callisto || Varies: Multiple Instruments Throughout World || Varies || Varies || Ongoing || Full Sun - Unresolved || Varies || Varies || [https://www.e-callisto.org/Data/data.html E-Callisto] || N/A
|-

|}

== Other Useful Resources ==

# The [https://lesia.obspm.fr/cesra/sites.html CESRA-related data centers] is another place where you can find a list of different Radio instruments.

# The plots below show the Observation times of different instruments:

[[file:Solar_ghz.png|thumb|center|600px|'''Figure 1:''' Radio Instruments (Solar Dedicated and Proposal based observing between 1 - 40 GHz) (Plot Credit: Anshu Kumari)]]

File:Solar ghz.png

2023-08-08T16:31:45Z

Schhabra:

Radio Data from Around the World

2023-08-08T16:23:27Z

Schhabra: /* Other Useful Resources */

== Radio Data Access ==

{| class="wikitable" style="text-align: center; width: 1300px; height: 200px;"
|-
| '''Instrument''' || style="width: 400pt;" | '''Frequency Range/Central Frequencies''' ||style="width: 300pt;" |'''Cadence''' || '''Observing Time (UT)'''|| style="width: 250pt;" |'''Dates of Operation (start/end)''' || '''FOV''' || style="width: 130pt;" | '''Imaging FOV / Resolution''' || '''Spectroscopy Frequency Resolution''' || '''Useful Links''' || '''Points of Contact for Data '''
|-

| ALMA || 92-96, 104-108, 190-194, 202-206, 229-233, 245-249, 338.6-342.6, 350.6-345.6 GHz || 1s || 13 - 21 || 2016 - ongoing || Sun, sub active region scales || 6.54-66″ / 0.62-11.6″ || 2 GHz || [https://ui.adsabs.harvard.edu/abs/2022FrASS...9.7368B/abstract Harvard] || Stephen White (stephen.white.24@spaceforce.mil) Tim Bastian (tbastian@nrao.edu)
|-



| EOVSA || 1-18 GHz || Sample time: 20ms: Full Sweep: 1s || ~14-24 || 2017 - ongoing || Point ||57″/<math>\nu_{GHz}</math> x 51″/<math>\nu_{GHz}</math> || 0.500 GHz || [http://ovsa.njit.edu//wiki/index.php You Are Here] || Dale Gary (dgary@njit.edu) Bin Chen (bin.chen@njit.edu)
|-

| F10.7|| 2.8 GHz || 3 times daily || 17 (18), 20, 23 (22) || 1947 - ongoing || Full Sun, unresolved || N/A || N/A || [https://www.spaceweather.gc.ca/forecast-prevision/solar-solaire/solarflux/sx-en.php Space Weather Monitor] || Sam Schonfeld (schonfsj@gmail.com)
|-



| LOFAR || LBA: 10 - 80 MHz, HBA: 120 - 240 MHz || 10ms || 8.5-16 || Time Allocation by Proposal|| [https://drive.google.com/file/d/1YaBnSdzuEvxWee-wUAUZcUzxiaO1524J/view?usp=sharing FOV Info] || [https://drive.google.com/file/d/1YaBnSdzuEvxWee-wUAUZcUzxiaO1524J/view?usp=sharing FOV Info] || 10 kHz || [https://docs.google.com/document/d/103S1ANjTPXDIWnOAlhEAbQy2oLm5ZOcqV8x1eR0hvxs/edit LOFAR Info Doc] || Anshu Kumari (anshusingh628@gmail.com)
|-

| LWA-NM || 10-88 MHz || < 1s || 15 - 23 || April 2011 - ongoing|| Full-sky || All-sky, ~2-15° (zenith) || 7.8 - 612 kHz || [https://leo.phys.unm.edu/~lwa/specs.html LWA Spec] || Stephen White (stephen.white.24@spaceforce.mil)
|-



| MWA || 80-300MHz || 0.5s || 01:00-08:00 || 2014 - ongoing || 30deg || depends on configuration; ~1-2′/200MHz || 10/40 kHz || [link MWA]|| Surajit Mondal (surajit.mondal@njit.edu)
|-

| NoRP || 1, 2, 3.75, 9.4, 17, 35, 80 GHz || 0.1s || 23 - 7 || ~1979 - ongoing (3.75 GHz since 1951) || Full Sun, unresolved || N/A || N/A || [https://solar.nro.nao.ac.jp/norp/index.html NRO] || Stephen White (stephen.white.24@spaceforce.mil)
|-

| NoRH || 17, 34 GHz || 1s, daily || 22.5 - 6.5 || 7/1/1992 - 3/31/2020|| Full Sun || 10″, 5″ || N/A || [https://hinode.isee.nagoya-u.ac.jp/ICCON/ Hinode Site] || Stephen White (stephen.white.24@spaceforce.mil), Nat Gopalswamy (nat.gopalswamy@nasa.gov)
|-

| OVRO-LWA || 15-88 MHz || 1ms (dynamic spectrum); 100ms (fast visibilities); 10s (regular mode) || 14-24:00 || under commissioning (some data already available) || Full Sky || 5′ at 80MHz || 24kHz (dynamic spectrum and regular mode), 100 kHz (fast visibilities) || [link to come] || Dale Gary (dgary@njit.edu) Bin Chen (bin.chen@njit.edu)
|-

| PSP/FIELDS || LFR: 10 kHz-2.4 MHz HFR:1.6 kHz-19.2 MHz || 7s || 0-24 || 2018/10/02 - Present during PSP Perihelia || N/A || N/A || Non-uniform, Δν/ν ~ 4.5% || [https://sprg.ssl.berkeley.edu/data/psp/data/sci/fields/ FIELDS data] [https://sprg.ssl.berkeley.edu/data/psp/data/sci/fields/ Data Release Notes] || Marc Pulupa (pulupa@berkeley.edu) Sam Badman (samuel.badman@cfa.harvard.edu)
|-



| RSTN || 245, 410, 610, 1415, 4995, 8800, 15400 MHz || 1s || 0 - 24 || 1980 - ongoing|| Full Sun, unresolved || N/A || N/A || [https://www.ngdc.noaa.gov/stp/space-weather/solar-data/solar-features/solar-radio/rstn-1-second/ NOAA], [https://www.sws.bom.gov.au/World_Data_Centre/2/8/9 SWS] || Stephen White (stephen.white.24@spaceforce.mil)
|-



| VLA || 1-18 GHz; 0.3 GHz (P-Band) || 50 ms - 5 s, depending on observing frequency || 15 - 23 || from 1990s; based on proposals || 42′ at 1 GHz || 1.3-46 ″ @ 1.3GHz depending on configuration|| Depends on configuration || [https://data.nrao.edu/portal/#/ NRAO Archive] || Tim Bastian (tbastian@nrao.edu)
|-
| NRH || 150, 173, 228, 270, 298, 327, 327.5, 408, 432, 445 MHz || 125 ms || 8 - 15 || Ongoing || Full Sun || 6′ @ 150 MHz, 2′ @ 450 MHz || || [https://secchirh.obspm.fr/ NRH] || L. Klein (ludwig.klein@obspm.fr) and A. Hamini (abdallah.hamini@obspm.fr)
|-

| E-Callisto || Varies: Multiple Instruments Throughout World || Varies || Varies || Ongoing || Full Sun || Varies || Varies || [https://www.e-callisto.org/Data/data.html E-Callisto] || N/A
|-

|}

== Other Useful Resources ==

# The [https://lesia.obspm.fr/cesra/sites.html CESRA-related data centers] is another place where you can find a list of different Radio instruments.

# The plots below show the Observation times of different instruments:

Radio Data from Around the World

2023-08-08T16:20:22Z

Schhabra: /* Other Useful Resources */

== Radio Data Access ==

{| class="wikitable" style="text-align: center; width: 1300px; height: 200px;"
|-
| '''Instrument''' || style="width: 400pt;" | '''Frequency Range/Central Frequencies''' ||style="width: 200pt;" |'''Cadence''' || '''Observing Time (UT)'''|| style="width: 250pt;" |'''Dates of Operation (start/end)''' || '''FOV''' || style="width: 130pt;" | '''Imaging FOV / Resolution''' || '''Spectroscopy Frequency Resolution''' || '''Useful Links''' || '''Points of Contact for Data '''
|-

| ALMA || 92-96, 104-108, 190-194, 202-206, 229-233, 245-249, 338.6-342.6, 350.6-345.6 GHz || 1s || 13 - 21 || 2016 - ongoing || Sun, sub active region scales || 6.54-66″ / 0.62-11.6″ || 2 GHz || [https://ui.adsabs.harvard.edu/abs/2022FrASS...9.7368B/abstract Harvard] || Stephen White (stephen.white.24@spaceforce.mil) Tim Bastian
|-



| EOVSA || 1-18 GHz || Sample time: 20ms: Full Sweep: 1s || ~14-24 || 2017 - ongoing || Point ||57″/<math>\nu_{GHz}</math> x 51″/<math>\nu_{GHz}</math> || 0.500 GHz || [http://ovsa.njit.edu//wiki/index.php You Are Here] || Dale Gary (dgary@njit.edu) Bin Chen (bin.chen@njit.edu)
|-

| F10.7|| 2.8 GHz || 3 times daily || 17 (18), 20, 23 (22) || 1947 - ongoing || Full Sun, unresolved || N/A || N/A || [https://www.spaceweather.gc.ca/forecast-prevision/solar-solaire/solarflux/sx-en.php Space Weather Monitor] || Sam Schonfeld (schonfsj@gmail.com)
|-



| LOFAR || LBA: 10 - 80 MHz, HBA: 120 - 240 MHz || 10ms || 8.5-16 || Time Allocation by Proposal|| [https://drive.google.com/file/d/1YaBnSdzuEvxWee-wUAUZcUzxiaO1524J/view?usp=sharing FOV Info] || [https://drive.google.com/file/d/1YaBnSdzuEvxWee-wUAUZcUzxiaO1524J/view?usp=sharing FOV Info] || 10 kHz || [https://docs.google.com/document/d/103S1ANjTPXDIWnOAlhEAbQy2oLm5ZOcqV8x1eR0hvxs/edit LOFAR Info Doc] || Anshu Kumari (anshusingh628@gmail.com)
|-

| LWA-NM || 10-88 MHz || < 1s || 15 - 23 || April 2011 - ongoing|| Full-sky || All-sky, ~2-15° (zenith) || 7.8 - 612 kHz || [https://leo.phys.unm.edu/~lwa/specs.html LWA Spec] || Stephen White (stephen.white.24@spaceforce.mil)
|-



| MWA || 80-300MHz || 0.5s || 01:00-08:00 || 2014 - ongoing || 30deg || depends on configuration; ~1-2′/200MHz || 10/40 kHz || [link MWA]|| Surajit Mondal (surajit.mondal@njit.edu)
|-

| NoRP || 1, 2, 3.75, 9.4, 17, 35, 80 GHz || 0.1s || 23 - 7 || ~1979 - ongoing (3.75 GHz since 1951) || Full Sun, unresolved || N/A || N/A || [https://solar.nro.nao.ac.jp/norp/index.html NRO] || Stephen White (stephen.white.24@spaceforce.mil)
|-

| NoRH || 17, 34 GHz || 1s, daily || 22.5 - 6.5 || 7/1/1992 - 3/31/2020|| Full Sun || 10″, 5″ || N/A || [https://hinode.isee.nagoya-u.ac.jp/ICCON/ Hinode Site] || Stephen White (stephen.white.24@spaceforce.mil), Nat Gopalswamy (nat.gopalswamy@nasa.gov)
|-

| OVRO-LWA || 15-88 MHz || 1ms (dynamic spectrum); 100ms (fast visibilities); 10s (regular mode) || 14-24:00 || under commissioning (some data already available) || Full Sky || 5′ at 80MHz || 24kHz (dynamic spectrum and regular mode), 100 kHz (fast visibilities) || [link to come] || Dale Gary (dgary@njit.edu) Bin Chen (bin.chen@njit.edu)
|-

| PSP/FIELDS || LFR: 10 kHz-2.4 MHz HFR:1.6 kHz-19.2 MHz || 7s || 0-24 || 2018/10/02 - Present during PSP Perihelia || N/A || N/A || Non-uniform, Δν/ν ~ 4.5% || [https://sprg.ssl.berkeley.edu/data/psp/data/sci/fields/ FIELDS data] [https://sprg.ssl.berkeley.edu/data/psp/data/sci/fields/ Data Release Notes] || Marc Pulupa (pulupa@berkeley.edu) Sam Badman (samuel.badman@cfa.harvard.edu)
|-



| RSTN || 245, 410, 610, 1415, 4995, 8800, 15400 MHz || 1s || 0 - 24 || 1980 - ongoing|| Full Sun, unresolved || N/A || N/A || [https://www.ngdc.noaa.gov/stp/space-weather/solar-data/solar-features/solar-radio/rstn-1-second/ NOAA], [https://www.sws.bom.gov.au/World_Data_Centre/2/8/9 SWS] || Stephen White (stephen.white.24@spaceforce.mil)
|-



| VLA || 1-18 GHz; 0.3 GHz (P-Band) || 50 ms - 5 s, depending on observing frequency || 15 - 23 || from 1990s; based on proposals || 42′ at 1 GHz || 1.3-46 ″ @ 1.3GHz depending on configuration|| Depends on configuration || [https://data.nrao.edu/portal/#/ NRAO Archive] || Tim Bastian (tbastian@nrao.edu)
|-
| NRH || 150, 173, 228, 270, 298, 327, 327.5, 408, 432, 445 MHz || 125 ms || 8 - 15 || Ongoing || Full Sun || 6′ @ 150 MHz, 2′ @ 450 MHz || || [https://secchirh.obspm.fr/ NRH] || L. Klein (ludwig.klein@obspm.fr) and A. Hamini (abdallah.hamini@obspm.fr)
|-

| E-Callisto || Varies || Varies || Varies || Ongoing || Full Sun || Varies || || [https://www.e-callisto.org/Data/data.html E-Callisto] || N/A
|-

|}

== Other Useful Resources ==

# The [https://lesia.obspm.fr/cesra/sites.html CESRA-related data centers] is another place where you can find a list of different Radio instruments.

Radio Data from Around the World

2023-08-08T16:15:30Z

Schhabra: /* Radio Data Access */

== Radio Data Access ==

{| class="wikitable" style="text-align: center; width: 1300px; height: 200px;"
|-
| '''Instrument''' || style="width: 400pt;" | '''Frequency Range/Central Frequencies''' ||'''Cadence''' || '''Observing Time (UT)'''|| style="width: 250pt;" |'''Dates of Operation (start/end)''' || '''FOV''' || style="width: 130pt;" | '''Imaging FOV / Resolution''' || '''Spectroscopy Frequency Resolution''' || '''Useful Links''' || '''Points of Contact for Data '''
|-

| ALMA || 92-96, 104-108, 190-194, 202-206, 229-233, 245-249, 338.6-342.6, 350.6-345.6 GHz || 1s || 13 - 21 || 2016 - ongoing || Sun, sub active region scales || 6.54-66″ / 0.62-11.6″ || 2 GHz || [https://ui.adsabs.harvard.edu/abs/2022FrASS...9.7368B/abstract Harvard] || Stephen White (stephen.white.24@spaceforce.mil) Tim Bastian
|-



| EOVSA || 1-18 GHz || Sample time: 20ms: Full Sweep: 1s || ~14-24 || 2017 - ongoing || Point ||57″/<math>\nu_{GHz}</math> x 51″/<math>\nu_{GHz}</math> || 0.500 GHz || [http://ovsa.njit.edu//wiki/index.php You Are Here] || Dale Gary (dgary@njit.edu) Bin Chen (bin.chen@njit.edu)
|-

| F10.7|| 2.8 GHz || 3 times daily || 17 (18), 20, 23 (22) || 1947 - ongoing || Full Sun, unresolved || N/A || N/A || [https://www.spaceweather.gc.ca/forecast-prevision/solar-solaire/solarflux/sx-en.php Space Weather Monitor] || Sam Schonfeld (schonfsj@gmail.com)
|-



| LOFAR || LBA: 10 - 80 MHz, HBA: 120 - 240 MHz || 10ms || 8.5-16 || Time Allocation by Proposal|| [https://drive.google.com/file/d/1YaBnSdzuEvxWee-wUAUZcUzxiaO1524J/view?usp=sharing FOV Info] || [https://drive.google.com/file/d/1YaBnSdzuEvxWee-wUAUZcUzxiaO1524J/view?usp=sharing FOV Info] || 10 kHz || [https://docs.google.com/document/d/103S1ANjTPXDIWnOAlhEAbQy2oLm5ZOcqV8x1eR0hvxs/edit LOFAR Info Doc] || Anshu Kumari (anshusingh628@gmail.com)
|-

| LWA-NM || 10-88 MHz || < 1s || 15 - 23 || April 2011 - ongoing|| Full-sky || All-sky, ~2-15° (zenith) || 7.8 - 612 kHz || [https://leo.phys.unm.edu/~lwa/specs.html LWA Spec] || Stephen White (stephen.white.24@spaceforce.mil)
|-



| MWA || 80-300MHz || 0.5s || 01:00-08:00 || 2014 - ongoing || 30deg || depends on configuration; ~1-2′/200MHz || 10/40 kHz || [link MWA]|| Surajit Mondal (surajit.mondal@njit.edu)
|-

| NoRP || 1, 2, 3.75, 9.4, 17, 35, 80 GHz || 0.1s || 23 - 7 || ~1979 - ongoing (3.75 GHz since 1951) || Full Sun, unresolved || N/A || N/A || [https://solar.nro.nao.ac.jp/norp/index.html NRO] || Stephen White (stephen.white.24@spaceforce.mil)
|-

| NoRH || 17, 34 GHz || 1s, daily || 22.5 - 6.5 || 7/1/1992 - 3/31/2020|| Full Sun || 10″, 5″ || N/A || [https://hinode.isee.nagoya-u.ac.jp/ICCON/ Hinode Site] || Stephen White (stephen.white.24@spaceforce.mil), Nat Gopalswamy (nat.gopalswamy@nasa.gov)
|-

| OVRO-LWA || 15-88 MHz || 1ms (dynamic spectrum); 100ms (fast visibilities); 10s (regular mode) || 14-24:00 || under commissioning (some data already available) || Full Sky || 5′ at 80MHz || 24kHz (dynamic spectrum and regular mode), 100 kHz (fast visibilities) || [link to come] || Dale Gary (dgary@njit.edu) Bin Chen (bin.chen@njit.edu)
|-

| PSP/FIELDS || LFR: 10 kHz-2.4 MHz HFR:1.6 kHz-19.2 MHz || 7s || 0-24 || 2018/10/02 - Present during PSP Perihelia || N/A || N/A || Non-uniform, Δν/ν ~ 4.5% || [https://sprg.ssl.berkeley.edu/data/psp/data/sci/fields/ FIELDS data] [https://sprg.ssl.berkeley.edu/data/psp/data/sci/fields/ Data Release Notes] || Marc Pulupa (pulupa@berkeley.edu) Sam Badman (samuel.badman@cfa.harvard.edu)
|-



| RSTN || 245, 410, 610, 1415, 4995, 8800, 15400 MHz || 1s || 0 - 24 || 1980 - ongoing|| Full Sun, unresolved || N/A || N/A || [https://www.ngdc.noaa.gov/stp/space-weather/solar-data/solar-features/solar-radio/rstn-1-second/ NOAA], [https://www.sws.bom.gov.au/World_Data_Centre/2/8/9 SWS] || Stephen White (stephen.white.24@spaceforce.mil)
|-



| VLA || 1-18 GHz; 0.3 GHz (P-Band) || 50 ms - 5 s, depending on observing frequency || 15 - 23 || from 1990s; based on proposals || 42′ at 1 GHz || 1.3-46 ″ @ 1.3GHz depending on configuration|| Depends on configuration || [https://data.nrao.edu/portal/#/ NRAO Archive] || Tim Bastian (tbastian@nrao.edu)
|-
| NRH || 150, 173, 228, 270, 298, 327, 327.5, 408, 432, 445 MHz || 125 ms || 8 - 15 || Ongoing || Full Sun || 6′ @ 150 MHz, 2′ @ 450 MHz || || [https://secchirh.obspm.fr/ NRH] || L. Klein (ludwig.klein@obspm.fr) and A. Hamini (abdallah.hamini@obspm.fr)
|-

|}

== Other Useful Resources ==

Radio Data from Around the World

2023-08-08T16:14:34Z

Schhabra: /* Radio Data Access */

== Radio Data Access ==

{| class="wikitable" style="text-align: center; width: 1300px; height: 200px;"
|-
| '''Instrument''' || style="width: 400pt;" | '''Frequency Range/Central Frequencies''' ||'''Cadence''' || '''Observing Time (UT)'''|| style="width: 250pt;" |'''Dates of Operation (start/end)''' || '''FOV''' || style="width: 130pt;" | '''Imaging FOV / Resolution''' || '''Spectroscopy Frequency Resolution''' || '''Useful Links''' || '''Points of Contact for Data '''
|-

| ALMA || 92-96, 104-108, 190-194, 202-206, 229-233, 245-249, 338.6-342.6, 350.6-345.6 GHz || 1s || 13 - 21 || 2016 - ongoing || Sun, sub active region scales || 6.54-66″ / 0.62-11.6″ || 2 GHz || [https://ui.adsabs.harvard.edu/abs/2022FrASS...9.7368B/abstract Harvard] || Stephen White (stephen.white.24@spaceforce.mil) Tim Bastian
|-



| EOVSA || 1-18 GHz || Sample time: 20ms: Full Sweep: 1s || ~14-24 || 2017 - ongoing || Point ||57″/<math>\nu_{GHz}</math> x 51″/<math>\nu_{GHz}</math> || 0.500 GHz || [http://ovsa.njit.edu//wiki/index.php You Are Here] || Dale Gary (dgary@njit.edu) Bin Chen (bin.chen@njit.edu)
|-

| F10.7|| 2.8 GHz || 3 times daily || 17 (18), 20, 23 (22) || 1947 - ongoing || Full Sun, unresolved || N/A || N/A || [https://www.spaceweather.gc.ca/forecast-prevision/solar-solaire/solarflux/sx-en.php Space Weather Monitor] || Sam Schonfeld (schonfsj@gmail.com)
|-



| LOFAR || LBA: 10 - 80 MHz, HBA: 120 - 240 MHz || 10ms || 8.5-16 || Time Allocation by Proposal|| [https://drive.google.com/file/d/1YaBnSdzuEvxWee-wUAUZcUzxiaO1524J/view?usp=sharing FOV Info] || [https://drive.google.com/file/d/1YaBnSdzuEvxWee-wUAUZcUzxiaO1524J/view?usp=sharing FOV Info] || 10 kHz || [https://docs.google.com/document/d/103S1ANjTPXDIWnOAlhEAbQy2oLm5ZOcqV8x1eR0hvxs/edit LOFAR Info Doc] || Anshu Kumari (anshusingh628@gmail.com)
|-

| LWA-NM || 10-88 MHz || < 1s || 15 - 23 || April 2011 - ongoing|| Full-sky || All-sky, ~2-15° (zenith) || 7.8 - 612 kHz || [https://leo.phys.unm.edu/~lwa/specs.html LWA Spec] || Stephen White (stephen.white.24@spaceforce.mil)
|-



| MWA || 80-300MHz || 0.5s || 01:00-08:00 || 2014 - ongoing || 30deg || depends on configuration; ~1-2′/200MHz || 10/40 kHz || [link MWA]|| Surajit Mondal (surajit.mondal@njit.edu)
|-

| NoRP || 1, 2, 3.75, 9.4, 17, 35, 80 GHz || 0.1s || 23 - 7 || ~1979 - ongoing (3.75 GHz since 1951) || Full Sun, unresolved || N/A || N/A || [https://solar.nro.nao.ac.jp/norp/index.html NRO] || Stephen White (stephen.white.24@spaceforce.mil)
|-

| NoRH || 17, 34 GHz || 1s, daily || 22.5 - 6.5 || 7/1/1992 - 3/31/2020|| Full Sun || 10″, 5″ || N/A || [https://hinode.isee.nagoya-u.ac.jp/ICCON/ Hinode Site] || Stephen White (stephen.white.24@spaceforce.mil), Nat Gopalswamy (nat.gopalswamy@nasa.gov)
|-

| OVRO-LWA || 15-88 MHz || 1ms (dynamic spectrum); 100ms (fast visibilities); 10s (regular mode) || 14-24:00 || under commissioning (some data already available) || Full Sky || 5′ at 80MHz || 24kHz (dynamic spectrum and regular mode), 100 kHz (fast visibilities) || [link to come] || Dale Gary (dgary@njit.edu) Bin Chen (bin.chen@njit.edu)
|-

| PSP/FIELDS || LFR: 10 kHz-2.4 MHz HFR:1.6 kHz-19.2 MHz || 7s || 0-24 || 2018/10/02 - Present during PSP Perihelia || N/A || N/A || Non-uniform, Δν/ν ~ 4.5% || [https://sprg.ssl.berkeley.edu/data/psp/data/sci/fields/ FIELDS data] [https://sprg.ssl.berkeley.edu/data/psp/data/sci/fields/ Data Release Notes] || Marc Pulupa (pulupa@berkeley.edu) Sam Badman (samuel.badman@cfa.harvard.edu)
|-



| RSTN || 245, 410, 610, 1415, 4995, 8800, 15400 MHz || 1s || 0 - 24 || 1980 - ongoing|| Full Sun, unresolved || N/A || N/A || [https://www.ngdc.noaa.gov/stp/space-weather/solar-data/solar-features/solar-radio/rstn-1-second/ NOAA], [https://www.sws.bom.gov.au/World_Data_Centre/2/8/9 SWS] || Stephen White (stephen.white.24@spaceforce.mil)
|-



| VLA || 1-18 GHz; 0.3 GHz (P-Band) || 50 ms - 5 s, depending on observing frequency || 15 - 23 || from 1990s; based on proposals || 42′ at 1 GHz || 1.3-46 ″ @ 1.3GHz depending on configuration|| Depends on configuration || [https://data.nrao.edu/portal/#/ NRAO Archive] || Tim Bastian (tbastian@nrao.edu)
|-
| NRH || 150, 173, 228, 270, 298, 327, 327.5, 408, 432, 445 MHz || 125 ms || 8 - 15 || Ongoing || Full Sun || 6′ @ 150 MHz, 2′ @ 450 MHz || || [https://secchirh.obspm.fr/ NRH] || L. Klein (ludwig.klein@obspm.fr) and A. Hamini (abdallah.hamini@obspm.fr)
|-

|}

Tohban EOVSA Imaging Tutorial A-Z V2

2022-07-07T23:16:19Z

Schhabra: Blanked the page

Tohban Guide to Self Calibration and Imaging for EOVSA

2022-07-07T23:14:36Z

Schhabra: Created page with "This tutorial describes the step-by-step procedure to download EOVSA IDB data, obtain and calibrate the measurement sets (.ms), and, transfer them to the inti server for self-calibration and further analysis. === Pre-requisites === Accounts on Pipeline and Inti or Baozi servers. Sijie and Bin can help with the account set-up for pipeline, inti and boazi. ==== How to connect to pipeline ==== '''Windows:''' On a windows machine Mobaxterm can be used to connect to Pipe..."

This tutorial describes the step-by-step procedure to download EOVSA IDB data, obtain and calibrate the measurement sets (.ms), and, transfer them to the inti server for self-calibration and further analysis.

=== Pre-requisites ===
Accounts on Pipeline and Inti or Baozi servers. Sijie and Bin can help with the account set-up for pipeline, inti and boazi.

==== How to connect to pipeline ====
'''Windows:''' On a windows machine Mobaxterm can be used to connect to Pipeline. The process to add rsa keys to access gues@ovsa is slightly complicated. Please contact Dale for the set up.

'''Linux/Unix:''' On a linux/Mac, the guest account will be used for people to log in as Tohban. Rather than using a password, access will be tied to specific computers that you use. One needs to generate a public + private ssh key pair using ssh-keygen or equivalent program that will result in a public key (plain text) file called id_rsa.pub, created in your ~/.ssh folder, the contents of which can be emailed to Dale or Owen to be added under "authorized keys". The ssh commands can then be used directly through the terminal:

<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">

ssh -L 5902:helios.solar.pvt:20000 guest@ovsa.njit.edu ### will not ask for a password for your computer's authorized key ###
ssh -XY your_username@pipeline ### add your username and enter password to your pipeline account, when prompted ###
</pre>

Alternatively, to avoid typing the longer commands, proxy commands can be added to the ~/.ssh/config file:

<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
vim ~/.ssh/config ### press 'i' to insert content and esc +':wq' to write and quit ###

Host *
ForwardAgent yes

Host *
ForwardX11 yes

Host ovsa
HostName ovsa.njit.edu
User guest

Host pipeline
Hostname pipeline.solar.pvt
User your_username ### Insert your pipeline username here ###
ProxyCommand ssh -W %h:%p guest@ovsa.njit.edu
</pre>

Upon saving the ~/.ssh/config file and restarting the terminal, you can connect to pipeline directly using:

<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
ssh -XY pipeline
</pre>

==== How to connect to the inti server ====
NOTE: To connect to inti you must have a NJIT UCID. The arcs have granted access to inti for all SRG members. If you encounter any issue with the login. Please let arcs folks (arcs@njit.edu) know.

'''Windows:''' As mentioned above, on a windows machine Mobaxterm can be used to connect to inti.
With your NJIT VPN connected, connect to one of the afsconnect servers (for example, afsaccess3.njit.edu) using your UCID and password. Next, ssh into inti.
<pre style="background-color: #FCEBD9">
ssh -XY UCID@afsaccess3.njit.edu ###Insert UCID password if prompted###

ssh -XY UCID@inti.hpcnet.campus.njit.edu ###Insert UCID/inti username here###
</pre>
Proxy commands for inti can also be added to the C:\Program Files\Git\etc\ssh\ssh_config file:
<pre style="background-color: #FCEBD9">
Host inti
Hostname inti.hpcnet.campus.njit.edu
User UCID ###Insert UCID/inti username here###
</pre>

'''Linux/Unix:''' On a linux/Mac the ssh command can be used:
<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
ssh -XY UCID@inti.hpcnet.campus.njit.edu ### insert UCID here and enter your NJIT UCID password, when prompted ###
</pre>

Proxy commands for inti can also be added to the ~/.ssh/config (on local machine) file for convenience:
<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
vim ~/.ssh/config ### press 'i' to insert content and esc +':wq' to write and quit ###

Host inti
Hostname inti.hpcnet.campus.njit.edu
User UCID ### insert NJIT UCID here ###
</pre>
after the proxy is set up, connect to inti by simply typing:
<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
ssh -XY inti ### enter your NJIT UCID password when prompted ###
</pre>

'''NOTE:'''When logging in for the first time, please add the following lines (demarcated by #) in your ~/.bashrc (on the inti server) file in order to use CASA on inti.

<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
#### setting start ####
if [ $HOSTNAME == "baozi.hpcnet.campus.njit.edu" ]; then
source /srg/.setenv_baozi
fi
if [ $HOSTNAME == "inti.hpcnet.campus.njit.edu" ]; then
source /inti/.setenv_inti
fi
if [ $HOSTNAME == "guko.resource.campus.njit.edu" ]; then
source /data/data/.setenv_guko
fi
#### setting end ####
</pre>

For sufficient disk space to analyze EOVSA data on Inti, use your dedicated directory YOURDIRECTORY at the location given below. If you do not have a directory, please ask Sijie to set one up.
<pre style="background-color: #FCEBD9">
cd /inti/data/users/YOURDIRECTORY
</pre>

'''Both CASA 5 and 6 are available on Inti.'''

Enter the bash environment on inti, and load the desired casa environment.

To load CASA 5:
<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
>> bash
>> loadcasa5
>> suncasa
</pre>

To load CASA 6:
<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
>> bash
>> loadcasa6
>> ipython
</pre>

=== Downloading data and Preliminary Calibration ===

==== Step 1: Downloading raw (IDB) data on Pipeline machine====
The initial downloading of data and applying refcal and phasecal to the .ms files is performed on the pipeline since the complete EOVSA SQL database is stored here for now.

If you are not using mobaxterm, directly SSH to Pipeline through your Linux or Mac computer (as shown above) and '''start tcsh to run CASA'''.

The following commands (run within CASA) can be used to import a single IDB file if the given time-range is less than 10 mins.
<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
from astropy.time import Time
import os
trange = Time(['2017-08-21 20:15:00', '2017-08-21 20:25:00']) ###Change accordingly###

#### (Optional) change output path, default current directory "./" #######
outpath = './msdata/' ###Change accordingly###
if not os.path.exists(outpath):
os.makedirs(outpath)

msfiles = importeovsa(idbfiles=trange, ncpu=1, visprefix=outpath)
</pre>

'''NOTE:''' The above step currently gives an error with importeovsa. This is because the data location was changed and the code doesn't know where to look for the IDB files. Sijie is looking into the problem. We will update the page when it is fixed. For now, please use the method below for all time ranges.

OR (for a time range longer than 10 minutes)

<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
from suncasa.tasks import task_calibeovsa as calibeovsa
from suncasa.tasks import task_importeovsa as timporteovsa
from split_cli import split_cli as split
import dump_tsys as dt
from util import Time
import numpy as np
import os
from glob import glob
from eovsapy import util

trange = Time(['2017-08-21 20:15:00', '2017-08-21 20:35:00']) ###Change accordingly###
idbdir = util.get_idbdir(trange[0])

info = dt.rd_fdb(trange[0])
for k, v in info.iteritems():
info[k] = info[k][~(info[k] == '')]

sidx = np.where(
np.logical_and(info['SOURCEID'] == 'Sun', info['PROJECTID'] == 'NormalObserving') & np.logical_and(
info['ST_TS'].astype(np.float) >= trange[0].lv,
info['ST_TS'].astype(np.float) <= trange[
1].lv))
filelist = info['FILE'][sidx]

outpath = './msdata/' ###Change accordingly###
if not os.path.exists(outpath):
os.makedirs(outpath)
inpath = idbdir + '{}/'.format(trange[0].datetime.strftime("%Y%m%d"))
ncpu = 1

msfiles = timporteovsa.importeovsa(idbfiles=[inpath + ll for ll in filelist], ncpu=ncpu, timebin="0s", width=1,
visprefix=outpath,
nocreatms=False, doconcat=False,
modelms="", doscaling=False, keep_nsclms=False, udb_corr=True)
</pre>

If you come across errors with calibeovsa, add the following lines to your ~/.casa/init.py (within pipeline) file.
<pre style="background-color: #FCEBD9">
vim ~/.ssh/config ### press 'i' to insert content and esc +':wq' to write and quit ###
import sys
sys.path.append('/common/python')
sys.path.append('/common/python/packages/pipeline_casa')
execfile('/common/python/suncasa/tasks/mytasks.py')
</pre>

==== Step 2: Concatenate multiple .ms files and Calibrate====
'''Follow this step if there are more than one .ms files for the chosen time-range. If not, move to Step 3.'''

<pre style="background-color: #FCEBD9">
concatvis = os.path.basename(msfiles[0])[:11] + '_concat_cal.ms' ### Set the path/name for the concatenated files ###

vis = calibeovsa.calibeovsa(msfiles, doconcat=True, concatvis=concatvis, caltype=['refpha','phacal'], doimage=True)
### If doimage=True, a quicklook image will be produced (by integrating over the entire time) as shown in Figure 1 ###

vis_str = str(' '.join(vis))
caled_vis=vis_str.replace('.ms','_cal.ms') ### Append '_cal' to the ms filename ###
split(vis=' '.join(vis),outputvis=caled_vis,datacolumn='corrected',timerange='',correlation='XX') ### Split the corrected column to the new 'caled' ms ###
</pre>

[[file:Figure1_imagingtutorial.png|thumb|right|300px|'''Figure 1:''' Quick-look full-Sun image after the initial calibration.]]

==== Step 3: Calibration ====
For a single .ms file from Step 1, this will calibrate the input visibility, write out calibration tables under /data1/eovsa/caltable/, and apply the calibration.
<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
vis = calibeovsa.calibeovsa('IDB20170821202020.ms', caltype=['refpha','phacal'], doimage=True) ###Change the vis filename accordingly###

vis_str = str(' '.join(vis))
caled_vis=vis_str.replace('.ms','_cal.ms') ### Append '_cal' to the ms filename ###

split(vis=' '.join(vis),outputvis=caled_vis,datacolumn='corrected',timerange='',correlation='XX') ### Split the corrected column to the new 'caled' ms ###
</pre>

The created 'caled' ms data files (xxx_concat_cal.ms or xxx_cal.ms) need to be transferred from pipeline to inti server, which has all the casa tasks installed, in order to run the rest of the imaging steps.

=== Transferring data files between servers ===
To directly transfer your calibrated .ms data from the Pipeline to Inti:

<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
1. Log into Inti using your username.
ssh -X USERNAME@inti.hpcnet.campus.njit.edu

Then create a tunnel into Pipeline from Inti.
ssh -L 8888:pipeline.solar.pvt:22 guest@ovsa.njit.edu

2. Log into Inti again from a new terminal. Change to your working directory (where you need the data transferred) and give the following command:

scp -v -C -r -P 8888 PipelineUSERNAME@localhost:PATH/TO/MSFILE ./

</pre>

===Advanced Calibration and Imaging===
====Step 4: Self-calibration====
[[file:Figure3_imagingtutorial.png|thumb|center|400px|Figure 2: Cotton-Schwab clean major and minor cycles. [Source: http://www.aoc.nrao.edu/~rurvashi/ImagingAlgorithmsInCasa/node2.html].]]
Follow the steps below to run the self-calibration of the imaging data and produce the calibrated images in .fits format. https://github.com/binchensun/casa-eovsa/blob/master/slfcal_example.py

<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
bash
loadcasa5
suncasa

from suncasa.utils import helioimage2fits as hf
import os
import numpy as np
import pickle
from matplotlib import gridspec as gridspec
from sunpy import map as smap
from matplotlib import pyplot as plt
from split_cli import split_cli as split
import time

# =========== task handlers =============
dofullsun = 1 # initial full-sun imaging ###Change accordingly###
domasks=1 # get masks ###Change accordingly###
doslfcal=1 # main cycle of doing selfcalibration ###Change accordingly###
doapply=1 # apply the results ###Change accordingly###
doclean_slfcaled=1 # perform clean for self-calibrated data ###Change accordingly###

# ============ declaring the working directories ============
workdir = os.getcwd()+'/' #main working directory. Using current directory in this example
slfcaldir = workdir+'slfcal/' #place to put all selfcalibration products
imagedir = slfcaldir+'images/' #place to put all selfcalibration images
maskdir = slfcaldir+'masks/' #place to put clean masks
imagedir_slfcaled = slfcaldir+'images_slfcaled/' #place to put final self-calibrated images
caltbdir = slfcaldir+'caltbs/' # place to put calibration tables
# make these directories if they do not already exist
dirs = [workdir, slfcaldir, imagedir, maskdir, imagedir_slfcaled, caltbdir]
for d in dirs:
if not os.path.exists(d):
os.makedirs(d)

# ============ Split a short time for self-calibration ===========
# input visibility
ms_in = workdir + 'IDB20170821202020_cal.ms' ###Change the initial calibrated (through calibeovsa) vis accordingly###
# output, selfcaled, visibility
ms_slfcaled = workdir + os.path.basename(ms_in).replace('cal','slfcaled')
# intermediate small visibility for selfcalbration
# selected time range for generating self-calibration solutions
trange='2017/08/21/20:21:10~2017/08/21/20:21:30' ###Change accordingly###
slfcalms = slfcaldir+'slfcalms.XX.slfcal'
slfcaledms = slfcaldir+'slfcalms.XX.slfcaled'
if not os.path.exists(slfcalms):
split(vis=ms_in,outputvis=slfcalms,datacolumn='data',timerange=trange,correlation='XX')

# ============ Prior definitions for spectral windows, antennas, pixel numbers =========
spws=[str(s+1) for s in range(30)] ###spws=[str(s+1) for s in range(49)] # For post-2020 data
antennas='0~12&0~12'
npix=512
nround=3 #number of slfcal cycles
</pre>

[[file:Figure2_imagingtutorial.png|thumb|right|300px|Figure 3: Full-Sun image after initial clean to find the flare location.]]
<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
# =========== Step 1, doing a full-Sun image to find out phasecenter and appropriate field of view =========
if dofullsun:
#initial mfs clean to find out the image phase center
im_init='fullsun_init'
os.system('rm -rf '+im_init+'*')
tclean(vis=slfcalms,
antenna=antennas,
imagename=im_init,
spw='2~15',
specmode='mfs',
timerange=trange,
imsize=[npix],
cell=['5arcsec'],
niter=1000,
gain=0.05,
stokes='I',
restoringbeam=['30arcsec'],
interactive=False,
pbcor=True)

hf.imreg(vis=slfcalms,imagefile=im_init+'.image.pbcor',fitsfile=im_init+'.fits',
timerange=trange,usephacenter=False,verbose=True)
clnjunks = ['.flux', '.mask', '.model', '.psf', '.residual','.sumwt','.pb','.image'] ###Do not run the next 4 lines if needed to view and assess the subset of clean process images
for clnjunk in clnjunks:
if os.path.exists(im_init + clnjunk):
os.system('rm -rf '+im_init + clnjunk)

from sunpy import map as smap
from matplotlib import pyplot as plt
fig = plt.figure(figsize=(6,6))
ax = fig.add_subplot(111)
eomap=smap.Map(im_init+'.fits')
#eomap.data=eomap.data.reshape((npix,npix))
eomap.plot_settings['cmap'] = plt.get_cmap('jet')
eomap.plot(axes = ax)
eomap.draw_limb()
plt.show()
viewer(im_init+'.image.pbcor')

# parameters specific to the event (found from step 1)
phasecenter='J2000 10h02m59 11d58m07' ###Change accordingly###
xran=[280,480] ###Change accordingly###
yran=[-50,150] ###Change accordingly###
</pre>

[[file:Figure4_imagingtutorial.PNG|thumb|right|300px|Figure 4: Interactive clean window to create masks over the source. The white outline surrounding the source is the mask selected by the polygon drawing option.]]
<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
# =========== Step 2 (optional), generate masks =========
# The outlines drawn for masks can be created by any of the icons with the letter 'R' in the Viewer window. The instructions for doing this can be found by hovering over those icons. See Figure 4.
# if skipped, will not use any masks
if domasks:
clearcal(slfcalms)
delmod(slfcalms)
antennas=antennas
pol='XX'
imgprefix=maskdir+'slf_t0'

# step 1: set up the clean masks
img_init=imgprefix+'_init_ar_'
os.system('rm -rf '+img_init+'*')
#spwrans_mask=['1~5','6~12','13~20','21~30']
spwrans_mask=['2~12']
#convert to a list of spws
spwrans_mask_list = [[str(i) for i in (np.arange(int(m.split('~')[0]),int(m.split('~')[1])))] for m in spwrans_mask] # Not used?
masks=[]
imnames=[]
for spwran in spwrans_mask:
imname=img_init+spwran.replace('~','-')
try:
tclean(vis=slfcalms,
antenna=antennas,
imagename=imname,
spw=spwran,
specmode='mfs',
timerange=trange,
imsize=[npix],
cell=['2arcsec'],
niter=1000,
gain=0.05,
stokes='XX',
restoringbeam=['20arcsec'],
phasecenter=phasecenter,
weighting='briggs',
robust=1.0,
interactive=True,
datacolumn='data',
pbcor=True,
savemodel='modelcolumn')
imnames.append(imname+'.image')
masks.append(imname+'.mask')
clnjunks = ['.flux', '.model', '.psf', '.residual'] #Do not run the next 4 lines if needed to view and assess the subset of clean process images
for clnjunk in clnjunks:
if os.path.exists(imname + clnjunk):
os.system('rm -rf '+ imname + clnjunk)
except:
print('error in cleaning spw: '+spwran)

pickle.dump(masks,open(slfcaldir+'masks.p','wb'))
</pre>

[[file:Slf.G0.png|thumb|right|300px|Figure 5: Phase before self-calibration]]
[[file:Slf.G1.png|thumb|right|300px|Figure 6: Phase after self-calibration]]
<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
# =========== Step 3, main step of selfcalibration =========
if doslfcal:
if os.path.exists(slfcaldir+'masks.p'):
masks=pickle.load(open(slfcaldir+'masks.p','rb'))
if not os.path.exists(slfcaldir+'masks.p'):
print 'masks do not exist. Use default mask'
masks=[]
os.system('rm -rf '+imagedir+'*')
os.system('rm -rf '+caltbdir+'*')
#first step: make a mock caltable for the entire database
print('Processing ' + trange)
slftbs=[]
calprefix=caltbdir+'slf'
imgprefix=imagedir+'slf'
tb.open(slfcalms+'/SPECTRAL_WINDOW')
reffreqs=tb.getcol('REF_FREQUENCY')
bdwds=tb.getcol('TOTAL_BANDWIDTH')
cfreqs=reffreqs+bdwds/2.
tb.close()
# starting beam size at 3.4 GHz in arcsec #Change accordingly
sbeam=40.
strtmp=[m.replace(':','') for m in trange.split('~')]
timestr='t'+strtmp[0]+'-'+strtmp[1]
refantenna='0'
# number of iterations for each round
niters=[100, 300, 500]
# roburst value for weighting the baselines
robusts=[1.0, 0.5, 0.0]
# apply calibration tables? Set to true for most cases
doapplycal=[1,1,1]
# modes for calibration, 'p' for phase-only, 'a' for amplitude only, 'ap' for both
calmodes=['p','p','a']
# setting uvranges for model image (optional, not used here)
uvranges=['','','']
for n in range(nround):
slfcal_tb_g= calprefix+'.G'+str(n)
fig = plt.figure(figsize=(8.4,7.)) # fig = plt.figure(figsize=(14, 7)) # For post-2020 data (50 spws)
gs = gridspec.GridSpec(5, 6) # gs = gridspec.GridSpec(5, 10) # For post-2020 data (50 spws)
for s,sp in enumerate(spws):
print 'processing spw: '+sp
cfreq=cfreqs[int(sp)]
# setting restoring beam size (not very useful for selfcal anyway, but just to see the results)
bm=max(sbeam*cfreqs[1]/cfreq, 6.)
slfcal_img = imgprefix+'.spw'+sp.zfill(2)+'.slfcal'+str(n)
# only the first round uses nearby spws for getting initial model
if n == 0:
spbg=max(int(sp)-2,1) # spbg=max(int(sp)-2,0) # For post-2020 data (50 spws)
sped=min(int(sp)+2,30) # sped=min(int(sp)+2,49) # For post-2020 data (50 spws)
spwran=str(spbg)+'~'+str(sped)
print('using spw {0:s} as model'.format(spwran))
if 'spwrans_mask' in vars():
for m, spwran_mask in enumerate(spwrans_mask):
if sp in spwran_mask:
mask = masks[m]
print('using mask {0:s}'.format(mask))
findmask = True
if not findmask:
print('mask not found. Do use any masks')
else:
spwran = sp
if 'spwrans_mask' in vars():
for m, spwran_mask in enumerate(spwrans_mask):
if sp in spwran_mask:
mask = masks[m]
print 'using mask {0:s}'.format(mask)
findmask = True
if not findmask:
print('mask not found. Do use any masks')
try:
tclean(vis=slfcalms,
antenna=antennas,
imagename=slfcal_img,
uvrange=uvranges[n],
spw=spwran,
specmode='mfs',
timerange=trange,
imsize=[npix],
cell=['2arcsec'],
niter=niters[n],
gain=0.05,
stokes='XX', #use pol XX image as the model
weighting='briggs',
robust=robusts[n],
phasecenter=phasecenter,
mask=mask,
restoringbeam=[str(bm)+'arcsec'],
pbcor=False,
interactive=False,
savemodel='modelcolumn')
if os.path.exists(slfcal_img+'.image'):
fitsfile=slfcal_img+'.fits'
hf.imreg(vis=slfcalms,imagefile=slfcal_img+'.image',fitsfile=fitsfile,
timerange=trange,usephacenter=False,toTb=True,verbose=False,overwrite=True)
clnjunks = ['.mask','.flux', '.model', '.psf', '.residual', '.image','.pb','.image.pbcor','.sumwt'] #Do not run the next 4 lines if needed to view and assess the subset of clean process images
for clnjunk in clnjunks:
if os.path.exists(slfcal_img + clnjunk):
os.system('rm -rf '+ slfcal_img + clnjunk)
ax = fig.add_subplot(gs[s])
eomap=smap.Map(fitsfile)
eomap.plot_settings['cmap'] = plt.get_cmap('jet')
eomap.plot(axes = ax)
eomap.draw_limb()
#eomap.draw_grid()
ax.set_title(' ')
ax.get_xaxis().set_visible(False)
ax.get_yaxis().set_visible(False)
ax.set_xlim(xran)
ax.set_ylim(yran)
plt.pause(0.5) # Allows viewing of each image as it is plotted.
os.system('rm -f '+ fitsfile)

except:
print 'error in cleaning spw: '+sp
print 'using nearby spws for initial model'
sp_e=int(sp)+2
sp_i=int(sp)-2
if sp_i < 1: # if sp < 0: # For post-2020 data (50 spws)
sp_i = 1 # sp_i = 0 # For post-2020 data (50 spws)
if sp_e > 30: # if sp_e > 49: # For post-2020 data (50 spws)
sp_e = 30 # sp_e = 49 # For post-2020 data (50 spws)
sp_=str(sp_i)+'~'+str(sp_e)
try:
tget(tclean)
spw=sp_
print('using spw {0:s} as model'.format(sp_))
tclean()
except:
print 'still not successful. abort...'
break

gaincal(vis=slfcalms, refant=refantenna,antenna=antennas,caltable=slfcal_tb_g,spw=sp, uvrange='',\
gaintable=[],selectdata=True,timerange=trange,solint='inf',gaintype='G',calmode=calmodes[n],\
combine='',minblperant=4,minsnr=2,append=True)
if not os.path.exists(slfcal_tb_g):
print 'No solution found in spw: '+sp
figname=imagedir+'slf_t0_n{:d}.png'.format(n)
plt.subplots_adjust(left=0, bottom=0, right=1, top=1, wspace=0, hspace=0)
plt.savefig(figname)
time.sleep(10)
plt.close()

if os.path.exists(slfcal_tb_g):
slftbs.append(slfcal_tb_g)
slftb=[slfcal_tb_g]
os.chdir(slfcaldir)
if calmodes[n] == 'p':
plotcal(caltable=slfcal_tb_g,antenna='1~12',xaxis='freq',yaxis='phase',\
subplot=431,plotrange=[-1,-1,-180,180],iteration='antenna',figfile=slfcal_tb_g+'.png',showgui=False)
if calmodes[n] == 'a':
plotcal(caltable=slfcal_tb_g,antenna='1~12',xaxis='freq',yaxis='amp',\
subplot=431,plotrange=[-1,-1,0,2.],iteration='antenna',figfile=slfcal_tb_g+'.png',showgui=False)
os.chdir(workdir)
plt.pause(0.5) # Allows viewing of the plot

if doapplycal[n]:
clearcal(slfcalms)
delmod(slfcalms)
applycal(vis=slfcalms,gaintable=slftb,spw=','.join(spws),selectdata=True,\
antenna=antennas,interp='nearest',flagbackup=False,applymode='calonly',calwt=False)

if n < nround-1:
prompt=raw_input('Continuing to selfcal?')
#prompt='y'
if prompt.lower() == 'n':
if os.path.exists(slfcaledms):
os.system('rm -rf '+slfcaledms)
split(slfcalms,slfcaledms,datacolumn='corrected')
print 'Final calibrated ms is {0:s}'.format(slfcaledms)
break
if prompt.lower() == 'y':
slfcalms_=slfcalms+str(n)
if os.path.exists(slfcalms_):
os.system('rm -rf '+slfcalms_)
split(slfcalms,slfcalms_,datacolumn='corrected')
slfcalms=slfcalms_
else:
if os.path.exists(slfcaledms):
os.system('rm -rf '+slfcaledms)
split(slfcalms,slfcaledms,datacolumn='corrected')
print 'Final calibrated ms is {0:s}'.format(slfcaledms)
</pre>

<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
# =========== Step 4: Apply self-calibration tables =========
if doapply:
import glob
os.chdir(workdir)
clearcal(ms_in)
clearcal(slfcalms)
applycal(vis=slfcalms,gaintable=slftbs,spw=','.join(spws),selectdata=True,\
antenna=antennas,interp='linear',flagbackup=False,applymode='calonly',calwt=False)
applycal(vis=ms_in,gaintable=slftbs,spw=','.join(spws),selectdata=True,\
antenna=antennas,interp='linear',flagbackup=False,applymode='calonly',calwt=False)
if os.path.exists(ms_slfcaled):
os.system('rm -rf '+ms_slfcaled)
split(ms_in, ms_slfcaled,datacolumn='corrected')
</pre>

[[file:Slf_t0_n0.png|thumb|right|300px|Figure 7: Multi-frequency images before self-calibration]]
[[file:Slf_t0_n1.png|thumb|right|300px|Figure 8: Multi-frequency images after self-calibration]]

<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
# =========== Step 5: Generate final self-calibrated images (optional) =========
if doclean_slfcaled:
import glob
pol='XX'
print('Processing ' + trange)
img_final=imagedir_slfcaled+'/slf_final_{0}_t0'.format(pol)
vis = ms_slfcaled
spws=[str(s+1) for s in range(30)]
tb.open(vis+'/SPECTRAL_WINDOW')
reffreqs=tb.getcol('REF_FREQUENCY')
bdwds=tb.getcol('TOTAL_BANDWIDTH')
cfreqs=reffreqs+bdwds/2.
tb.close()
sbeam=30.
from matplotlib import gridspec as gridspec
from sunpy import map as smap
from matplotlib import pyplot as plt
fitsfiles=[]
for s,sp in enumerate(spws):
cfreq=cfreqs[int(sp)]
bm=max(sbeam*cfreqs[1]/cfreq,4.)
imname=img_final+'_s'+sp.zfill(2)
fitsfile=imname+'.fits'
if not os.path.exists(fitsfile):
print 'cleaning spw {0:s} with beam size {1:.1f}"'.format(sp,bm)
try:
tclean(vis=vis,
antenna=antennas,
imagename=imname,
spw=sp,
specmode='mfs',
timerange=trange,
imsize=[npix],
cell=['1arcsec'],
niter=1000,
gain=0.05,
stokes=pol,
weighting='briggs',
robust=2.0,
restoringbeam=[str(bm)+'arcsec'],
phasecenter=phasecenter,
mask='',
pbcor=True,
interactive=False)
except:
print 'cleaning spw '+sp+' unsuccessful. Proceed to next spw'
continue
if os.path.exists(imname+'.image.pbcor'):
imn = imname+'.image.pbcor'
hf.imreg(vis=vis,imagefile=imn,fitsfile=fitsfile,
timerange=trange,usephacenter=False,toTb=True,verbose=False)
fitsfiles.append(fitsfile)
junks=['.flux','.model','.psf','.residual','.mask','.image','.pb','.image.pbcor','.sumwt'] #Do not run the next 4 lines if needed to view and assess the subset of clean process images
for junk in junks:
if os.path.exists(imname+junk):
os.system('rm -rf '+imname+junk)
else:
print('fits file '+fitsfile+' already exists, skip clean...')
fitsfiles.append(fitsfile)

fig = plt.figure(figsize=(8.4,7.)) # fig = plt.figure(figsize=(14, 7)) # For post-2020 data (50 spws)
gs = gridspec.GridSpec(5, 6) # gs = gridspec.GridSpec(5, 10) # For post-2020 data (50 spws)
for s,sp in enumerate(spws):
cfreq=cfreqs[int(sp)]
ax = fig.add_subplot(gs[s])
eomap=smap.Map(fitsfiles[s])
eomap.plot_settings['cmap'] = plt.get_cmap('jet')
eomap.plot(axes = ax)
eomap.draw_limb()
ax.set_title(' ')
ax.get_xaxis().set_visible(False)
ax.get_yaxis().set_visible(False)
ax.set_xlim(xran)
ax.set_ylim(yran)
plt.text(0.98,0.85,'{0:.1f} GHz'.format(cfreq/1e9),transform=ax.transAxes,ha='right',color='w',fontweight='bold')
plt.subplots_adjust(left=0, bottom=0, right=1, top=1, wspace=0, hspace=0)
plt.show()

</pre>
The .fits files of the self calibrated images at each frequency for the given time are saved at /slfcal/images_slfcaled in your working directory.

====Step 5: Quick-look imaging ==== 
For spectral imaging analysis of the event, follow [http://www.ovsa.njit.edu/wiki/index.php/EOVSA_Data_Analysis_Tutorial#Spectral_Imaging_with_SunCASA this tutorial] using the self-calibrated data obtained from the previous step or use [https://colab.research.google.com/drive/1lSLLxgOG6b8kgu9Sk6kSKvrViyubnXG6?usp=sharing#scrollTo=xbXyyLmCFCGL this link].

<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
from suncasa.utils import qlookplot as ql ## (Optional) Supply the npz file of the dynamic spectrum from previous step.
## If not provided, the program will generate a new one from the visibility.
## set the time interval
from suncasa import dspec as ds
import time
visibility_data = 'IDB20170821202020_slfcaled.ms'
specfile = visibility_data + '.dspec.npz'
d = ds.Dspec(visibility_data, bl='4&9', specfile=specfile)

timerange = '19:02:00~19:02:10' ## select frequency range from 2.5 GHz to 3.5 GHz
spw = '2~5' ## select stokes XX
stokes = 'XX' ## turn off AIA image plotting, default is True
plotaia = False
xycen = [375, 45] ## image center for clean in solar X-Y in arcsec
cell=['2.0arcsec'] ## pixel size
imsize=[128] ## x and y image size in pixels.
fov = [100,100] ## field of view of the zoomed-in panels in unit of arcsec
spw = ['{}'.format(s) for s in range(1,31)]
clevels = [0.5, 1.0] ## contour levels to fill in between.
calpha=0.35 ## now tune down the alpha
restoringbeam=['6arcsec']
ql.qlookplot(vis=msfile, specfile=specfile, timerange=timerange, spw=spw, stokes=stokes, \
restoringbeam=restoringbeam,imsize=imsize,cell=cell, \
xycen=xycen,fov=fov,clevels=clevels,calpha=calpha)
</pre>

Page Last Updated: 07/07/2022

Owens Valley Solar Arrays

2022-07-07T23:12:34Z

Schhabra: /* Tohbans */

[[File:Eovsa1.png|border|text-top|800px]]

<big>[http://ovsa.njit.edu/ EOVSA] (Expanded Owens Valley Solar Array) is a new solar-dedicated radio interferometer operated by the New Jersey Institute of Technology. This wiki serves as the site for EOVSA documentation. </big>

== EOVSA Documentation ==

* <big>General</big>
** [[Downconversion and Frequency Tuning]]
** [[Dealing with Radio Frequency Interference]]
** [[Switching between 200 MHz and 300 MHz Correlator]]

* <big>Computer-Network</big>
** [[Computing Systems]]
** [[Network]]

* <big>Control System</big>
** [[27-m Antenna Commands]]
** [[Schedule Commands]]
** [[Control Commands]]
** [[Attenuation and Level Control]]

* <big>Hardware</big>
** [[Hardware Overview]]
** [[2.1-m Antennas]]
** [[27-m Antennas]]

* <big>System Software</big>
** [[Calibration Database]]
** [[Stateframe Database]]
** [[Database Maintenance]]
** [[Create CASA measurement sets]]

* <big>Calibration</big>
**[[Calibration Overview]]
**[[Pointing Calibration]]
**[[Total Power Calibration]]
**[[System Gain Calibration]]
**[[Antenna Position]] (Baseline Calibration)
**[[Reference Gain Calibration]]
**[[Daily Gain Calibration]]
**[[Delay Calibration]]
**[[Bandpass Calibration]]
**[[Polarization Calibration]]
**[[Calibrator Survey]]
**[[Practical Calibration Tutorial]]

* <big>[[Starburst]]</big>

== Using EOVSA Data ==
* <big>[[EOVSA Data products]]</big>
* <big>Analysis Software</big>
** [https://github.com/suncasa/suncasa SunCASA] A wrapper around [https://casa.nrao.edu/ CASA (the Common Astronomy Software Applications package)] for synthesis imaging and visualizing solar spectral imaging data. CASA is one of the leading software tool for "supporting the data post-processing needs of the next generation of radio astronomical telescopes such as ALMA and VLA", an international effort led by the [https://public.nrao.edu/ National Radio Astronomy Observatory]. The current version of CASA uses Python (2.7) interface. More information about CASA can be found on [https://casa.nrao.edu/ NRAO's CASA website ]. Note, CASA is available ONLY on UNIX-BASED PLATFORMS (and therefore, so is SunCASA).
** [https://github.com/Gelu-Nita/GSFIT GSFIT] A IDL-widget(GUI)-based spectral fitting package called gsfit, which provides a user-friendly display of EOVSA image cubes and an interface to fast fitting codes (via platform-dependent shared-object libraries).
** [[Spectrogram Software]]
** [[Mapping Software]]
* <big>Data Analysis Guides</big>
** <big>[[EOVSA Data Analysis Tutorial 2022]]</big> and <big>[https://colab.research.google.com/drive/19NQb6Emb9HvKX4QHq9ZYCP3RM6nT7sDL#scrollTo=cLdDVptBGG-X EOVSA Workspace]</big> at [https://sphere.boulder.swri.edu/ SPHERE 2022 Workshop]
** <big>[https://colab.research.google.com/drive/1lSLLxgOG6b8kgu9Sk6kSKvrViyubnXG6?usp=sharing#scrollTo=xbXyyLmCFCGL EOVSA Data Analysis Tutorial at RHESSI 19 Workshop]</big>
** <big>[[EOVSA Data Analysis Tutorial]]</big> at [http://rhessi18.umn.edu/ RHESSI XVIII Workshop]
** [[Self-Calibrating Flare Data]] Example script and guides for self-calibrating EOVSA flare data (to be completed)


**[[IDB flare pipeline]] Tutorial to run the flare pipeline for quicklook images



* <big>EOVSA Modeling Guide</big>
**[[GX Simulator]]

* Other helpful links
** [https://casaguides.nrao.edu CASA Guides]
** [http://www.lmsal.com/solarsoft/ SolarSoft IDL]
** [http://www.atnf.csiro.au/computing/software/miriad/userguide/userhtml.html Miriad Guides]
** [https://sites.google.com/site/fgscodes/ Fast Gyrosynchrotron Codes (Alexey Kuznetsov's website)]
** [[Basic GitHub Tutorial]]


* <big>[[Full Disk Simulations]]</big>
* <big>[[All-Day Synthesis Issues]]</big>
* <big>[[Analyzing Pre-2017 Data]]</big>
* <big>[[Fixing Pipeline Problems pre-2021-Feb-07]]</big>

== System Software ==

* LabVIEW software
* Python code [https://github.com/dgary50/eovsa Github repository]
* [[Python3 Code Installation]]

== Observing Log ==
[[2016 November]]; [[2016 December| December]]

[[2017 January]]; [[2017 February | February]]; [[2017 March | March]]; [[2017 April | April]]; [[2017 May | May]]; [[2017 June | June]];
[[2017 July | July]]; [[2017 August | August]]; [[2017 September | September]]; [[2017 October | October]]; [[2017 November | November]]; [[2017 December | December]]

[[2018 January]]; [[2018 February | February]]; [[2018 March | March]]; [[2018 April | April]]; [[2018 May | May]]; [[2018 June | June]];
[[2018 July | July]]; [[2018 August | August]]; [[2018 September | September]]; [[2018 October | October]]; [[2018 November | November]]; [[2018 December | December]]

[[2019 January]]; [[2019 February | February]]; [[2019 March | March]]; [[2019 April | April]]; [[2019 May | May]]; [[2019 June | June]];
[[2019 July | July]]; [[2019 August | August]]; [[2019 September | September]]; [[2019 October | October]]; [[2019 November | November]]; [[2019 December | December]]

[[2020 January]]; [[2020 February | February]]; [[2020 March | March]]; [[2020 April | April]]; [[2020 May | May]]; [[2020 June | June]];
[[2020 July | July]]; [[2020 August | August]]; [[2020 September | September]]; [[2020 October | October]]; [[2020 November | November]]; [[2020 December | December]]

[[2021 January]]; [[2021 February | February]]; [[2021 March | March]]; [[2021 April | April]]; [[2021 May | May]]; [[2021 June | June]];
[[2021 July | July]]; [[2021 August | August]]; [[2021 September | September]]; [[2021 October | October]]; [[2021 November | November]]; [[2021 December | December]]

[[2022 SQL Outage]]

== Tohbans ==

[[Trouble Shooting Guide]]

[[Tohban Records]]

[[Owen's Notes]]

[[Tohban EOVSA Imaging Tutorial A-Z]]

[[Tohban Guide to Self Calibration and Imaging for EOVSA]]

[[Guide to Upgrade SolarSoft(SSW)]]

== EOVSA Flare List ==

See [https://docs.google.com/spreadsheets/d/1P8jHuDRF93dMflU6RMQcsJqVepD9vFkPkofV8Imj4xA/edit?usp=sharing this link] for a list of EOVSA flares as a Google Spreadsheet.

[[Recent Flare List (2021-)]]

[http://ovsa.njit.edu/jay/rd_db.php An older link] is available at the EOVSA website.

== EOVSA Publications ==
Here is a (partial) list of publications that utilize EOVSA data. See also the collection of EOVSA publications at [https://ui.adsabs.harvard.edu/public-libraries/eQ7HfPkySqydu-B8BCt6QQ this NASA/ADS Library].
; 2022
: [https://ui.adsabs.harvard.edu/abs/2022arXiv220503518Z/abstract Zhang et al. (2022), arXiv 2205.03518] ''Implications for additional plasma heating driving the extreme-ultraviolet late phase of a solar flare with microwave imaging spectroscopy''
: [https://ui.adsabs.harvard.edu/abs/2022A%26A...657A..51L/abstract Lopez et al. (2021), A&A, 657, A51] ''A solar flare driven by thermal conduction observed in mid-infrared''
; 2021
: [https://ui.adsabs.harvard.edu/abs/2021ApJ...923..213W/abstract Wei et al. (2021), ApJ, 923, 213] ''Coronal Magnetic Field Measurements along a Partially Erupting Filament in a Solar Flare''
: [https://ui.adsabs.harvard.edu/abs/2021ApJ...919...44S/abstract Shaik & Gary (2021), ApJ, 919, 44] ''Implications of Flat Optically Thick Microwave Spectra in Solar Flares for Source Size and Morphology''
: [https://ui.adsabs.harvard.edu/abs/2021ApJ...915...12K/abstract Kocharov et al. (2021), ApJ, 915, 12] ''Multiple Sources of Solar High-energy Protons''
: [https://ui.adsabs.harvard.edu/abs/2021ApJ...908L..55C/abstract Chen et al. (2021), ApJL, 908, L55] ''Energetic Electron Distribution of the Coronal Acceleration Region: First results from Joint Microwave and Hard X-ray Imaging Spectroscopy''
: [https://ui.adsabs.harvard.edu/abs/2021ApJ...906..132C/abstract Chhabra et al. (2021), ApJ, 906, 132] ''Imaging Spectroscopy of CME-Associated Solar Radio Bursts''
; 2020
: [https://ui.adsabs.harvard.edu/abs/2020ApJ...905..165R/abstract Reeves et al. (2020), ApJ, 905, 165] ''Hot Plasma Flows and Oscillations in the Loop-top Region During the September 10 2017 X8.2 Solar Flare''
: [https://ui.adsabs.harvard.edu/abs/2020ApJ...900...17Y/abstract Yu et al. (2020), ApJ, 900, 17] ''Magnetic Reconnection During the Post Impulsive Phase of the X8.2 Solar Flare: Bi-Directional Outflows as a Cause of Microwave and X-ray Bursts''
: [https://ui.adsabs.harvard.edu/abs/2020NatAs...4.1140C/abstract Chen et al. (2020b), Nature Astronomy, 4, 1140] ''Measurement of magnetic field and relativistic electrons along a solar flare current sheet''
: [https://ui.adsabs.harvard.edu/abs/2020ApJ...895L..50C/abstract Chen et al. (2020a), ApJL, 895, 50] ''Microwave Spectral Imaging of an Erupting Magnetic Flux Rope: Implications for the Standard Solar Flare Model in Three Dimensions''
: [https://ui.adsabs.harvard.edu/abs/2020FrASS...7...22K/abstract Kuroda et al. (2020), Frontiers, 7, 22] ''Evolution of Flare-accelerated Electrons Quantified by Spatially Resolved Analysis''
: [https://ui.adsabs.harvard.edu/abs/2020ApJ...891L..34G/abstract Glesener et al. (2020), ApJL, 891, 34] ''Accelerated Electrons Observed Down to <7 keV in a NuSTAR Solar Microflare''
: [https://ui.adsabs.harvard.edu/abs/2020ApJ...889...72K/abstract Karlicky at al. (2020), ApJ, 889, 72] ''Drifting Pulsation Structure at the Very Beginning of the 2017 September 10 Limb Flare''
: [https://ui.adsabs.harvard.edu/abs/2020Sci...367..278F/abstract Fleishman et al. (2020), Science, 367, 278] ''Decay of the coronal magnetic field can release sufficient energy to power a solar flare''
: [https://ui.adsabs.harvard.edu/abs/2020AAS...23538501G/abstract Gary et al. (2020), BAAS 52, 385.01] [https://aas235-aas.ipostersessions.com/default.aspx?s=97-69-9E-4B-34-19-68-53-1B-C6-21-0C-16-1C-5C-82&guestview=true Direct link to AAS iPoster] ''A new view of the solar atmosphere: daily full-disk multifrequency radio images from EOVSA''
; 2018
: [https://ui.adsabs.harvard.edu/#abs/2018ApJ...864...63P/abstract Polito et al. (2018), ApJ, 864, 63] ''Broad Non-Gaussian Fe XXIV Line Profiles in the Impulsive Phase of the 2017 September 10 X8.3-class Flare Observed by Hinode/EIS''
: [https://ui.adsabs.harvard.edu/#abs/2018ApJ...863...83G/abstract Gary et al. (2018), ApJ, 863, 83] ''Microwave and Hard X-Ray Observations of the 2017 September 10 Solar Limb Flare''
: [https://ui.adsabs.harvard.edu/#abs/2018ApJ...852...32K/abstract Kuroda et al. (2018), ApJ, 852, 32] ''Three-dimensional Forward-fit Modeling of the Hard X-ray and the Microwave Emissions of the 2015 June 22 M6.5 flare''
; 2017
: [https://ui.adsabs.harvard.edu/abs/2017NatAs...1E..85W/abstract Wang et al. (2017), Nature Astronomy, 1, 85] ''High-resolution observations of flare precursors in the low solar atmosphere''
; 2016
: [https://ui.adsabs.harvard.edu/abs/2016JAI.....541009N/abstract Nita et al. (2016), J. Astron. Instr., 5, 1641009-7366] ''EOVSA Implementation of a Spectral Kurtosis Correlator for Transient Detection and Classification''

== VLA Flare List and Publications ==
See [http://www.ovsa.njit.edu/wiki/index.php/VLA_Data_Survey#List_of_Jansky_VLA_Solar_Observations this link] for a list of flare observations made by the [https://science.nrao.edu/facilities/vla/ Karl G. Jansky Very Large Array] (VLA). Below is a partial list of publications that utilize VLA solar data (see also [https://ui.adsabs.harvard.edu/public-libraries/ZwbjpLo9RS-viufWEoQ95Q this NASA/ADS Library]).
* [https://ui.adsabs.harvard.edu/abs/2021ApJ...911....4L/abstract Luo et al. (2021), ApJ, 911, 4] ''Radio Spectral Imaging of an M8.4 Eruptive Solar Flare: Possible Evidence of a Termination Shock''
* [https://ui.adsabs.harvard.edu/abs/2021ApJ...910...40Z/abstract Zhang et al. (2021), ApJ, 910, 40] ''Multiwavelength Observations of the Formation and Eruption of a Complex Filament''
* [https://ui.adsabs.harvard.edu/abs/2020ApJ...904...94S/abstract Sharma et al. (2020), ApJ, 904, 94] ''Radio and X-Ray Observations of Short-lived Episodes of Electron Acceleration in a Solar Microflare''
* [https://ui.adsabs.harvard.edu/abs/2019ApJ...884...63C/abstract Chen et al. (2019), ApJ, 884, 63] ''Radio Spectroscopic Imaging of a Solar Flare Termination Shock: Split-band Feature as Evidence for Shock Compression''
* [https://ui.adsabs.harvard.edu/abs/2019ApJ...872...71Y/abstract Yu & Chen (2019), ApJ, 872, 71] ''Possible Detection of Subsecond-period Propagating Magnetohydrodynamics Waves in Post-reconnection Magnetic Loops during a Two-ribbon Solar Flare''
* [https://ui.adsabs.harvard.edu/abs/2018ApJ...866...62C/abstract Chen et al. (2018), ApJ, 866, 62] ''Magnetic Reconnection Null Points as the Origin of Semirelativistic Electron Beams in a Solar Jet
''
* [https://ui.adsabs.harvard.edu/abs/2017ApJ...848...77W/abstract Wang et al. (2016), ApJ, 848, 77] ''Dynamic Spectral Imaging of Decimetric Fiber Bursts in an Eruptive Solar Flare''
* [https://ui.adsabs.harvard.edu/abs/2015Sci...350.1238C/abstract Chen et al. (2015), Science, 350, 1238] ''Particle acceleration by a solar flare termination shock''
* [https://ui.adsabs.harvard.edu/abs/2014ApJ...794..149C/abstract Chen et al. (2014), ApJ, 794, 149] ''Direct Evidence of an Eruptive, Filament-hosting Magnetic Flux Rope Leading to a Fast Solar Coronal Mass Ejection''
* [https://ui.adsabs.harvard.edu/abs/2013ApJ...763L..21C/abstract Chen et al. (2013), ApJL, 763, 21] ''Tracing Electron Beams in the Sun's Corona with Radio Dynamic Imaging Spectroscopy''

Tohban EOVSA Imaging Tutorial A-Z V2

2022-07-07T08:41:18Z

Schhabra:

Tohban EOVSA Imaging Tutorial A-Z V2

2022-07-07T08:38:13Z

Schhabra:

Tohban EOVSA Imaging Tutorial A-Z V2

2022-07-07T08:36:58Z

Schhabra:

This tutorial describes the step-by-step procedure to download EOVSA IDB data, obtain and calibrate the measurement sets (.ms), and, transfer them to the inti server for self-calibration and further analysis.

=== Pre-requisites ===
Accounts on Pipeline and Inti or Baozi servers. Sijie and Bin can help with the account set-up for pipeline, inti and boazi.

==== How to connect to pipeline ====
'''Windows:''' On a windows machine Mobaxterm can be used to connect to Pipeline. For details, see ...

'''Linux/Unix:''' On a linux/Mac, the guest account will be used for people to log in as Tohban. Rather than using a password, access will be tied to specific computers that you use. One needs to generate a public + private ssh key pair using ssh-keygen or equivalent program that will result in a public key (plain text) file called id_rsa.pub, created in your ~/.ssh folder, the contents of which can be emailed to Dale or Owen to be added under "authorized keys". The ssh commands can then be used directly through the terminal:

<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">

ssh -L 5902:helios.solar.pvt:20000 guest@ovsa.njit.edu ### will not ask for a password for your computer's authorized key ###
ssh -XY your_username@pipeline ### add your username and enter password to your pipeline account, when prompted ###
</pre>

Alternatively, to avoid typing the longer commands, proxy commands can be added to the ~/.ssh/config file:

<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
vim ~/.ssh/config ### press 'i' to insert content and esc +':wq' to write and quit ###

Host *
ForwardAgent yes

Host *
ForwardX11 yes

Host ovsa
HostName ovsa.njit.edu
User guest

Host pipeline
Hostname pipeline.solar.pvt
User your_username ### Insert your pipeline username here ###
ProxyCommand ssh -W %h:%p guest@ovsa.njit.edu
</pre>

Upon saving the ~/.ssh/config file and restarting the terminal, you can connect to pipeline directly using:

<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
ssh -XY pipeline
</pre>

==== How to connect to the inti server ====
NOTE: To connect to inti you must have a NJIT UCID. The arcs have granted access to inti for all SRG members. If you encounter any issue with the login. Please let arcs folks (arcs@njit.edu) know.

'''Windows:''' As mentioned above, on a windows machine Mobaxterm can be used to connect to inti.
With your NJIT VPN connected, connect to one of the afsconnect servers (for example, afsaccess3.njit.edu) using your UCID and password. Next, ssh into inti.
<pre style="background-color: #FCEBD9">
ssh -XY UCID@afsaccess3.njit.edu ###Insert UCID password if prompted###

ssh -XY UCID@inti.hpcnet.campus.njit.edu ###Insert UCID/inti username here###
</pre>
Proxy commands for inti can also be added to the C:\Program Files\Git\etc\ssh\ssh_config file:
<pre style="background-color: #FCEBD9">
Host inti
Hostname inti.hpcnet.campus.njit.edu
User UCID ###Insert UCID/inti username here###
</pre>

'''Linux/Unix:''' On a linux/Mac the ssh command can be used:
<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
ssh -XY UCID@inti.hpcnet.campus.njit.edu ### insert UCID here and enter your NJIT UCID password, when prompted ###
</pre>

Proxy commands for inti can also be added to the ~/.ssh/config (on local machine) file for convenience:
<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
vim ~/.ssh/config ### press 'i' to insert content and esc +':wq' to write and quit ###

Host inti
Hostname inti.hpcnet.campus.njit.edu
User UCID ### insert NJIT UCID here ###
</pre>
after the proxy is set up, connect to inti by simply typing:
<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
ssh -XY inti ### enter your NJIT UCID password when prompted ###
</pre>

'''NOTE:'''When logging in for the first time, please add the following lines (demarcated by #) in your ~/.bashrc (on the inti server) file in order to use CASA on inti.

<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
#### setting start ####
if [ $HOSTNAME == "baozi.hpcnet.campus.njit.edu" ]; then
source /srg/.setenv_baozi
fi
if [ $HOSTNAME == "inti.hpcnet.campus.njit.edu" ]; then
source /inti/.setenv_inti
fi
if [ $HOSTNAME == "guko.resource.campus.njit.edu" ]; then
source /data/data/.setenv_guko
fi
#### setting end ####
</pre>

For sufficient disk space to analyze EOVSA data on Inti, use your dedicated directory YOURDIRECTORY at the location given below. If you do not have a directory, please ask Sijie to set one up.
<pre style="background-color: #FCEBD9">
cd /inti/data/users/YOURDIRECTORY
</pre>

'''Both CASA 5 and 6 are available on Inti.'''

Enter the bash environment on inti, and load the desired casa environment.

To load CASA 5:
<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
>> bash
>> loadcasa5
>> suncasa
</pre>

To load CASA 6:
<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
>> bash
>> loadcasa6
>> ipython
</pre>

=== Downloading data and Preliminary Calibration ===

==== Step 1: Downloading raw (IDB) data on Pipeline machine====
The initial downloading of data and applying refcal and phasecal to the .ms files is performed on the pipeline since the complete EOVSA SQL database is stored here for now.

If you are not using mobaxterm, directly SSH to Pipeline through your Linux or Mac computer (as shown above) and '''start tcsh to run CASA'''.

The following commands (run within CASA) can be used to import a single IDB file if the given time-range is less than 10 mins.
<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
from astropy.time import Time
import os
trange = Time(['2017-08-21 20:15:00', '2017-08-21 20:25:00']) ###Change accordingly###

#### (Optional) change output path, default current directory "./" #######
outpath = './msdata/' ###Change accordingly###
if not os.path.exists(outpath):
os.makedirs(outpath)

msfiles = importeovsa(idbfiles=trange, ncpu=1, visprefix=outpath)
</pre>

'''NOTE:''' The above step currently gives an error with importeovsa. This is because the data location was changed and the code doesn't know where to look for the IDB files. Sijie is looking into the problem. We will update the page when it is fixed. For now, please use the method below for all time ranges.

OR (for a time range longer than 10 minutes)

<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
from suncasa.tasks import task_calibeovsa as calibeovsa
from suncasa.tasks import task_importeovsa as timporteovsa
from split_cli import split_cli as split
import dump_tsys as dt
from util import Time
import numpy as np
import os
from glob import glob
from eovsapy import util

trange = Time(['2017-08-21 20:15:00', '2017-08-21 20:35:00']) ###Change accordingly###
idbdir = util.get_idbdir(trange[0])

info = dt.rd_fdb(trange[0])
for k, v in info.iteritems():
info[k] = info[k][~(info[k] == '')]

sidx = np.where(
np.logical_and(info['SOURCEID'] == 'Sun', info['PROJECTID'] == 'NormalObserving') & np.logical_and(
info['ST_TS'].astype(np.float) >= trange[0].lv,
info['ST_TS'].astype(np.float) <= trange[
1].lv))
filelist = info['FILE'][sidx]

outpath = './msdata/' ###Change accordingly###
if not os.path.exists(outpath):
os.makedirs(outpath)
inpath = idbdir + '{}/'.format(trange[0].datetime.strftime("%Y%m%d"))
ncpu = 1

msfiles = timporteovsa.importeovsa(idbfiles=[inpath + ll for ll in filelist], ncpu=ncpu, timebin="0s", width=1,
visprefix=outpath,
nocreatms=False, doconcat=False,
modelms="", doscaling=False, keep_nsclms=False, udb_corr=True)
</pre>

If you come across errors with calibeovsa, add the following lines to your ~/.casa/init.py (within pipeline) file.
<pre style="background-color: #FCEBD9">
vim ~/.ssh/config ### press 'i' to insert content and esc +':wq' to write and quit ###
import sys
sys.path.append('/common/python')
sys.path.append('/common/python/packages/pipeline_casa')
execfile('/common/python/suncasa/tasks/mytasks.py')
</pre>

==== Step 2: Concatenate multiple .ms files and Calibrate====
'''Follow this step if there are more than one .ms files for the chosen time-range. If not, move to Step 3.'''

<pre style="background-color: #FCEBD9">
concatvis = os.path.basename(msfiles[0])[:11] + '_concat_cal.ms' ### Set the path/name for the concatenated files ###

vis = calibeovsa.calibeovsa(msfiles, doconcat=True, concatvis=concatvis, caltype=['refpha','phacal'], doimage=True)
### If doimage=True, a quicklook image will be produced (by integrating over the entire time) as shown in Figure 1 ###

vis_str = str(' '.join(vis))
caled_vis=vis_str.replace('.ms','_cal.ms') ### Append '_cal' to the ms filename ###
split(vis=' '.join(vis),outputvis=caled_vis,datacolumn='corrected',timerange='',correlation='XX') ### Split the corrected column to the new 'caled' ms ###
</pre>

[[file:Figure1_imagingtutorial.png|thumb|right|300px|'''Figure 1:''' Quick-look full-Sun image after the initial calibration.]]

==== Step 3: Calibration ====
For a single .ms file from Step 1, this will calibrate the input visibility, write out calibration tables under /data1/eovsa/caltable/, and apply the calibration.
<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
vis = calibeovsa.calibeovsa('IDB20170821202020.ms', caltype=['refpha','phacal'], doimage=True) ###Change the vis filename accordingly###

vis_str = str(' '.join(vis))
caled_vis=vis_str.replace('.ms','_cal.ms') ### Append '_cal' to the ms filename ###

split(vis=' '.join(vis),outputvis=caled_vis,datacolumn='corrected',timerange='',correlation='XX') ### Split the corrected column to the new 'caled' ms ###
</pre>

The created 'caled' ms data files (xxx_concat_cal.ms or xxx_cal.ms) need to be transferred from pipeline to inti server, which has all the casa tasks installed, in order to run the rest of the imaging steps.

=== Transferring data files between servers ===
To directly transfer your calibrated .ms data from the Pipeline to Inti:

<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
1. Log into Inti using your username.
ssh -X USERNAME@inti.hpcnet.campus.njit.edu

Then create a tunnel into Pipeline from Inti.
ssh -L 8888:pipeline.solar.pvt:22 guest@ovsa.njit.edu

2. Log into Inti again from a new terminal. Change to your working directory (where you need the data transferred) and give the following command:

scp -v -C -r -P 8888 PipelineUSERNAME@localhost:PATH/TO/MSFILE ./

</pre>

===Advanced Calibration and Imaging===
====Step 4: Self-calibration====
[[file:Figure3_imagingtutorial.png|thumb|center|400px|Figure 2: Cotton-Schwab clean major and minor cycles. [Source: http://www.aoc.nrao.edu/~rurvashi/ImagingAlgorithmsInCasa/node2.html].]]
Follow the steps below to run the self-calibration of the imaging data and produce the calibrated images in .fits format. https://github.com/binchensun/casa-eovsa/blob/master/slfcal_example.py

<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
bash
loadcasa5
suncasa

from suncasa.utils import helioimage2fits as hf
import os
import numpy as np
import pickle
from matplotlib import gridspec as gridspec
from sunpy import map as smap
from matplotlib import pyplot as plt
from split_cli import split_cli as split
import time

# =========== task handlers =============
dofullsun = 1 # initial full-sun imaging ###Change accordingly###
domasks=1 # get masks ###Change accordingly###
doslfcal=1 # main cycle of doing selfcalibration ###Change accordingly###
doapply=1 # apply the results ###Change accordingly###
doclean_slfcaled=1 # perform clean for self-calibrated data ###Change accordingly###

# ============ declaring the working directories ============
workdir = os.getcwd()+'/' #main working directory. Using current directory in this example
slfcaldir = workdir+'slfcal/' #place to put all selfcalibration products
imagedir = slfcaldir+'images/' #place to put all selfcalibration images
maskdir = slfcaldir+'masks/' #place to put clean masks
imagedir_slfcaled = slfcaldir+'images_slfcaled/' #place to put final self-calibrated images
caltbdir = slfcaldir+'caltbs/' # place to put calibration tables
# make these directories if they do not already exist
dirs = [workdir, slfcaldir, imagedir, maskdir, imagedir_slfcaled, caltbdir]
for d in dirs:
if not os.path.exists(d):
os.makedirs(d)

# ============ Split a short time for self-calibration ===========
# input visibility
ms_in = workdir + 'IDB20170821202020_cal.ms' ###Change the initial calibrated (through calibeovsa) vis accordingly###
# output, selfcaled, visibility
ms_slfcaled = workdir + os.path.basename(ms_in).replace('cal','slfcaled')
# intermediate small visibility for selfcalbration
# selected time range for generating self-calibration solutions
trange='2017/08/21/20:21:10~2017/08/21/20:21:30' ###Change accordingly###
slfcalms = slfcaldir+'slfcalms.XX.slfcal'
slfcaledms = slfcaldir+'slfcalms.XX.slfcaled'
if not os.path.exists(slfcalms):
split(vis=ms_in,outputvis=slfcalms,datacolumn='data',timerange=trange,correlation='XX')

# ============ Prior definitions for spectral windows, antennas, pixel numbers =========
spws=[str(s+1) for s in range(30)] ###spws=[str(s+1) for s in range(49)] # For post-2020 data
antennas='0~12&0~12'
npix=512
nround=3 #number of slfcal cycles
</pre>

[[file:Figure2_imagingtutorial.png|thumb|right|300px|Figure 3: Full-Sun image after initial clean to find the flare location.]]
<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
# =========== Step 1, doing a full-Sun image to find out phasecenter and appropriate field of view =========
if dofullsun:
#initial mfs clean to find out the image phase center
im_init='fullsun_init'
os.system('rm -rf '+im_init+'*')
tclean(vis=slfcalms,
antenna=antennas,
imagename=im_init,
spw='2~15',
specmode='mfs',
timerange=trange,
imsize=[npix],
cell=['5arcsec'],
niter=1000,
gain=0.05,
stokes='I',
restoringbeam=['30arcsec'],
interactive=False,
pbcor=True)

hf.imreg(vis=slfcalms,imagefile=im_init+'.image.pbcor',fitsfile=im_init+'.fits',
timerange=trange,usephacenter=False,verbose=True)
clnjunks = ['.flux', '.mask', '.model', '.psf', '.residual','.sumwt','.pb','.image'] ###Do not run the next 4 lines if needed to view and assess the subset of clean process images
for clnjunk in clnjunks:
if os.path.exists(im_init + clnjunk):
os.system('rm -rf '+im_init + clnjunk)

from sunpy import map as smap
from matplotlib import pyplot as plt
fig = plt.figure(figsize=(6,6))
ax = fig.add_subplot(111)
eomap=smap.Map(im_init+'.fits')
#eomap.data=eomap.data.reshape((npix,npix))
eomap.plot_settings['cmap'] = plt.get_cmap('jet')
eomap.plot(axes = ax)
eomap.draw_limb()
plt.show()
viewer(im_init+'.image.pbcor')

# parameters specific to the event (found from step 1)
phasecenter='J2000 10h02m59 11d58m07' ###Change accordingly###
xran=[280,480] ###Change accordingly###
yran=[-50,150] ###Change accordingly###
</pre>

[[file:Figure4_imagingtutorial.PNG|thumb|right|300px|Figure 4: Interactive clean window to create masks over the source. The white outline surrounding the source is the mask selected by the polygon drawing option.]]
<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
# =========== Step 2 (optional), generate masks =========
# The outlines drawn for masks can be created by any of the icons with the letter 'R' in the Viewer window. The instructions for doing this can be found by hovering over those icons. See Figure 4.
# if skipped, will not use any masks
if domasks:
clearcal(slfcalms)
delmod(slfcalms)
antennas=antennas
pol='XX'
imgprefix=maskdir+'slf_t0'

# step 1: set up the clean masks
img_init=imgprefix+'_init_ar_'
os.system('rm -rf '+img_init+'*')
#spwrans_mask=['1~5','6~12','13~20','21~30']
spwrans_mask=['2~12']
#convert to a list of spws
spwrans_mask_list = [[str(i) for i in (np.arange(int(m.split('~')[0]),int(m.split('~')[1])))] for m in spwrans_mask] # Not used?
masks=[]
imnames=[]
for spwran in spwrans_mask:
imname=img_init+spwran.replace('~','-')
try:
tclean(vis=slfcalms,
antenna=antennas,
imagename=imname,
spw=spwran,
specmode='mfs',
timerange=trange,
imsize=[npix],
cell=['2arcsec'],
niter=1000,
gain=0.05,
stokes='XX',
restoringbeam=['20arcsec'],
phasecenter=phasecenter,
weighting='briggs',
robust=1.0,
interactive=True,
datacolumn='data',
pbcor=True,
savemodel='modelcolumn')
imnames.append(imname+'.image')
masks.append(imname+'.mask')
clnjunks = ['.flux', '.model', '.psf', '.residual'] #Do not run the next 4 lines if needed to view and assess the subset of clean process images
for clnjunk in clnjunks:
if os.path.exists(imname + clnjunk):
os.system('rm -rf '+ imname + clnjunk)
except:
print('error in cleaning spw: '+spwran)

pickle.dump(masks,open(slfcaldir+'masks.p','wb'))
</pre>

[[file:Slf.G0.png|thumb|right|300px|Figure 5: Phase before self-calibration]]
[[file:Slf.G1.png|thumb|right|300px|Figure 6: Phase after self-calibration]]
<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
# =========== Step 3, main step of selfcalibration =========
if doslfcal:
if os.path.exists(slfcaldir+'masks.p'):
masks=pickle.load(open(slfcaldir+'masks.p','rb'))
if not os.path.exists(slfcaldir+'masks.p'):
print 'masks do not exist. Use default mask'
masks=[]
os.system('rm -rf '+imagedir+'*')
os.system('rm -rf '+caltbdir+'*')
#first step: make a mock caltable for the entire database
print('Processing ' + trange)
slftbs=[]
calprefix=caltbdir+'slf'
imgprefix=imagedir+'slf'
tb.open(slfcalms+'/SPECTRAL_WINDOW')
reffreqs=tb.getcol('REF_FREQUENCY')
bdwds=tb.getcol('TOTAL_BANDWIDTH')
cfreqs=reffreqs+bdwds/2.
tb.close()
# starting beam size at 3.4 GHz in arcsec #Change accordingly
sbeam=40.
strtmp=[m.replace(':','') for m in trange.split('~')]
timestr='t'+strtmp[0]+'-'+strtmp[1]
refantenna='0'
# number of iterations for each round
niters=[100, 300, 500]
# roburst value for weighting the baselines
robusts=[1.0, 0.5, 0.0]
# apply calibration tables? Set to true for most cases
doapplycal=[1,1,1]
# modes for calibration, 'p' for phase-only, 'a' for amplitude only, 'ap' for both
calmodes=['p','p','a']
# setting uvranges for model image (optional, not used here)
uvranges=['','','']
for n in range(nround):
slfcal_tb_g= calprefix+'.G'+str(n)
fig = plt.figure(figsize=(8.4,7.)) # fig = plt.figure(figsize=(14, 7)) # For post-2020 data (50 spws)
gs = gridspec.GridSpec(5, 6) # gs = gridspec.GridSpec(5, 10) # For post-2020 data (50 spws)
for s,sp in enumerate(spws):
print 'processing spw: '+sp
cfreq=cfreqs[int(sp)]
# setting restoring beam size (not very useful for selfcal anyway, but just to see the results)
bm=max(sbeam*cfreqs[1]/cfreq, 6.)
slfcal_img = imgprefix+'.spw'+sp.zfill(2)+'.slfcal'+str(n)
# only the first round uses nearby spws for getting initial model
if n == 0:
spbg=max(int(sp)-2,1) # spbg=max(int(sp)-2,0) # For post-2020 data (50 spws)
sped=min(int(sp)+2,30) # sped=min(int(sp)+2,49) # For post-2020 data (50 spws)
spwran=str(spbg)+'~'+str(sped)
print('using spw {0:s} as model'.format(spwran))
if 'spwrans_mask' in vars():
for m, spwran_mask in enumerate(spwrans_mask):
if sp in spwran_mask:
mask = masks[m]
print('using mask {0:s}'.format(mask))
findmask = True
if not findmask:
print('mask not found. Do use any masks')
else:
spwran = sp
if 'spwrans_mask' in vars():
for m, spwran_mask in enumerate(spwrans_mask):
if sp in spwran_mask:
mask = masks[m]
print 'using mask {0:s}'.format(mask)
findmask = True
if not findmask:
print('mask not found. Do use any masks')
try:
tclean(vis=slfcalms,
antenna=antennas,
imagename=slfcal_img,
uvrange=uvranges[n],
spw=spwran,
specmode='mfs',
timerange=trange,
imsize=[npix],
cell=['2arcsec'],
niter=niters[n],
gain=0.05,
stokes='XX', #use pol XX image as the model
weighting='briggs',
robust=robusts[n],
phasecenter=phasecenter,
mask=mask,
restoringbeam=[str(bm)+'arcsec'],
pbcor=False,
interactive=False,
savemodel='modelcolumn')
if os.path.exists(slfcal_img+'.image'):
fitsfile=slfcal_img+'.fits'
hf.imreg(vis=slfcalms,imagefile=slfcal_img+'.image',fitsfile=fitsfile,
timerange=trange,usephacenter=False,toTb=True,verbose=False,overwrite=True)
clnjunks = ['.mask','.flux', '.model', '.psf', '.residual', '.image','.pb','.image.pbcor','.sumwt'] #Do not run the next 4 lines if needed to view and assess the subset of clean process images
for clnjunk in clnjunks:
if os.path.exists(slfcal_img + clnjunk):
os.system('rm -rf '+ slfcal_img + clnjunk)
ax = fig.add_subplot(gs[s])
eomap=smap.Map(fitsfile)
eomap.plot_settings['cmap'] = plt.get_cmap('jet')
eomap.plot(axes = ax)
eomap.draw_limb()
#eomap.draw_grid()
ax.set_title(' ')
ax.get_xaxis().set_visible(False)
ax.get_yaxis().set_visible(False)
ax.set_xlim(xran)
ax.set_ylim(yran)
plt.pause(0.5) # Allows viewing of each image as it is plotted.
os.system('rm -f '+ fitsfile)

except:
print 'error in cleaning spw: '+sp
print 'using nearby spws for initial model'
sp_e=int(sp)+2
sp_i=int(sp)-2
if sp_i < 1: # if sp < 0: # For post-2020 data (50 spws)
sp_i = 1 # sp_i = 0 # For post-2020 data (50 spws)
if sp_e > 30: # if sp_e > 49: # For post-2020 data (50 spws)
sp_e = 30 # sp_e = 49 # For post-2020 data (50 spws)
sp_=str(sp_i)+'~'+str(sp_e)
try:
tget(tclean)
spw=sp_
print('using spw {0:s} as model'.format(sp_))
tclean()
except:
print 'still not successful. abort...'
break

gaincal(vis=slfcalms, refant=refantenna,antenna=antennas,caltable=slfcal_tb_g,spw=sp, uvrange='',\
gaintable=[],selectdata=True,timerange=trange,solint='inf',gaintype='G',calmode=calmodes[n],\
combine='',minblperant=4,minsnr=2,append=True)
if not os.path.exists(slfcal_tb_g):
print 'No solution found in spw: '+sp
figname=imagedir+'slf_t0_n{:d}.png'.format(n)
plt.subplots_adjust(left=0, bottom=0, right=1, top=1, wspace=0, hspace=0)
plt.savefig(figname)
time.sleep(10)
plt.close()

if os.path.exists(slfcal_tb_g):
slftbs.append(slfcal_tb_g)
slftb=[slfcal_tb_g]
os.chdir(slfcaldir)
if calmodes[n] == 'p':
plotcal(caltable=slfcal_tb_g,antenna='1~12',xaxis='freq',yaxis='phase',\
subplot=431,plotrange=[-1,-1,-180,180],iteration='antenna',figfile=slfcal_tb_g+'.png',showgui=False)
if calmodes[n] == 'a':
plotcal(caltable=slfcal_tb_g,antenna='1~12',xaxis='freq',yaxis='amp',\
subplot=431,plotrange=[-1,-1,0,2.],iteration='antenna',figfile=slfcal_tb_g+'.png',showgui=False)
os.chdir(workdir)
plt.pause(0.5) # Allows viewing of the plot

if doapplycal[n]:
clearcal(slfcalms)
delmod(slfcalms)
applycal(vis=slfcalms,gaintable=slftb,spw=','.join(spws),selectdata=True,\
antenna=antennas,interp='nearest',flagbackup=False,applymode='calonly',calwt=False)

if n < nround-1:
prompt=raw_input('Continuing to selfcal?')
#prompt='y'
if prompt.lower() == 'n':
if os.path.exists(slfcaledms):
os.system('rm -rf '+slfcaledms)
split(slfcalms,slfcaledms,datacolumn='corrected')
print 'Final calibrated ms is {0:s}'.format(slfcaledms)
break
if prompt.lower() == 'y':
slfcalms_=slfcalms+str(n)
if os.path.exists(slfcalms_):
os.system('rm -rf '+slfcalms_)
split(slfcalms,slfcalms_,datacolumn='corrected')
slfcalms=slfcalms_
else:
if os.path.exists(slfcaledms):
os.system('rm -rf '+slfcaledms)
split(slfcalms,slfcaledms,datacolumn='corrected')
print 'Final calibrated ms is {0:s}'.format(slfcaledms)
</pre>

<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
# =========== Step 4: Apply self-calibration tables =========
if doapply:
import glob
os.chdir(workdir)
clearcal(ms_in)
clearcal(slfcalms)
applycal(vis=slfcalms,gaintable=slftbs,spw=','.join(spws),selectdata=True,\
antenna=antennas,interp='linear',flagbackup=False,applymode='calonly',calwt=False)
applycal(vis=ms_in,gaintable=slftbs,spw=','.join(spws),selectdata=True,\
antenna=antennas,interp='linear',flagbackup=False,applymode='calonly',calwt=False)
if os.path.exists(ms_slfcaled):
os.system('rm -rf '+ms_slfcaled)
split(ms_in, ms_slfcaled,datacolumn='corrected')
</pre>

[[file:Slf_t0_n0.png|thumb|right|300px|Figure 7: Multi-frequency images before self-calibration]]
[[file:Slf_t0_n1.png|thumb|right|300px|Figure 8: Multi-frequency images after self-calibration]]

<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
# =========== Step 5: Generate final self-calibrated images (optional) =========
if doclean_slfcaled:
import glob
pol='XX'
print('Processing ' + trange)
img_final=imagedir_slfcaled+'/slf_final_{0}_t0'.format(pol)
vis = ms_slfcaled
spws=[str(s+1) for s in range(30)]
tb.open(vis+'/SPECTRAL_WINDOW')
reffreqs=tb.getcol('REF_FREQUENCY')
bdwds=tb.getcol('TOTAL_BANDWIDTH')
cfreqs=reffreqs+bdwds/2.
tb.close()
sbeam=30.
from matplotlib import gridspec as gridspec
from sunpy import map as smap
from matplotlib import pyplot as plt
fitsfiles=[]
for s,sp in enumerate(spws):
cfreq=cfreqs[int(sp)]
bm=max(sbeam*cfreqs[1]/cfreq,4.)
imname=img_final+'_s'+sp.zfill(2)
fitsfile=imname+'.fits'
if not os.path.exists(fitsfile):
print 'cleaning spw {0:s} with beam size {1:.1f}"'.format(sp,bm)
try:
tclean(vis=vis,
antenna=antennas,
imagename=imname,
spw=sp,
specmode='mfs',
timerange=trange,
imsize=[npix],
cell=['1arcsec'],
niter=1000,
gain=0.05,
stokes=pol,
weighting='briggs',
robust=2.0,
restoringbeam=[str(bm)+'arcsec'],
phasecenter=phasecenter,
mask='',
pbcor=True,
interactive=False)
except:
print 'cleaning spw '+sp+' unsuccessful. Proceed to next spw'
continue
if os.path.exists(imname+'.image.pbcor'):
imn = imname+'.image.pbcor'
hf.imreg(vis=vis,imagefile=imn,fitsfile=fitsfile,
timerange=trange,usephacenter=False,toTb=True,verbose=False)
fitsfiles.append(fitsfile)
junks=['.flux','.model','.psf','.residual','.mask','.image','.pb','.image.pbcor','.sumwt'] #Do not run the next 4 lines if needed to view and assess the subset of clean process images
for junk in junks:
if os.path.exists(imname+junk):
os.system('rm -rf '+imname+junk)
else:
print('fits file '+fitsfile+' already exists, skip clean...')
fitsfiles.append(fitsfile)

fig = plt.figure(figsize=(8.4,7.)) # fig = plt.figure(figsize=(14, 7)) # For post-2020 data (50 spws)
gs = gridspec.GridSpec(5, 6) # gs = gridspec.GridSpec(5, 10) # For post-2020 data (50 spws)
for s,sp in enumerate(spws):
cfreq=cfreqs[int(sp)]
ax = fig.add_subplot(gs[s])
eomap=smap.Map(fitsfiles[s])
eomap.plot_settings['cmap'] = plt.get_cmap('jet')
eomap.plot(axes = ax)
eomap.draw_limb()
ax.set_title(' ')
ax.get_xaxis().set_visible(False)
ax.get_yaxis().set_visible(False)
ax.set_xlim(xran)
ax.set_ylim(yran)
plt.text(0.98,0.85,'{0:.1f} GHz'.format(cfreq/1e9),transform=ax.transAxes,ha='right',color='w',fontweight='bold')
plt.subplots_adjust(left=0, bottom=0, right=1, top=1, wspace=0, hspace=0)
plt.show()

</pre>
The .fits files of the self calibrated images at each frequency for the given time are saved at /slfcal/images_slfcaled in your working directory.

====Step 5: Quick-look imaging ==== 
For spectral imaging analysis of the event, follow [http://www.ovsa.njit.edu/wiki/index.php/EOVSA_Data_Analysis_Tutorial#Spectral_Imaging_with_SunCASA this tutorial] using the self-calibrated data obtained from the previous step or use [https://colab.research.google.com/drive/1lSLLxgOG6b8kgu9Sk6kSKvrViyubnXG6?usp=sharing#scrollTo=xbXyyLmCFCGL this link].

<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
from suncasa.utils import qlookplot as ql ## (Optional) Supply the npz file of the dynamic spectrum from previous step.
## If not provided, the program will generate a new one from the visibility.
## set the time interval
from suncasa import dspec as ds
import time
visibility_data = 'IDB20170821202020_slfcaled.ms'
specfile = visibility_data + '.dspec.npz'
d = ds.Dspec(visibility_data, bl='4&9', specfile=specfile)

timerange = '19:02:00~19:02:10' ## select frequency range from 2.5 GHz to 3.5 GHz
spw = '2~5' ## select stokes XX
stokes = 'XX' ## turn off AIA image plotting, default is True
plotaia = False
xycen = [375, 45] ## image center for clean in solar X-Y in arcsec
cell=['2.0arcsec'] ## pixel size
imsize=[128] ## x and y image size in pixels.
fov = [100,100] ## field of view of the zoomed-in panels in unit of arcsec
spw = ['{}'.format(s) for s in range(1,31)]
clevels = [0.5, 1.0] ## contour levels to fill in between.
calpha=0.35 ## now tune down the alpha
restoringbeam=['6arcsec']
ql.qlookplot(vis=msfile, specfile=specfile, timerange=timerange, spw=spw, stokes=stokes, \
restoringbeam=restoringbeam,imsize=imsize,cell=cell, \
xycen=xycen,fov=fov,clevels=clevels,calpha=calpha)
</pre>

Page Last Updated: 07/06/2022

Tohban EOVSA Imaging Tutorial A-Z V2

2022-07-07T08:22:34Z

Schhabra:

This tutorial describes the step-by-step procedure to download EOVSA IDB data, obtain and calibrate the measurement sets (.ms), and, transfer them to the inti server for self-calibration and further analysis.

=== Pre-requisites ===
Accounts on Pipeline and Inti or Baozi servers. Sijie and Bin can help with the account set-up for pipeline, inti and boazi.

==== How to connect to pipeline ====
'''Windows:''' On a windows machine Mobaxterm can be used to connect to Pipeline. For details, see ...

'''Linux/Unix:''' On a linux/Mac the ssh commands can be used directly through the terminal:

<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">

ssh -L 5902:helios.solar.pvt:20000 guest@ovsa.njit.edu ### when prompted, enter password to guest@ovsa ###
ssh -XY your_username@pipeline ### add your username and enter password to your pipeline account, when prompted ###
</pre>

Alternatively, to avoid typing the longer commands, proxy commands can be added to the ~/.ssh/config file:

<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
vim ~/.ssh/config ### press 'i' to insert content and esc +':wq' to write and quit ###

Host *
ForwardAgent yes

Host *
ForwardX11 yes

Host ovsa
HostName ovsa.njit.edu
User guest

Host pipeline
Hostname pipeline.solar.pvt
User your_username ### Insert your pipeline username here ###
ProxyCommand ssh -W %h:%p guest@ovsa.njit.edu
</pre>

Upon saving the ~/.ssh/config file and restarting the terminal, you can connect to pipeline directly using:

<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
ssh -XY pipeline
</pre>

==== How to connect to the inti server ====
NOTE: To connect to inti you must have a NJIT UCID. The arcs have granted access to inti for all SRG members. If you encounter any issue with the login. Please let arcs folks (arcs@njit.edu) know.

'''Windows:''' As mentioned above, on a windows machine Mobaxterm can be used to connect to inti.
With your NJIT VPN connected, connect to one of the afsconnect servers (for example, afsaccess3.njit.edu) using your UCID and password. Next, ssh into inti.
<pre style="background-color: #FCEBD9">
ssh -XY UCID@afsaccess3.njit.edu ###Insert UCID password if prompted###

ssh -XY UCID@inti.hpcnet.campus.njit.edu ###Insert UCID/inti username here###
</pre>
Proxy commands for inti can also be added to the C:\Program Files\Git\etc\ssh\ssh_config file:
<pre style="background-color: #FCEBD9">
Host inti
Hostname inti.hpcnet.campus.njit.edu
User UCID ###Insert UCID/inti username here###
</pre>

'''Linux/Unix:''' On a linux/Mac the ssh command can be used:
<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
ssh -XY UCID@inti.hpcnet.campus.njit.edu ### insert UCID here and enter your NJIT UCID password, when prompted ###
</pre>

Proxy commands for inti can also be added to the ~/.ssh/config (on local machine) file for convenience:
<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
vim ~/.ssh/config ### press 'i' to insert content and esc +':wq' to write and quit ###

Host inti
Hostname inti.hpcnet.campus.njit.edu
User UCID ### insert NJIT UCID here ###
</pre>
after the proxy is set up, connect to inti by simply typing:
<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
ssh -XY inti ### enter your NJIT UCID password when prompted ###
</pre>

'''NOTE:'''When logging in for the first time, please add the following lines (demarcated by #) in your ~/.bashrc (on the inti server) file in order to use CASA on inti.

<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
#### setting start ####
if [ $HOSTNAME == "baozi.hpcnet.campus.njit.edu" ]; then
source /srg/.setenv_baozi
fi
if [ $HOSTNAME == "inti.hpcnet.campus.njit.edu" ]; then
source /inti/.setenv_inti
fi
if [ $HOSTNAME == "guko.resource.campus.njit.edu" ]; then
source /data/data/.setenv_guko
fi
#### setting end ####
</pre>

For sufficient disk space to analyze EOVSA data on Inti, use your dedicated directory YOURDIRECTORY at the location given below. If you do not have a directory, please ask Sijie to set one up.
<pre style="background-color: #FCEBD9">
cd /inti/data/users/YOURDIRECTORY
</pre>

'''Both CASA 5 and 6 are available on Inti.'''

Enter the bash environment on inti, and load the desired casa environment.

To load CASA 5:
<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
>> bash
>> loadcasa5
>> suncasa
</pre>

To load CASA 6:
<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
>> bash
>> loadcasa6
>> ipython
</pre>

=== Downloading data and Preliminary Calibration ===

==== Step 1: Downloading raw (IDB) data on Pipeline machine====
The initial downloading of data and applying refcal and phasecal to the .ms files is performed on the pipeline since the complete EOVSA SQL database is stored here for now.

If you are not using mobaxterm, directly SSH to Pipeline through your Linux or Mac computer (as shown above) and '''start tcsh to run CASA'''.

The following commands (run within CASA) can be used to import a single IDB file if the given time-range is less than 10 mins.
<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
from astropy.time import Time
import os
trange = Time(['2017-08-21 20:15:00', '2017-08-21 20:25:00']) ###Change accordingly###

#### (Optional) change output path, default current directory "./" #######
outpath = './msdata/' ###Change accordingly###
if not os.path.exists(outpath):
os.makedirs(outpath)

msfiles = importeovsa(idbfiles=trange, ncpu=1, visprefix=outpath)
</pre>

'''NOTE:''' The above step currently gives an error with importeovsa. This is because the data location was changed and the code doesn't know where to look for the IDB files. Sijie is looking into the problem. We will update the page when it is fixed. For now, please use the method below for all time ranges.

OR (for a time range longer than 10 minutes)

<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
from suncasa.tasks import task_calibeovsa as calibeovsa
from suncasa.tasks import task_importeovsa as timporteovsa
from split_cli import split_cli as split
import dump_tsys as dt
from util import Time
import numpy as np
import os
from glob import glob
from eovsapy import util

trange = Time(['2017-08-21 20:15:00', '2017-08-21 20:35:00']) ###Change accordingly###
idbdir = util.get_idbdir(trange[0])

info = dt.rd_fdb(trange[0])
for k, v in info.iteritems():
info[k] = info[k][~(info[k] == '')]

sidx = np.where(
np.logical_and(info['SOURCEID'] == 'Sun', info['PROJECTID'] == 'NormalObserving') & np.logical_and(
info['ST_TS'].astype(np.float) >= trange[0].lv,
info['ST_TS'].astype(np.float) <= trange[
1].lv))
filelist = info['FILE'][sidx]

outpath = './msdata/' ###Change accordingly###
if not os.path.exists(outpath):
os.makedirs(outpath)
inpath = idbdir + '{}/'.format(trange[0].datetime.strftime("%Y%m%d"))
ncpu = 1

msfiles = timporteovsa.importeovsa(idbfiles=[inpath + ll for ll in filelist], ncpu=ncpu, timebin="0s", width=1,
visprefix=outpath,
nocreatms=False, doconcat=False,
modelms="", doscaling=False, keep_nsclms=False, udb_corr=True)
</pre>

If you come across errors with calibeovsa, add the following lines to your ~/.casa/init.py (within pipeline) file.
<pre style="background-color: #FCEBD9">
vim ~/.ssh/config ### press 'i' to insert content and esc +':wq' to write and quit ###
import sys
sys.path.append('/common/python')
sys.path.append('/common/python/packages/pipeline_casa')
execfile('/common/python/suncasa/tasks/mytasks.py')
</pre>

==== Step 2: Concatenate multiple .ms files and Calibrate====
'''Follow this step if there are more than one .ms files for the chosen time-range. If not, move to Step 3.'''

<pre style="background-color: #FCEBD9">
concatvis = os.path.basename(msfiles[0])[:11] + '_concat_cal.ms' ### Set the path/name for the concatenated files ###

vis = calibeovsa.calibeovsa(msfiles, doconcat=True, concatvis=concatvis, caltype=['refpha','phacal'], doimage=True)
### If doimage=True, a quicklook image will be produced (by integrating over the entire time) as shown in Figure 1 ###

vis_str = str(' '.join(vis))
caled_vis=vis_str.replace('.ms','_cal.ms') ### Append '_cal' to the ms filename ###
split(vis=' '.join(vis),outputvis=caled_vis,datacolumn='corrected',timerange='',correlation='XX') ### Split the corrected column to the new 'caled' ms ###
</pre>

[[file:Figure1_imagingtutorial.png|thumb|right|300px|'''Figure 1:''' Quick-look full-Sun image after the initial calibration.]]

==== Step 3: Calibration ====
For a single .ms file from Step 1, this will calibrate the input visibility, write out calibration tables under /data1/eovsa/caltable/, and apply the calibration.
<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
vis = calibeovsa.calibeovsa('IDB20170821202020.ms', caltype=['refpha','phacal'], doimage=True) ###Change the vis filename accordingly###

vis_str = str(' '.join(vis))
caled_vis=vis_str.replace('.ms','_cal.ms') ### Append '_cal' to the ms filename ###

split(vis=' '.join(vis),outputvis=caled_vis,datacolumn='corrected',timerange='',correlation='XX') ### Split the corrected column to the new 'caled' ms ###
</pre>

The created 'caled' ms data files (xxx_concat_cal.ms or xxx_cal.ms) need to be transferred from pipeline to inti server, which has all the casa tasks installed, in order to run the rest of the imaging steps.

=== Transferring data files between servers ===
To directly transfer your calibrated .ms data from the Pipeline to Inti:

<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
1. Log into Inti using your username.
ssh -X USERNAME@inti.hpcnet.campus.njit.edu

Then create a tunnel into Pipeline from Inti.
ssh -L 8888:pipeline.solar.pvt:22 guest@ovsa.njit.edu

2. Log into Inti again from a new terminal. Change to your working directory (where you need the data transferred) and give the following command:

scp -v -C -r -P 8888 PipelineUSERNAME@localhost:PATH/TO/MSFILE ./

</pre>

===Advanced Calibration and Imaging===
====Step 4: Self-calibration====
[[file:Figure3_imagingtutorial.png|thumb|center|400px|Figure 2: Cotton-Schwab clean major and minor cycles. [Source: http://www.aoc.nrao.edu/~rurvashi/ImagingAlgorithmsInCasa/node2.html].]]
Follow the steps below to run the self-calibration of the imaging data and produce the calibrated images in .fits format. https://github.com/binchensun/casa-eovsa/blob/master/slfcal_example.py

<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
bash
loadcasa5
suncasa

from suncasa.utils import helioimage2fits as hf
import os
import numpy as np
import pickle
from matplotlib import gridspec as gridspec
from sunpy import map as smap
from matplotlib import pyplot as plt
from split_cli import split_cli as split
import time

# =========== task handlers =============
dofullsun = 1 # initial full-sun imaging ###Change accordingly###
domasks=1 # get masks ###Change accordingly###
doslfcal=1 # main cycle of doing selfcalibration ###Change accordingly###
doapply=1 # apply the results ###Change accordingly###
doclean_slfcaled=1 # perform clean for self-calibrated data ###Change accordingly###

# ============ declaring the working directories ============
workdir = os.getcwd()+'/' #main working directory. Using current directory in this example
slfcaldir = workdir+'slfcal/' #place to put all selfcalibration products
imagedir = slfcaldir+'images/' #place to put all selfcalibration images
maskdir = slfcaldir+'masks/' #place to put clean masks
imagedir_slfcaled = slfcaldir+'images_slfcaled/' #place to put final self-calibrated images
caltbdir = slfcaldir+'caltbs/' # place to put calibration tables
# make these directories if they do not already exist
dirs = [workdir, slfcaldir, imagedir, maskdir, imagedir_slfcaled, caltbdir]
for d in dirs:
if not os.path.exists(d):
os.makedirs(d)

# ============ Split a short time for self-calibration ===========
# input visibility
ms_in = workdir + 'IDB20170821202020_cal.ms' ###Change the initial calibrated (through calibeovsa) vis accordingly###
# output, selfcaled, visibility
ms_slfcaled = workdir + os.path.basename(ms_in).replace('cal','slfcaled')
# intermediate small visibility for selfcalbration
# selected time range for generating self-calibration solutions
trange='2017/08/21/20:21:10~2017/08/21/20:21:30' ###Change accordingly###
slfcalms = slfcaldir+'slfcalms.XX.slfcal'
slfcaledms = slfcaldir+'slfcalms.XX.slfcaled'
if not os.path.exists(slfcalms):
split(vis=ms_in,outputvis=slfcalms,datacolumn='data',timerange=trange,correlation='XX')

# ============ Prior definitions for spectral windows, antennas, pixel numbers =========
spws=[str(s+1) for s in range(30)] # spws=[str(s+1) for s in range(49)] # For post-2020 data
antennas='0~12&0~12'
npix=512
nround=3 #number of slfcal cycles
</pre>

[[file:Figure2_imagingtutorial.png|thumb|right|300px|Figure 3: Full-Sun image after initial clean to find the flare location.]]
<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
# =========== Step 1, doing a full-Sun image to find out phasecenter and appropriate field of view =========
if dofullsun:
#initial mfs clean to find out the image phase center
im_init='fullsun_init'
os.system('rm -rf '+im_init+'*')
tclean(vis=slfcalms,
antenna=antennas,
imagename=im_init,
spw='2~15',
specmode='mfs',
timerange=trange,
imsize=[npix],
cell=['5arcsec'],
niter=1000,
gain=0.05,
stokes='I',
restoringbeam=['30arcsec'],
interactive=False,
pbcor=True)

hf.imreg(vis=slfcalms,imagefile=im_init+'.image.pbcor',fitsfile=im_init+'.fits',
timerange=trange,usephacenter=False,verbose=True)
clnjunks = ['.flux', '.mask', '.model', '.psf', '.residual','.sumwt','.pb','.image'] #Do not run the next 4 lines if needed to view and assess the subset of clean process images
for clnjunk in clnjunks:
if os.path.exists(im_init + clnjunk):
os.system('rm -rf '+im_init + clnjunk)

from sunpy import map as smap
from matplotlib import pyplot as plt
fig = plt.figure(figsize=(6,6))
ax = fig.add_subplot(111)
eomap=smap.Map(im_init+'.fits')
#eomap.data=eomap.data.reshape((npix,npix))
eomap.plot_settings['cmap'] = plt.get_cmap('jet')
eomap.plot(axes = ax)
eomap.draw_limb()
plt.show()
viewer(im_init+'.image.pbcor')

# parameters specific to the event (found from step 1)
phasecenter='J2000 10h02m59 11d58m07' ###Change accordingly###
xran=[280,480] ###Change accordingly###
yran=[-50,150] ###Change accordingly###
</pre>

[[file:Figure4_imagingtutorial.PNG|thumb|right|300px|Figure 4: Interactive clean window to create masks over the source. The white outline surrounding the source is the mask selected by the polygon drawing option.]]
<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
# =========== Step 2 (optional), generate masks =========
# The outlines drawn for masks can be created by any of the icons with the letter 'R' in the Viewer window. The instructions for doing this can be found by hovering over those icons. See Figure 4.
# if skipped, will not use any masks
if domasks:
clearcal(slfcalms)
delmod(slfcalms)
antennas=antennas
pol='XX'
imgprefix=maskdir+'slf_t0'

# step 1: set up the clean masks
img_init=imgprefix+'_init_ar_'
os.system('rm -rf '+img_init+'*')
#spwrans_mask=['1~5','6~12','13~20','21~30']
spwrans_mask=['2~12']
#convert to a list of spws
spwrans_mask_list = [[str(i) for i in (np.arange(int(m.split('~')[0]),int(m.split('~')[1])))] for m in spwrans_mask] # Not used?
masks=[]
imnames=[]
for spwran in spwrans_mask:
imname=img_init+spwran.replace('~','-')
try:
tclean(vis=slfcalms,
antenna=antennas,
imagename=imname,
spw=spwran,
specmode='mfs',
timerange=trange,
imsize=[npix],
cell=['2arcsec'],
niter=1000,
gain=0.05,
stokes='XX',
restoringbeam=['20arcsec'],
phasecenter=phasecenter,
weighting='briggs',
robust=1.0,
interactive=True,
datacolumn='data',
pbcor=True,
savemodel='modelcolumn')
imnames.append(imname+'.image')
masks.append(imname+'.mask')
clnjunks = ['.flux', '.model', '.psf', '.residual'] #Do not run the next 4 lines if needed to view and assess the subset of clean process images
for clnjunk in clnjunks:
if os.path.exists(imname + clnjunk):
os.system('rm -rf '+ imname + clnjunk)
except:
print('error in cleaning spw: '+spwran)

pickle.dump(masks,open(slfcaldir+'masks.p','wb'))
</pre>

[[file:Slf.G0.png|thumb|right|300px|Figure 5: Phase before self-calibration]]
[[file:Slf.G1.png|thumb|right|300px|Figure 6: Phase after self-calibration]]
<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
# =========== Step 3, main step of selfcalibration =========
if doslfcal:
if os.path.exists(slfcaldir+'masks.p'):
masks=pickle.load(open(slfcaldir+'masks.p','rb'))
if not os.path.exists(slfcaldir+'masks.p'):
print 'masks do not exist. Use default mask'
masks=[]
os.system('rm -rf '+imagedir+'*')
os.system('rm -rf '+caltbdir+'*')
#first step: make a mock caltable for the entire database
print('Processing ' + trange)
slftbs=[]
calprefix=caltbdir+'slf'
imgprefix=imagedir+'slf'
tb.open(slfcalms+'/SPECTRAL_WINDOW')
reffreqs=tb.getcol('REF_FREQUENCY')
bdwds=tb.getcol('TOTAL_BANDWIDTH')
cfreqs=reffreqs+bdwds/2.
tb.close()
# starting beam size at 3.4 GHz in arcsec #Change accordingly
sbeam=40.
strtmp=[m.replace(':','') for m in trange.split('~')]
timestr='t'+strtmp[0]+'-'+strtmp[1]
refantenna='0'
# number of iterations for each round
niters=[100, 300, 500]
# roburst value for weighting the baselines
robusts=[1.0, 0.5, 0.0]
# apply calibration tables? Set to true for most cases
doapplycal=[1,1,1]
# modes for calibration, 'p' for phase-only, 'a' for amplitude only, 'ap' for both
calmodes=['p','p','a']
# setting uvranges for model image (optional, not used here)
uvranges=['','','']
for n in range(nround):
slfcal_tb_g= calprefix+'.G'+str(n)
fig = plt.figure(figsize=(8.4,7.)) # fig = plt.figure(figsize=(14, 7)) # For post-2020 data (50 spws)
gs = gridspec.GridSpec(5, 6) # gs = gridspec.GridSpec(5, 10) # For post-2020 data (50 spws)
for s,sp in enumerate(spws):
print 'processing spw: '+sp
cfreq=cfreqs[int(sp)]
# setting restoring beam size (not very useful for selfcal anyway, but just to see the results)
bm=max(sbeam*cfreqs[1]/cfreq, 6.)
slfcal_img = imgprefix+'.spw'+sp.zfill(2)+'.slfcal'+str(n)
# only the first round uses nearby spws for getting initial model
if n == 0:
spbg=max(int(sp)-2,1) # spbg=max(int(sp)-2,0) # For post-2020 data (50 spws)
sped=min(int(sp)+2,30) # sped=min(int(sp)+2,49) # For post-2020 data (50 spws)
spwran=str(spbg)+'~'+str(sped)
print('using spw {0:s} as model'.format(spwran))
if 'spwrans_mask' in vars():
for m, spwran_mask in enumerate(spwrans_mask):
if sp in spwran_mask:
mask = masks[m]
print('using mask {0:s}'.format(mask))
findmask = True
if not findmask:
print('mask not found. Do use any masks')
else:
spwran = sp
if 'spwrans_mask' in vars():
for m, spwran_mask in enumerate(spwrans_mask):
if sp in spwran_mask:
mask = masks[m]
print 'using mask {0:s}'.format(mask)
findmask = True
if not findmask:
print('mask not found. Do use any masks')
try:
tclean(vis=slfcalms,
antenna=antennas,
imagename=slfcal_img,
uvrange=uvranges[n],
spw=spwran,
specmode='mfs',
timerange=trange,
imsize=[npix],
cell=['2arcsec'],
niter=niters[n],
gain=0.05,
stokes='XX', #use pol XX image as the model
weighting='briggs',
robust=robusts[n],
phasecenter=phasecenter,
mask=mask,
restoringbeam=[str(bm)+'arcsec'],
pbcor=False,
interactive=False,
savemodel='modelcolumn')
if os.path.exists(slfcal_img+'.image'):
fitsfile=slfcal_img+'.fits'
hf.imreg(vis=slfcalms,imagefile=slfcal_img+'.image',fitsfile=fitsfile,
timerange=trange,usephacenter=False,toTb=True,verbose=False,overwrite=True)
clnjunks = ['.mask','.flux', '.model', '.psf', '.residual', '.image','.pb','.image.pbcor','.sumwt'] #Do not run the next 4 lines if needed to view and assess the subset of clean process images
for clnjunk in clnjunks:
if os.path.exists(slfcal_img + clnjunk):
os.system('rm -rf '+ slfcal_img + clnjunk)
ax = fig.add_subplot(gs[s])
eomap=smap.Map(fitsfile)
eomap.plot_settings['cmap'] = plt.get_cmap('jet')
eomap.plot(axes = ax)
eomap.draw_limb()
#eomap.draw_grid()
ax.set_title(' ')
ax.get_xaxis().set_visible(False)
ax.get_yaxis().set_visible(False)
ax.set_xlim(xran)
ax.set_ylim(yran)
plt.pause(0.5) # Allows viewing of each image as it is plotted.
os.system('rm -f '+ fitsfile)

except:
print 'error in cleaning spw: '+sp
print 'using nearby spws for initial model'
sp_e=int(sp)+2
sp_i=int(sp)-2
if sp_i < 1: # if sp < 0: # For post-2020 data (50 spws)
sp_i = 1 # sp_i = 0 # For post-2020 data (50 spws)
if sp_e > 30: # if sp_e > 49: # For post-2020 data (50 spws)
sp_e = 30 # sp_e = 49 # For post-2020 data (50 spws)
sp_=str(sp_i)+'~'+str(sp_e)
try:
tget(tclean)
spw=sp_
print('using spw {0:s} as model'.format(sp_))
tclean()
except:
print 'still not successful. abort...'
break

gaincal(vis=slfcalms, refant=refantenna,antenna=antennas,caltable=slfcal_tb_g,spw=sp, uvrange='',\
gaintable=[],selectdata=True,timerange=trange,solint='inf',gaintype='G',calmode=calmodes[n],\
combine='',minblperant=4,minsnr=2,append=True)
if not os.path.exists(slfcal_tb_g):
print 'No solution found in spw: '+sp
figname=imagedir+'slf_t0_n{:d}.png'.format(n)
plt.subplots_adjust(left=0, bottom=0, right=1, top=1, wspace=0, hspace=0)
plt.savefig(figname)
time.sleep(10)
plt.close()

if os.path.exists(slfcal_tb_g):
slftbs.append(slfcal_tb_g)
slftb=[slfcal_tb_g]
os.chdir(slfcaldir)
if calmodes[n] == 'p':
plotcal(caltable=slfcal_tb_g,antenna='1~12',xaxis='freq',yaxis='phase',\
subplot=431,plotrange=[-1,-1,-180,180],iteration='antenna',figfile=slfcal_tb_g+'.png',showgui=False)
if calmodes[n] == 'a':
plotcal(caltable=slfcal_tb_g,antenna='1~12',xaxis='freq',yaxis='amp',\
subplot=431,plotrange=[-1,-1,0,2.],iteration='antenna',figfile=slfcal_tb_g+'.png',showgui=False)
os.chdir(workdir)
plt.pause(0.5) # Allows viewing of the plot

if doapplycal[n]:
clearcal(slfcalms)
delmod(slfcalms)
applycal(vis=slfcalms,gaintable=slftb,spw=','.join(spws),selectdata=True,\
antenna=antennas,interp='nearest',flagbackup=False,applymode='calonly',calwt=False)

if n < nround-1:
prompt=raw_input('Continuing to selfcal?')
#prompt='y'
if prompt.lower() == 'n':
if os.path.exists(slfcaledms):
os.system('rm -rf '+slfcaledms)
split(slfcalms,slfcaledms,datacolumn='corrected')
print 'Final calibrated ms is {0:s}'.format(slfcaledms)
break
if prompt.lower() == 'y':
slfcalms_=slfcalms+str(n)
if os.path.exists(slfcalms_):
os.system('rm -rf '+slfcalms_)
split(slfcalms,slfcalms_,datacolumn='corrected')
slfcalms=slfcalms_
else:
if os.path.exists(slfcaledms):
os.system('rm -rf '+slfcaledms)
split(slfcalms,slfcaledms,datacolumn='corrected')
print 'Final calibrated ms is {0:s}'.format(slfcaledms)
</pre>

<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
# =========== Step 4: Apply self-calibration tables =========
if doapply:
import glob
os.chdir(workdir)
clearcal(ms_in)
clearcal(slfcalms)
applycal(vis=slfcalms,gaintable=slftbs,spw=','.join(spws),selectdata=True,\
antenna=antennas,interp='linear',flagbackup=False,applymode='calonly',calwt=False)
applycal(vis=ms_in,gaintable=slftbs,spw=','.join(spws),selectdata=True,\
antenna=antennas,interp='linear',flagbackup=False,applymode='calonly',calwt=False)
if os.path.exists(ms_slfcaled):
os.system('rm -rf '+ms_slfcaled)
split(ms_in, ms_slfcaled,datacolumn='corrected')
</pre>

[[file:Slf_t0_n0.png|thumb|right|300px|Figure 7: Multi-frequency images before self-calibration]]
[[file:Slf_t0_n1.png|thumb|right|300px|Figure 8: Multi-frequency images after self-calibration]]

<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
# =========== Step 5: Generate final self-calibrated images (optional) =========
if doclean_slfcaled:
import glob
pol='XX'
print('Processing ' + trange)
img_final=imagedir_slfcaled+'/slf_final_{0}_t0'.format(pol)
vis = ms_slfcaled
spws=[str(s+1) for s in range(30)]
tb.open(vis+'/SPECTRAL_WINDOW')
reffreqs=tb.getcol('REF_FREQUENCY')
bdwds=tb.getcol('TOTAL_BANDWIDTH')
cfreqs=reffreqs+bdwds/2.
tb.close()
sbeam=30.
from matplotlib import gridspec as gridspec
from sunpy import map as smap
from matplotlib import pyplot as plt
fitsfiles=[]
for s,sp in enumerate(spws):
cfreq=cfreqs[int(sp)]
bm=max(sbeam*cfreqs[1]/cfreq,4.)
imname=img_final+'_s'+sp.zfill(2)
fitsfile=imname+'.fits'
if not os.path.exists(fitsfile):
print 'cleaning spw {0:s} with beam size {1:.1f}"'.format(sp,bm)
try:
tclean(vis=vis,
antenna=antennas,
imagename=imname,
spw=sp,
specmode='mfs',
timerange=trange,
imsize=[npix],
cell=['1arcsec'],
niter=1000,
gain=0.05,
stokes=pol,
weighting='briggs',
robust=2.0,
restoringbeam=[str(bm)+'arcsec'],
phasecenter=phasecenter,
mask='',
pbcor=True,
interactive=False)
except:
print 'cleaning spw '+sp+' unsuccessful. Proceed to next spw'
continue
if os.path.exists(imname+'.image.pbcor'):
imn = imname+'.image.pbcor'
hf.imreg(vis=vis,imagefile=imn,fitsfile=fitsfile,
timerange=trange,usephacenter=False,toTb=True,verbose=False)
fitsfiles.append(fitsfile)
junks=['.flux','.model','.psf','.residual','.mask','.image','.pb','.image.pbcor','.sumwt'] #Do not run the next 4 lines if needed to view and assess the subset of clean process images
for junk in junks:
if os.path.exists(imname+junk):
os.system('rm -rf '+imname+junk)
else:
print('fits file '+fitsfile+' already exists, skip clean...')
fitsfiles.append(fitsfile)

fig = plt.figure(figsize=(8.4,7.)) # fig = plt.figure(figsize=(14, 7)) # For post-2020 data (50 spws)
gs = gridspec.GridSpec(5, 6) # gs = gridspec.GridSpec(5, 10) # For post-2020 data (50 spws)
for s,sp in enumerate(spws):
cfreq=cfreqs[int(sp)]
ax = fig.add_subplot(gs[s])
eomap=smap.Map(fitsfiles[s])
eomap.plot_settings['cmap'] = plt.get_cmap('jet')
eomap.plot(axes = ax)
eomap.draw_limb()
ax.set_title(' ')
ax.get_xaxis().set_visible(False)
ax.get_yaxis().set_visible(False)
ax.set_xlim(xran)
ax.set_ylim(yran)
plt.text(0.98,0.85,'{0:.1f} GHz'.format(cfreq/1e9),transform=ax.transAxes,ha='right',color='w',fontweight='bold')
plt.subplots_adjust(left=0, bottom=0, right=1, top=1, wspace=0, hspace=0)
plt.show()

</pre>
The .fits files of the self calibrated images at each frequency for the given time are saved at /slfcal/images_slfcaled in your working directory.

====Step 5: Quick-look imaging ==== 
For spectral imaging analysis of the event, follow [http://www.ovsa.njit.edu/wiki/index.php/EOVSA_Data_Analysis_Tutorial#Spectral_Imaging_with_SunCASA this tutorial] using the self-calibrated data obtained from the previous step or use [https://colab.research.google.com/drive/1lSLLxgOG6b8kgu9Sk6kSKvrViyubnXG6?usp=sharing#scrollTo=xbXyyLmCFCGL this link].

<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
from suncasa.utils import qlookplot as ql ## (Optional) Supply the npz file of the dynamic spectrum from previous step.
## If not provided, the program will generate a new one from the visibility.
## set the time interval
from suncasa import dspec as ds
import time
visibility_data = 'IDB20170821202020_slfcaled.ms'
specfile = visibility_data + '.dspec.npz'
d = ds.Dspec(visibility_data, bl='4&9', specfile=specfile)

timerange = '19:02:00~19:02:10' ## select frequency range from 2.5 GHz to 3.5 GHz
spw = '2~5' ## select stokes XX
stokes = 'XX' ## turn off AIA image plotting, default is True
plotaia = False
xycen = [375, 45] ## image center for clean in solar X-Y in arcsec
cell=['2.0arcsec'] ## pixel size
imsize=[128] ## x and y image size in pixels.
fov = [100,100] ## field of view of the zoomed-in panels in unit of arcsec
spw = ['{}'.format(s) for s in range(1,31)]
clevels = [0.5, 1.0] ## contour levels to fill in between.
calpha=0.35 ## now tune down the alpha
restoringbeam=['6arcsec']
ql.qlookplot(vis=msfile, specfile=specfile, timerange=timerange, spw=spw, stokes=stokes, \
restoringbeam=restoringbeam,imsize=imsize,cell=cell, \
xycen=xycen,fov=fov,clevels=clevels,calpha=calpha)
</pre>

Page Last Updated: 07/06/2022

Tohban EOVSA Imaging Tutorial A-Z V2

2022-07-06T11:47:22Z

Schhabra:

This tutorial describes the step-by-step procedure to download EOVSA IDB data, obtain and calibrate the measurement sets (.ms), and, transfer them to the inti server for self-calibration and further analysis.

=== Pre-requisites ===
Accounts on Pipeline and Inti or Baozi servers. Sijie and Bin can help with the account set-up for pipeline, inti and boazi.

==== How to connect to pipeline ====
'''Windows:''' On a windows machine Mobaxterm can be used to connect to Pipeline. For details, see ...

'''Linux/Unix:''' On a linux/Mac the ssh commands can be used directly through the terminal:

<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">

ssh -L 5902:helios.solar.pvt:20000 guest@ovsa.njit.edu ### when prompted, enter password to guest@ovsa ###
ssh -XY your_username@pipeline ### add your username and enter password to your pipeline account, when prompted ###
</pre>

Alternatively, to avoid typing the longer commands, proxy commands can be added to the ~/.ssh/config file:

<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
vim ~/.ssh/config ### press 'i' to insert content and esc +':wq' to write and quit ###

Host *
ForwardAgent yes

Host *
ForwardX11 yes

Host ovsa
HostName ovsa.njit.edu
User guest

Host pipeline
Hostname pipeline.solar.pvt
User your_username ### Insert your pipeline username here ###
ProxyCommand ssh -W %h:%p guest@ovsa.njit.edu
</pre>

Upon saving the ~/.ssh/config file and restarting the terminal, you can connect to pipeline directly using:

<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
ssh -XY pipeline
</pre>

==== How to connect to the inti server ====
NOTE: To connect to inti you must have a NJIT UCID. The arcs have granted access to inti for all SRG members. If you encounter any issue with the login. Please let arcs folks (arcs@njit.edu) know.

'''Windows:''' As mentioned above, on a windows machine Mobaxterm can be used to connect to inti.
With your NJIT VPN connected, connect to one of the afsconnect servers (for example, afsaccess3.njit.edu) using your UCID and password. Next, ssh into inti.
<pre style="background-color: #FCEBD9">
ssh -XY UCID@afsaccess3.njit.edu ###Insert UCID password if prompted###

ssh -XY UCID@inti.hpcnet.campus.njit.edu ###Insert UCID/inti username here###
</pre>
Proxy commands for inti can also be added to the C:\Program Files\Git\etc\ssh\ssh_config file:
<pre style="background-color: #FCEBD9">
Host inti
Hostname inti.hpcnet.campus.njit.edu
User UCID ###Insert UCID/inti username here###
</pre>

'''Linux/Unix:''' On a linux/Mac the ssh command can be used:
<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
ssh -XY UCID@inti.hpcnet.campus.njit.edu ### insert UCID here and enter your NJIT UCID password, when prompted ###
</pre>

Proxy commands for inti can also be added to the ~/.ssh/config (on local machine) file for convenience:
<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
vim ~/.ssh/config ### press 'i' to insert content and esc +':wq' to write and quit ###

Host inti
Hostname inti.hpcnet.campus.njit.edu
User UCID ### insert NJIT UCID here ###
</pre>
after the proxy is set up, connect to inti by simply typing:
<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
ssh -XY inti ### enter your NJIT UCID password when prompted ###
</pre>

'''NOTE:'''When logging in for the first time, please add the following lines (demarcated by #) in your ~/.bashrc (on the inti server) file in order to use CASA on inti.

<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
#### setting start ####
if [ $HOSTNAME == "baozi.hpcnet.campus.njit.edu" ]; then
source /srg/.setenv_baozi
fi
if [ $HOSTNAME == "inti.hpcnet.campus.njit.edu" ]; then
source /inti/.setenv_inti
fi
if [ $HOSTNAME == "guko.resource.campus.njit.edu" ]; then
source /data/data/.setenv_guko
fi
#### setting end ####
</pre>

For sufficient disk space to analyze EOVSA data on Inti, use your dedicated directory YOURDIRECTORY at the location given below. If you do not have a directory, please ask Sijie to set one up.
<pre style="background-color: #FCEBD9">
cd /inti/data/users/YOURDIRECTORY
</pre>

'''Both CASA 5 and 6 are available on Inti.'''

Enter the bash environment on inti, and load the desired casa environment.

To load CASA 5:
<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
>> bash
>> loadcasa5
>> suncasa
</pre>

To load CASA 6:
<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
>> bash
>> loadcasa6
>> ipython
</pre>

=== Downloading data and Preliminary Calibration ===

==== Step 1: Downloading raw (IDB) data on Pipeline machine====
The initial downloading of data and applying refcal and phasecal to the .ms files is performed on the pipeline since the complete EOVSA SQL database is stored here for now.

If you are not using mobaxterm, directly SSH to Pipeline through your Linux or Mac computer (as shown above) and '''start tcsh to run CASA'''.

The following commands (run within CASA) can be used to import a single IDB file if the given time-range is less than 10 mins.
<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
from astropy.time import Time
import os
trange = Time(['2017-08-21 20:15:00', '2017-08-21 20:25:00']) ###Change accordingly###

#### (Optional) change output path, default current directory "./" #######
outpath = './msdata/' ###Change accordingly###
if not os.path.exists(outpath):
os.makedirs(outpath)

msfiles = importeovsa(idbfiles=trange, ncpu=1, visprefix=outpath)
</pre>

'''NOTE:''' The above step currently gives an error with importeovsa. This is because the data location was changed and the code doesn't know where to look for the IDB files. Sijie is looking into the problem. We will update the page when it is fixed. For now, please use the method below for all time ranges.

OR (for a time range longer than 10 minutes)

<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
from suncasa.tasks import task_calibeovsa as calibeovsa
from suncasa.tasks import task_importeovsa as timporteovsa
from split_cli import split_cli as split
import dump_tsys as dt
from util import Time
import numpy as np
import os
from glob import glob
from eovsapy import util

trange = Time(['2017-08-21 20:15:00', '2017-08-21 20:35:00']) ###Change accordingly###
idbdir = util.get_idbdir(trange[0])

info = dt.rd_fdb(trange[0])
for k, v in info.iteritems():
info[k] = info[k][~(info[k] == '')]

sidx = np.where(
np.logical_and(info['SOURCEID'] == 'Sun', info['PROJECTID'] == 'NormalObserving') & np.logical_and(
info['ST_TS'].astype(np.float) >= trange[0].lv,
info['ST_TS'].astype(np.float) <= trange[
1].lv))
filelist = info['FILE'][sidx]

outpath = './msdata/' ###Change accordingly###
if not os.path.exists(outpath):
os.makedirs(outpath)
inpath = idbdir + '{}/'.format(trange[0].datetime.strftime("%Y%m%d"))
ncpu = 1

msfiles = timporteovsa.importeovsa(idbfiles=[inpath + ll for ll in filelist], ncpu=ncpu, timebin="0s", width=1,
visprefix=outpath,
nocreatms=False, doconcat=False,
modelms="", doscaling=False, keep_nsclms=False, udb_corr=True)
</pre>

If you come across errors with calibeovsa, add the following lines to your ~/.casa/init.py (within pipeline) file.
<pre style="background-color: #FCEBD9">
vim ~/.ssh/config ### press 'i' to insert content and esc +':wq' to write and quit ###
import sys
sys.path.append('/common/python')
sys.path.append('/common/python/packages/pipeline_casa')
execfile('/common/python/suncasa/tasks/mytasks.py')
</pre>

==== Step 2: Concatenate multiple .ms files and Calibrate====
'''Follow this step if there are more than one .ms files for the chosen time-range. If not, move to Step 3.'''

<pre style="background-color: #FCEBD9">
concatvis = os.path.basename(msfiles[0])[:11] + '_concat_cal.ms' ### Set the path/name for the concatenated files ###

vis = calibeovsa.calibeovsa(msfiles, doconcat=True, concatvis=concatvis, caltype=['refpha','phacal'], doimage=True)
### If doimage=True, a quicklook image will be produced (by integrating over the entire time) as shown in Figure 1 ###

vis_str = str(' '.join(vis))
caled_vis=vis_str.replace('.ms','_cal.ms') ### Append '_cal' to the ms filename ###
split(vis=' '.join(vis),outputvis=caled_vis,datacolumn='corrected',timerange='',correlation='XX') ### Split the corrected column to the new 'caled' ms ###
</pre>

[[file:Figure1_imagingtutorial.png|thumb|right|250px|'''Figure 1:''' Quick-look full-Sun image after the initial calibration.]]

==== Step 3: Calibration ====
For a single .ms file from Step 1, this will calibrate the input visibility, write out calibration tables under /data1/eovsa/caltable/, and apply the calibration.
<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
vis = calibeovsa.calibeovsa('IDB20170821202020.ms', caltype=['refpha','phacal'], doimage=True) ###Change the vis filename accordingly###

vis_str = str(' '.join(vis))
caled_vis=vis_str.replace('.ms','_cal.ms') ### Append '_cal' to the ms filename ###

split(vis=' '.join(vis),outputvis=caled_vis,datacolumn='corrected',timerange='',correlation='XX') ### Split the corrected column to the new 'caled' ms ###
</pre>

The created 'caled' ms data files (xxx_concat_cal.ms or xxx_cal.ms) need to be transferred from pipeline to inti server, which has all the casa tasks installed, in order to run the rest of the imaging steps.

=== Transferring data files between servers ===
To directly transfer your calibrated .ms data from the Pipeline to Inti:

<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
1. Log into Inti using your username.
ssh -X USERNAME@inti.hpcnet.campus.njit.edu

Then create a tunnel into Pipeline from Inti.
ssh -L 8888:pipeline.solar.pvt:22 guest@ovsa.njit.edu

2. Log into Inti again from a new terminal. Change to your working directory (where you need the data transferred) and give the following command:

scp -v -C -r -P 8888 PipelineUSERNAME@localhost:PATH/TO/MSFILE ./

</pre>

===Advanced Calibration and Imaging===
====Step 4: Self-calibration====
[[file:Figure3_imagingtutorial.png|thumb|center|400px|Figure 2: Cotton-Schwab clean major and minor cycles. [Source: http://www.aoc.nrao.edu/~rurvashi/ImagingAlgorithmsInCasa/node2.html].]]
Follow the steps below to run the self-calibration of the imaging data and produce the calibrated images in .fits format. https://github.com/binchensun/casa-eovsa/blob/master/slfcal_example.py

<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
bash
loadcasa5
suncasa

from suncasa.utils import helioimage2fits as hf
import os
import numpy as np
import pickle
from matplotlib import gridspec as gridspec
from sunpy import map as smap
from matplotlib import pyplot as plt
from split_cli import split_cli as split
import time

# =========== task handlers =============
dofullsun = 1 # initial full-sun imaging ###Change accordingly###
domasks=1 # get masks ###Change accordingly###
doslfcal=1 # main cycle of doing selfcalibration ###Change accordingly###
doapply=1 # apply the results ###Change accordingly###
doclean_slfcaled=1 # perform clean for self-calibrated data ###Change accordingly###

# ============ declaring the working directories ============
workdir = os.getcwd()+'/' #main working directory. Using current directory in this example
slfcaldir = workdir+'slfcal/' #place to put all selfcalibration products
imagedir = slfcaldir+'images/' #place to put all selfcalibration images
maskdir = slfcaldir+'masks/' #place to put clean masks
imagedir_slfcaled = slfcaldir+'images_slfcaled/' #place to put final self-calibrated images
caltbdir = slfcaldir+'caltbs/' # place to put calibration tables
# make these directories if they do not already exist
dirs = [workdir, slfcaldir, imagedir, maskdir, imagedir_slfcaled, caltbdir]
for d in dirs:
if not os.path.exists(d):
os.makedirs(d)

# ============ Split a short time for self-calibration ===========
# input visibility
ms_in = workdir + 'IDB20170821202020_cal.ms' ###Change the initial calibrated (through calibeovsa) vis accordingly###
# output, selfcaled, visibility
ms_slfcaled = workdir + os.path.basename(ms_in).replace('cal','slfcaled')
# intermediate small visibility for selfcalbration
# selected time range for generating self-calibration solutions
trange='2017/08/21/20:21:10~2017/08/21/20:21:30' ###Change accordingly###
slfcalms = slfcaldir+'slfcalms.XX.slfcal'
slfcaledms = slfcaldir+'slfcalms.XX.slfcaled'
if not os.path.exists(slfcalms):
split(vis=ms_in,outputvis=slfcalms,datacolumn='data',timerange=trange,correlation='XX')

# ============ Prior definitions for spectral windows, antennas, pixel numbers =========
spws=[str(s+1) for s in range(30)] # spws=[str(s+1) for s in range(49)] # For post-2020 data
antennas='0~12&0~12'
npix=512
nround=3 #number of slfcal cycles
</pre>

[[file:Figure2_imagingtutorial.png|thumb|right|250px|Figure 3: Full-Sun image after initial clean to find the flare location.]]
<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
# =========== Step 1, doing a full-Sun image to find out phasecenter and appropriate field of view =========
if dofullsun:
#initial mfs clean to find out the image phase center
im_init='fullsun_init'
os.system('rm -rf '+im_init+'*')
tclean(vis=slfcalms,
antenna=antennas,
imagename=im_init,
spw='2~15',
specmode='mfs',
timerange=trange,
imsize=[npix],
cell=['5arcsec'],
niter=1000,
gain=0.05,
stokes='I',
restoringbeam=['30arcsec'],
interactive=False,
pbcor=True)

hf.imreg(vis=slfcalms,imagefile=im_init+'.image.pbcor',fitsfile=im_init+'.fits',
timerange=trange,usephacenter=False,verbose=True)
clnjunks = ['.flux', '.mask', '.model', '.psf', '.residual','.sumwt','.pb','.image'] #Do not run the next 4 lines if needed to view and assess the subset of clean process images
for clnjunk in clnjunks:
if os.path.exists(im_init + clnjunk):
os.system('rm -rf '+im_init + clnjunk)

from sunpy import map as smap
from matplotlib import pyplot as plt
fig = plt.figure(figsize=(6,6))
ax = fig.add_subplot(111)
eomap=smap.Map(im_init+'.fits')
#eomap.data=eomap.data.reshape((npix,npix))
eomap.plot_settings['cmap'] = plt.get_cmap('jet')
eomap.plot(axes = ax)
eomap.draw_limb()
plt.show()
viewer(im_init+'.image.pbcor')

# parameters specific to the event (found from step 1)
phasecenter='J2000 10h02m59 11d58m07' ###Change accordingly###
xran=[280,480] ###Change accordingly###
yran=[-50,150] ###Change accordingly###
</pre>

[[file:Figure4_imagingtutorial.PNG|thumb|right|250px|Figure 4: Interactive clean window to create masks over the source. The white outline surrounding the source is the mask selected by the polygon drawing option.]]
<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
# =========== Step 2 (optional), generate masks =========
# The outlines drawn for masks can be created by any of the icons with the letter 'R' in the Viewer window. The instructions for doing this can be found by hovering over those icons. See Figure 4.
# if skipped, will not use any masks
if domasks:
clearcal(slfcalms)
delmod(slfcalms)
antennas=antennas
pol='XX'
imgprefix=maskdir+'slf_t0'

# step 1: set up the clean masks
img_init=imgprefix+'_init_ar_'
os.system('rm -rf '+img_init+'*')
#spwrans_mask=['1~5','6~12','13~20','21~30']
spwrans_mask=['2~12']
#convert to a list of spws
spwrans_mask_list = [[str(i) for i in (np.arange(int(m.split('~')[0]),int(m.split('~')[1])))] for m in spwrans_mask] # Not used?
masks=[]
imnames=[]
for spwran in spwrans_mask:
imname=img_init+spwran.replace('~','-')
try:
tclean(vis=slfcalms,
antenna=antennas,
imagename=imname,
spw=spwran,
specmode='mfs',
timerange=trange,
imsize=[npix],
cell=['2arcsec'],
niter=1000,
gain=0.05,
stokes='XX',
restoringbeam=['20arcsec'],
phasecenter=phasecenter,
weighting='briggs',
robust=1.0,
interactive=True,
datacolumn='data',
pbcor=True,
savemodel='modelcolumn')
imnames.append(imname+'.image')
masks.append(imname+'.mask')
clnjunks = ['.flux', '.model', '.psf', '.residual'] #Do not run the next 4 lines if needed to view and assess the subset of clean process images
for clnjunk in clnjunks:
if os.path.exists(imname + clnjunk):
os.system('rm -rf '+ imname + clnjunk)
except:
print('error in cleaning spw: '+spwran)

pickle.dump(masks,open(slfcaldir+'masks.p','wb'))
</pre>

[[file:Slf.G0.png|thumb|right|250px|Figure 5: Phase before self-calibration]]
[[file:Slf.G1.png|thumb|right|250px|Figure 6: Phase after self-calibration]]
<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
# =========== Step 3, main step of selfcalibration =========
if doslfcal:
if os.path.exists(slfcaldir+'masks.p'):
masks=pickle.load(open(slfcaldir+'masks.p','rb'))
if not os.path.exists(slfcaldir+'masks.p'):
print 'masks do not exist. Use default mask'
masks=[]
os.system('rm -rf '+imagedir+'*')
os.system('rm -rf '+caltbdir+'*')
#first step: make a mock caltable for the entire database
print('Processing ' + trange)
slftbs=[]
calprefix=caltbdir+'slf'
imgprefix=imagedir+'slf'
tb.open(slfcalms+'/SPECTRAL_WINDOW')
reffreqs=tb.getcol('REF_FREQUENCY')
bdwds=tb.getcol('TOTAL_BANDWIDTH')
cfreqs=reffreqs+bdwds/2.
tb.close()
# starting beam size at 3.4 GHz in arcsec #Change accordingly
sbeam=40.
strtmp=[m.replace(':','') for m in trange.split('~')]
timestr='t'+strtmp[0]+'-'+strtmp[1]
refantenna='0'
# number of iterations for each round
niters=[100, 300, 500]
# roburst value for weighting the baselines
robusts=[1.0, 0.5, 0.0]
# apply calibration tables? Set to true for most cases
doapplycal=[1,1,1]
# modes for calibration, 'p' for phase-only, 'a' for amplitude only, 'ap' for both
calmodes=['p','p','a']
# setting uvranges for model image (optional, not used here)
uvranges=['','','']
for n in range(nround):
slfcal_tb_g= calprefix+'.G'+str(n)
fig = plt.figure(figsize=(8.4,7.)) # fig = plt.figure(figsize=(14, 7)) # For post-2020 data (50 spws)
gs = gridspec.GridSpec(5, 6) # gs = gridspec.GridSpec(5, 10) # For post-2020 data (50 spws)
for s,sp in enumerate(spws):
print 'processing spw: '+sp
cfreq=cfreqs[int(sp)]
# setting restoring beam size (not very useful for selfcal anyway, but just to see the results)
bm=max(sbeam*cfreqs[1]/cfreq, 6.)
slfcal_img = imgprefix+'.spw'+sp.zfill(2)+'.slfcal'+str(n)
# only the first round uses nearby spws for getting initial model
if n == 0:
spbg=max(int(sp)-2,1) # spbg=max(int(sp)-2,0) # For post-2020 data (50 spws)
sped=min(int(sp)+2,30) # sped=min(int(sp)+2,49) # For post-2020 data (50 spws)
spwran=str(spbg)+'~'+str(sped)
print('using spw {0:s} as model'.format(spwran))
if 'spwrans_mask' in vars():
for m, spwran_mask in enumerate(spwrans_mask):
if sp in spwran_mask:
mask = masks[m]
print('using mask {0:s}'.format(mask))
findmask = True
if not findmask:
print('mask not found. Do use any masks')
else:
spwran = sp
if 'spwrans_mask' in vars():
for m, spwran_mask in enumerate(spwrans_mask):
if sp in spwran_mask:
mask = masks[m]
print 'using mask {0:s}'.format(mask)
findmask = True
if not findmask:
print('mask not found. Do use any masks')
try:
tclean(vis=slfcalms,
antenna=antennas,
imagename=slfcal_img,
uvrange=uvranges[n],
spw=spwran,
specmode='mfs',
timerange=trange,
imsize=[npix],
cell=['2arcsec'],
niter=niters[n],
gain=0.05,
stokes='XX', #use pol XX image as the model
weighting='briggs',
robust=robusts[n],
phasecenter=phasecenter,
mask=mask,
restoringbeam=[str(bm)+'arcsec'],
pbcor=False,
interactive=False,
savemodel='modelcolumn')
if os.path.exists(slfcal_img+'.image'):
fitsfile=slfcal_img+'.fits'
hf.imreg(vis=slfcalms,imagefile=slfcal_img+'.image',fitsfile=fitsfile,
timerange=trange,usephacenter=False,toTb=True,verbose=False,overwrite=True)
clnjunks = ['.mask','.flux', '.model', '.psf', '.residual', '.image','.pb','.image.pbcor','.sumwt'] #Do not run the next 4 lines if needed to view and assess the subset of clean process images
for clnjunk in clnjunks:
if os.path.exists(slfcal_img + clnjunk):
os.system('rm -rf '+ slfcal_img + clnjunk)
ax = fig.add_subplot(gs[s])
eomap=smap.Map(fitsfile)
eomap.plot_settings['cmap'] = plt.get_cmap('jet')
eomap.plot(axes = ax)
eomap.draw_limb()
#eomap.draw_grid()
ax.set_title(' ')
ax.get_xaxis().set_visible(False)
ax.get_yaxis().set_visible(False)
ax.set_xlim(xran)
ax.set_ylim(yran)
plt.pause(0.5) # Allows viewing of each image as it is plotted.
os.system('rm -f '+ fitsfile)

except:
print 'error in cleaning spw: '+sp
print 'using nearby spws for initial model'
sp_e=int(sp)+2
sp_i=int(sp)-2
if sp_i < 1: # if sp < 0: # For post-2020 data (50 spws)
sp_i = 1 # sp_i = 0 # For post-2020 data (50 spws)
if sp_e > 30: # if sp_e > 49: # For post-2020 data (50 spws)
sp_e = 30 # sp_e = 49 # For post-2020 data (50 spws)
sp_=str(sp_i)+'~'+str(sp_e)
try:
tget(tclean)
spw=sp_
print('using spw {0:s} as model'.format(sp_))
tclean()
except:
print 'still not successful. abort...'
break

gaincal(vis=slfcalms, refant=refantenna,antenna=antennas,caltable=slfcal_tb_g,spw=sp, uvrange='',\
gaintable=[],selectdata=True,timerange=trange,solint='inf',gaintype='G',calmode=calmodes[n],\
combine='',minblperant=4,minsnr=2,append=True)
if not os.path.exists(slfcal_tb_g):
print 'No solution found in spw: '+sp
figname=imagedir+'slf_t0_n{:d}.png'.format(n)
plt.subplots_adjust(left=0, bottom=0, right=1, top=1, wspace=0, hspace=0)
plt.savefig(figname)
time.sleep(10)
plt.close()

if os.path.exists(slfcal_tb_g):
slftbs.append(slfcal_tb_g)
slftb=[slfcal_tb_g]
os.chdir(slfcaldir)
if calmodes[n] == 'p':
plotcal(caltable=slfcal_tb_g,antenna='1~12',xaxis='freq',yaxis='phase',\
subplot=431,plotrange=[-1,-1,-180,180],iteration='antenna',figfile=slfcal_tb_g+'.png',showgui=False)
if calmodes[n] == 'a':
plotcal(caltable=slfcal_tb_g,antenna='1~12',xaxis='freq',yaxis='amp',\
subplot=431,plotrange=[-1,-1,0,2.],iteration='antenna',figfile=slfcal_tb_g+'.png',showgui=False)
os.chdir(workdir)
plt.pause(0.5) # Allows viewing of the plot

if doapplycal[n]:
clearcal(slfcalms)
delmod(slfcalms)
applycal(vis=slfcalms,gaintable=slftb,spw=','.join(spws),selectdata=True,\
antenna=antennas,interp='nearest',flagbackup=False,applymode='calonly',calwt=False)

if n < nround-1:
prompt=raw_input('Continuing to selfcal?')
#prompt='y'
if prompt.lower() == 'n':
if os.path.exists(slfcaledms):
os.system('rm -rf '+slfcaledms)
split(slfcalms,slfcaledms,datacolumn='corrected')
print 'Final calibrated ms is {0:s}'.format(slfcaledms)
break
if prompt.lower() == 'y':
slfcalms_=slfcalms+str(n)
if os.path.exists(slfcalms_):
os.system('rm -rf '+slfcalms_)
split(slfcalms,slfcalms_,datacolumn='corrected')
slfcalms=slfcalms_
else:
if os.path.exists(slfcaledms):
os.system('rm -rf '+slfcaledms)
split(slfcalms,slfcaledms,datacolumn='corrected')
print 'Final calibrated ms is {0:s}'.format(slfcaledms)
</pre>

<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
# =========== Step 4: Apply self-calibration tables =========
if doapply:
import glob
os.chdir(workdir)
clearcal(ms_in)
clearcal(slfcalms)
applycal(vis=slfcalms,gaintable=slftbs,spw=','.join(spws),selectdata=True,\
antenna=antennas,interp='linear',flagbackup=False,applymode='calonly',calwt=False)
applycal(vis=ms_in,gaintable=slftbs,spw=','.join(spws),selectdata=True,\
antenna=antennas,interp='linear',flagbackup=False,applymode='calonly',calwt=False)
if os.path.exists(ms_slfcaled):
os.system('rm -rf '+ms_slfcaled)
split(ms_in, ms_slfcaled,datacolumn='corrected')
</pre>

[[file:Slf_t0_n0.png|thumb|right|250px|Figure 7: Multi-frequency images before self-calibration]]
[[file:Slf_t0_n1.png|thumb|right|250px|Figure 8: Multi-frequency images after self-calibration]]

<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
# =========== Step 5: Generate final self-calibrated images (optional) =========
if doclean_slfcaled:
import glob
pol='XX'
print('Processing ' + trange)
img_final=imagedir_slfcaled+'/slf_final_{0}_t0'.format(pol)
vis = ms_slfcaled
spws=[str(s+1) for s in range(30)]
tb.open(vis+'/SPECTRAL_WINDOW')
reffreqs=tb.getcol('REF_FREQUENCY')
bdwds=tb.getcol('TOTAL_BANDWIDTH')
cfreqs=reffreqs+bdwds/2.
tb.close()
sbeam=30.
from matplotlib import gridspec as gridspec
from sunpy import map as smap
from matplotlib import pyplot as plt
fitsfiles=[]
for s,sp in enumerate(spws):
cfreq=cfreqs[int(sp)]
bm=max(sbeam*cfreqs[1]/cfreq,4.)
imname=img_final+'_s'+sp.zfill(2)
fitsfile=imname+'.fits'
if not os.path.exists(fitsfile):
print 'cleaning spw {0:s} with beam size {1:.1f}"'.format(sp,bm)
try:
tclean(vis=vis,
antenna=antennas,
imagename=imname,
spw=sp,
specmode='mfs',
timerange=trange,
imsize=[npix],
cell=['1arcsec'],
niter=1000,
gain=0.05,
stokes=pol,
weighting='briggs',
robust=2.0,
restoringbeam=[str(bm)+'arcsec'],
phasecenter=phasecenter,
mask='',
pbcor=True,
interactive=False)
except:
print 'cleaning spw '+sp+' unsuccessful. Proceed to next spw'
continue
if os.path.exists(imname+'.image.pbcor'):
imn = imname+'.image.pbcor'
hf.imreg(vis=vis,imagefile=imn,fitsfile=fitsfile,
timerange=trange,usephacenter=False,toTb=True,verbose=False)
fitsfiles.append(fitsfile)
junks=['.flux','.model','.psf','.residual','.mask','.image','.pb','.image.pbcor','.sumwt'] #Do not run the next 4 lines if needed to view and assess the subset of clean process images
for junk in junks:
if os.path.exists(imname+junk):
os.system('rm -rf '+imname+junk)
else:
print('fits file '+fitsfile+' already exists, skip clean...')
fitsfiles.append(fitsfile)

fig = plt.figure(figsize=(8.4,7.)) # fig = plt.figure(figsize=(14, 7)) # For post-2020 data (50 spws)
gs = gridspec.GridSpec(5, 6) # gs = gridspec.GridSpec(5, 10) # For post-2020 data (50 spws)
for s,sp in enumerate(spws):
cfreq=cfreqs[int(sp)]
ax = fig.add_subplot(gs[s])
eomap=smap.Map(fitsfiles[s])
eomap.plot_settings['cmap'] = plt.get_cmap('jet')
eomap.plot(axes = ax)
eomap.draw_limb()
ax.set_title(' ')
ax.get_xaxis().set_visible(False)
ax.get_yaxis().set_visible(False)
ax.set_xlim(xran)
ax.set_ylim(yran)
plt.text(0.98,0.85,'{0:.1f} GHz'.format(cfreq/1e9),transform=ax.transAxes,ha='right',color='w',fontweight='bold')
plt.subplots_adjust(left=0, bottom=0, right=1, top=1, wspace=0, hspace=0)
plt.show()

</pre>
The .fits files of the self calibrated images at each frequency for the given time are saved at /slfcal/images_slfcaled in your working directory.

====Step 5: Quick-look imaging ==== 
For spectral imaging analysis of the event, follow [http://www.ovsa.njit.edu/wiki/index.php/EOVSA_Data_Analysis_Tutorial#Spectral_Imaging_with_SunCASA this tutorial] using the self-calibrated data obtained from the previous step or use [https://colab.research.google.com/drive/1lSLLxgOG6b8kgu9Sk6kSKvrViyubnXG6?usp=sharing#scrollTo=xbXyyLmCFCGL this link].

<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
from suncasa.utils import qlookplot as ql ## (Optional) Supply the npz file of the dynamic spectrum from previous step.
## If not provided, the program will generate a new one from the visibility.
## set the time interval
from suncasa import dspec as ds
import time
visibility_data = 'IDB20170821202020_slfcaled.ms'
specfile = visibility_data + '.dspec.npz'
d = ds.Dspec(visibility_data, bl='4&9', specfile=specfile)

timerange = '19:02:00~19:02:10' ## select frequency range from 2.5 GHz to 3.5 GHz
spw = '2~5' ## select stokes XX
stokes = 'XX' ## turn off AIA image plotting, default is True
plotaia = False
xycen = [375, 45] ## image center for clean in solar X-Y in arcsec
cell=['2.0arcsec'] ## pixel size
imsize=[128] ## x and y image size in pixels.
fov = [100,100] ## field of view of the zoomed-in panels in unit of arcsec
spw = ['{}'.format(s) for s in range(1,31)]
clevels = [0.5, 1.0] ## contour levels to fill in between.
calpha=0.35 ## now tune down the alpha
restoringbeam=['6arcsec']
ql.qlookplot(vis=msfile, specfile=specfile, timerange=timerange, spw=spw, stokes=stokes, \
restoringbeam=restoringbeam,imsize=imsize,cell=cell, \
xycen=xycen,fov=fov,clevels=clevels,calpha=calpha)
</pre>

Page Last Updated: 07/06/2022

Tohban EOVSA Imaging Tutorial A-Z V2

2022-07-06T11:36:33Z

Schhabra:

This tutorial describes the step-by-step procedure to download EOVSA IDB data, obtain and calibrate the measurement sets (.ms), and, transfer them to the inti server for self-calibration and further analysis.

=== Pre-requisites ===
Accounts on Pipeline and Inti or Baozi servers. Sijie and Bin can help with the account set-up for pipeline, inti and boazi.

==== How to connect to pipeline ====
'''Windows:''' On a windows machine Mobaxterm can be used to connect to Pipeline. For details, see ...

'''Linux/Unix:''' On a linux/Mac the ssh commands can be used directly through the terminal:

<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">

ssh -L 5902:helios.solar.pvt:20000 guest@ovsa.njit.edu ### when prompted, enter password to guest@ovsa ###
ssh -XY your_username@pipeline ### add your username and enter password to your pipeline account, when prompted ###
</pre>

Alternatively, to avoid typing the longer commands, proxy commands can be added to the ~/.ssh/config file:

<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
vim ~/.ssh/config ### press 'i' to insert content and esc +':wq' to write and quit ###

Host *
ForwardAgent yes

Host *
ForwardX11 yes

Host ovsa
HostName ovsa.njit.edu
User guest

Host pipeline
Hostname pipeline.solar.pvt
User your_username ### Insert your pipeline username here ###
ProxyCommand ssh -W %h:%p guest@ovsa.njit.edu
</pre>

Upon saving the ~/.ssh/config file and restarting the terminal, you can connect to pipeline directly using:

<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
ssh -XY pipeline
</pre>

==== How to connect to the inti server ====
NOTE: To connect to inti you must have a NJIT UCID. The arcs have granted access to inti for all SRG members. If you encounter any issue with the login. Please let arcs folks (arcs@njit.edu) know.

'''Windows:''' As mentioned above, on a windows machine Mobaxterm can be used to connect to inti.
With your NJIT VPN connected, connect to one of the afsconnect servers (for example, afsaccess3.njit.edu) using your UCID and password. Next, ssh into inti.
<pre style="background-color: #FCEBD9">
ssh -XY UCID@afsaccess3.njit.edu ###Insert UCID password if prompted###

ssh -XY UCID@inti.hpcnet.campus.njit.edu ###Insert UCID/inti username here###
</pre>
Proxy commands for inti can also be added to the C:\Program Files\Git\etc\ssh\ssh_config file:
<pre style="background-color: #FCEBD9">
Host inti
Hostname inti.hpcnet.campus.njit.edu
User UCID ###Insert UCID/inti username here###
</pre>

'''Linux/Unix:''' On a linux/Mac the ssh command can be used:
<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
ssh -XY UCID@inti.hpcnet.campus.njit.edu ### insert UCID here and enter your NJIT UCID password, when prompted ###
</pre>

Proxy commands for inti can also be added to the ~/.ssh/config (on local machine) file for convenience:
<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
vim ~/.ssh/config ### press 'i' to insert content and esc +':wq' to write and quit ###

Host inti
Hostname inti.hpcnet.campus.njit.edu
User UCID ### insert NJIT UCID here ###
</pre>
after the proxy is set up, connect to inti by simply typing:
<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
ssh -XY inti ### enter your NJIT UCID password when prompted ###
</pre>

'''NOTE:'''When logging in for the first time, please add the following lines (demarcated by #) in your ~/.bashrc (on the inti server) file in order to use CASA on inti.

<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
#### setting start ####
if [ $HOSTNAME == "baozi.hpcnet.campus.njit.edu" ]; then
source /srg/.setenv_baozi
fi
if [ $HOSTNAME == "inti.hpcnet.campus.njit.edu" ]; then
source /inti/.setenv_inti
fi
if [ $HOSTNAME == "guko.resource.campus.njit.edu" ]; then
source /data/data/.setenv_guko
fi
#### setting end ####
</pre>

For sufficient disk space to analyze EOVSA data on Inti, use your dedicated directory YOURDIRECTORY at the location given below. If you do not have a directory, please ask Sijie to set one up.
<pre style="background-color: #FCEBD9">
cd /inti/data/users/YOURDIRECTORY
</pre>

'''Both CASA 5 and 6 are available on Inti.'''

Enter the bash environment on inti, and load the desired casa environment.

To load CASA 5:
<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
>> bash
>> loadcasa5
>> suncasa
</pre>

To load CASA 6:
<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
>> bash
>> loadcasa6
>> ipython
</pre>

=== Downloading data and Preliminary Calibration ===

==== Step 1: Downloading raw (IDB) data on Pipeline machine====
The initial downloading of data and applying refcal and phasecal to the .ms files is performed on the pipeline since the complete EOVSA SQL database is stored here for now.

If you are not using mobaxterm, directly SSH to Pipeline through your Linux or Mac computer (as shown above) and '''start tcsh to run CASA'''.

The following commands (run within CASA) can be used to import a single IDB file if the given time-range is less than 10 mins.
<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
from astropy.time import Time
import os
trange = Time(['2017-08-21 20:15:00', '2017-08-21 20:25:00']) ###Change accordingly###

#### (Optional) change output path, default current directory "./" #######
outpath = './msdata/' ###Change accordingly###
if not os.path.exists(outpath):
os.makedirs(outpath)

msfiles = importeovsa(idbfiles=trange, ncpu=1, visprefix=outpath)
</pre>

'''NOTE:''' The above step currently gives an error with importeovsa. This is because the data location was changed and the code doesn't know where to look for the IDB files. Sijie is looking into the problem. We will update the page when it is fixed. For now, please use the method below for all time ranges.

OR (for a time range longer than 10 minutes)

<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
from suncasa.tasks import task_calibeovsa as calibeovsa
from suncasa.tasks import task_importeovsa as timporteovsa
from split_cli import split_cli as split
import dump_tsys as dt
from util import Time
import numpy as np
import os
from glob import glob
from eovsapy import util

trange = Time(['2017-08-21 20:15:00', '2017-08-21 20:35:00']) ###Change accordingly###
idbdir = util.get_idbdir(trange[0])

info = dt.rd_fdb(trange[0])
for k, v in info.iteritems():
info[k] = info[k][~(info[k] == '')]

sidx = np.where(
np.logical_and(info['SOURCEID'] == 'Sun', info['PROJECTID'] == 'NormalObserving') & np.logical_and(
info['ST_TS'].astype(np.float) >= trange[0].lv,
info['ST_TS'].astype(np.float) <= trange[
1].lv))
filelist = info['FILE'][sidx]

outpath = './msdata/' ###Change accordingly###
if not os.path.exists(outpath):
os.makedirs(outpath)
inpath = idbdir + '{}/'.format(trange[0].datetime.strftime("%Y%m%d"))
ncpu = 1

msfiles = timporteovsa.importeovsa(idbfiles=[inpath + ll for ll in filelist], ncpu=ncpu, timebin="0s", width=1,
visprefix=outpath,
nocreatms=False, doconcat=False,
modelms="", doscaling=False, keep_nsclms=False, udb_corr=True)
</pre>

If you come across errors with calibeovsa, add the following lines to your ~/.casa/init.py (within pipeline) file.
<pre style="background-color: #FCEBD9">
vim ~/.ssh/config ### press 'i' to insert content and esc +':wq' to write and quit ###
import sys
sys.path.append('/common/python')
sys.path.append('/common/python/packages/pipeline_casa')
execfile('/common/python/suncasa/tasks/mytasks.py')
</pre>

==== Step 2: Concatenate multiple .ms files and Calibrate====
'''Follow this step if there are more than one .ms files for the chosen time-range. If not, move to Step 3.'''

<pre style="background-color: #FCEBD9">
concatvis = os.path.basename(msfiles[0])[:11] + '_concat_cal.ms' ### Set the path/name for the concatenated files ###

vis = calibeovsa.calibeovsa(msfiles, doconcat=True, concatvis=concatvis, caltype=['refpha','phacal'], doimage=True)
### If doimage=True, a quicklook image will be produced (by integrating over the entire time) as shown in Figure 1 ###

vis_str = str(' '.join(vis))
caled_vis=vis_str.replace('.ms','_cal.ms') ### Append '_cal' to the ms filename ###
split(vis=' '.join(vis),outputvis=caled_vis,datacolumn='corrected',timerange='',correlation='XX') ### Split the corrected column to the new 'caled' ms ###
</pre>

[[file:Figure1_imagingtutorial.png|thumb|right|250px|'''Figure 1:''' Quick-look full-Sun image after the initial calibration.]]

==== Step 3: Calibration ====
For a single .ms file from Step 1, this will calibrate the input visibility, write out calibration tables under /data1/eovsa/caltable/, and apply the calibration.
<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
vis = calibeovsa.calibeovsa('IDB20170821202020.ms', caltype=['refpha','phacal'], doimage=True) ###Change the vis filename accordingly###

vis_str = str(' '.join(vis))
caled_vis=vis_str.replace('.ms','_cal.ms') ### Append '_cal' to the ms filename ###

split(vis=' '.join(vis),outputvis=caled_vis,datacolumn='corrected',timerange='',correlation='XX') ### Split the corrected column to the new 'caled' ms ###
</pre>

The created 'caled' ms data files (xxx_concat_cal.ms or xxx_cal.ms) need to be transferred from pipeline to inti server, which has all the casa tasks installed, in order to run the rest of the imaging steps.

=== Transferring data files between servers ===
To directly transfer your calibrated .ms data from the Pipeline to Inti:

<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
1. Log into Inti using your username.
ssh -X USERNAME@inti.hpcnet.campus.njit.edu

Then create a tunnel into Pipeline from Inti.
ssh -L 8888:pipeline.solar.pvt:22 guest@ovsa.njit.edu

2. Log into Inti again from a new terminal. Change to your working directory (where you need the data transferred) and give the following command:

scp -v -C -r -P 8888 PipelineUSERNAME@localhost:PATH/TO/MSFILE ./

</pre>

===Advanced Calibration and Imaging===
====Step 4: Self-calibration====
[[file:Figure3_imagingtutorial.png|thumb|right|250px|Figure 2: Cotton-Schwab clean major and minor cycles. [Source: http://www.aoc.nrao.edu/~rurvashi/ImagingAlgorithmsInCasa/node2.html].]]
Follow the steps below to run the self-calibration of the imaging data and produce the calibrated images in .fits format. https://github.com/binchensun/casa-eovsa/blob/master/slfcal_example.py

<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
bash
loadcasa5
suncasa

from suncasa.utils import helioimage2fits as hf
import os
import numpy as np
import pickle
from matplotlib import gridspec as gridspec
from sunpy import map as smap
from matplotlib import pyplot as plt
from split_cli import split_cli as split
import time

# =========== task handlers =============
dofullsun = 1 # initial full-sun imaging ###Change accordingly###
domasks=1 # get masks ###Change accordingly###
doslfcal=1 # main cycle of doing selfcalibration ###Change accordingly###
doapply=1 # apply the results ###Change accordingly###
doclean_slfcaled=1 # perform clean for self-calibrated data ###Change accordingly###

# ============ declaring the working directories ============
workdir = os.getcwd()+'/' #main working directory. Using current directory in this example
slfcaldir = workdir+'slfcal/' #place to put all selfcalibration products
imagedir = slfcaldir+'images/' #place to put all selfcalibration images
maskdir = slfcaldir+'masks/' #place to put clean masks
imagedir_slfcaled = slfcaldir+'images_slfcaled/' #place to put final self-calibrated images
caltbdir = slfcaldir+'caltbs/' # place to put calibration tables
# make these directories if they do not already exist
dirs = [workdir, slfcaldir, imagedir, maskdir, imagedir_slfcaled, caltbdir]
for d in dirs:
if not os.path.exists(d):
os.makedirs(d)

# ============ Split a short time for self-calibration ===========
# input visibility
ms_in = workdir + 'IDB20170821202020_cal.ms' ###Change the initial calibrated (through calibeovsa) vis accordingly###
# output, selfcaled, visibility
ms_slfcaled = workdir + os.path.basename(ms_in).replace('cal','slfcaled')
# intermediate small visibility for selfcalbration
# selected time range for generating self-calibration solutions
trange='2017/08/21/20:21:10~2017/08/21/20:21:30' ###Change accordingly###
slfcalms = slfcaldir+'slfcalms.XX.slfcal'
slfcaledms = slfcaldir+'slfcalms.XX.slfcaled'
if not os.path.exists(slfcalms):
split(vis=ms_in,outputvis=slfcalms,datacolumn='data',timerange=trange,correlation='XX')

# ============ Prior definitions for spectral windows, antennas, pixel numbers =========
spws=[str(s+1) for s in range(30)] # spws=[str(s+1) for s in range(49)] # For post-2020 data
antennas='0~12&0~12'
npix=512
nround=3 #number of slfcal cycles

# =========== Step 1, doing a full-Sun image to find out phasecenter and appropriate field of view =========
if dofullsun:
#initial mfs clean to find out the image phase center
im_init='fullsun_init'
os.system('rm -rf '+im_init+'*')
tclean(vis=slfcalms,
antenna=antennas,
imagename=im_init,
spw='2~15',
specmode='mfs',
timerange=trange,
imsize=[npix],
cell=['5arcsec'],
niter=1000,
gain=0.05,
stokes='I',
restoringbeam=['30arcsec'],
interactive=False,
pbcor=True)

hf.imreg(vis=slfcalms,imagefile=im_init+'.image.pbcor',fitsfile=im_init+'.fits',
timerange=trange,usephacenter=False,verbose=True)
clnjunks = ['.flux', '.mask', '.model', '.psf', '.residual','.sumwt','.pb','.image'] #Do not run the next 4 lines if needed to view and assess the subset of clean process images
for clnjunk in clnjunks:
if os.path.exists(im_init + clnjunk):
os.system('rm -rf '+im_init + clnjunk)

from sunpy import map as smap
from matplotlib import pyplot as plt
fig = plt.figure(figsize=(6,6))
ax = fig.add_subplot(111)
eomap=smap.Map(im_init+'.fits')
#eomap.data=eomap.data.reshape((npix,npix))
eomap.plot_settings['cmap'] = plt.get_cmap('jet')
eomap.plot(axes = ax)
eomap.draw_limb()
plt.show()
viewer(im_init+'.image.pbcor')
</pre>
[[file:Figure2_imagingtutorial.png|thumb|right|250px|Figure 3: Full-Sun image after initial clean to find the flare location.]]
[[file:Figure4_imagingtutorial.PNG|thumb|right|250px|Figure 4: Interactive clean window to create masks over the source. The white outline surrounding the source is the mask selected by the polygon drawing option.]]
<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
# parameters specific to the event (found from step 1)
phasecenter='J2000 10h02m59 11d58m07' ###Change accordingly###
xran=[280,480] ###Change accordingly###
yran=[-50,150] ###Change accordingly###

# =========== Step 2 (optional), generate masks =========
# if skipped, will not use any masks
if domasks:
clearcal(slfcalms)
delmod(slfcalms)
antennas=antennas
pol='XX'
imgprefix=maskdir+'slf_t0'

# step 1: set up the clean masks
img_init=imgprefix+'_init_ar_'
os.system('rm -rf '+img_init+'*')
#spwrans_mask=['1~5','6~12','13~20','21~30']
spwrans_mask=['2~12']
#convert to a list of spws
spwrans_mask_list = [[str(i) for i in (np.arange(int(m.split('~')[0]),int(m.split('~')[1])))] for m in spwrans_mask] # Not used?
masks=[]
imnames=[]
for spwran in spwrans_mask:
imname=img_init+spwran.replace('~','-')
try:
tclean(vis=slfcalms,
antenna=antennas,
imagename=imname,
spw=spwran,
specmode='mfs',
timerange=trange,
imsize=[npix],
cell=['2arcsec'],
niter=1000,
gain=0.05,
stokes='XX',
restoringbeam=['20arcsec'],
phasecenter=phasecenter,
weighting='briggs',
robust=1.0,
interactive=True,
datacolumn='data',
pbcor=True,
savemodel='modelcolumn')
imnames.append(imname+'.image')
masks.append(imname+'.mask')
clnjunks = ['.flux', '.model', '.psf', '.residual'] #Do not run the next 4 lines if needed to view and assess the subset of clean process images
for clnjunk in clnjunks:
if os.path.exists(imname + clnjunk):
os.system('rm -rf '+ imname + clnjunk)
except:
print('error in cleaning spw: '+spwran)

pickle.dump(masks,open(slfcaldir+'masks.p','wb'))
</pre>

The outlines drawn for masks can be created by any of the icons with the letter 'R' in the Viewer window. The instructions for doing this can be found by hovering over those icons. See Figure 4.

<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
# =========== Step 3, main step of selfcalibration =========
if doslfcal:
if os.path.exists(slfcaldir+'masks.p'):
masks=pickle.load(open(slfcaldir+'masks.p','rb'))
if not os.path.exists(slfcaldir+'masks.p'):
print 'masks do not exist. Use default mask'
masks=[]
os.system('rm -rf '+imagedir+'*')
os.system('rm -rf '+caltbdir+'*')
#first step: make a mock caltable for the entire database
print('Processing ' + trange)
slftbs=[]
calprefix=caltbdir+'slf'
imgprefix=imagedir+'slf'
tb.open(slfcalms+'/SPECTRAL_WINDOW')
reffreqs=tb.getcol('REF_FREQUENCY')
bdwds=tb.getcol('TOTAL_BANDWIDTH')
cfreqs=reffreqs+bdwds/2.
tb.close()
# starting beam size at 3.4 GHz in arcsec #Change accordingly
sbeam=40.
strtmp=[m.replace(':','') for m in trange.split('~')]
timestr='t'+strtmp[0]+'-'+strtmp[1]
refantenna='0'
# number of iterations for each round
niters=[100, 300, 500]
# roburst value for weighting the baselines
robusts=[1.0, 0.5, 0.0]
# apply calibration tables? Set to true for most cases
doapplycal=[1,1,1]
# modes for calibration, 'p' for phase-only, 'a' for amplitude only, 'ap' for both
calmodes=['p','p','a']
# setting uvranges for model image (optional, not used here)
uvranges=['','','']
for n in range(nround):
slfcal_tb_g= calprefix+'.G'+str(n)
fig = plt.figure(figsize=(8.4,7.)) # fig = plt.figure(figsize=(14, 7)) # For post-2020 data (50 spws)
gs = gridspec.GridSpec(5, 6) # gs = gridspec.GridSpec(5, 10) # For post-2020 data (50 spws)
for s,sp in enumerate(spws):
print 'processing spw: '+sp
cfreq=cfreqs[int(sp)]
# setting restoring beam size (not very useful for selfcal anyway, but just to see the results)
bm=max(sbeam*cfreqs[1]/cfreq, 6.)
slfcal_img = imgprefix+'.spw'+sp.zfill(2)+'.slfcal'+str(n)
# only the first round uses nearby spws for getting initial model
if n == 0:
spbg=max(int(sp)-2,1) # spbg=max(int(sp)-2,0) # For post-2020 data (50 spws)
sped=min(int(sp)+2,30) # sped=min(int(sp)+2,49) # For post-2020 data (50 spws)
spwran=str(spbg)+'~'+str(sped)
print('using spw {0:s} as model'.format(spwran))
if 'spwrans_mask' in vars():
for m, spwran_mask in enumerate(spwrans_mask):
if sp in spwran_mask:
mask = masks[m]
print('using mask {0:s}'.format(mask))
findmask = True
if not findmask:
print('mask not found. Do use any masks')
else:
spwran = sp
if 'spwrans_mask' in vars():
for m, spwran_mask in enumerate(spwrans_mask):
if sp in spwran_mask:
mask = masks[m]
print 'using mask {0:s}'.format(mask)
findmask = True
if not findmask:
print('mask not found. Do use any masks')
try:
tclean(vis=slfcalms,
antenna=antennas,
imagename=slfcal_img,
uvrange=uvranges[n],
spw=spwran,
specmode='mfs',
timerange=trange,
imsize=[npix],
cell=['2arcsec'],
niter=niters[n],
gain=0.05,
stokes='XX', #use pol XX image as the model
weighting='briggs',
robust=robusts[n],
phasecenter=phasecenter,
mask=mask,
restoringbeam=[str(bm)+'arcsec'],
pbcor=False,
interactive=False,
savemodel='modelcolumn')
if os.path.exists(slfcal_img+'.image'):
fitsfile=slfcal_img+'.fits'
hf.imreg(vis=slfcalms,imagefile=slfcal_img+'.image',fitsfile=fitsfile,
timerange=trange,usephacenter=False,toTb=True,verbose=False,overwrite=True)
clnjunks = ['.mask','.flux', '.model', '.psf', '.residual', '.image','.pb','.image.pbcor','.sumwt'] #Do not run the next 4 lines if needed to view and assess the subset of clean process images
for clnjunk in clnjunks:
if os.path.exists(slfcal_img + clnjunk):
os.system('rm -rf '+ slfcal_img + clnjunk)
ax = fig.add_subplot(gs[s])
eomap=smap.Map(fitsfile)
eomap.plot_settings['cmap'] = plt.get_cmap('jet')
eomap.plot(axes = ax)
eomap.draw_limb()
#eomap.draw_grid()
ax.set_title(' ')
ax.get_xaxis().set_visible(False)
ax.get_yaxis().set_visible(False)
ax.set_xlim(xran)
ax.set_ylim(yran)
plt.pause(0.5) # Allows viewing of each image as it is plotted.
os.system('rm -f '+ fitsfile)

except:
print 'error in cleaning spw: '+sp
print 'using nearby spws for initial model'
sp_e=int(sp)+2
sp_i=int(sp)-2
if sp_i < 1: # if sp < 0: # For post-2020 data (50 spws)
sp_i = 1 # sp_i = 0 # For post-2020 data (50 spws)
if sp_e > 30: # if sp_e > 49: # For post-2020 data (50 spws)
sp_e = 30 # sp_e = 49 # For post-2020 data (50 spws)
sp_=str(sp_i)+'~'+str(sp_e)
try:
tget(tclean)
spw=sp_
print('using spw {0:s} as model'.format(sp_))
tclean()
except:
print 'still not successful. abort...'
break

gaincal(vis=slfcalms, refant=refantenna,antenna=antennas,caltable=slfcal_tb_g,spw=sp, uvrange='',\
gaintable=[],selectdata=True,timerange=trange,solint='inf',gaintype='G',calmode=calmodes[n],\
combine='',minblperant=4,minsnr=2,append=True)
if not os.path.exists(slfcal_tb_g):
print 'No solution found in spw: '+sp
figname=imagedir+'slf_t0_n{:d}.png'.format(n)
plt.subplots_adjust(left=0, bottom=0, right=1, top=1, wspace=0, hspace=0)
plt.savefig(figname)
time.sleep(10)
plt.close()

if os.path.exists(slfcal_tb_g):
slftbs.append(slfcal_tb_g)
slftb=[slfcal_tb_g]
os.chdir(slfcaldir)
if calmodes[n] == 'p':
plotcal(caltable=slfcal_tb_g,antenna='1~12',xaxis='freq',yaxis='phase',\
subplot=431,plotrange=[-1,-1,-180,180],iteration='antenna',figfile=slfcal_tb_g+'.png',showgui=False)
if calmodes[n] == 'a':
plotcal(caltable=slfcal_tb_g,antenna='1~12',xaxis='freq',yaxis='amp',\
subplot=431,plotrange=[-1,-1,0,2.],iteration='antenna',figfile=slfcal_tb_g+'.png',showgui=False)
os.chdir(workdir)
plt.pause(0.5) # Allows viewing of the plot

if doapplycal[n]:
clearcal(slfcalms)
delmod(slfcalms)
applycal(vis=slfcalms,gaintable=slftb,spw=','.join(spws),selectdata=True,\
antenna=antennas,interp='nearest',flagbackup=False,applymode='calonly',calwt=False)

if n < nround-1:
prompt=raw_input('Continuing to selfcal?')
#prompt='y'
if prompt.lower() == 'n':
if os.path.exists(slfcaledms):
os.system('rm -rf '+slfcaledms)
split(slfcalms,slfcaledms,datacolumn='corrected')
print 'Final calibrated ms is {0:s}'.format(slfcaledms)
break
if prompt.lower() == 'y':
slfcalms_=slfcalms+str(n)
if os.path.exists(slfcalms_):
os.system('rm -rf '+slfcalms_)
split(slfcalms,slfcalms_,datacolumn='corrected')
slfcalms=slfcalms_
else:
if os.path.exists(slfcaledms):
os.system('rm -rf '+slfcaledms)
split(slfcalms,slfcaledms,datacolumn='corrected')
print 'Final calibrated ms is {0:s}'.format(slfcaledms)
</pre>

<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
# =========== Step 4: Apply self-calibration tables =========
if doapply:
import glob
os.chdir(workdir)
clearcal(ms_in)
clearcal(slfcalms)
applycal(vis=slfcalms,gaintable=slftbs,spw=','.join(spws),selectdata=True,\
antenna=antennas,interp='linear',flagbackup=False,applymode='calonly',calwt=False)
applycal(vis=ms_in,gaintable=slftbs,spw=','.join(spws),selectdata=True,\
antenna=antennas,interp='linear',flagbackup=False,applymode='calonly',calwt=False)
if os.path.exists(ms_slfcaled):
os.system('rm -rf '+ms_slfcaled)
split(ms_in, ms_slfcaled,datacolumn='corrected')
</pre>
[[file:Slf.G0.png|thumb|right|250px|Figure 1: Phase before self-calibration]]
[[file:Slf.G1.png|thumb|right|250px|Figure 2: Phase after self-calibration]]

<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
# =========== Step 5: Generate final self-calibrated images (optional) =========
if doclean_slfcaled:
import glob
pol='XX'
print('Processing ' + trange)
img_final=imagedir_slfcaled+'/slf_final_{0}_t0'.format(pol)
vis = ms_slfcaled
spws=[str(s+1) for s in range(30)]
tb.open(vis+'/SPECTRAL_WINDOW')
reffreqs=tb.getcol('REF_FREQUENCY')
bdwds=tb.getcol('TOTAL_BANDWIDTH')
cfreqs=reffreqs+bdwds/2.
tb.close()
sbeam=30.
from matplotlib import gridspec as gridspec
from sunpy import map as smap
from matplotlib import pyplot as plt
fitsfiles=[]
for s,sp in enumerate(spws):
cfreq=cfreqs[int(sp)]
bm=max(sbeam*cfreqs[1]/cfreq,4.)
imname=img_final+'_s'+sp.zfill(2)
fitsfile=imname+'.fits'
if not os.path.exists(fitsfile):
print 'cleaning spw {0:s} with beam size {1:.1f}"'.format(sp,bm)
try:
tclean(vis=vis,
antenna=antennas,
imagename=imname,
spw=sp,
specmode='mfs',
timerange=trange,
imsize=[npix],
cell=['1arcsec'],
niter=1000,
gain=0.05,
stokes=pol,
weighting='briggs',
robust=2.0,
restoringbeam=[str(bm)+'arcsec'],
phasecenter=phasecenter,
mask='',
pbcor=True,
interactive=False)
except:
print 'cleaning spw '+sp+' unsuccessful. Proceed to next spw'
continue
if os.path.exists(imname+'.image.pbcor'):
imn = imname+'.image.pbcor'
hf.imreg(vis=vis,imagefile=imn,fitsfile=fitsfile,
timerange=trange,usephacenter=False,toTb=True,verbose=False)
fitsfiles.append(fitsfile)
junks=['.flux','.model','.psf','.residual','.mask','.image','.pb','.image.pbcor','.sumwt'] #Do not run the next 4 lines if needed to view and assess the subset of clean process images
for junk in junks:
if os.path.exists(imname+junk):
os.system('rm -rf '+imname+junk)
else:
print('fits file '+fitsfile+' already exists, skip clean...')
fitsfiles.append(fitsfile)

fig = plt.figure(figsize=(8.4,7.)) # fig = plt.figure(figsize=(14, 7)) # For post-2020 data (50 spws)
gs = gridspec.GridSpec(5, 6) # gs = gridspec.GridSpec(5, 10) # For post-2020 data (50 spws)
for s,sp in enumerate(spws):
cfreq=cfreqs[int(sp)]
ax = fig.add_subplot(gs[s])
eomap=smap.Map(fitsfiles[s])
eomap.plot_settings['cmap'] = plt.get_cmap('jet')
eomap.plot(axes = ax)
eomap.draw_limb()
ax.set_title(' ')
ax.get_xaxis().set_visible(False)
ax.get_yaxis().set_visible(False)
ax.set_xlim(xran)
ax.set_ylim(yran)
plt.text(0.98,0.85,'{0:.1f} GHz'.format(cfreq/1e9),transform=ax.transAxes,ha='right',color='w',fontweight='bold')
plt.subplots_adjust(left=0, bottom=0, right=1, top=1, wspace=0, hspace=0)
plt.show()

</pre>
The .fits files of the self calibrated images at each frequency for the given time are saved at /slfcal/images_slfcaled in your working directory.

[[file:Slf_t0_n0.png|thumb|right|250px|Figure 3: Multi-frequency images before self-calibration]]
[[file:Slf_t0_n1.png|thumb|right|250px|Figure 4: Multi-frequency images after self-calibration]]

====Step 5: Quick-look imaging ==== 
For spectral imaging analysis of the event, follow [http://www.ovsa.njit.edu/wiki/index.php/EOVSA_Data_Analysis_Tutorial#Spectral_Imaging_with_SunCASA this tutorial] using the self-calibrated data obtained from the previous step or use [https://colab.research.google.com/drive/1lSLLxgOG6b8kgu9Sk6kSKvrViyubnXG6?usp=sharing#scrollTo=xbXyyLmCFCGL this link].

<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
from suncasa.utils import qlookplot as ql ## (Optional) Supply the npz file of the dynamic spectrum from previous step.
## If not provided, the program will generate a new one from the visibility.
## set the time interval
from suncasa import dspec as ds
import time
visibility_data = 'IDB20170821202020_slfcaled.ms'
specfile = visibility_data + '.dspec.npz'
d = ds.Dspec(visibility_data, bl='4&9', specfile=specfile)

timerange = '19:02:00~19:02:10' ## select frequency range from 2.5 GHz to 3.5 GHz
spw = '2~5' ## select stokes XX
stokes = 'XX' ## turn off AIA image plotting, default is True
plotaia = False
xycen = [375, 45] ## image center for clean in solar X-Y in arcsec
cell=['2.0arcsec'] ## pixel size
imsize=[128] ## x and y image size in pixels.
fov = [100,100] ## field of view of the zoomed-in panels in unit of arcsec
spw = ['{}'.format(s) for s in range(1,31)]
clevels = [0.5, 1.0] ## contour levels to fill in between.
calpha=0.35 ## now tune down the alpha
restoringbeam=['6arcsec']
ql.qlookplot(vis=msfile, specfile=specfile, timerange=timerange, spw=spw, stokes=stokes, \
restoringbeam=restoringbeam,imsize=imsize,cell=cell, \
xycen=xycen,fov=fov,clevels=clevels,calpha=calpha)
</pre>

Page Last Updated: 07/06/2022

Tohban EOVSA Imaging Tutorial A-Z V2

2022-07-06T11:34:13Z

Schhabra:

This tutorial describes the step-by-step procedure to download EOVSA IDB data, obtain and calibrate the measurement sets (.ms), and, transfer them to the inti server for self-calibration and further analysis.

=== Pre-requisites ===
Accounts on Pipeline and Inti or Baozi servers. Sijie and Bin can help with the account set-up for pipeline, inti and boazi.

==== How to connect to pipeline ====
'''Windows:''' On a windows machine Mobaxterm can be used to connect to Pipeline. For details, see ...

'''Linux/Unix:''' On a linux/Mac the ssh commands can be used directly through the terminal:

<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">

ssh -L 5902:helios.solar.pvt:20000 guest@ovsa.njit.edu ### when prompted, enter password to guest@ovsa ###
ssh -XY your_username@pipeline ### add your username and enter password to your pipeline account, when prompted ###
</pre>

Alternatively, to avoid typing the longer commands, proxy commands can be added to the ~/.ssh/config file:

<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
vim ~/.ssh/config ### press 'i' to insert content and esc +':wq' to write and quit ###

Host *
ForwardAgent yes

Host *
ForwardX11 yes

Host ovsa
HostName ovsa.njit.edu
User guest

Host pipeline
Hostname pipeline.solar.pvt
User your_username ### Insert your pipeline username here ###
ProxyCommand ssh -W %h:%p guest@ovsa.njit.edu
</pre>

Upon saving the ~/.ssh/config file and restarting the terminal, you can connect to pipeline directly using:

<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
ssh -XY pipeline
</pre>

==== How to connect to the inti server ====
NOTE: To connect to inti you must have a NJIT UCID. The arcs have granted access to inti for all SRG members. If you encounter any issue with the login. Please let arcs folks (arcs@njit.edu) know.

'''Windows:''' As mentioned above, on a windows machine Mobaxterm can be used to connect to inti.
With your NJIT VPN connected, connect to one of the afsconnect servers (for example, afsaccess3.njit.edu) using your UCID and password. Next, ssh into inti.
<pre style="background-color: #FCEBD9">
ssh -XY UCID@afsaccess3.njit.edu ###Insert UCID password if prompted###

ssh -XY UCID@inti.hpcnet.campus.njit.edu ###Insert UCID/inti username here###
</pre>
Proxy commands for inti can also be added to the C:\Program Files\Git\etc\ssh\ssh_config file:
<pre style="background-color: #FCEBD9">
Host inti
Hostname inti.hpcnet.campus.njit.edu
User UCID ###Insert UCID/inti username here###
</pre>

'''Linux/Unix:''' On a linux/Mac the ssh command can be used:
<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
ssh -XY UCID@inti.hpcnet.campus.njit.edu ### insert UCID here and enter your NJIT UCID password, when prompted ###
</pre>

Proxy commands for inti can also be added to the ~/.ssh/config (on local machine) file for convenience:
<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
vim ~/.ssh/config ### press 'i' to insert content and esc +':wq' to write and quit ###

Host inti
Hostname inti.hpcnet.campus.njit.edu
User UCID ### insert NJIT UCID here ###
</pre>
after the proxy is set up, connect to inti by simply typing:
<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
ssh -XY inti ### enter your NJIT UCID password when prompted ###
</pre>

'''NOTE:'''When logging in for the first time, please add the following lines (demarcated by #) in your ~/.bashrc (on the inti server) file in order to use CASA on inti.

<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
#### setting start ####
if [ $HOSTNAME == "baozi.hpcnet.campus.njit.edu" ]; then
source /srg/.setenv_baozi
fi
if [ $HOSTNAME == "inti.hpcnet.campus.njit.edu" ]; then
source /inti/.setenv_inti
fi
if [ $HOSTNAME == "guko.resource.campus.njit.edu" ]; then
source /data/data/.setenv_guko
fi
#### setting end ####
</pre>

For sufficient disk space to analyze EOVSA data on Inti, use your dedicated directory YOURDIRECTORY at the location given below. If you do not have a directory, please ask Sijie to set one up.
<pre style="background-color: #FCEBD9">
cd /inti/data/users/YOURDIRECTORY
</pre>

'''Both CASA 5 and 6 are available on Inti.'''

Enter the bash environment on inti, and load the desired casa environment.

To load CASA 5:
<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
>> bash
>> loadcasa5
>> suncasa
</pre>

To load CASA 6:
<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
>> bash
>> loadcasa6
>> ipython
</pre>

=== Downloading data and Preliminary Calibration ===

==== Step 1: Downloading raw (IDB) data on Pipeline machine====
The initial downloading of data and applying refcal and phasecal to the .ms files is performed on the pipeline since the complete EOVSA SQL database is stored here for now.

If you are not using mobaxterm, directly SSH to Pipeline through your Linux or Mac computer (as shown above) and '''start tcsh to run CASA'''.

The following commands (run within CASA) can be used to import a single IDB file if the given time-range is less than 10 mins.
<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
from astropy.time import Time
import os
trange = Time(['2017-08-21 20:15:00', '2017-08-21 20:25:00']) ###Change accordingly###

#### (Optional) change output path, default current directory "./" #######
outpath = './msdata/' ###Change accordingly###
if not os.path.exists(outpath):
os.makedirs(outpath)

msfiles = importeovsa(idbfiles=trange, ncpu=1, visprefix=outpath)
</pre>

'''NOTE:''' The above step currently gives an error with importeovsa. This is because the data location was changed and the code doesn't know where to look for the IDB files. Sijie is looking into the problem. We will update the page when it is fixed. For now, please use the method below for all time ranges.

OR (for a time range longer than 10 minutes)

<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
from suncasa.tasks import task_calibeovsa as calibeovsa
from suncasa.tasks import task_importeovsa as timporteovsa
from split_cli import split_cli as split
import dump_tsys as dt
from util import Time
import numpy as np
import os
from glob import glob
from eovsapy import util

trange = Time(['2017-08-21 20:15:00', '2017-08-21 20:35:00']) ###Change accordingly###
idbdir = util.get_idbdir(trange[0])

info = dt.rd_fdb(trange[0])
for k, v in info.iteritems():
info[k] = info[k][~(info[k] == '')]

sidx = np.where(
np.logical_and(info['SOURCEID'] == 'Sun', info['PROJECTID'] == 'NormalObserving') & np.logical_and(
info['ST_TS'].astype(np.float) >= trange[0].lv,
info['ST_TS'].astype(np.float) <= trange[
1].lv))
filelist = info['FILE'][sidx]

outpath = './msdata/' ###Change accordingly###
if not os.path.exists(outpath):
os.makedirs(outpath)
inpath = idbdir + '{}/'.format(trange[0].datetime.strftime("%Y%m%d"))
ncpu = 1

msfiles = timporteovsa.importeovsa(idbfiles=[inpath + ll for ll in filelist], ncpu=ncpu, timebin="0s", width=1,
visprefix=outpath,
nocreatms=False, doconcat=False,
modelms="", doscaling=False, keep_nsclms=False, udb_corr=True)
</pre>

If you come across errors with calibeovsa, add the following lines to your ~/.casa/init.py (within pipeline) file.
<pre style="background-color: #FCEBD9">
vim ~/.ssh/config ### press 'i' to insert content and esc +':wq' to write and quit ###
import sys
sys.path.append('/common/python')
sys.path.append('/common/python/packages/pipeline_casa')
execfile('/common/python/suncasa/tasks/mytasks.py')
</pre>

==== Step 2: Concatenate multiple .ms files and Calibrate====
'''Follow this step if there are more than one .ms files for the chosen time-range. If not, move to Step 3.'''

<pre style="background-color: #FCEBD9">
concatvis = os.path.basename(msfiles[0])[:11] + '_concat_cal.ms' ### Set the path/name for the concatenated files ###

vis = calibeovsa.calibeovsa(msfiles, doconcat=True, concatvis=concatvis, caltype=['refpha','phacal'], doimage=True)
### If doimage=True, a quicklook image will be produced (by integrating over the entire time) as shown in Figure 1 ###

vis_str = str(' '.join(vis))
caled_vis=vis_str.replace('.ms','_cal.ms') ### Append '_cal' to the ms filename ###
split(vis=' '.join(vis),outputvis=caled_vis,datacolumn='corrected',timerange='',correlation='XX') ### Split the corrected column to the new 'caled' ms ###
</pre>

[[file:Figure1_imagingtutorial.png|thumb|right|250px|'''Figure 1:''' Quick-look full-Sun image after the initial calibration.]]

==== Step 3: Calibration ====
For a single .ms file from Step 1, this will calibrate the input visibility, write out calibration tables under /data1/eovsa/caltable/, and apply the calibration.
<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
vis = calibeovsa.calibeovsa('IDB20170821202020.ms', caltype=['refpha','phacal'], doimage=True) ###Change the vis filename accordingly###

vis_str = str(' '.join(vis))
caled_vis=vis_str.replace('.ms','_cal.ms') ### Append '_cal' to the ms filename ###

split(vis=' '.join(vis),outputvis=caled_vis,datacolumn='corrected',timerange='',correlation='XX') ### Split the corrected column to the new 'caled' ms ###
</pre>

The created 'caled' ms data files (xxx_concat_cal.ms or xxx_cal.ms) need to be transferred from pipeline to inti server, which has all the casa tasks installed, in order to run the rest of the imaging steps.

=== Transferring data files between servers ===
To directly transfer your calibrated .ms data from the Pipeline to Inti:

<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
1. Log into Inti using your username.
ssh -X USERNAME@inti.hpcnet.campus.njit.edu

Then create a tunnel into Pipeline from Inti.
ssh -L 8888:pipeline.solar.pvt:22 guest@ovsa.njit.edu

2. Log into Inti again from a new terminal. Change to your working directory (where you need the data transferred) and give the following command:

scp -v -C -r -P 8888 PipelineUSERNAME@localhost:PATH/TO/MSFILE ./

</pre>

===Advanced Calibration and Imaging===
====Step 4: Self-calibration====
[[file:Figure3_imagingtutorial.png|thumb|right|250px|Figure 2: Cotton-Schwab clean major and minor cycles. [Source: http://www.aoc.nrao.edu/~rurvashi/ImagingAlgorithmsInCasa/node2.html].]]
Follow the steps below to run the self-calibration of the imaging data and produce the calibrated images in .fits format. https://github.com/binchensun/casa-eovsa/blob/master/slfcal_example.py

<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
bash
loadcasa5
suncasa

from suncasa.utils import helioimage2fits as hf
import os
import numpy as np
import pickle
from matplotlib import gridspec as gridspec
from sunpy import map as smap
from matplotlib import pyplot as plt
from split_cli import split_cli as split
import time

# =========== task handlers =============
dofullsun = 1 # initial full-sun imaging ###Change accordingly###
domasks=1 # get masks ###Change accordingly###
doslfcal=1 # main cycle of doing selfcalibration ###Change accordingly###
doapply=1 # apply the results ###Change accordingly###
doclean_slfcaled=1 # perform clean for self-calibrated data ###Change accordingly###

# ============ declaring the working directories ============
workdir = os.getcwd()+'/' #main working directory. Using current directory in this example
slfcaldir = workdir+'slfcal/' #place to put all selfcalibration products
imagedir = slfcaldir+'images/' #place to put all selfcalibration images
maskdir = slfcaldir+'masks/' #place to put clean masks
imagedir_slfcaled = slfcaldir+'images_slfcaled/' #place to put final self-calibrated images
caltbdir = slfcaldir+'caltbs/' # place to put calibration tables
# make these directories if they do not already exist
dirs = [workdir, slfcaldir, imagedir, maskdir, imagedir_slfcaled, caltbdir]
for d in dirs:
if not os.path.exists(d):
os.makedirs(d)

# ============ Split a short time for self-calibration ===========
# input visibility
ms_in = workdir + 'IDB20170821202020_cal.ms' ###Change the initial calibrated (through calibeovsa) vis accordingly###
# output, selfcaled, visibility
ms_slfcaled = workdir + os.path.basename(ms_in).replace('cal','slfcaled')
# intermediate small visibility for selfcalbration
# selected time range for generating self-calibration solutions
trange='2017/08/21/20:21:10~2017/08/21/20:21:30' ###Change accordingly###
slfcalms = slfcaldir+'slfcalms.XX.slfcal'
slfcaledms = slfcaldir+'slfcalms.XX.slfcaled'
if not os.path.exists(slfcalms):
split(vis=ms_in,outputvis=slfcalms,datacolumn='data',timerange=trange,correlation='XX')

# ============ Prior definitions for spectral windows, antennas, pixel numbers =========
spws=[str(s+1) for s in range(30)] # spws=[str(s+1) for s in range(49)] # For post-2020 data
antennas='0~12&0~12'
npix=512
nround=3 #number of slfcal cycles

# =========== Step 1, doing a full-Sun image to find out phasecenter and appropriate field of view =========
if dofullsun:
#initial mfs clean to find out the image phase center
im_init='fullsun_init'
os.system('rm -rf '+im_init+'*')
tclean(vis=slfcalms,
antenna=antennas,
imagename=im_init,
spw='2~15',
specmode='mfs',
timerange=trange,
imsize=[npix],
cell=['5arcsec'],
niter=1000,
gain=0.05,
stokes='I',
restoringbeam=['30arcsec'],
interactive=False,
pbcor=True)

hf.imreg(vis=slfcalms,imagefile=im_init+'.image.pbcor',fitsfile=im_init+'.fits',
timerange=trange,usephacenter=False,verbose=True)
clnjunks = ['.flux', '.mask', '.model', '.psf', '.residual','.sumwt','.pb','.image'] #Do not run the next 4 lines if needed to view and assess the subset of clean process images
for clnjunk in clnjunks:
if os.path.exists(im_init + clnjunk):
os.system('rm -rf '+im_init + clnjunk)

from sunpy import map as smap
from matplotlib import pyplot as plt
fig = plt.figure(figsize=(6,6))
ax = fig.add_subplot(111)
eomap=smap.Map(im_init+'.fits')
#eomap.data=eomap.data.reshape((npix,npix))
eomap.plot_settings['cmap'] = plt.get_cmap('jet')
eomap.plot(axes = ax)
eomap.draw_limb()
plt.show()
viewer(im_init+'.image.pbcor')
</pre>
[[file:Figure2_imagingtutorial.png|thumb|right|250px|Figure 3: Full-Sun image after initial clean to find the flare location.]]

<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
# parameters specific to the event (found from step 1)
phasecenter='J2000 10h02m59 11d58m07' ###Change accordingly###
xran=[280,480] ###Change accordingly###
yran=[-50,150] ###Change accordingly###

# =========== Step 2 (optional), generate masks =========
# if skipped, will not use any masks
if domasks:
clearcal(slfcalms)
delmod(slfcalms)
antennas=antennas
pol='XX'
imgprefix=maskdir+'slf_t0'

# step 1: set up the clean masks
img_init=imgprefix+'_init_ar_'
os.system('rm -rf '+img_init+'*')
#spwrans_mask=['1~5','6~12','13~20','21~30']
spwrans_mask=['2~12']
#convert to a list of spws
spwrans_mask_list = [[str(i) for i in (np.arange(int(m.split('~')[0]),int(m.split('~')[1])))] for m in spwrans_mask] # Not used?
masks=[]
imnames=[]
for spwran in spwrans_mask:
imname=img_init+spwran.replace('~','-')
try:
tclean(vis=slfcalms,
antenna=antennas,
imagename=imname,
spw=spwran,
specmode='mfs',
timerange=trange,
imsize=[npix],
cell=['2arcsec'],
niter=1000,
gain=0.05,
stokes='XX',
restoringbeam=['20arcsec'],
phasecenter=phasecenter,
weighting='briggs',
robust=1.0,
interactive=True,
datacolumn='data',
pbcor=True,
savemodel='modelcolumn')
imnames.append(imname+'.image')
masks.append(imname+'.mask')
clnjunks = ['.flux', '.model', '.psf', '.residual'] #Do not run the next 4 lines if needed to view and assess the subset of clean process images
for clnjunk in clnjunks:
if os.path.exists(imname + clnjunk):
os.system('rm -rf '+ imname + clnjunk)
except:
print('error in cleaning spw: '+spwran)

pickle.dump(masks,open(slfcaldir+'masks.p','wb'))
</pre>
[[file:Figure4_imagingtutorial.PNG|thumb|right|250px|Figure 4: Interactive clean window to create masks over the source. The white outline surrounding the source is the mask selected by the polygon drawing option.]]

The outlines drawn for masks can be created by any of the icons with the letter 'R' in the Viewer window. The instructions for doing this can be found by hovering over those icons.

<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
# =========== Step 3, main step of selfcalibration =========
if doslfcal:
if os.path.exists(slfcaldir+'masks.p'):
masks=pickle.load(open(slfcaldir+'masks.p','rb'))
if not os.path.exists(slfcaldir+'masks.p'):
print 'masks do not exist. Use default mask'
masks=[]
os.system('rm -rf '+imagedir+'*')
os.system('rm -rf '+caltbdir+'*')
#first step: make a mock caltable for the entire database
print('Processing ' + trange)
slftbs=[]
calprefix=caltbdir+'slf'
imgprefix=imagedir+'slf'
tb.open(slfcalms+'/SPECTRAL_WINDOW')
reffreqs=tb.getcol('REF_FREQUENCY')
bdwds=tb.getcol('TOTAL_BANDWIDTH')
cfreqs=reffreqs+bdwds/2.
tb.close()
# starting beam size at 3.4 GHz in arcsec #Change accordingly
sbeam=40.
strtmp=[m.replace(':','') for m in trange.split('~')]
timestr='t'+strtmp[0]+'-'+strtmp[1]
refantenna='0'
# number of iterations for each round
niters=[100, 300, 500]
# roburst value for weighting the baselines
robusts=[1.0, 0.5, 0.0]
# apply calibration tables? Set to true for most cases
doapplycal=[1,1,1]
# modes for calibration, 'p' for phase-only, 'a' for amplitude only, 'ap' for both
calmodes=['p','p','a']
# setting uvranges for model image (optional, not used here)
uvranges=['','','']
for n in range(nround):
slfcal_tb_g= calprefix+'.G'+str(n)
fig = plt.figure(figsize=(8.4,7.)) # fig = plt.figure(figsize=(14, 7)) # For post-2020 data (50 spws)
gs = gridspec.GridSpec(5, 6) # gs = gridspec.GridSpec(5, 10) # For post-2020 data (50 spws)
for s,sp in enumerate(spws):
print 'processing spw: '+sp
cfreq=cfreqs[int(sp)]
# setting restoring beam size (not very useful for selfcal anyway, but just to see the results)
bm=max(sbeam*cfreqs[1]/cfreq, 6.)
slfcal_img = imgprefix+'.spw'+sp.zfill(2)+'.slfcal'+str(n)
# only the first round uses nearby spws for getting initial model
if n == 0:
spbg=max(int(sp)-2,1) # spbg=max(int(sp)-2,0) # For post-2020 data (50 spws)
sped=min(int(sp)+2,30) # sped=min(int(sp)+2,49) # For post-2020 data (50 spws)
spwran=str(spbg)+'~'+str(sped)
print('using spw {0:s} as model'.format(spwran))
if 'spwrans_mask' in vars():
for m, spwran_mask in enumerate(spwrans_mask):
if sp in spwran_mask:
mask = masks[m]
print('using mask {0:s}'.format(mask))
findmask = True
if not findmask:
print('mask not found. Do use any masks')
else:
spwran = sp
if 'spwrans_mask' in vars():
for m, spwran_mask in enumerate(spwrans_mask):
if sp in spwran_mask:
mask = masks[m]
print 'using mask {0:s}'.format(mask)
findmask = True
if not findmask:
print('mask not found. Do use any masks')
try:
tclean(vis=slfcalms,
antenna=antennas,
imagename=slfcal_img,
uvrange=uvranges[n],
spw=spwran,
specmode='mfs',
timerange=trange,
imsize=[npix],
cell=['2arcsec'],
niter=niters[n],
gain=0.05,
stokes='XX', #use pol XX image as the model
weighting='briggs',
robust=robusts[n],
phasecenter=phasecenter,
mask=mask,
restoringbeam=[str(bm)+'arcsec'],
pbcor=False,
interactive=False,
savemodel='modelcolumn')
if os.path.exists(slfcal_img+'.image'):
fitsfile=slfcal_img+'.fits'
hf.imreg(vis=slfcalms,imagefile=slfcal_img+'.image',fitsfile=fitsfile,
timerange=trange,usephacenter=False,toTb=True,verbose=False,overwrite=True)
clnjunks = ['.mask','.flux', '.model', '.psf', '.residual', '.image','.pb','.image.pbcor','.sumwt'] #Do not run the next 4 lines if needed to view and assess the subset of clean process images
for clnjunk in clnjunks:
if os.path.exists(slfcal_img + clnjunk):
os.system('rm -rf '+ slfcal_img + clnjunk)
ax = fig.add_subplot(gs[s])
eomap=smap.Map(fitsfile)
eomap.plot_settings['cmap'] = plt.get_cmap('jet')
eomap.plot(axes = ax)
eomap.draw_limb()
#eomap.draw_grid()
ax.set_title(' ')
ax.get_xaxis().set_visible(False)
ax.get_yaxis().set_visible(False)
ax.set_xlim(xran)
ax.set_ylim(yran)
plt.pause(0.5) # Allows viewing of each image as it is plotted.
os.system('rm -f '+ fitsfile)

except:
print 'error in cleaning spw: '+sp
print 'using nearby spws for initial model'
sp_e=int(sp)+2
sp_i=int(sp)-2
if sp_i < 1: # if sp < 0: # For post-2020 data (50 spws)
sp_i = 1 # sp_i = 0 # For post-2020 data (50 spws)
if sp_e > 30: # if sp_e > 49: # For post-2020 data (50 spws)
sp_e = 30 # sp_e = 49 # For post-2020 data (50 spws)
sp_=str(sp_i)+'~'+str(sp_e)
try:
tget(tclean)
spw=sp_
print('using spw {0:s} as model'.format(sp_))
tclean()
except:
print 'still not successful. abort...'
break

gaincal(vis=slfcalms, refant=refantenna,antenna=antennas,caltable=slfcal_tb_g,spw=sp, uvrange='',\
gaintable=[],selectdata=True,timerange=trange,solint='inf',gaintype='G',calmode=calmodes[n],\
combine='',minblperant=4,minsnr=2,append=True)
if not os.path.exists(slfcal_tb_g):
print 'No solution found in spw: '+sp
figname=imagedir+'slf_t0_n{:d}.png'.format(n)
plt.subplots_adjust(left=0, bottom=0, right=1, top=1, wspace=0, hspace=0)
plt.savefig(figname)
time.sleep(10)
plt.close()

if os.path.exists(slfcal_tb_g):
slftbs.append(slfcal_tb_g)
slftb=[slfcal_tb_g]
os.chdir(slfcaldir)
if calmodes[n] == 'p':
plotcal(caltable=slfcal_tb_g,antenna='1~12',xaxis='freq',yaxis='phase',\
subplot=431,plotrange=[-1,-1,-180,180],iteration='antenna',figfile=slfcal_tb_g+'.png',showgui=False)
if calmodes[n] == 'a':
plotcal(caltable=slfcal_tb_g,antenna='1~12',xaxis='freq',yaxis='amp',\
subplot=431,plotrange=[-1,-1,0,2.],iteration='antenna',figfile=slfcal_tb_g+'.png',showgui=False)
os.chdir(workdir)
plt.pause(0.5) # Allows viewing of the plot

if doapplycal[n]:
clearcal(slfcalms)
delmod(slfcalms)
applycal(vis=slfcalms,gaintable=slftb,spw=','.join(spws),selectdata=True,\
antenna=antennas,interp='nearest',flagbackup=False,applymode='calonly',calwt=False)

if n < nround-1:
prompt=raw_input('Continuing to selfcal?')
#prompt='y'
if prompt.lower() == 'n':
if os.path.exists(slfcaledms):
os.system('rm -rf '+slfcaledms)
split(slfcalms,slfcaledms,datacolumn='corrected')
print 'Final calibrated ms is {0:s}'.format(slfcaledms)
break
if prompt.lower() == 'y':
slfcalms_=slfcalms+str(n)
if os.path.exists(slfcalms_):
os.system('rm -rf '+slfcalms_)
split(slfcalms,slfcalms_,datacolumn='corrected')
slfcalms=slfcalms_
else:
if os.path.exists(slfcaledms):
os.system('rm -rf '+slfcaledms)
split(slfcalms,slfcaledms,datacolumn='corrected')
print 'Final calibrated ms is {0:s}'.format(slfcaledms)
</pre>

<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
# =========== Step 4: Apply self-calibration tables =========
if doapply:
import glob
os.chdir(workdir)
clearcal(ms_in)
clearcal(slfcalms)
applycal(vis=slfcalms,gaintable=slftbs,spw=','.join(spws),selectdata=True,\
antenna=antennas,interp='linear',flagbackup=False,applymode='calonly',calwt=False)
applycal(vis=ms_in,gaintable=slftbs,spw=','.join(spws),selectdata=True,\
antenna=antennas,interp='linear',flagbackup=False,applymode='calonly',calwt=False)
if os.path.exists(ms_slfcaled):
os.system('rm -rf '+ms_slfcaled)
split(ms_in, ms_slfcaled,datacolumn='corrected')
</pre>
[[file:Slf.G0.png|thumb|right|250px|Figure 1: Phase before self-calibration]]
[[file:Slf.G1.png|thumb|right|250px|Figure 2: Phase after self-calibration]]

<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
# =========== Step 5: Generate final self-calibrated images (optional) =========
if doclean_slfcaled:
import glob
pol='XX'
print('Processing ' + trange)
img_final=imagedir_slfcaled+'/slf_final_{0}_t0'.format(pol)
vis = ms_slfcaled
spws=[str(s+1) for s in range(30)]
tb.open(vis+'/SPECTRAL_WINDOW')
reffreqs=tb.getcol('REF_FREQUENCY')
bdwds=tb.getcol('TOTAL_BANDWIDTH')
cfreqs=reffreqs+bdwds/2.
tb.close()
sbeam=30.
from matplotlib import gridspec as gridspec
from sunpy import map as smap
from matplotlib import pyplot as plt
fitsfiles=[]
for s,sp in enumerate(spws):
cfreq=cfreqs[int(sp)]
bm=max(sbeam*cfreqs[1]/cfreq,4.)
imname=img_final+'_s'+sp.zfill(2)
fitsfile=imname+'.fits'
if not os.path.exists(fitsfile):
print 'cleaning spw {0:s} with beam size {1:.1f}"'.format(sp,bm)
try:
tclean(vis=vis,
antenna=antennas,
imagename=imname,
spw=sp,
specmode='mfs',
timerange=trange,
imsize=[npix],
cell=['1arcsec'],
niter=1000,
gain=0.05,
stokes=pol,
weighting='briggs',
robust=2.0,
restoringbeam=[str(bm)+'arcsec'],
phasecenter=phasecenter,
mask='',
pbcor=True,
interactive=False)
except:
print 'cleaning spw '+sp+' unsuccessful. Proceed to next spw'
continue
if os.path.exists(imname+'.image.pbcor'):
imn = imname+'.image.pbcor'
hf.imreg(vis=vis,imagefile=imn,fitsfile=fitsfile,
timerange=trange,usephacenter=False,toTb=True,verbose=False)
fitsfiles.append(fitsfile)
junks=['.flux','.model','.psf','.residual','.mask','.image','.pb','.image.pbcor','.sumwt'] #Do not run the next 4 lines if needed to view and assess the subset of clean process images
for junk in junks:
if os.path.exists(imname+junk):
os.system('rm -rf '+imname+junk)
else:
print('fits file '+fitsfile+' already exists, skip clean...')
fitsfiles.append(fitsfile)

fig = plt.figure(figsize=(8.4,7.)) # fig = plt.figure(figsize=(14, 7)) # For post-2020 data (50 spws)
gs = gridspec.GridSpec(5, 6) # gs = gridspec.GridSpec(5, 10) # For post-2020 data (50 spws)
for s,sp in enumerate(spws):
cfreq=cfreqs[int(sp)]
ax = fig.add_subplot(gs[s])
eomap=smap.Map(fitsfiles[s])
eomap.plot_settings['cmap'] = plt.get_cmap('jet')
eomap.plot(axes = ax)
eomap.draw_limb()
ax.set_title(' ')
ax.get_xaxis().set_visible(False)
ax.get_yaxis().set_visible(False)
ax.set_xlim(xran)
ax.set_ylim(yran)
plt.text(0.98,0.85,'{0:.1f} GHz'.format(cfreq/1e9),transform=ax.transAxes,ha='right',color='w',fontweight='bold')
plt.subplots_adjust(left=0, bottom=0, right=1, top=1, wspace=0, hspace=0)
plt.show()

</pre>
The .fits files of the self calibrated images at each frequency for the given time are saved at /slfcal/images_slfcaled in your working directory.

[[file:Slf_t0_n0.png|thumb|right|250px|Figure 3: Multi-frequency images before self-calibration]]
[[file:Slf_t0_n1.png|thumb|right|250px|Figure 4: Multi-frequency images after self-calibration]]

====Step 5: Quick-look imaging ==== 
For spectral imaging analysis of the event, follow [http://www.ovsa.njit.edu/wiki/index.php/EOVSA_Data_Analysis_Tutorial#Spectral_Imaging_with_SunCASA this tutorial] using the self-calibrated data obtained from the previous step or use [https://colab.research.google.com/drive/1lSLLxgOG6b8kgu9Sk6kSKvrViyubnXG6?usp=sharing#scrollTo=xbXyyLmCFCGL this link].

<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
from suncasa.utils import qlookplot as ql ## (Optional) Supply the npz file of the dynamic spectrum from previous step.
## If not provided, the program will generate a new one from the visibility.
## set the time interval
from suncasa import dspec as ds
import time
visibility_data = 'IDB20170821202020_slfcaled.ms'
specfile = visibility_data + '.dspec.npz'
d = ds.Dspec(visibility_data, bl='4&9', specfile=specfile)

timerange = '19:02:00~19:02:10' ## select frequency range from 2.5 GHz to 3.5 GHz
spw = '2~5' ## select stokes XX
stokes = 'XX' ## turn off AIA image plotting, default is True
plotaia = False
xycen = [375, 45] ## image center for clean in solar X-Y in arcsec
cell=['2.0arcsec'] ## pixel size
imsize=[128] ## x and y image size in pixels.
fov = [100,100] ## field of view of the zoomed-in panels in unit of arcsec
spw = ['{}'.format(s) for s in range(1,31)]
clevels = [0.5, 1.0] ## contour levels to fill in between.
calpha=0.35 ## now tune down the alpha
restoringbeam=['6arcsec']
ql.qlookplot(vis=msfile, specfile=specfile, timerange=timerange, spw=spw, stokes=stokes, \
restoringbeam=restoringbeam,imsize=imsize,cell=cell, \
xycen=xycen,fov=fov,clevels=clevels,calpha=calpha)
</pre>

Page Last Updated: 07/06/2022

Tohban EOVSA Imaging Tutorial A-Z V2

2022-07-06T11:27:29Z

Schhabra:

This tutorial describes the step-by-step procedure to download EOVSA IDB data, obtain and calibrate the measurement sets (.ms), and, transfer them to the inti server for self-calibration and further analysis.

=== Pre-requisites ===
Accounts on Pipeline and Inti or Baozi servers. Sijie and Bin can help with the account set-up for pipeline, inti and boazi.

==== How to connect to pipeline ====
'''Windows:''' On a windows machine Mobaxterm can be used to connect to Pipeline. For details, see ...

'''Linux/Unix:''' On a linux/Mac the ssh commands can be used directly through the terminal:

<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">

ssh -L 5902:helios.solar.pvt:20000 guest@ovsa.njit.edu ### when prompted, enter password to guest@ovsa ###
ssh -XY your_username@pipeline ### add your username and enter password to your pipeline account, when prompted ###
</pre>

Alternatively, to avoid typing the longer commands, proxy commands can be added to the ~/.ssh/config file:

<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
vim ~/.ssh/config ### press 'i' to insert content and esc +':wq' to write and quit ###

Host *
ForwardAgent yes

Host *
ForwardX11 yes

Host ovsa
HostName ovsa.njit.edu
User guest

Host pipeline
Hostname pipeline.solar.pvt
User your_username ### Insert your pipeline username here ###
ProxyCommand ssh -W %h:%p guest@ovsa.njit.edu
</pre>

Upon saving the ~/.ssh/config file and restarting the terminal, you can connect to pipeline directly using:

<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
ssh -XY pipeline
</pre>

==== How to connect to the inti server ====
NOTE: To connect to inti you must have a NJIT UCID. The arcs have granted access to inti for all SRG members. If you encounter any issue with the login. Please let arcs folks (arcs@njit.edu) know.

'''Windows:''' As mentioned above, on a windows machine Mobaxterm can be used to connect to inti.
With your NJIT VPN connected, connect to one of the afsconnect servers (for example, afsaccess3.njit.edu) using your UCID and password. Next, ssh into inti.
<pre style="background-color: #FCEBD9">
ssh -XY UCID@afsaccess3.njit.edu ###Insert UCID password if prompted###

ssh -XY UCID@inti.hpcnet.campus.njit.edu ###Insert UCID/inti username here###
</pre>
Proxy commands for inti can also be added to the C:\Program Files\Git\etc\ssh\ssh_config file:
<pre style="background-color: #FCEBD9">
Host inti
Hostname inti.hpcnet.campus.njit.edu
User UCID ###Insert UCID/inti username here###
</pre>

'''Linux/Unix:''' On a linux/Mac the ssh command can be used:
<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
ssh -XY UCID@inti.hpcnet.campus.njit.edu ### insert UCID here and enter your NJIT UCID password, when prompted ###
</pre>

Proxy commands for inti can also be added to the ~/.ssh/config (on local machine) file for convenience:
<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
vim ~/.ssh/config ### press 'i' to insert content and esc +':wq' to write and quit ###

Host inti
Hostname inti.hpcnet.campus.njit.edu
User UCID ### insert NJIT UCID here ###
</pre>
after the proxy is set up, connect to inti by simply typing:
<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
ssh -XY inti ### enter your NJIT UCID password when prompted ###
</pre>

'''NOTE:'''When logging in for the first time, please add the following lines (demarcated by #) in your ~/.bashrc (on the inti server) file in order to use CASA on inti.

<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
#### setting start ####
if [ $HOSTNAME == "baozi.hpcnet.campus.njit.edu" ]; then
source /srg/.setenv_baozi
fi
if [ $HOSTNAME == "inti.hpcnet.campus.njit.edu" ]; then
source /inti/.setenv_inti
fi
if [ $HOSTNAME == "guko.resource.campus.njit.edu" ]; then
source /data/data/.setenv_guko
fi
#### setting end ####
</pre>

For sufficient disk space to analyze EOVSA data on Inti, use your dedicated directory YOURDIRECTORY at the location given below. If you do not have a directory, please ask Sijie to set one up.
<pre style="background-color: #FCEBD9">
cd /inti/data/users/YOURDIRECTORY
</pre>

'''Both CASA 5 and 6 are available on Inti.'''

Enter the bash environment on inti, and load the desired casa environment.

To load CASA 5:
<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
>> bash
>> loadcasa5
>> suncasa
</pre>

To load CASA 6:
<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
>> bash
>> loadcasa6
>> ipython
</pre>

=== Downloading data and Preliminary Calibration ===

==== Step 1: Downloading raw (IDB) data on Pipeline machine====
The initial downloading of data and applying refcal and phasecal to the .ms files is performed on the pipeline since the complete EOVSA SQL database is stored here for now.

If you are not using mobaxterm, directly SSH to Pipeline through your Linux or Mac computer (as shown above) and '''start tcsh to run CASA'''.

The following commands (run within CASA) can be used to import a single IDB file if the given time-range is less than 10 mins.
<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
from astropy.time import Time
import os
trange = Time(['2017-08-21 20:15:00', '2017-08-21 20:25:00']) ###Change accordingly###

#### (Optional) change output path, default current directory "./" #######
outpath = './msdata/' ###Change accordingly###
if not os.path.exists(outpath):
os.makedirs(outpath)

msfiles = importeovsa(idbfiles=trange, ncpu=1, visprefix=outpath)
</pre>

'''NOTE:''' The above step currently gives an error with importeovsa. This is because the data location was changed and the code doesn't know where to look for the IDB files. Sijie is looking into the problem. We will update the page when it is fixed. For now, please use the method below for all time ranges.

OR (for a time range longer than 10 minutes)

<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
from suncasa.tasks import task_calibeovsa as calibeovsa
from suncasa.tasks import task_importeovsa as timporteovsa
from split_cli import split_cli as split
import dump_tsys as dt
from util import Time
import numpy as np
import os
from glob import glob
from eovsapy import util

trange = Time(['2017-08-21 20:15:00', '2017-08-21 20:35:00']) ###Change accordingly###
idbdir = util.get_idbdir(trange[0])

info = dt.rd_fdb(trange[0])
for k, v in info.iteritems():
info[k] = info[k][~(info[k] == '')]

sidx = np.where(
np.logical_and(info['SOURCEID'] == 'Sun', info['PROJECTID'] == 'NormalObserving') & np.logical_and(
info['ST_TS'].astype(np.float) >= trange[0].lv,
info['ST_TS'].astype(np.float) <= trange[
1].lv))
filelist = info['FILE'][sidx]

outpath = './msdata/' ###Change accordingly###
if not os.path.exists(outpath):
os.makedirs(outpath)
inpath = idbdir + '{}/'.format(trange[0].datetime.strftime("%Y%m%d"))
ncpu = 1

msfiles = timporteovsa.importeovsa(idbfiles=[inpath + ll for ll in filelist], ncpu=ncpu, timebin="0s", width=1,
visprefix=outpath,
nocreatms=False, doconcat=False,
modelms="", doscaling=False, keep_nsclms=False, udb_corr=True)
</pre>

If you come across errors with calibeovsa, add the following lines to your ~/.casa/init.py (within pipeline) file.
<pre style="background-color: #FCEBD9">
vim ~/.ssh/config ### press 'i' to insert content and esc +':wq' to write and quit ###
import sys
sys.path.append('/common/python')
sys.path.append('/common/python/packages/pipeline_casa')
execfile('/common/python/suncasa/tasks/mytasks.py')
</pre>

==== Step 2: Concatenate multiple .ms files and Calibrate====
'''Follow this step if there are more than one .ms files for the chosen time-range. If not, move to Step 3.'''

<pre style="background-color: #FCEBD9">
concatvis = os.path.basename(msfiles[0])[:11] + '_concat_cal.ms' ### Set the path/name for the concatenated files ###

vis = calibeovsa.calibeovsa(msfiles, doconcat=True, concatvis=concatvis, caltype=['refpha','phacal'], doimage=True)
### If doimage=True, a quicklook image will be produced (by integrating over the entire time) as shown below ###

vis_str = str(' '.join(vis))
caled_vis=vis_str.replace('.ms','_cal.ms') ### Append '_cal' to the ms filename ###
split(vis=' '.join(vis),outputvis=caled_vis,datacolumn='corrected',timerange='',correlation='XX') ### Split the corrected column to the new 'caled' ms ###
</pre>

[[file:Figure1_imagingtutorial.png|thumb|right|350px|'''Figure 1:''' Quick-look full-Sun image after the initial calibration.]]

==== Step 3: Calibration ====
For a single .ms file from Step 1, this will calibrate the input visibility, write out calibration tables under /data1/eovsa/caltable/, and apply the calibration.
<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
vis = calibeovsa.calibeovsa('IDB20170821202020.ms', caltype=['refpha','phacal'], doimage=True) ###Change the vis filename accordingly###

vis_str = str(' '.join(vis))
caled_vis=vis_str.replace('.ms','_cal.ms') ### Append '_cal' to the ms filename ###
split(vis=' '.join(vis),outputvis=caled_vis,datacolumn='corrected',timerange='',correlation='XX') ### Split the corrected column to the new 'caled' ms ###
</pre>

The created 'caled' ms data files (xxx_concat_cal.ms or xxx_cal.ms) need to be transferred from pipeline to inti server, which has all the casa tasks installed, in order to run the rest of the imaging steps.

=== Transferring data files between servers ===
To directly transfer your calibrated .ms data from the Pipeline to Inti:

<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
1. Log into Inti using your username.
ssh -X USERNAME@inti.hpcnet.campus.njit.edu

Then create a tunnel into Pipeline from Inti.
ssh -L 8888:pipeline.solar.pvt:22 guest@ovsa.njit.edu

2. Log into Inti again from a new terminal. Change to your working directory (where you need the data transferred) and give the following command:

scp -v -C -r -P 8888 PipelineUSERNAME@localhost:PATH/TO/MSFILE ./

</pre>

===Advanced Calibration and Imaging===
====Step 4: Self-calibration====
[[file:Figure3_imagingtutorial.png|thumb|center|500px|Figure 2: Cotton-Schwab clean major and minor cycles. [Source: http://www.aoc.nrao.edu/~rurvashi/ImagingAlgorithmsInCasa/node2.html].]]
Follow the below given steps to run the self-calibration of the imaging data and produce the calibrated images in .fits format. https://github.com/binchensun/casa-eovsa/blob/master/slfcal_example.py

<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
bash
loadcasa5
suncasa

from suncasa.utils import helioimage2fits as hf
import os
import numpy as np
import pickle
from matplotlib import gridspec as gridspec
from sunpy import map as smap
from matplotlib import pyplot as plt
from split_cli import split_cli as split
import time

# =========== task handlers =============
dofullsun = 1 # initial full-sun imaging ###Change accordingly###
domasks=1 # get masks ###Change accordingly###
doslfcal=1 # main cycle of doing selfcalibration ###Change accordingly###
doapply=1 # apply the results ###Change accordingly###
doclean_slfcaled=1 # perform clean for self-calibrated data ###Change accordingly###

# ============ declaring the working directories ============
workdir = os.getcwd()+'/' #main working directory. Using current directory in this example
slfcaldir = workdir+'slfcal/' #place to put all selfcalibration products
imagedir = slfcaldir+'images/' #place to put all selfcalibration images
maskdir = slfcaldir+'masks/' #place to put clean masks
imagedir_slfcaled = slfcaldir+'images_slfcaled/' #place to put final self-calibrated images
caltbdir = slfcaldir+'caltbs/' # place to put calibration tables
# make these directories if they do not already exist
dirs = [workdir, slfcaldir, imagedir, maskdir, imagedir_slfcaled, caltbdir]
for d in dirs:
if not os.path.exists(d):
os.makedirs(d)

# ============ Split a short time for self-calibration ===========
# input visibility
ms_in = workdir + 'IDB20170821202020_cal.ms' ###Change the initial calibrated (through calibeovsa) vis accordingly###
# output, selfcaled, visibility
ms_slfcaled = workdir + os.path.basename(ms_in).replace('cal','slfcaled')
# intermediate small visibility for selfcalbration
# selected time range for generating self-calibration solutions
trange='2017/08/21/20:21:10~2017/08/21/20:21:30' ###Change accordingly###
slfcalms = slfcaldir+'slfcalms.XX.slfcal'
slfcaledms = slfcaldir+'slfcalms.XX.slfcaled'
if not os.path.exists(slfcalms):
split(vis=ms_in,outputvis=slfcalms,datacolumn='data',timerange=trange,correlation='XX')

# ============ Prior definitions for spectral windows, antennas, pixel numbers =========
spws=[str(s+1) for s in range(30)] # spws=[str(s+1) for s in range(49)] # For post-2020 data
antennas='0~12&0~12'
npix=512
nround=3 #number of slfcal cycles

# =========== Step 1, doing a full-Sun image to find out phasecenter and appropriate field of view =========
if dofullsun:
#initial mfs clean to find out the image phase center
im_init='fullsun_init'
os.system('rm -rf '+im_init+'*')
tclean(vis=slfcalms,
antenna=antennas,
imagename=im_init,
spw='2~15',
specmode='mfs',
timerange=trange,
imsize=[npix],
cell=['5arcsec'],
niter=1000,
gain=0.05,
stokes='I',
restoringbeam=['30arcsec'],
interactive=False,
pbcor=True)

hf.imreg(vis=slfcalms,imagefile=im_init+'.image.pbcor',fitsfile=im_init+'.fits',
timerange=trange,usephacenter=False,verbose=True)
clnjunks = ['.flux', '.mask', '.model', '.psf', '.residual','.sumwt','.pb','.image'] #Do not run the next 4 lines if needed to view and assess the subset of clean process images
for clnjunk in clnjunks:
if os.path.exists(im_init + clnjunk):
os.system('rm -rf '+im_init + clnjunk)

from sunpy import map as smap
from matplotlib import pyplot as plt
fig = plt.figure(figsize=(6,6))
ax = fig.add_subplot(111)
eomap=smap.Map(im_init+'.fits')
#eomap.data=eomap.data.reshape((npix,npix))
eomap.plot_settings['cmap'] = plt.get_cmap('jet')
eomap.plot(axes = ax)
eomap.draw_limb()
plt.show()
viewer(im_init+'.image.pbcor')
</pre>
[[file:Figure2_imagingtutorial.png|thumb|center|500px|Figure 3: Full-Sun image after initial clean to find the flare location.]]

<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
# parameters specific to the event (found from step 1)
phasecenter='J2000 10h02m59 11d58m07' ###Change accordingly###
xran=[280,480] ###Change accordingly###
yran=[-50,150] ###Change accordingly###

# =========== Step 2 (optional), generate masks =========
# if skipped, will not use any masks
if domasks:
clearcal(slfcalms)
delmod(slfcalms)
antennas=antennas
pol='XX'
imgprefix=maskdir+'slf_t0'

# step 1: set up the clean masks
img_init=imgprefix+'_init_ar_'
os.system('rm -rf '+img_init+'*')
#spwrans_mask=['1~5','6~12','13~20','21~30']
spwrans_mask=['2~12']
#convert to a list of spws
spwrans_mask_list = [[str(i) for i in (np.arange(int(m.split('~')[0]),int(m.split('~')[1])))] for m in spwrans_mask] # Not used?
masks=[]
imnames=[]
for spwran in spwrans_mask:
imname=img_init+spwran.replace('~','-')
try:
tclean(vis=slfcalms,
antenna=antennas,
imagename=imname,
spw=spwran,
specmode='mfs',
timerange=trange,
imsize=[npix],
cell=['2arcsec'],
niter=1000,
gain=0.05,
stokes='XX',
restoringbeam=['20arcsec'],
phasecenter=phasecenter,
weighting='briggs',
robust=1.0,
interactive=True,
datacolumn='data',
pbcor=True,
savemodel='modelcolumn')
imnames.append(imname+'.image')
masks.append(imname+'.mask')
clnjunks = ['.flux', '.model', '.psf', '.residual'] #Do not run the next 4 lines if needed to view and assess the subset of clean process images
for clnjunk in clnjunks:
if os.path.exists(imname + clnjunk):
os.system('rm -rf '+ imname + clnjunk)
except:
print('error in cleaning spw: '+spwran)

pickle.dump(masks,open(slfcaldir+'masks.p','wb'))
</pre>
[[file:Figure4_imagingtutorial.PNG|thumb|center|800px|Figure 4: Interactive clean window to create masks over the source. The white outline surrounding the source is the mask selected by the polygon drawing option.]]

The outlines drawn for masks can be created by any of the icons with the letter 'R' in the Viewer window. The instructions for doing this can be found by hovering over those icons.

<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
# =========== Step 3, main step of selfcalibration =========
if doslfcal:
if os.path.exists(slfcaldir+'masks.p'):
masks=pickle.load(open(slfcaldir+'masks.p','rb'))
if not os.path.exists(slfcaldir+'masks.p'):
print 'masks do not exist. Use default mask'
masks=[]
os.system('rm -rf '+imagedir+'*')
os.system('rm -rf '+caltbdir+'*')
#first step: make a mock caltable for the entire database
print('Processing ' + trange)
slftbs=[]
calprefix=caltbdir+'slf'
imgprefix=imagedir+'slf'
tb.open(slfcalms+'/SPECTRAL_WINDOW')
reffreqs=tb.getcol('REF_FREQUENCY')
bdwds=tb.getcol('TOTAL_BANDWIDTH')
cfreqs=reffreqs+bdwds/2.
tb.close()
# starting beam size at 3.4 GHz in arcsec #Change accordingly
sbeam=40.
strtmp=[m.replace(':','') for m in trange.split('~')]
timestr='t'+strtmp[0]+'-'+strtmp[1]
refantenna='0'
# number of iterations for each round
niters=[100, 300, 500]
# roburst value for weighting the baselines
robusts=[1.0, 0.5, 0.0]
# apply calibration tables? Set to true for most cases
doapplycal=[1,1,1]
# modes for calibration, 'p' for phase-only, 'a' for amplitude only, 'ap' for both
calmodes=['p','p','a']
# setting uvranges for model image (optional, not used here)
uvranges=['','','']
for n in range(nround):
slfcal_tb_g= calprefix+'.G'+str(n)
fig = plt.figure(figsize=(8.4,7.)) # fig = plt.figure(figsize=(14, 7)) # For post-2020 data (50 spws)
gs = gridspec.GridSpec(5, 6) # gs = gridspec.GridSpec(5, 10) # For post-2020 data (50 spws)
for s,sp in enumerate(spws):
print 'processing spw: '+sp
cfreq=cfreqs[int(sp)]
# setting restoring beam size (not very useful for selfcal anyway, but just to see the results)
bm=max(sbeam*cfreqs[1]/cfreq, 6.)
slfcal_img = imgprefix+'.spw'+sp.zfill(2)+'.slfcal'+str(n)
# only the first round uses nearby spws for getting initial model
if n == 0:
spbg=max(int(sp)-2,1) # spbg=max(int(sp)-2,0) # For post-2020 data (50 spws)
sped=min(int(sp)+2,30) # sped=min(int(sp)+2,49) # For post-2020 data (50 spws)
spwran=str(spbg)+'~'+str(sped)
print('using spw {0:s} as model'.format(spwran))
if 'spwrans_mask' in vars():
for m, spwran_mask in enumerate(spwrans_mask):
if sp in spwran_mask:
mask = masks[m]
print('using mask {0:s}'.format(mask))
findmask = True
if not findmask:
print('mask not found. Do use any masks')
else:
spwran = sp
if 'spwrans_mask' in vars():
for m, spwran_mask in enumerate(spwrans_mask):
if sp in spwran_mask:
mask = masks[m]
print 'using mask {0:s}'.format(mask)
findmask = True
if not findmask:
print('mask not found. Do use any masks')
try:
tclean(vis=slfcalms,
antenna=antennas,
imagename=slfcal_img,
uvrange=uvranges[n],
spw=spwran,
specmode='mfs',
timerange=trange,
imsize=[npix],
cell=['2arcsec'],
niter=niters[n],
gain=0.05,
stokes='XX', #use pol XX image as the model
weighting='briggs',
robust=robusts[n],
phasecenter=phasecenter,
mask=mask,
restoringbeam=[str(bm)+'arcsec'],
pbcor=False,
interactive=False,
savemodel='modelcolumn')
if os.path.exists(slfcal_img+'.image'):
fitsfile=slfcal_img+'.fits'
hf.imreg(vis=slfcalms,imagefile=slfcal_img+'.image',fitsfile=fitsfile,
timerange=trange,usephacenter=False,toTb=True,verbose=False,overwrite=True)
clnjunks = ['.mask','.flux', '.model', '.psf', '.residual', '.image','.pb','.image.pbcor','.sumwt'] #Do not run the next 4 lines if needed to view and assess the subset of clean process images
for clnjunk in clnjunks:
if os.path.exists(slfcal_img + clnjunk):
os.system('rm -rf '+ slfcal_img + clnjunk)
ax = fig.add_subplot(gs[s])
eomap=smap.Map(fitsfile)
eomap.plot_settings['cmap'] = plt.get_cmap('jet')
eomap.plot(axes = ax)
eomap.draw_limb()
#eomap.draw_grid()
ax.set_title(' ')
ax.get_xaxis().set_visible(False)
ax.get_yaxis().set_visible(False)
ax.set_xlim(xran)
ax.set_ylim(yran)
plt.pause(0.5) # Allows viewing of each image as it is plotted.
os.system('rm -f '+ fitsfile)

except:
print 'error in cleaning spw: '+sp
print 'using nearby spws for initial model'
sp_e=int(sp)+2
sp_i=int(sp)-2
if sp_i < 1: # if sp < 0: # For post-2020 data (50 spws)
sp_i = 1 # sp_i = 0 # For post-2020 data (50 spws)
if sp_e > 30: # if sp_e > 49: # For post-2020 data (50 spws)
sp_e = 30 # sp_e = 49 # For post-2020 data (50 spws)
sp_=str(sp_i)+'~'+str(sp_e)
try:
tget(tclean)
spw=sp_
print('using spw {0:s} as model'.format(sp_))
tclean()
except:
print 'still not successful. abort...'
break

gaincal(vis=slfcalms, refant=refantenna,antenna=antennas,caltable=slfcal_tb_g,spw=sp, uvrange='',\
gaintable=[],selectdata=True,timerange=trange,solint='inf',gaintype='G',calmode=calmodes[n],\
combine='',minblperant=4,minsnr=2,append=True)
if not os.path.exists(slfcal_tb_g):
print 'No solution found in spw: '+sp
figname=imagedir+'slf_t0_n{:d}.png'.format(n)
plt.subplots_adjust(left=0, bottom=0, right=1, top=1, wspace=0, hspace=0)
plt.savefig(figname)
time.sleep(10)
plt.close()

if os.path.exists(slfcal_tb_g):
slftbs.append(slfcal_tb_g)
slftb=[slfcal_tb_g]
os.chdir(slfcaldir)
if calmodes[n] == 'p':
plotcal(caltable=slfcal_tb_g,antenna='1~12',xaxis='freq',yaxis='phase',\
subplot=431,plotrange=[-1,-1,-180,180],iteration='antenna',figfile=slfcal_tb_g+'.png',showgui=False)
if calmodes[n] == 'a':
plotcal(caltable=slfcal_tb_g,antenna='1~12',xaxis='freq',yaxis='amp',\
subplot=431,plotrange=[-1,-1,0,2.],iteration='antenna',figfile=slfcal_tb_g+'.png',showgui=False)
os.chdir(workdir)
plt.pause(0.5) # Allows viewing of the plot

if doapplycal[n]:
clearcal(slfcalms)
delmod(slfcalms)
applycal(vis=slfcalms,gaintable=slftb,spw=','.join(spws),selectdata=True,\
antenna=antennas,interp='nearest',flagbackup=False,applymode='calonly',calwt=False)

if n < nround-1:
prompt=raw_input('Continuing to selfcal?')
#prompt='y'
if prompt.lower() == 'n':
if os.path.exists(slfcaledms):
os.system('rm -rf '+slfcaledms)
split(slfcalms,slfcaledms,datacolumn='corrected')
print 'Final calibrated ms is {0:s}'.format(slfcaledms)
break
if prompt.lower() == 'y':
slfcalms_=slfcalms+str(n)
if os.path.exists(slfcalms_):
os.system('rm -rf '+slfcalms_)
split(slfcalms,slfcalms_,datacolumn='corrected')
slfcalms=slfcalms_
else:
if os.path.exists(slfcaledms):
os.system('rm -rf '+slfcaledms)
split(slfcalms,slfcaledms,datacolumn='corrected')
print 'Final calibrated ms is {0:s}'.format(slfcaledms)
</pre>

<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
# =========== Step 4: Apply self-calibration tables =========
if doapply:
import glob
os.chdir(workdir)
clearcal(ms_in)
clearcal(slfcalms)
applycal(vis=slfcalms,gaintable=slftbs,spw=','.join(spws),selectdata=True,\
antenna=antennas,interp='linear',flagbackup=False,applymode='calonly',calwt=False)
applycal(vis=ms_in,gaintable=slftbs,spw=','.join(spws),selectdata=True,\
antenna=antennas,interp='linear',flagbackup=False,applymode='calonly',calwt=False)
if os.path.exists(ms_slfcaled):
os.system('rm -rf '+ms_slfcaled)
split(ms_in, ms_slfcaled,datacolumn='corrected')
</pre>
[[file:Slf.G0.png|thumb|center|500px|Figure 1: Phase before self-calibration]]
[[file:Slf.G1.png|thumb|center|500px|Figure 2: Phase after self-calibration]]

<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
# =========== Step 5: Generate final self-calibrated images (optional) =========
if doclean_slfcaled:
import glob
pol='XX'
print('Processing ' + trange)
img_final=imagedir_slfcaled+'/slf_final_{0}_t0'.format(pol)
vis = ms_slfcaled
spws=[str(s+1) for s in range(30)]
tb.open(vis+'/SPECTRAL_WINDOW')
reffreqs=tb.getcol('REF_FREQUENCY')
bdwds=tb.getcol('TOTAL_BANDWIDTH')
cfreqs=reffreqs+bdwds/2.
tb.close()
sbeam=30.
from matplotlib import gridspec as gridspec
from sunpy import map as smap
from matplotlib import pyplot as plt
fitsfiles=[]
for s,sp in enumerate(spws):
cfreq=cfreqs[int(sp)]
bm=max(sbeam*cfreqs[1]/cfreq,4.)
imname=img_final+'_s'+sp.zfill(2)
fitsfile=imname+'.fits'
if not os.path.exists(fitsfile):
print 'cleaning spw {0:s} with beam size {1:.1f}"'.format(sp,bm)
try:
tclean(vis=vis,
antenna=antennas,
imagename=imname,
spw=sp,
specmode='mfs',
timerange=trange,
imsize=[npix],
cell=['1arcsec'],
niter=1000,
gain=0.05,
stokes=pol,
weighting='briggs',
robust=2.0,
restoringbeam=[str(bm)+'arcsec'],
phasecenter=phasecenter,
mask='',
pbcor=True,
interactive=False)
except:
print 'cleaning spw '+sp+' unsuccessful. Proceed to next spw'
continue
if os.path.exists(imname+'.image.pbcor'):
imn = imname+'.image.pbcor'
hf.imreg(vis=vis,imagefile=imn,fitsfile=fitsfile,
timerange=trange,usephacenter=False,toTb=True,verbose=False)
fitsfiles.append(fitsfile)
junks=['.flux','.model','.psf','.residual','.mask','.image','.pb','.image.pbcor','.sumwt'] #Do not run the next 4 lines if needed to view and assess the subset of clean process images
for junk in junks:
if os.path.exists(imname+junk):
os.system('rm -rf '+imname+junk)
else:
print('fits file '+fitsfile+' already exists, skip clean...')
fitsfiles.append(fitsfile)

fig = plt.figure(figsize=(8.4,7.)) # fig = plt.figure(figsize=(14, 7)) # For post-2020 data (50 spws)
gs = gridspec.GridSpec(5, 6) # gs = gridspec.GridSpec(5, 10) # For post-2020 data (50 spws)
for s,sp in enumerate(spws):
cfreq=cfreqs[int(sp)]
ax = fig.add_subplot(gs[s])
eomap=smap.Map(fitsfiles[s])
eomap.plot_settings['cmap'] = plt.get_cmap('jet')
eomap.plot(axes = ax)
eomap.draw_limb()
ax.set_title(' ')
ax.get_xaxis().set_visible(False)
ax.get_yaxis().set_visible(False)
ax.set_xlim(xran)
ax.set_ylim(yran)
plt.text(0.98,0.85,'{0:.1f} GHz'.format(cfreq/1e9),transform=ax.transAxes,ha='right',color='w',fontweight='bold')
plt.subplots_adjust(left=0, bottom=0, right=1, top=1, wspace=0, hspace=0)
plt.show()

</pre>
The .fits files of the self calibrated images at each frequency for the given time are saved at /slfcal/images_slfcaled in your working directory.

[[file:Slf_t0_n0.png|thumb|center|500px|Figure 3: Multi-frequency images before self-calibration]]
[[file:Slf_t0_n1.png|thumb|center|500px|Figure 4: Multi-frequency images after self-calibration]]

====Step 5: Quick-look imaging ==== 
For spectral imaging analysis of the event, follow [http://www.ovsa.njit.edu/wiki/index.php/EOVSA_Data_Analysis_Tutorial#Spectral_Imaging_with_SunCASA this tutorial] using the self-calibrated data obtained from the previous step or use [https://colab.research.google.com/drive/1lSLLxgOG6b8kgu9Sk6kSKvrViyubnXG6?usp=sharing#scrollTo=xbXyyLmCFCGL this link].

<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
from suncasa.utils import qlookplot as ql ## (Optional) Supply the npz file of the dynamic spectrum from previous step.
## If not provided, the program will generate a new one from the visibility.
## set the time interval
from suncasa import dspec as ds
import time
visibility_data = 'IDB20170821202020_slfcaled.ms'
specfile = visibility_data + '.dspec.npz'
d = ds.Dspec(visibility_data, bl='4&9', specfile=specfile)

timerange = '19:02:00~19:02:10' ## select frequency range from 2.5 GHz to 3.5 GHz
spw = '2~5' ## select stokes XX
stokes = 'XX' ## turn off AIA image plotting, default is True
plotaia = False
xycen = [375, 45] ## image center for clean in solar X-Y in arcsec
cell=['2.0arcsec'] ## pixel size
imsize=[128] ## x and y image size in pixels.
fov = [100,100] ## field of view of the zoomed-in panels in unit of arcsec
spw = ['{}'.format(s) for s in range(1,31)]
clevels = [0.5, 1.0] ## contour levels to fill in between.
calpha=0.35 ## now tune down the alpha
restoringbeam=['6arcsec']
ql.qlookplot(vis=msfile, specfile=specfile, timerange=timerange, spw=spw, stokes=stokes, \
restoringbeam=restoringbeam,imsize=imsize,cell=cell, \
xycen=xycen,fov=fov,clevels=clevels,calpha=calpha)
</pre>

Page Last Updated: 07/06/2022

Tohban EOVSA Imaging Tutorial A-Z V2

2022-07-06T11:24:21Z

Schhabra:

This tutorial describes the step-by-step procedure to download EOVSA IDB data, obtain and calibrate the measurement sets (.ms), and, transfer them to the inti server for self-calibration and further analysis.

=== Pre-requisites ===
Accounts on Pipeline and Inti or Baozi servers. Sijie and Bin can help with the account set-up for pipeline, inti and boazi.

==== How to connect to pipeline ====
'''Windows:''' On a windows machine Mobaxterm can be used to connect to Pipeline. For details, see ...

'''Linux/Unix:''' On a linux/Mac the ssh commands can be used directly through the terminal:

<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">

ssh -L 5902:helios.solar.pvt:20000 guest@ovsa.njit.edu ### when prompted, enter password to guest@ovsa ###
ssh -XY your_username@pipeline ### add your username and enter password to your pipeline account, when prompted ###
</pre>

Alternatively, to avoid typing the longer commands, proxy commands can be added to the ~/.ssh/config file:

<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
vim ~/.ssh/config ### press 'i' to insert content and esc +':wq' to write and quit ###

Host *
ForwardAgent yes

Host *
ForwardX11 yes

Host ovsa
HostName ovsa.njit.edu
User guest

Host pipeline
Hostname pipeline.solar.pvt
User your_username ### Insert your pipeline username here ###
ProxyCommand ssh -W %h:%p guest@ovsa.njit.edu
</pre>

Upon saving the ~/.ssh/config file and restarting the terminal, you can connect to pipeline directly using:

<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
ssh -XY pipeline
</pre>

==== How to connect to the inti server ====
NOTE: To connect to inti you must have a NJIT UCID. The arcs have granted access to inti for all SRG members. If you encounter any issue with the login. Please let arcs folks (arcs@njit.edu) know.

'''Windows:''' As mentioned above, on a windows machine Mobaxterm can be used to connect to inti.
With your NJIT VPN connected, connect to one of the afsconnect servers (for example, afsaccess3.njit.edu) using your UCID and password. Next, ssh into inti.
<pre style="background-color: #FCEBD9">
ssh -XY UCID@afsaccess3.njit.edu ###Insert UCID password if prompted###

ssh -XY UCID@inti.hpcnet.campus.njit.edu ###Insert UCID/inti username here###
</pre>
Proxy commands for inti can also be added to the C:\Program Files\Git\etc\ssh\ssh_config file:
<pre style="background-color: #FCEBD9">
Host inti
Hostname inti.hpcnet.campus.njit.edu
User UCID ###Insert UCID/inti username here###
</pre>

'''Linux/Unix:''' On a linux/Mac the ssh command can be used:
<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
ssh -XY UCID@inti.hpcnet.campus.njit.edu ### insert UCID here and enter your NJIT UCID password, when prompted ###
</pre>

Proxy commands for inti can also be added to the ~/.ssh/config (on local machine) file for convenience:
<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
vim ~/.ssh/config ### press 'i' to insert content and esc +':wq' to write and quit ###

Host inti
Hostname inti.hpcnet.campus.njit.edu
User UCID ### insert NJIT UCID here ###
</pre>
after the proxy is set up, connect to inti by simply typing:
<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
ssh -XY inti ### enter your NJIT UCID password when prompted ###
</pre>

'''NOTE:'''When logging in for the first time, please add the following lines (demarcated by #) in your ~/.bashrc (on the inti server) file in order to use CASA on inti.

<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
#### setting start ####
if [ $HOSTNAME == "baozi.hpcnet.campus.njit.edu" ]; then
source /srg/.setenv_baozi
fi
if [ $HOSTNAME == "inti.hpcnet.campus.njit.edu" ]; then
source /inti/.setenv_inti
fi
if [ $HOSTNAME == "guko.resource.campus.njit.edu" ]; then
source /data/data/.setenv_guko
fi
#### setting end ####
</pre>

For sufficient disk space to analyze EOVSA data on Inti, use your dedicated directory YOURDIRECTORY at the location given below. If you do not have a directory, please ask Sijie to set one up.
<pre style="background-color: #FCEBD9">
cd /inti/data/users/YOURDIRECTORY
</pre>

'''Both CASA 5 and 6 are available on Inti.'''

Enter the bash environment on inti, and load the desired casa environment.

To load CASA 5:
<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
>> bash
>> loadcasa5
>> suncasa
</pre>

To load CASA 6:
<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
>> bash
>> loadcasa6
>> ipython
</pre>

=== Downloading data and Preliminary Calibration ===

==== Step 1: Downloading raw (IDB) data on Pipeline machine====
The initial downloading of data and applying refcal and phasecal to the .ms files is performed on the pipeline since the complete EOVSA SQL database is stored here for now.

If you are not using mobaxterm, directly SSH to Pipeline through your Linux or Mac computer (as shown above) and '''start tcsh to run CASA'''.

The following commands (run within CASA) can be used to import a single IDB file if the given time-range is less than 10 mins.
<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
from astropy.time import Time
import os
trange = Time(['2017-08-21 20:15:00', '2017-08-21 20:25:00']) ###Change accordingly###

#### (Optional) change output path, default current directory "./" #######
outpath = './msdata/' ###Change accordingly###
if not os.path.exists(outpath):
os.makedirs(outpath)

msfiles = importeovsa(idbfiles=trange, ncpu=1, visprefix=outpath)
</pre>

'''NOTE:''' The above step currently gives an error with importeovsa. This is because the data location was changed and the code doesn't know where to look for the IDB files. Sijie is looking into the problem. We will update the page when it is fixed. For now, please use the method below for all time ranges.

OR (for a time range longer than 10 minutes)

<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
from suncasa.tasks import task_calibeovsa as calibeovsa
from suncasa.tasks import task_importeovsa as timporteovsa
from split_cli import split_cli as split
import dump_tsys as dt
from util import Time
import numpy as np
import os
from glob import glob
from eovsapy import util

trange = Time(['2017-08-21 20:15:00', '2017-08-21 20:35:00']) ###Change accordingly###
idbdir = util.get_idbdir(trange[0])

info = dt.rd_fdb(trange[0])
for k, v in info.iteritems():
info[k] = info[k][~(info[k] == '')]

sidx = np.where(
np.logical_and(info['SOURCEID'] == 'Sun', info['PROJECTID'] == 'NormalObserving') & np.logical_and(
info['ST_TS'].astype(np.float) >= trange[0].lv,
info['ST_TS'].astype(np.float) <= trange[
1].lv))
filelist = info['FILE'][sidx]

outpath = './msdata/' ###Change accordingly###
if not os.path.exists(outpath):
os.makedirs(outpath)
inpath = idbdir + '{}/'.format(trange[0].datetime.strftime("%Y%m%d"))
ncpu = 1

msfiles = timporteovsa.importeovsa(idbfiles=[inpath + ll for ll in filelist], ncpu=ncpu, timebin="0s", width=1,
visprefix=outpath,
nocreatms=False, doconcat=False,
modelms="", doscaling=False, keep_nsclms=False, udb_corr=True)
</pre>

If you come across errors with calibeovsa, add the following lines to your ~/.casa/init.py (within pipeline) file.
<pre style="background-color: #FCEBD9">
vim ~/.ssh/config ### press 'i' to insert content and esc +':wq' to write and quit ###
import sys
sys.path.append('/common/python')
sys.path.append('/common/python/packages/pipeline_casa')
execfile('/common/python/suncasa/tasks/mytasks.py')
</pre>

==== Step 2: Concatenate multiple .ms files and Calibrate====
'''Follow this step if there are more than one .ms files for the chosen time-range. If not, move to Step 3.'''

<pre style="background-color: #FCEBD9">
concatvis = os.path.basename(msfiles[0])[:11] + '_concat_cal.ms' ### Set the path/name for the concatenated files ###

vis = calibeovsa.calibeovsa(msfiles, doconcat=True, concatvis=concatvis, caltype=['refpha','phacal'], doimage=True)
### If doimage=True, a quicklook image will be produced (by integrating over the entire time) as shown below ###

vis_str = str(' '.join(vis))
caled_vis=vis_str.replace('.ms','_cal.ms') ### Append '_cal' to the ms filename ###
split(vis=' '.join(vis),outputvis=caled_vis,datacolumn='corrected',timerange='',correlation='XX') ### Split the corrected column to the new 'caled' ms ###
</pre>

[[file:Figure1_imagingtutorial.png|frame|center|800px|'''Figure 1:''' Quick-look full-Sun image after the initial calibration.]]

==== Step 3: Calibration ====
For a single .ms file from Step 1, this will calibrate the input visibility, write out calibration tables under /data1/eovsa/caltable/, and apply the calibration.
<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
vis = calibeovsa.calibeovsa('IDB20170821202020.ms', caltype=['refpha','phacal'], doimage=True) ###Change the vis filename accordingly###

vis_str = str(' '.join(vis))
caled_vis=vis_str.replace('.ms','_cal.ms') ### Append '_cal' to the ms filename ###
split(vis=' '.join(vis),outputvis=caled_vis,datacolumn='corrected',timerange='',correlation='XX') ### Split the corrected column to the new 'caled' ms ###
</pre>

The created 'caled' ms data files (xxx_concat_cal.ms or xxx_cal.ms) need to be transferred from pipeline to inti server, which has all the casa tasks installed, in order to run the rest of the imaging steps.

=== Transferring data files between servers ===
To directly transfer your calibrated .ms data from the Pipeline to Inti:

<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
1. Log into Inti using your username.
ssh -X USERNAME@inti.hpcnet.campus.njit.edu

Then create a tunnel into Pipeline from Inti.
ssh -L 8888:pipeline.solar.pvt:22 guest@ovsa.njit.edu

2. Log into Inti again from a new terminal. Change to your working directory (where you need the data transferred) and give the following command:

scp -v -C -r -P 8888 PipelineUSERNAME@localhost:PATH/TO/MSFILE ./

</pre>

===Advanced Calibration and Imaging===
====Step 4: Self-calibration====
[[file:Figure3_imagingtutorial.png|thumb|center|500px|Figure 2: Cotton-Schwab clean major and minor cycles. [Source: http://www.aoc.nrao.edu/~rurvashi/ImagingAlgorithmsInCasa/node2.html].]]
Follow the below given steps to run the self-calibration of the imaging data and produce the calibrated images in .fits format. https://github.com/binchensun/casa-eovsa/blob/master/slfcal_example.py

<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
bash
loadcasa5
suncasa

from suncasa.utils import helioimage2fits as hf
import os
import numpy as np
import pickle
from matplotlib import gridspec as gridspec
from sunpy import map as smap
from matplotlib import pyplot as plt
from split_cli import split_cli as split
import time

# =========== task handlers =============
dofullsun = 1 # initial full-sun imaging ###Change accordingly###
domasks=1 # get masks ###Change accordingly###
doslfcal=1 # main cycle of doing selfcalibration ###Change accordingly###
doapply=1 # apply the results ###Change accordingly###
doclean_slfcaled=1 # perform clean for self-calibrated data ###Change accordingly###

# ============ declaring the working directories ============
workdir = os.getcwd()+'/' #main working directory. Using current directory in this example
slfcaldir = workdir+'slfcal/' #place to put all selfcalibration products
imagedir = slfcaldir+'images/' #place to put all selfcalibration images
maskdir = slfcaldir+'masks/' #place to put clean masks
imagedir_slfcaled = slfcaldir+'images_slfcaled/' #place to put final self-calibrated images
caltbdir = slfcaldir+'caltbs/' # place to put calibration tables
# make these directories if they do not already exist
dirs = [workdir, slfcaldir, imagedir, maskdir, imagedir_slfcaled, caltbdir]
for d in dirs:
if not os.path.exists(d):
os.makedirs(d)

# ============ Split a short time for self-calibration ===========
# input visibility
ms_in = workdir + 'IDB20170821202020_cal.ms' ###Change the initial calibrated (through calibeovsa) vis accordingly###
# output, selfcaled, visibility
ms_slfcaled = workdir + os.path.basename(ms_in).replace('cal','slfcaled')
# intermediate small visibility for selfcalbration
# selected time range for generating self-calibration solutions
trange='2017/08/21/20:21:10~2017/08/21/20:21:30' ###Change accordingly###
slfcalms = slfcaldir+'slfcalms.XX.slfcal'
slfcaledms = slfcaldir+'slfcalms.XX.slfcaled'
if not os.path.exists(slfcalms):
split(vis=ms_in,outputvis=slfcalms,datacolumn='data',timerange=trange,correlation='XX')

# ============ Prior definitions for spectral windows, antennas, pixel numbers =========
spws=[str(s+1) for s in range(30)] # spws=[str(s+1) for s in range(49)] # For post-2020 data
antennas='0~12&0~12'
npix=512
nround=3 #number of slfcal cycles

# =========== Step 1, doing a full-Sun image to find out phasecenter and appropriate field of view =========
if dofullsun:
#initial mfs clean to find out the image phase center
im_init='fullsun_init'
os.system('rm -rf '+im_init+'*')
tclean(vis=slfcalms,
antenna=antennas,
imagename=im_init,
spw='2~15',
specmode='mfs',
timerange=trange,
imsize=[npix],
cell=['5arcsec'],
niter=1000,
gain=0.05,
stokes='I',
restoringbeam=['30arcsec'],
interactive=False,
pbcor=True)

hf.imreg(vis=slfcalms,imagefile=im_init+'.image.pbcor',fitsfile=im_init+'.fits',
timerange=trange,usephacenter=False,verbose=True)
clnjunks = ['.flux', '.mask', '.model', '.psf', '.residual','.sumwt','.pb','.image'] #Do not run the next 4 lines if needed to view and assess the subset of clean process images
for clnjunk in clnjunks:
if os.path.exists(im_init + clnjunk):
os.system('rm -rf '+im_init + clnjunk)

from sunpy import map as smap
from matplotlib import pyplot as plt
fig = plt.figure(figsize=(6,6))
ax = fig.add_subplot(111)
eomap=smap.Map(im_init+'.fits')
#eomap.data=eomap.data.reshape((npix,npix))
eomap.plot_settings['cmap'] = plt.get_cmap('jet')
eomap.plot(axes = ax)
eomap.draw_limb()
plt.show()
viewer(im_init+'.image.pbcor')
</pre>
[[file:Figure2_imagingtutorial.png|thumb|center|500px|Figure 3: Full-Sun image after initial clean to find the flare location.]]

<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
# parameters specific to the event (found from step 1)
phasecenter='J2000 10h02m59 11d58m07' ###Change accordingly###
xran=[280,480] ###Change accordingly###
yran=[-50,150] ###Change accordingly###

# =========== Step 2 (optional), generate masks =========
# if skipped, will not use any masks
if domasks:
clearcal(slfcalms)
delmod(slfcalms)
antennas=antennas
pol='XX'
imgprefix=maskdir+'slf_t0'

# step 1: set up the clean masks
img_init=imgprefix+'_init_ar_'
os.system('rm -rf '+img_init+'*')
#spwrans_mask=['1~5','6~12','13~20','21~30']
spwrans_mask=['2~12']
#convert to a list of spws
spwrans_mask_list = [[str(i) for i in (np.arange(int(m.split('~')[0]),int(m.split('~')[1])))] for m in spwrans_mask] # Not used?
masks=[]
imnames=[]
for spwran in spwrans_mask:
imname=img_init+spwran.replace('~','-')
try:
tclean(vis=slfcalms,
antenna=antennas,
imagename=imname,
spw=spwran,
specmode='mfs',
timerange=trange,
imsize=[npix],
cell=['2arcsec'],
niter=1000,
gain=0.05,
stokes='XX',
restoringbeam=['20arcsec'],
phasecenter=phasecenter,
weighting='briggs',
robust=1.0,
interactive=True,
datacolumn='data',
pbcor=True,
savemodel='modelcolumn')
imnames.append(imname+'.image')
masks.append(imname+'.mask')
clnjunks = ['.flux', '.model', '.psf', '.residual'] #Do not run the next 4 lines if needed to view and assess the subset of clean process images
for clnjunk in clnjunks:
if os.path.exists(imname + clnjunk):
os.system('rm -rf '+ imname + clnjunk)
except:
print('error in cleaning spw: '+spwran)

pickle.dump(masks,open(slfcaldir+'masks.p','wb'))
</pre>
[[file:Figure4_imagingtutorial.PNG|thumb|center|800px|Figure 4: Interactive clean window to create masks over the source. The white outline surrounding the source is the mask selected by the polygon drawing option.]]

The outlines drawn for masks can be created by any of the icons with the letter 'R' in the Viewer window. The instructions for doing this can be found by hovering over those icons.

<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
# =========== Step 3, main step of selfcalibration =========
if doslfcal:
if os.path.exists(slfcaldir+'masks.p'):
masks=pickle.load(open(slfcaldir+'masks.p','rb'))
if not os.path.exists(slfcaldir+'masks.p'):
print 'masks do not exist. Use default mask'
masks=[]
os.system('rm -rf '+imagedir+'*')
os.system('rm -rf '+caltbdir+'*')
#first step: make a mock caltable for the entire database
print('Processing ' + trange)
slftbs=[]
calprefix=caltbdir+'slf'
imgprefix=imagedir+'slf'
tb.open(slfcalms+'/SPECTRAL_WINDOW')
reffreqs=tb.getcol('REF_FREQUENCY')
bdwds=tb.getcol('TOTAL_BANDWIDTH')
cfreqs=reffreqs+bdwds/2.
tb.close()
# starting beam size at 3.4 GHz in arcsec #Change accordingly
sbeam=40.
strtmp=[m.replace(':','') for m in trange.split('~')]
timestr='t'+strtmp[0]+'-'+strtmp[1]
refantenna='0'
# number of iterations for each round
niters=[100, 300, 500]
# roburst value for weighting the baselines
robusts=[1.0, 0.5, 0.0]
# apply calibration tables? Set to true for most cases
doapplycal=[1,1,1]
# modes for calibration, 'p' for phase-only, 'a' for amplitude only, 'ap' for both
calmodes=['p','p','a']
# setting uvranges for model image (optional, not used here)
uvranges=['','','']
for n in range(nround):
slfcal_tb_g= calprefix+'.G'+str(n)
fig = plt.figure(figsize=(8.4,7.)) # fig = plt.figure(figsize=(14, 7)) # For post-2020 data (50 spws)
gs = gridspec.GridSpec(5, 6) # gs = gridspec.GridSpec(5, 10) # For post-2020 data (50 spws)
for s,sp in enumerate(spws):
print 'processing spw: '+sp
cfreq=cfreqs[int(sp)]
# setting restoring beam size (not very useful for selfcal anyway, but just to see the results)
bm=max(sbeam*cfreqs[1]/cfreq, 6.)
slfcal_img = imgprefix+'.spw'+sp.zfill(2)+'.slfcal'+str(n)
# only the first round uses nearby spws for getting initial model
if n == 0:
spbg=max(int(sp)-2,1) # spbg=max(int(sp)-2,0) # For post-2020 data (50 spws)
sped=min(int(sp)+2,30) # sped=min(int(sp)+2,49) # For post-2020 data (50 spws)
spwran=str(spbg)+'~'+str(sped)
print('using spw {0:s} as model'.format(spwran))
if 'spwrans_mask' in vars():
for m, spwran_mask in enumerate(spwrans_mask):
if sp in spwran_mask:
mask = masks[m]
print('using mask {0:s}'.format(mask))
findmask = True
if not findmask:
print('mask not found. Do use any masks')
else:
spwran = sp
if 'spwrans_mask' in vars():
for m, spwran_mask in enumerate(spwrans_mask):
if sp in spwran_mask:
mask = masks[m]
print 'using mask {0:s}'.format(mask)
findmask = True
if not findmask:
print('mask not found. Do use any masks')
try:
tclean(vis=slfcalms,
antenna=antennas,
imagename=slfcal_img,
uvrange=uvranges[n],
spw=spwran,
specmode='mfs',
timerange=trange,
imsize=[npix],
cell=['2arcsec'],
niter=niters[n],
gain=0.05,
stokes='XX', #use pol XX image as the model
weighting='briggs',
robust=robusts[n],
phasecenter=phasecenter,
mask=mask,
restoringbeam=[str(bm)+'arcsec'],
pbcor=False,
interactive=False,
savemodel='modelcolumn')
if os.path.exists(slfcal_img+'.image'):
fitsfile=slfcal_img+'.fits'
hf.imreg(vis=slfcalms,imagefile=slfcal_img+'.image',fitsfile=fitsfile,
timerange=trange,usephacenter=False,toTb=True,verbose=False,overwrite=True)
clnjunks = ['.mask','.flux', '.model', '.psf', '.residual', '.image','.pb','.image.pbcor','.sumwt'] #Do not run the next 4 lines if needed to view and assess the subset of clean process images
for clnjunk in clnjunks:
if os.path.exists(slfcal_img + clnjunk):
os.system('rm -rf '+ slfcal_img + clnjunk)
ax = fig.add_subplot(gs[s])
eomap=smap.Map(fitsfile)
eomap.plot_settings['cmap'] = plt.get_cmap('jet')
eomap.plot(axes = ax)
eomap.draw_limb()
#eomap.draw_grid()
ax.set_title(' ')
ax.get_xaxis().set_visible(False)
ax.get_yaxis().set_visible(False)
ax.set_xlim(xran)
ax.set_ylim(yran)
plt.pause(0.5) # Allows viewing of each image as it is plotted.
os.system('rm -f '+ fitsfile)

except:
print 'error in cleaning spw: '+sp
print 'using nearby spws for initial model'
sp_e=int(sp)+2
sp_i=int(sp)-2
if sp_i < 1: # if sp < 0: # For post-2020 data (50 spws)
sp_i = 1 # sp_i = 0 # For post-2020 data (50 spws)
if sp_e > 30: # if sp_e > 49: # For post-2020 data (50 spws)
sp_e = 30 # sp_e = 49 # For post-2020 data (50 spws)
sp_=str(sp_i)+'~'+str(sp_e)
try:
tget(tclean)
spw=sp_
print('using spw {0:s} as model'.format(sp_))
tclean()
except:
print 'still not successful. abort...'
break

gaincal(vis=slfcalms, refant=refantenna,antenna=antennas,caltable=slfcal_tb_g,spw=sp, uvrange='',\
gaintable=[],selectdata=True,timerange=trange,solint='inf',gaintype='G',calmode=calmodes[n],\
combine='',minblperant=4,minsnr=2,append=True)
if not os.path.exists(slfcal_tb_g):
print 'No solution found in spw: '+sp
figname=imagedir+'slf_t0_n{:d}.png'.format(n)
plt.subplots_adjust(left=0, bottom=0, right=1, top=1, wspace=0, hspace=0)
plt.savefig(figname)
time.sleep(10)
plt.close()

if os.path.exists(slfcal_tb_g):
slftbs.append(slfcal_tb_g)
slftb=[slfcal_tb_g]
os.chdir(slfcaldir)
if calmodes[n] == 'p':
plotcal(caltable=slfcal_tb_g,antenna='1~12',xaxis='freq',yaxis='phase',\
subplot=431,plotrange=[-1,-1,-180,180],iteration='antenna',figfile=slfcal_tb_g+'.png',showgui=False)
if calmodes[n] == 'a':
plotcal(caltable=slfcal_tb_g,antenna='1~12',xaxis='freq',yaxis='amp',\
subplot=431,plotrange=[-1,-1,0,2.],iteration='antenna',figfile=slfcal_tb_g+'.png',showgui=False)
os.chdir(workdir)
plt.pause(0.5) # Allows viewing of the plot

if doapplycal[n]:
clearcal(slfcalms)
delmod(slfcalms)
applycal(vis=slfcalms,gaintable=slftb,spw=','.join(spws),selectdata=True,\
antenna=antennas,interp='nearest',flagbackup=False,applymode='calonly',calwt=False)

if n < nround-1:
prompt=raw_input('Continuing to selfcal?')
#prompt='y'
if prompt.lower() == 'n':
if os.path.exists(slfcaledms):
os.system('rm -rf '+slfcaledms)
split(slfcalms,slfcaledms,datacolumn='corrected')
print 'Final calibrated ms is {0:s}'.format(slfcaledms)
break
if prompt.lower() == 'y':
slfcalms_=slfcalms+str(n)
if os.path.exists(slfcalms_):
os.system('rm -rf '+slfcalms_)
split(slfcalms,slfcalms_,datacolumn='corrected')
slfcalms=slfcalms_
else:
if os.path.exists(slfcaledms):
os.system('rm -rf '+slfcaledms)
split(slfcalms,slfcaledms,datacolumn='corrected')
print 'Final calibrated ms is {0:s}'.format(slfcaledms)
</pre>

<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
# =========== Step 4: Apply self-calibration tables =========
if doapply:
import glob
os.chdir(workdir)
clearcal(ms_in)
clearcal(slfcalms)
applycal(vis=slfcalms,gaintable=slftbs,spw=','.join(spws),selectdata=True,\
antenna=antennas,interp='linear',flagbackup=False,applymode='calonly',calwt=False)
applycal(vis=ms_in,gaintable=slftbs,spw=','.join(spws),selectdata=True,\
antenna=antennas,interp='linear',flagbackup=False,applymode='calonly',calwt=False)
if os.path.exists(ms_slfcaled):
os.system('rm -rf '+ms_slfcaled)
split(ms_in, ms_slfcaled,datacolumn='corrected')
</pre>
[[file:Slf.G0.png|thumb|center|500px|Figure 1: Phase before self-calibration]]
[[file:Slf.G1.png|thumb|center|500px|Figure 2: Phase after self-calibration]]

<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
# =========== Step 5: Generate final self-calibrated images (optional) =========
if doclean_slfcaled:
import glob
pol='XX'
print('Processing ' + trange)
img_final=imagedir_slfcaled+'/slf_final_{0}_t0'.format(pol)
vis = ms_slfcaled
spws=[str(s+1) for s in range(30)]
tb.open(vis+'/SPECTRAL_WINDOW')
reffreqs=tb.getcol('REF_FREQUENCY')
bdwds=tb.getcol('TOTAL_BANDWIDTH')
cfreqs=reffreqs+bdwds/2.
tb.close()
sbeam=30.
from matplotlib import gridspec as gridspec
from sunpy import map as smap
from matplotlib import pyplot as plt
fitsfiles=[]
for s,sp in enumerate(spws):
cfreq=cfreqs[int(sp)]
bm=max(sbeam*cfreqs[1]/cfreq,4.)
imname=img_final+'_s'+sp.zfill(2)
fitsfile=imname+'.fits'
if not os.path.exists(fitsfile):
print 'cleaning spw {0:s} with beam size {1:.1f}"'.format(sp,bm)
try:
tclean(vis=vis,
antenna=antennas,
imagename=imname,
spw=sp,
specmode='mfs',
timerange=trange,
imsize=[npix],
cell=['1arcsec'],
niter=1000,
gain=0.05,
stokes=pol,
weighting='briggs',
robust=2.0,
restoringbeam=[str(bm)+'arcsec'],
phasecenter=phasecenter,
mask='',
pbcor=True,
interactive=False)
except:
print 'cleaning spw '+sp+' unsuccessful. Proceed to next spw'
continue
if os.path.exists(imname+'.image.pbcor'):
imn = imname+'.image.pbcor'
hf.imreg(vis=vis,imagefile=imn,fitsfile=fitsfile,
timerange=trange,usephacenter=False,toTb=True,verbose=False)
fitsfiles.append(fitsfile)
junks=['.flux','.model','.psf','.residual','.mask','.image','.pb','.image.pbcor','.sumwt'] #Do not run the next 4 lines if needed to view and assess the subset of clean process images
for junk in junks:
if os.path.exists(imname+junk):
os.system('rm -rf '+imname+junk)
else:
print('fits file '+fitsfile+' already exists, skip clean...')
fitsfiles.append(fitsfile)

fig = plt.figure(figsize=(8.4,7.)) # fig = plt.figure(figsize=(14, 7)) # For post-2020 data (50 spws)
gs = gridspec.GridSpec(5, 6) # gs = gridspec.GridSpec(5, 10) # For post-2020 data (50 spws)
for s,sp in enumerate(spws):
cfreq=cfreqs[int(sp)]
ax = fig.add_subplot(gs[s])
eomap=smap.Map(fitsfiles[s])
eomap.plot_settings['cmap'] = plt.get_cmap('jet')
eomap.plot(axes = ax)
eomap.draw_limb()
ax.set_title(' ')
ax.get_xaxis().set_visible(False)
ax.get_yaxis().set_visible(False)
ax.set_xlim(xran)
ax.set_ylim(yran)
plt.text(0.98,0.85,'{0:.1f} GHz'.format(cfreq/1e9),transform=ax.transAxes,ha='right',color='w',fontweight='bold')
plt.subplots_adjust(left=0, bottom=0, right=1, top=1, wspace=0, hspace=0)
plt.show()

</pre>
The .fits files of the self calibrated images at each frequency for the given time are saved at /slfcal/images_slfcaled in your working directory.

[[file:Slf_t0_n0.png|thumb|center|500px|Figure 3: Multi-frequency images before self-calibration]]
[[file:Slf_t0_n1.png|thumb|center|500px|Figure 4: Multi-frequency images after self-calibration]]

====Step 5: Quick-look imaging ==== 
For spectral imaging analysis of the event, follow [http://www.ovsa.njit.edu/wiki/index.php/EOVSA_Data_Analysis_Tutorial#Spectral_Imaging_with_SunCASA this tutorial] using the self-calibrated data obtained from the previous step or use [https://colab.research.google.com/drive/1lSLLxgOG6b8kgu9Sk6kSKvrViyubnXG6?usp=sharing#scrollTo=xbXyyLmCFCGL this link].

<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
from suncasa.utils import qlookplot as ql ## (Optional) Supply the npz file of the dynamic spectrum from previous step.
## If not provided, the program will generate a new one from the visibility.
## set the time interval
from suncasa import dspec as ds
import time
visibility_data = 'IDB20170821202020_slfcaled.ms'
specfile = visibility_data + '.dspec.npz'
d = ds.Dspec(visibility_data, bl='4&9', specfile=specfile)

timerange = '19:02:00~19:02:10' ## select frequency range from 2.5 GHz to 3.5 GHz
spw = '2~5' ## select stokes XX
stokes = 'XX' ## turn off AIA image plotting, default is True
plotaia = False
xycen = [375, 45] ## image center for clean in solar X-Y in arcsec
cell=['2.0arcsec'] ## pixel size
imsize=[128] ## x and y image size in pixels.
fov = [100,100] ## field of view of the zoomed-in panels in unit of arcsec
spw = ['{}'.format(s) for s in range(1,31)]
clevels = [0.5, 1.0] ## contour levels to fill in between.
calpha=0.35 ## now tune down the alpha
restoringbeam=['6arcsec']
ql.qlookplot(vis=msfile, specfile=specfile, timerange=timerange, spw=spw, stokes=stokes, \
restoringbeam=restoringbeam,imsize=imsize,cell=cell, \
xycen=xycen,fov=fov,clevels=clevels,calpha=calpha)
</pre>

Page Last Updated: 07/06/2022

Tohban EOVSA Imaging Tutorial A-Z V2

2022-07-06T11:21:37Z

Schhabra:

This tutorial describes the step-by-step procedure to download EOVSA IDB data, obtain and calibrate the measurement sets (.ms), and, transfer them to the inti server for self-calibration and further analysis.

=== Pre-requisites ===
Accounts on Pipeline and Inti or Baozi servers. Sijie and Bin can help with the account set-up for pipeline, inti and boazi.

==== How to connect to pipeline ====
'''Windows:''' On a windows machine Mobaxterm can be used to connect to Pipeline. For details, see ...

'''Linux/Unix:''' On a linux/Mac the ssh commands can be used directly through the terminal:

<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">

ssh -L 5902:helios.solar.pvt:20000 guest@ovsa.njit.edu ### when prompted, enter password to guest@ovsa ###
ssh -XY your_username@pipeline ### add your username and enter password to your pipeline account, when prompted ###
</pre>

Alternatively, to avoid typing the longer commands, proxy commands can be added to the ~/.ssh/config file:

<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
vim ~/.ssh/config ### press 'i' to insert content and esc +':wq' to write and quit ###

Host *
ForwardAgent yes

Host *
ForwardX11 yes

Host ovsa
HostName ovsa.njit.edu
User guest

Host pipeline
Hostname pipeline.solar.pvt
User your_username ### Insert your pipeline username here ###
ProxyCommand ssh -W %h:%p guest@ovsa.njit.edu
</pre>

Upon saving the ~/.ssh/config file and restarting the terminal, you can connect to pipeline directly using:

<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
ssh -XY pipeline
</pre>

==== How to connect to the inti server ====
NOTE: To connect to inti you must have a NJIT UCID. The arcs have granted access to inti for all SRG members. If you encounter any issue with the login. Please let arcs folks (arcs@njit.edu) know.

'''Windows:''' As mentioned above, on a windows machine Mobaxterm can be used to connect to inti.
With your NJIT VPN connected, connect to one of the afsconnect servers (for example, afsaccess3.njit.edu) using your UCID and password. Next, ssh into inti.
<pre style="background-color: #FCEBD9">
ssh -XY UCID@afsaccess3.njit.edu ###Insert UCID password if prompted###

ssh -XY UCID@inti.hpcnet.campus.njit.edu ###Insert UCID/inti username here###
</pre>
Proxy commands for inti can also be added to the C:\Program Files\Git\etc\ssh\ssh_config file:
<pre style="background-color: #FCEBD9">
Host inti
Hostname inti.hpcnet.campus.njit.edu
User UCID ###Insert UCID/inti username here###
</pre>

'''Linux/Unix:''' On a linux/Mac the ssh command can be used:
<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
ssh -XY UCID@inti.hpcnet.campus.njit.edu ### insert UCID here and enter your NJIT UCID password, when prompted ###
</pre>

Proxy commands for inti can also be added to the ~/.ssh/config (on local machine) file for convenience:
<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
vim ~/.ssh/config ### press 'i' to insert content and esc +':wq' to write and quit ###

Host inti
Hostname inti.hpcnet.campus.njit.edu
User UCID ### insert NJIT UCID here ###
</pre>
after the proxy is set up, connect to inti by simply typing:
<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
ssh -XY inti ### enter your NJIT UCID password when prompted ###
</pre>

'''NOTE:'''When logging in for the first time, please add the following lines (demarcated by #) in your ~/.bashrc (on the inti server) file in order to use CASA on inti.

<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
#### setting start ####
if [ $HOSTNAME == "baozi.hpcnet.campus.njit.edu" ]; then
source /srg/.setenv_baozi
fi
if [ $HOSTNAME == "inti.hpcnet.campus.njit.edu" ]; then
source /inti/.setenv_inti
fi
if [ $HOSTNAME == "guko.resource.campus.njit.edu" ]; then
source /data/data/.setenv_guko
fi
#### setting end ####
</pre>

For sufficient disk space to analyze EOVSA data on Inti, use your dedicated directory YOURDIRECTORY at the location given below. If you do not have a directory, please ask Sijie to set one up.
<pre style="background-color: #FCEBD9">
cd /inti/data/users/YOURDIRECTORY
</pre>

'''Both CASA 5 and 6 are available on Inti.'''

Enter the bash environment on inti, and load the desired casa environment.

To load CASA 5:
<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
>> bash
>> loadcasa5
>> suncasa
</pre>

To load CASA 6:
<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
>> bash
>> loadcasa6
>> ipython
</pre>

=== Downloading data and Preliminary Calibration ===

==== Step 1: Downloading raw (IDB) data on Pipeline machine====
The initial downloading of data and applying refcal and phasecal to the .ms files is performed on the pipeline since the complete EOVSA SQL database is stored here for now.

If you are not using mobaxterm, directly SSH to Pipeline through your Linux or Mac computer (as shown above) and '''start tcsh to run CASA'''.

The following commands (run within CASA) can be used to import a single IDB file if the given time-range is less than 10 mins.
<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
from astropy.time import Time
import os
trange = Time(['2017-08-21 20:15:00', '2017-08-21 20:25:00']) ###Change accordingly###

#### (Optional) change output path, default current directory "./" #######
outpath = './msdata/' ###Change accordingly###
if not os.path.exists(outpath):
os.makedirs(outpath)

msfiles = importeovsa(idbfiles=trange, ncpu=1, visprefix=outpath)
</pre>

'''NOTE:''' The above step currently gives an error with importeovsa. This is because the data location was changed and the code doesn't know where to look for the IDB files. Sijie is looking into the problem. We will update the page when it is fixed. For now, please use the method below for all time ranges.

OR (for a time range longer than 10 minutes)

<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
from suncasa.tasks import task_calibeovsa as calibeovsa
from suncasa.tasks import task_importeovsa as timporteovsa
from split_cli import split_cli as split
import dump_tsys as dt
from util import Time
import numpy as np
import os
from glob import glob
from eovsapy import util

trange = Time(['2017-08-21 20:15:00', '2017-08-21 20:35:00']) ###Change accordingly###
idbdir = util.get_idbdir(trange[0])

info = dt.rd_fdb(trange[0])
for k, v in info.iteritems():
info[k] = info[k][~(info[k] == '')]

sidx = np.where(
np.logical_and(info['SOURCEID'] == 'Sun', info['PROJECTID'] == 'NormalObserving') & np.logical_and(
info['ST_TS'].astype(np.float) >= trange[0].lv,
info['ST_TS'].astype(np.float) <= trange[
1].lv))
filelist = info['FILE'][sidx]

outpath = './msdata/' ###Change accordingly###
if not os.path.exists(outpath):
os.makedirs(outpath)
inpath = idbdir + '{}/'.format(trange[0].datetime.strftime("%Y%m%d"))
ncpu = 1

msfiles = timporteovsa.importeovsa(idbfiles=[inpath + ll for ll in filelist], ncpu=ncpu, timebin="0s", width=1,
visprefix=outpath,
nocreatms=False, doconcat=False,
modelms="", doscaling=False, keep_nsclms=False, udb_corr=True)
</pre>

If you come across errors with calibeovsa, add the following lines to your ~/.casa/init.py (within pipeline) file.
<pre style="background-color: #FCEBD9">
vim ~/.ssh/config ### press 'i' to insert content and esc +':wq' to write and quit ###
import sys
sys.path.append('/common/python')
sys.path.append('/common/python/packages/pipeline_casa')
execfile('/common/python/suncasa/tasks/mytasks.py')
</pre>

==== Step 2: Concatenate multiple .ms files and Calibrate====
'''Follow this step if there are more than one .ms files for the chosen time-range. If not, move to Step 3.'''

<pre style="background-color: #FCEBD9">
concatvis = os.path.basename(msfiles[0])[:11] + '_concat_cal.ms' ### Set the path/name for the concatenated files ###

vis = calibeovsa.calibeovsa(msfiles, doconcat=True, concatvis=concatvis, caltype=['refpha','phacal'], doimage=True)
### If doimage=True, a quicklook image will be produced (by integrating over the entire time) as shown below ###

vis_str = str(' '.join(vis))
caled_vis=vis_str.replace('.ms','_cal.ms') ### Append '_cal' to the ms filename ###
split(vis=' '.join(vis),outputvis=caled_vis,datacolumn='corrected',timerange='',correlation='XX') ### Split the corrected column to the new 'caled' ms ###
</pre>

[[file:Figure1_imagingtutorial.png|frame|center|800px|'''Figure 1:''' Quick-look full-Sun image after the initial calibration.]]

==== Step 3: Calibration ====
For a single .ms file from Step 1, this will calibrate the input visibility, write out calibration tables under /data1/eovsa/caltable/, and apply the calibration.
<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
vis = calibeovsa.calibeovsa('IDB20170821202020.ms', caltype=['refpha','phacal'], doimage=True) ###Change the vis filename accordingly###

vis_str = str(' '.join(vis))
caled_vis=vis_str.replace('.ms','_cal.ms') ### Append '_cal' to the ms filename ###
split(vis=' '.join(vis),outputvis=caled_vis,datacolumn='corrected',timerange='',correlation='XX') ### Split the corrected column to the new 'caled' ms ###
</pre>

The created 'caled' ms data files (xxx_concat_cal.ms or xxx_cal.ms) need to be transferred from pipeline to inti server, which has all the casa tasks installed, in order to run the rest of the imaging steps.

=== Transferring data files between servers ===
To directly transfer your calibrated .ms data from the Pipeline to Inti:

<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
1. Log into Inti using your username.
ssh -X USERNAME@inti.hpcnet.campus.njit.edu

Then create a tunnel into Pipeline from Inti.
ssh -L 8888:pipeline.solar.pvt:22 guest@ovsa.njit.edu

2. Log into Inti again from a new terminal. Change to your working directory (where you need the data transferred) and give the following command:

scp -v -C -r -P 8888 PipelineUSERNAME@localhost:PATH/TO/MSFILE ./

</pre>

===Advanced Calibration and Imaging===
====Step 4: Self-calibration====
[[file:Figure3_imagingtutorial.png|thumb|center|500px|Figure 2: Cotton-Schwab clean major and minor cycles. [Source: http://www.aoc.nrao.edu/~rurvashi/ImagingAlgorithmsInCasa/node2.html].]]
Follow the below given steps to run the self-calibration of the imaging data and produce the calibrated images in .fits format. https://github.com/binchensun/casa-eovsa/blob/master/slfcal_example.py

<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
from suncasa.utils import helioimage2fits as hf
import os
import numpy as np
import pickle
from matplotlib import gridspec as gridspec
from sunpy import map as smap
from matplotlib import pyplot as plt
from split_cli import split_cli as split
import time

# =========== task handlers =============
dofullsun = 1 # initial full-sun imaging ###Change accordingly###
domasks=1 # get masks ###Change accordingly###
doslfcal=1 # main cycle of doing selfcalibration ###Change accordingly###
doapply=1 # apply the results ###Change accordingly###
doclean_slfcaled=1 # perform clean for self-calibrated data ###Change accordingly###

# ============ declaring the working directories ============
workdir = os.getcwd()+'/' #main working directory. Using current directory in this example
slfcaldir = workdir+'slfcal/' #place to put all selfcalibration products
imagedir = slfcaldir+'images/' #place to put all selfcalibration images
maskdir = slfcaldir+'masks/' #place to put clean masks
imagedir_slfcaled = slfcaldir+'images_slfcaled/' #place to put final self-calibrated images
caltbdir = slfcaldir+'caltbs/' # place to put calibration tables
# make these directories if they do not already exist
dirs = [workdir, slfcaldir, imagedir, maskdir, imagedir_slfcaled, caltbdir]
for d in dirs:
if not os.path.exists(d):
os.makedirs(d)

# ============ Split a short time for self-calibration ===========
# input visibility
ms_in = workdir + 'IDB20170821202020_cal.ms' ###Change the initial calibrated (through calibeovsa) vis accordingly###
# output, selfcaled, visibility
ms_slfcaled = workdir + os.path.basename(ms_in).replace('cal','slfcaled')
# intermediate small visibility for selfcalbration
# selected time range for generating self-calibration solutions
trange='2017/08/21/20:21:10~2017/08/21/20:21:30' ###Change accordingly###
slfcalms = slfcaldir+'slfcalms.XX.slfcal'
slfcaledms = slfcaldir+'slfcalms.XX.slfcaled'
if not os.path.exists(slfcalms):
split(vis=ms_in,outputvis=slfcalms,datacolumn='data',timerange=trange,correlation='XX')

# ============ Prior definitions for spectral windows, antennas, pixel numbers =========
spws=[str(s+1) for s in range(30)] # spws=[str(s+1) for s in range(49)] # For post-2020 data
antennas='0~12&0~12'
npix=512
nround=3 #number of slfcal cycles

# =========== Step 1, doing a full-Sun image to find out phasecenter and appropriate field of view =========
if dofullsun:
#initial mfs clean to find out the image phase center
im_init='fullsun_init'
os.system('rm -rf '+im_init+'*')
tclean(vis=slfcalms,
antenna=antennas,
imagename=im_init,
spw='2~15',
specmode='mfs',
timerange=trange,
imsize=[npix],
cell=['5arcsec'],
niter=1000,
gain=0.05,
stokes='I',
restoringbeam=['30arcsec'],
interactive=False,
pbcor=True)

hf.imreg(vis=slfcalms,imagefile=im_init+'.image.pbcor',fitsfile=im_init+'.fits',
timerange=trange,usephacenter=False,verbose=True)
clnjunks = ['.flux', '.mask', '.model', '.psf', '.residual','.sumwt','.pb','.image'] #Do not run the next 4 lines if needed to view and assess the subset of clean process images
for clnjunk in clnjunks:
if os.path.exists(im_init + clnjunk):
os.system('rm -rf '+im_init + clnjunk)

from sunpy import map as smap
from matplotlib import pyplot as plt
fig = plt.figure(figsize=(6,6))
ax = fig.add_subplot(111)
eomap=smap.Map(im_init+'.fits')
#eomap.data=eomap.data.reshape((npix,npix))
eomap.plot_settings['cmap'] = plt.get_cmap('jet')
eomap.plot(axes = ax)
eomap.draw_limb()
plt.show()
viewer(im_init+'.image.pbcor')
</pre>
[[file:Figure2_imagingtutorial.png|thumb|center|500px|Figure 3: Full-Sun image after initial clean to find the flare location.]]

<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
# parameters specific to the event (found from step 1)
phasecenter='J2000 10h02m59 11d58m07' ###Change accordingly###
xran=[280,480] ###Change accordingly###
yran=[-50,150] ###Change accordingly###

# =========== Step 2 (optional), generate masks =========
# if skipped, will not use any masks
if domasks:
clearcal(slfcalms)
delmod(slfcalms)
antennas=antennas
pol='XX'
imgprefix=maskdir+'slf_t0'

# step 1: set up the clean masks
img_init=imgprefix+'_init_ar_'
os.system('rm -rf '+img_init+'*')
#spwrans_mask=['1~5','6~12','13~20','21~30']
spwrans_mask=['2~12']
#convert to a list of spws
spwrans_mask_list = [[str(i) for i in (np.arange(int(m.split('~')[0]),int(m.split('~')[1])))] for m in spwrans_mask] # Not used?
masks=[]
imnames=[]
for spwran in spwrans_mask:
imname=img_init+spwran.replace('~','-')
try:
tclean(vis=slfcalms,
antenna=antennas,
imagename=imname,
spw=spwran,
specmode='mfs',
timerange=trange,
imsize=[npix],
cell=['2arcsec'],
niter=1000,
gain=0.05,
stokes='XX',
restoringbeam=['20arcsec'],
phasecenter=phasecenter,
weighting='briggs',
robust=1.0,
interactive=True,
datacolumn='data',
pbcor=True,
savemodel='modelcolumn')
imnames.append(imname+'.image')
masks.append(imname+'.mask')
clnjunks = ['.flux', '.model', '.psf', '.residual'] #Do not run the next 4 lines if needed to view and assess the subset of clean process images
for clnjunk in clnjunks:
if os.path.exists(imname + clnjunk):
os.system('rm -rf '+ imname + clnjunk)
except:
print('error in cleaning spw: '+spwran)

pickle.dump(masks,open(slfcaldir+'masks.p','wb'))
</pre>
[[file:Figure4_imagingtutorial.PNG|thumb|center|800px|Figure 4: Interactive clean window to create masks over the source. The white outline surrounding the source is the mask selected by the polygon drawing option.]]

The outlines drawn for masks can be created by any of the icons with the letter 'R' in the Viewer window. The instructions for doing this can be found by hovering over those icons.

<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
# =========== Step 3, main step of selfcalibration =========
if doslfcal:
if os.path.exists(slfcaldir+'masks.p'):
masks=pickle.load(open(slfcaldir+'masks.p','rb'))
if not os.path.exists(slfcaldir+'masks.p'):
print 'masks do not exist. Use default mask'
masks=[]
os.system('rm -rf '+imagedir+'*')
os.system('rm -rf '+caltbdir+'*')
#first step: make a mock caltable for the entire database
print('Processing ' + trange)
slftbs=[]
calprefix=caltbdir+'slf'
imgprefix=imagedir+'slf'
tb.open(slfcalms+'/SPECTRAL_WINDOW')
reffreqs=tb.getcol('REF_FREQUENCY')
bdwds=tb.getcol('TOTAL_BANDWIDTH')
cfreqs=reffreqs+bdwds/2.
tb.close()
# starting beam size at 3.4 GHz in arcsec #Change accordingly
sbeam=40.
strtmp=[m.replace(':','') for m in trange.split('~')]
timestr='t'+strtmp[0]+'-'+strtmp[1]
refantenna='0'
# number of iterations for each round
niters=[100, 300, 500]
# roburst value for weighting the baselines
robusts=[1.0, 0.5, 0.0]
# apply calibration tables? Set to true for most cases
doapplycal=[1,1,1]
# modes for calibration, 'p' for phase-only, 'a' for amplitude only, 'ap' for both
calmodes=['p','p','a']
# setting uvranges for model image (optional, not used here)
uvranges=['','','']
for n in range(nround):
slfcal_tb_g= calprefix+'.G'+str(n)
fig = plt.figure(figsize=(8.4,7.)) # fig = plt.figure(figsize=(14, 7)) # For post-2020 data (50 spws)
gs = gridspec.GridSpec(5, 6) # gs = gridspec.GridSpec(5, 10) # For post-2020 data (50 spws)
for s,sp in enumerate(spws):
print 'processing spw: '+sp
cfreq=cfreqs[int(sp)]
# setting restoring beam size (not very useful for selfcal anyway, but just to see the results)
bm=max(sbeam*cfreqs[1]/cfreq, 6.)
slfcal_img = imgprefix+'.spw'+sp.zfill(2)+'.slfcal'+str(n)
# only the first round uses nearby spws for getting initial model
if n == 0:
spbg=max(int(sp)-2,1) # spbg=max(int(sp)-2,0) # For post-2020 data (50 spws)
sped=min(int(sp)+2,30) # sped=min(int(sp)+2,49) # For post-2020 data (50 spws)
spwran=str(spbg)+'~'+str(sped)
print('using spw {0:s} as model'.format(spwran))
if 'spwrans_mask' in vars():
for m, spwran_mask in enumerate(spwrans_mask):
if sp in spwran_mask:
mask = masks[m]
print('using mask {0:s}'.format(mask))
findmask = True
if not findmask:
print('mask not found. Do use any masks')
else:
spwran = sp
if 'spwrans_mask' in vars():
for m, spwran_mask in enumerate(spwrans_mask):
if sp in spwran_mask:
mask = masks[m]
print 'using mask {0:s}'.format(mask)
findmask = True
if not findmask:
print('mask not found. Do use any masks')
try:
tclean(vis=slfcalms,
antenna=antennas,
imagename=slfcal_img,
uvrange=uvranges[n],
spw=spwran,
specmode='mfs',
timerange=trange,
imsize=[npix],
cell=['2arcsec'],
niter=niters[n],
gain=0.05,
stokes='XX', #use pol XX image as the model
weighting='briggs',
robust=robusts[n],
phasecenter=phasecenter,
mask=mask,
restoringbeam=[str(bm)+'arcsec'],
pbcor=False,
interactive=False,
savemodel='modelcolumn')
if os.path.exists(slfcal_img+'.image'):
fitsfile=slfcal_img+'.fits'
hf.imreg(vis=slfcalms,imagefile=slfcal_img+'.image',fitsfile=fitsfile,
timerange=trange,usephacenter=False,toTb=True,verbose=False,overwrite=True)
clnjunks = ['.mask','.flux', '.model', '.psf', '.residual', '.image','.pb','.image.pbcor','.sumwt'] #Do not run the next 4 lines if needed to view and assess the subset of clean process images
for clnjunk in clnjunks:
if os.path.exists(slfcal_img + clnjunk):
os.system('rm -rf '+ slfcal_img + clnjunk)
ax = fig.add_subplot(gs[s])
eomap=smap.Map(fitsfile)
eomap.plot_settings['cmap'] = plt.get_cmap('jet')
eomap.plot(axes = ax)
eomap.draw_limb()
#eomap.draw_grid()
ax.set_title(' ')
ax.get_xaxis().set_visible(False)
ax.get_yaxis().set_visible(False)
ax.set_xlim(xran)
ax.set_ylim(yran)
plt.pause(0.5) # Allows viewing of each image as it is plotted.
os.system('rm -f '+ fitsfile)

except:
print 'error in cleaning spw: '+sp
print 'using nearby spws for initial model'
sp_e=int(sp)+2
sp_i=int(sp)-2
if sp_i < 1: # if sp < 0: # For post-2020 data (50 spws)
sp_i = 1 # sp_i = 0 # For post-2020 data (50 spws)
if sp_e > 30: # if sp_e > 49: # For post-2020 data (50 spws)
sp_e = 30 # sp_e = 49 # For post-2020 data (50 spws)
sp_=str(sp_i)+'~'+str(sp_e)
try:
tget(tclean)
spw=sp_
print('using spw {0:s} as model'.format(sp_))
tclean()
except:
print 'still not successful. abort...'
break

gaincal(vis=slfcalms, refant=refantenna,antenna=antennas,caltable=slfcal_tb_g,spw=sp, uvrange='',\
gaintable=[],selectdata=True,timerange=trange,solint='inf',gaintype='G',calmode=calmodes[n],\
combine='',minblperant=4,minsnr=2,append=True)
if not os.path.exists(slfcal_tb_g):
print 'No solution found in spw: '+sp
figname=imagedir+'slf_t0_n{:d}.png'.format(n)
plt.subplots_adjust(left=0, bottom=0, right=1, top=1, wspace=0, hspace=0)
plt.savefig(figname)
time.sleep(10)
plt.close()

if os.path.exists(slfcal_tb_g):
slftbs.append(slfcal_tb_g)
slftb=[slfcal_tb_g]
os.chdir(slfcaldir)
if calmodes[n] == 'p':
plotcal(caltable=slfcal_tb_g,antenna='1~12',xaxis='freq',yaxis='phase',\
subplot=431,plotrange=[-1,-1,-180,180],iteration='antenna',figfile=slfcal_tb_g+'.png',showgui=False)
if calmodes[n] == 'a':
plotcal(caltable=slfcal_tb_g,antenna='1~12',xaxis='freq',yaxis='amp',\
subplot=431,plotrange=[-1,-1,0,2.],iteration='antenna',figfile=slfcal_tb_g+'.png',showgui=False)
os.chdir(workdir)
plt.pause(0.5) # Allows viewing of the plot

if doapplycal[n]:
clearcal(slfcalms)
delmod(slfcalms)
applycal(vis=slfcalms,gaintable=slftb,spw=','.join(spws),selectdata=True,\
antenna=antennas,interp='nearest',flagbackup=False,applymode='calonly',calwt=False)

if n < nround-1:
prompt=raw_input('Continuing to selfcal?')
#prompt='y'
if prompt.lower() == 'n':
if os.path.exists(slfcaledms):
os.system('rm -rf '+slfcaledms)
split(slfcalms,slfcaledms,datacolumn='corrected')
print 'Final calibrated ms is {0:s}'.format(slfcaledms)
break
if prompt.lower() == 'y':
slfcalms_=slfcalms+str(n)
if os.path.exists(slfcalms_):
os.system('rm -rf '+slfcalms_)
split(slfcalms,slfcalms_,datacolumn='corrected')
slfcalms=slfcalms_
else:
if os.path.exists(slfcaledms):
os.system('rm -rf '+slfcaledms)
split(slfcalms,slfcaledms,datacolumn='corrected')
print 'Final calibrated ms is {0:s}'.format(slfcaledms)
</pre>

<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
# =========== Step 4: Apply self-calibration tables =========
if doapply:
import glob
os.chdir(workdir)
clearcal(ms_in)
clearcal(slfcalms)
applycal(vis=slfcalms,gaintable=slftbs,spw=','.join(spws),selectdata=True,\
antenna=antennas,interp='linear',flagbackup=False,applymode='calonly',calwt=False)
applycal(vis=ms_in,gaintable=slftbs,spw=','.join(spws),selectdata=True,\
antenna=antennas,interp='linear',flagbackup=False,applymode='calonly',calwt=False)
if os.path.exists(ms_slfcaled):
os.system('rm -rf '+ms_slfcaled)
split(ms_in, ms_slfcaled,datacolumn='corrected')
</pre>
[[file:Slf.G0.png|thumb|center|500px|Figure 1: Phase before self-calibration]]
[[file:Slf.G1.png|thumb|center|500px|Figure 2: Phase after self-calibration]]

<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
# =========== Step 5: Generate final self-calibrated images (optional) =========
if doclean_slfcaled:
import glob
pol='XX'
print('Processing ' + trange)
img_final=imagedir_slfcaled+'/slf_final_{0}_t0'.format(pol)
vis = ms_slfcaled
spws=[str(s+1) for s in range(30)]
tb.open(vis+'/SPECTRAL_WINDOW')
reffreqs=tb.getcol('REF_FREQUENCY')
bdwds=tb.getcol('TOTAL_BANDWIDTH')
cfreqs=reffreqs+bdwds/2.
tb.close()
sbeam=30.
from matplotlib import gridspec as gridspec
from sunpy import map as smap
from matplotlib import pyplot as plt
fitsfiles=[]
for s,sp in enumerate(spws):
cfreq=cfreqs[int(sp)]
bm=max(sbeam*cfreqs[1]/cfreq,4.)
imname=img_final+'_s'+sp.zfill(2)
fitsfile=imname+'.fits'
if not os.path.exists(fitsfile):
print 'cleaning spw {0:s} with beam size {1:.1f}"'.format(sp,bm)
try:
tclean(vis=vis,
antenna=antennas,
imagename=imname,
spw=sp,
specmode='mfs',
timerange=trange,
imsize=[npix],
cell=['1arcsec'],
niter=1000,
gain=0.05,
stokes=pol,
weighting='briggs',
robust=2.0,
restoringbeam=[str(bm)+'arcsec'],
phasecenter=phasecenter,
mask='',
pbcor=True,
interactive=False)
except:
print 'cleaning spw '+sp+' unsuccessful. Proceed to next spw'
continue
if os.path.exists(imname+'.image.pbcor'):
imn = imname+'.image.pbcor'
hf.imreg(vis=vis,imagefile=imn,fitsfile=fitsfile,
timerange=trange,usephacenter=False,toTb=True,verbose=False)
fitsfiles.append(fitsfile)
junks=['.flux','.model','.psf','.residual','.mask','.image','.pb','.image.pbcor','.sumwt'] #Do not run the next 4 lines if needed to view and assess the subset of clean process images
for junk in junks:
if os.path.exists(imname+junk):
os.system('rm -rf '+imname+junk)
else:
print('fits file '+fitsfile+' already exists, skip clean...')
fitsfiles.append(fitsfile)

fig = plt.figure(figsize=(8.4,7.)) # fig = plt.figure(figsize=(14, 7)) # For post-2020 data (50 spws)
gs = gridspec.GridSpec(5, 6) # gs = gridspec.GridSpec(5, 10) # For post-2020 data (50 spws)
for s,sp in enumerate(spws):
cfreq=cfreqs[int(sp)]
ax = fig.add_subplot(gs[s])
eomap=smap.Map(fitsfiles[s])
eomap.plot_settings['cmap'] = plt.get_cmap('jet')
eomap.plot(axes = ax)
eomap.draw_limb()
ax.set_title(' ')
ax.get_xaxis().set_visible(False)
ax.get_yaxis().set_visible(False)
ax.set_xlim(xran)
ax.set_ylim(yran)
plt.text(0.98,0.85,'{0:.1f} GHz'.format(cfreq/1e9),transform=ax.transAxes,ha='right',color='w',fontweight='bold')
plt.subplots_adjust(left=0, bottom=0, right=1, top=1, wspace=0, hspace=0)
plt.show()

</pre>
The .fits files of the self calibrated images at each frequency for the given time are saved at /slfcal/images_slfcaled in your working directory.

[[file:Slf_t0_n0.png|thumb|center|500px|Figure 3: Multi-frequency images before self-calibration]]
[[file:Slf_t0_n1.png|thumb|center|500px|Figure 4: Multi-frequency images after self-calibration]]

====Step 5: Quick-look imaging ==== 
For spectral imaging analysis of the event, follow [http://www.ovsa.njit.edu/wiki/index.php/EOVSA_Data_Analysis_Tutorial#Spectral_Imaging_with_SunCASA this tutorial] using the self-calibrated data obtained from the previous step or use [https://colab.research.google.com/drive/1lSLLxgOG6b8kgu9Sk6kSKvrViyubnXG6?usp=sharing#scrollTo=xbXyyLmCFCGL this link].

<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
from suncasa.utils import qlookplot as ql ## (Optional) Supply the npz file of the dynamic spectrum from previous step.
## If not provided, the program will generate a new one from the visibility.
## set the time interval
from suncasa import dspec as ds
import time
visibility_data = 'IDB20170821202020_slfcaled.ms'
specfile = visibility_data + '.dspec.npz'
d = ds.Dspec(visibility_data, bl='4&9', specfile=specfile)

timerange = '19:02:00~19:02:10' ## select frequency range from 2.5 GHz to 3.5 GHz
spw = '2~5' ## select stokes XX
stokes = 'XX' ## turn off AIA image plotting, default is True
plotaia = False
xycen = [375, 45] ## image center for clean in solar X-Y in arcsec
cell=['2.0arcsec'] ## pixel size
imsize=[128] ## x and y image size in pixels.
fov = [100,100] ## field of view of the zoomed-in panels in unit of arcsec
spw = ['{}'.format(s) for s in range(1,31)]
clevels = [0.5, 1.0] ## contour levels to fill in between.
calpha=0.35 ## now tune down the alpha
restoringbeam=['6arcsec']
ql.qlookplot(vis=msfile, specfile=specfile, timerange=timerange, spw=spw, stokes=stokes, \
restoringbeam=restoringbeam,imsize=imsize,cell=cell, \
xycen=xycen,fov=fov,clevels=clevels,calpha=calpha)
</pre>

Page Last Updated: 07/06/2022

Tohban EOVSA Imaging Tutorial A-Z V2

2022-07-06T11:20:56Z

Schhabra:

This tutorial describes the step-by-step procedure to download EOVSA IDB data, obtain and calibrate the measurement sets (.ms), and, transfer them to the inti server for self-calibration and further analysis.

=== Pre-requisites ===
Accounts on Pipeline and Inti or Baozi servers. Sijie and Bin can help with the account set-up for pipeline, inti and boazi.

==== How to connect to pipeline ====
'''Windows:''' On a windows machine Mobaxterm can be used to connect to Pipeline. For details, see ...

'''Linux/Unix:''' On a linux/Mac the ssh commands can be used directly through the terminal:

<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">

ssh -L 5902:helios.solar.pvt:20000 guest@ovsa.njit.edu ### when prompted, enter password to guest@ovsa ###
ssh -XY your_username@pipeline ### add your username and enter password to your pipeline account, when prompted ###
</pre>

Alternatively, to avoid typing the longer commands, proxy commands can be added to the ~/.ssh/config file:

<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
vim ~/.ssh/config ### press 'i' to insert content and esc +':wq' to write and quit ###

Host *
ForwardAgent yes

Host *
ForwardX11 yes

Host ovsa
HostName ovsa.njit.edu
User guest

Host pipeline
Hostname pipeline.solar.pvt
User your_username ### Insert your pipeline username here ###
ProxyCommand ssh -W %h:%p guest@ovsa.njit.edu
</pre>

Upon saving the ~/.ssh/config file and restarting the terminal, you can connect to pipeline directly using:

<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
ssh -XY pipeline
</pre>

==== How to connect to the inti server ====
NOTE: To connect to inti you must have a NJIT UCID. The arcs have granted access to inti for all SRG members. If you encounter any issue with the login. Please let arcs folks (arcs@njit.edu) know.

'''Windows:''' As mentioned above, on a windows machine Mobaxterm can be used to connect to inti.
With your NJIT VPN connected, connect to one of the afsconnect servers (for example, afsaccess3.njit.edu) using your UCID and password. Next, ssh into inti.
<pre style="background-color: #FCEBD9">
ssh -XY UCID@afsaccess3.njit.edu ###Insert UCID password if prompted###

ssh -XY UCID@inti.hpcnet.campus.njit.edu ###Insert UCID/inti username here###
</pre>
Proxy commands for inti can also be added to the C:\Program Files\Git\etc\ssh\ssh_config file:
<pre style="background-color: #FCEBD9">
Host inti
Hostname inti.hpcnet.campus.njit.edu
User UCID ###Insert UCID/inti username here###
</pre>

'''Linux/Unix:''' On a linux/Mac the ssh command can be used:
<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
ssh -XY UCID@inti.hpcnet.campus.njit.edu ### insert UCID here and enter your NJIT UCID password, when prompted ###
</pre>

Proxy commands for inti can also be added to the ~/.ssh/config (on local machine) file for convenience:
<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
vim ~/.ssh/config ### press 'i' to insert content and esc +':wq' to write and quit ###

Host inti
Hostname inti.hpcnet.campus.njit.edu
User UCID ### insert NJIT UCID here ###
</pre>
after the proxy is set up, connect to inti by simply typing:
<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
ssh -XY inti ### enter your NJIT UCID password when prompted ###
</pre>

'''NOTE:'''When logging in for the first time, please add the following lines (demarcated by #) in your ~/.bashrc (on the inti server) file in order to use CASA on inti.

<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
#### setting start ####
if [ $HOSTNAME == "baozi.hpcnet.campus.njit.edu" ]; then
source /srg/.setenv_baozi
fi
if [ $HOSTNAME == "inti.hpcnet.campus.njit.edu" ]; then
source /inti/.setenv_inti
fi
if [ $HOSTNAME == "guko.resource.campus.njit.edu" ]; then
source /data/data/.setenv_guko
fi
#### setting end ####
</pre>

For sufficient disk space to analyze EOVSA data on Inti, use your dedicated directory YOURDIRECTORY at the location given below. If you do not have a directory, please ask Sijie to set one up.
<pre style="background-color: #FCEBD9">
cd /inti/data/users/YOURDIRECTORY
</pre>

'''Both CASA 5 and 6 are available on Inti.'''

Enter the bash environment on inti, and load the desired casa environment.

To load CASA 5:
<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
>> bash
>> loadcasa5
>> suncasa
</pre>

To load CASA 6:
<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
>> bash
>> loadcasa6
>> ipython
</pre>

=== Downloading data and Preliminary Calibration ===

==== Step 1: Downloading raw (IDB) data on Pipeline machine====
The initial downloading of data and applying refcal and phasecal to the .ms files is performed on the pipeline since the complete EOVSA SQL database is stored here for now.

If you are not using mobaxterm, directly SSH to Pipeline through your Linux or Mac computer (as shown above) and '''start tcsh to run CASA'''.

The following commands (run within CASA) can be used to import a single IDB file if the given time-range is less than 10 mins.
<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
from astropy.time import Time
import os
trange = Time(['2017-08-21 20:15:00', '2017-08-21 20:25:00']) ###Change accordingly###

#### (Optional) change output path, default current directory "./" #######
outpath = './msdata/' ###Change accordingly###
if not os.path.exists(outpath):
os.makedirs(outpath)

msfiles = importeovsa(idbfiles=trange, ncpu=1, visprefix=outpath)
</pre>

'''NOTE:''' The above step currently gives an error with importeovsa. This is because the data location was changed and the code doesn't know where to look for the IDB files. Sijie is looking into the problem. We will update the page when it is fixed. For now, please use the method below for all time ranges.

OR (for a time range longer than 10 minutes)

<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
from suncasa.tasks import task_calibeovsa as calibeovsa
from suncasa.tasks import task_importeovsa as timporteovsa
from split_cli import split_cli as split
import dump_tsys as dt
from util import Time
import numpy as np
import os
from glob import glob
from eovsapy import util

trange = Time(['2017-08-21 20:15:00', '2017-08-21 20:35:00']) ###Change accordingly###
idbdir = util.get_idbdir(trange[0])

info = dt.rd_fdb(trange[0])
for k, v in info.iteritems():
info[k] = info[k][~(info[k] == '')]

sidx = np.where(
np.logical_and(info['SOURCEID'] == 'Sun', info['PROJECTID'] == 'NormalObserving') & np.logical_and(
info['ST_TS'].astype(np.float) >= trange[0].lv,
info['ST_TS'].astype(np.float) <= trange[
1].lv))
filelist = info['FILE'][sidx]

outpath = './msdata/' ###Change accordingly###
if not os.path.exists(outpath):
os.makedirs(outpath)
inpath = idbdir + '{}/'.format(trange[0].datetime.strftime("%Y%m%d"))
ncpu = 1

msfiles = timporteovsa.importeovsa(idbfiles=[inpath + ll for ll in filelist], ncpu=ncpu, timebin="0s", width=1,
visprefix=outpath,
nocreatms=False, doconcat=False,
modelms="", doscaling=False, keep_nsclms=False, udb_corr=True)
</pre>

If you come across errors with calibeovsa, add the following lines to your ~/.casa/init.py (within pipeline) file.
<pre style="background-color: #FCEBD9">
vim ~/.ssh/config ### press 'i' to insert content and esc +':wq' to write and quit ###
import sys
sys.path.append('/common/python')
sys.path.append('/common/python/packages/pipeline_casa')
execfile('/common/python/suncasa/tasks/mytasks.py')
</pre>

==== Step 2: Concatenate multiple .ms files and Calibrate====
'''Follow this step if there are more than one .ms files for the chosen time-range. If not, move to Step 3.'''

<pre style="background-color: #FCEBD9">
concatvis = os.path.basename(msfiles[0])[:11] + '_concat_cal.ms' ### Set the path/name for the concatenated files ###

vis = calibeovsa.calibeovsa(msfiles, doconcat=True, concatvis=concatvis, caltype=['refpha','phacal'], doimage=True)
### If doimage=True, a quicklook image will be produced (by integrating over the entire time) as shown below ###

vis_str = str(' '.join(vis))
caled_vis=vis_str.replace('.ms','_cal.ms') ### Append '_cal' to the ms filename ###
split(vis=' '.join(vis),outputvis=caled_vis,datacolumn='corrected',timerange='',correlation='XX') ### Split the corrected column to the new 'caled' ms ###
</pre>

[[file:Figure1_imagingtutorial.png|frame|center|800px|'''Figure 1:''' Quick-look full-Sun image after the initial calibration.]]

==== Step 3: Calibration ====
For a single .ms file from Step 1, this will calibrate the input visibility, write out calibration tables under /data1/eovsa/caltable/, and apply the calibration.
<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
vis = calibeovsa.calibeovsa('IDB20170821202020.ms', caltype=['refpha','phacal'], doimage=True) ###Change the vis filename accordingly###

vis_str = str(' '.join(vis))
caled_vis=vis_str.replace('.ms','_cal.ms') ### Append '_cal' to the ms filename ###
split(vis=' '.join(vis),outputvis=caled_vis,datacolumn='corrected',timerange='',correlation='XX') ### Split the corrected column to the new 'caled' ms ###
</pre>

The created 'caled' ms data files (xxx_concat_cal.ms or xxx_cal.ms) need to be transferred from pipeline to inti server, which has all the casa tasks installed, in order to run the rest of the imaging steps.

=== Transferring data files between servers ===
To directly transfer your calibrated .ms data from the Pipeline to Inti:

<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
1. Log into Inti using your username.
ssh -X USERNAME@inti.hpcnet.campus.njit.edu

Then create a tunnel into Pipeline from Inti.
ssh -L 8888:pipeline.solar.pvt:22 guest@ovsa.njit.edu

2. Log into Inti again from a new terminal. Change to your working directory (where you need the data transferred) and give the following command:

scp -v -C -r -P 8888 PipelineUSERNAME@localhost:PATH/TO/MSFILE ./

</pre>

===Advanced Calibration and Imaging===
====Step 4: Self-calibration====
[[file:Figure3_imagingtutorial.png|thumb|center|500px|Figure 2: Cotton-Schwab clean major and minor cycles. [Source: http://www.aoc.nrao.edu/~rurvashi/ImagingAlgorithmsInCasa/node2.html].]]
Follow the below given steps to run the self-calibration of the imaging data and produce the calibrated images in .fits format. https://github.com/binchensun/casa-eovsa/blob/master/slfcal_example.py

<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
from suncasa.utils import helioimage2fits as hf
import os
import numpy as np
import pickle
from matplotlib import gridspec as gridspec
from sunpy import map as smap
from matplotlib import pyplot as plt
from split_cli import split_cli as split
import time

# =========== task handlers =============
dofullsun = 1 # initial full-sun imaging ###Change accordingly###
domasks=1 # get masks ###Change accordingly###
doslfcal=1 # main cycle of doing selfcalibration ###Change accordingly###
doapply=1 # apply the results ###Change accordingly###
doclean_slfcaled=1 # perform clean for self-calibrated data ###Change accordingly###

# ============ declaring the working directories ============
workdir = os.getcwd()+'/' #main working directory. Using current directory in this example
slfcaldir = workdir+'slfcal/' #place to put all selfcalibration products
imagedir = slfcaldir+'images/' #place to put all selfcalibration images
maskdir = slfcaldir+'masks/' #place to put clean masks
imagedir_slfcaled = slfcaldir+'images_slfcaled/' #place to put final self-calibrated images
caltbdir = slfcaldir+'caltbs/' # place to put calibration tables
# make these directories if they do not already exist
dirs = [workdir, slfcaldir, imagedir, maskdir, imagedir_slfcaled, caltbdir]
for d in dirs:
if not os.path.exists(d):
os.makedirs(d)

# ============ Split a short time for self-calibration ===========
# input visibility
ms_in = workdir + 'IDB20170821202020_cal.ms' ###Change the initial calibrated (through calibeovsa) vis accordingly###
# output, selfcaled, visibility
ms_slfcaled = workdir + os.path.basename(ms_in).replace('cal','slfcaled')
# intermediate small visibility for selfcalbration
# selected time range for generating self-calibration solutions
trange='2017/08/21/20:21:10~2017/08/21/20:21:30' ###Change accordingly###
slfcalms = slfcaldir+'slfcalms.XX.slfcal'
slfcaledms = slfcaldir+'slfcalms.XX.slfcaled'
if not os.path.exists(slfcalms):
split(vis=ms_in,outputvis=slfcalms,datacolumn='data',timerange=trange,correlation='XX')

# ============ Prior definitions for spectral windows, antennas, pixel numbers =========
spws=[str(s+1) for s in range(30)] # spws=[str(s+1) for s in range(49)] # For post-2020 data
antennas='0~12&0~12'
npix=512
nround=3 #number of slfcal cycles

# =========== Step 1, doing a full-Sun image to find out phasecenter and appropriate field of view =========
if dofullsun:
#initial mfs clean to find out the image phase center
im_init='fullsun_init'
os.system('rm -rf '+im_init+'*')
tclean(vis=slfcalms,
antenna=antennas,
imagename=im_init,
spw='2~15',
specmode='mfs',
timerange=trange,
imsize=[npix],
cell=['5arcsec'],
niter=1000,
gain=0.05,
stokes='I',
restoringbeam=['30arcsec'],
interactive=False,
pbcor=True)

hf.imreg(vis=slfcalms,imagefile=im_init+'.image.pbcor',fitsfile=im_init+'.fits',
timerange=trange,usephacenter=False,verbose=True)
clnjunks = ['.flux', '.mask', '.model', '.psf', '.residual','.sumwt','.pb','.image'] #Do not run the next 4 lines if needed to view and assess the subset of clean process images
for clnjunk in clnjunks:
if os.path.exists(im_init + clnjunk):
os.system('rm -rf '+im_init + clnjunk)

from sunpy import map as smap
from matplotlib import pyplot as plt
fig = plt.figure(figsize=(6,6))
ax = fig.add_subplot(111)
eomap=smap.Map(im_init+'.fits')
#eomap.data=eomap.data.reshape((npix,npix))
eomap.plot_settings['cmap'] = plt.get_cmap('jet')
eomap.plot(axes = ax)
eomap.draw_limb()
plt.show()
viewer(im_init+'.image.pbcor')
</pre>
[[file:Figure2_imagingtutorial.png|thumb|center|500px|Figure 3: Full-Sun image after initial clean to find the flare location.]]

<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
# parameters specific to the event (found from step 1)
phasecenter='J2000 10h02m59 11d58m07' ###Change accordingly###
xran=[280,480] ###Change accordingly###
yran=[-50,150] ###Change accordingly###

# =========== Step 2 (optional), generate masks =========
# if skipped, will not use any masks
if domasks:
clearcal(slfcalms)
delmod(slfcalms)
antennas=antennas
pol='XX'
imgprefix=maskdir+'slf_t0'

# step 1: set up the clean masks
img_init=imgprefix+'_init_ar_'
os.system('rm -rf '+img_init+'*')
#spwrans_mask=['1~5','6~12','13~20','21~30']
spwrans_mask=['2~12']
#convert to a list of spws
spwrans_mask_list = [[str(i) for i in (np.arange(int(m.split('~')[0]),int(m.split('~')[1])))] for m in spwrans_mask] # Not used?
masks=[]
imnames=[]
for spwran in spwrans_mask:
imname=img_init+spwran.replace('~','-')
try:
tclean(vis=slfcalms,
antenna=antennas,
imagename=imname,
spw=spwran,
specmode='mfs',
timerange=trange,
imsize=[npix],
cell=['2arcsec'],
niter=1000,
gain=0.05,
stokes='XX',
restoringbeam=['20arcsec'],
phasecenter=phasecenter,
weighting='briggs',
robust=1.0,
interactive=True,
datacolumn='data',
pbcor=True,
savemodel='modelcolumn')
imnames.append(imname+'.image')
masks.append(imname+'.mask')
clnjunks = ['.flux', '.model', '.psf', '.residual'] #Do not run the next 4 lines if needed to view and assess the subset of clean process images
for clnjunk in clnjunks:
if os.path.exists(imname + clnjunk):
os.system('rm -rf '+ imname + clnjunk)
except:
print('error in cleaning spw: '+spwran)

pickle.dump(masks,open(slfcaldir+'masks.p','wb'))
</pre>
[[file:Figure4_imagingtutorial.PNG|thumb|center|800px|Figure 4: Interactive clean window to create masks over the source. The white outline surrounding the source is the mask selected by the polygon drawing option.]]

The outlines drawn for masks can be created by any of the icons with the letter 'R' in the Viewer window. The instructions for doing this can be found by hovering over those icons.

<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
# =========== Step 3, main step of selfcalibration =========
if doslfcal:
if os.path.exists(slfcaldir+'masks.p'):
masks=pickle.load(open(slfcaldir+'masks.p','rb'))
if not os.path.exists(slfcaldir+'masks.p'):
print 'masks do not exist. Use default mask'
masks=[]
os.system('rm -rf '+imagedir+'*')
os.system('rm -rf '+caltbdir+'*')
#first step: make a mock caltable for the entire database
print('Processing ' + trange)
slftbs=[]
calprefix=caltbdir+'slf'
imgprefix=imagedir+'slf'
tb.open(slfcalms+'/SPECTRAL_WINDOW')
reffreqs=tb.getcol('REF_FREQUENCY')
bdwds=tb.getcol('TOTAL_BANDWIDTH')
cfreqs=reffreqs+bdwds/2.
tb.close()
# starting beam size at 3.4 GHz in arcsec #Change accordingly
sbeam=40.
strtmp=[m.replace(':','') for m in trange.split('~')]
timestr='t'+strtmp[0]+'-'+strtmp[1]
refantenna='0'
# number of iterations for each round
niters=[100, 300, 500]
# roburst value for weighting the baselines
robusts=[1.0, 0.5, 0.0]
# apply calibration tables? Set to true for most cases
doapplycal=[1,1,1]
# modes for calibration, 'p' for phase-only, 'a' for amplitude only, 'ap' for both
calmodes=['p','p','a']
# setting uvranges for model image (optional, not used here)
uvranges=['','','']
for n in range(nround):
slfcal_tb_g= calprefix+'.G'+str(n)
fig = plt.figure(figsize=(8.4,7.)) # fig = plt.figure(figsize=(14, 7)) # For post-2020 data (50 spws)
gs = gridspec.GridSpec(5, 6) # gs = gridspec.GridSpec(5, 10) # For post-2020 data (50 spws)
for s,sp in enumerate(spws):
print 'processing spw: '+sp
cfreq=cfreqs[int(sp)]
# setting restoring beam size (not very useful for selfcal anyway, but just to see the results)
bm=max(sbeam*cfreqs[1]/cfreq, 6.)
slfcal_img = imgprefix+'.spw'+sp.zfill(2)+'.slfcal'+str(n)
# only the first round uses nearby spws for getting initial model
if n == 0:
spbg=max(int(sp)-2,1) # spbg=max(int(sp)-2,0) # For post-2020 data (50 spws)
sped=min(int(sp)+2,30) # sped=min(int(sp)+2,49) # For post-2020 data (50 spws)
spwran=str(spbg)+'~'+str(sped)
print('using spw {0:s} as model'.format(spwran))
if 'spwrans_mask' in vars():
for m, spwran_mask in enumerate(spwrans_mask):
if sp in spwran_mask:
mask = masks[m]
print('using mask {0:s}'.format(mask))
findmask = True
if not findmask:
print('mask not found. Do use any masks')
else:
spwran = sp
if 'spwrans_mask' in vars():
for m, spwran_mask in enumerate(spwrans_mask):
if sp in spwran_mask:
mask = masks[m]
print 'using mask {0:s}'.format(mask)
findmask = True
if not findmask:
print('mask not found. Do use any masks')
try:
tclean(vis=slfcalms,
antenna=antennas,
imagename=slfcal_img,
uvrange=uvranges[n],
spw=spwran,
specmode='mfs',
timerange=trange,
imsize=[npix],
cell=['2arcsec'],
niter=niters[n],
gain=0.05,
stokes='XX', #use pol XX image as the model
weighting='briggs',
robust=robusts[n],
phasecenter=phasecenter,
mask=mask,
restoringbeam=[str(bm)+'arcsec'],
pbcor=False,
interactive=False,
savemodel='modelcolumn')
if os.path.exists(slfcal_img+'.image'):
fitsfile=slfcal_img+'.fits'
hf.imreg(vis=slfcalms,imagefile=slfcal_img+'.image',fitsfile=fitsfile,
timerange=trange,usephacenter=False,toTb=True,verbose=False,overwrite=True)
clnjunks = ['.mask','.flux', '.model', '.psf', '.residual', '.image','.pb','.image.pbcor','.sumwt'] #Do not run the next 4 lines if needed to view and assess the subset of clean process images
for clnjunk in clnjunks:
if os.path.exists(slfcal_img + clnjunk):
os.system('rm -rf '+ slfcal_img + clnjunk)
ax = fig.add_subplot(gs[s])
eomap=smap.Map(fitsfile)
eomap.plot_settings['cmap'] = plt.get_cmap('jet')
eomap.plot(axes = ax)
eomap.draw_limb()
#eomap.draw_grid()
ax.set_title(' ')
ax.get_xaxis().set_visible(False)
ax.get_yaxis().set_visible(False)
ax.set_xlim(xran)
ax.set_ylim(yran)
plt.pause(0.5) # Allows viewing of each image as it is plotted.
os.system('rm -f '+ fitsfile)

except:
print 'error in cleaning spw: '+sp
print 'using nearby spws for initial model'
sp_e=int(sp)+2
sp_i=int(sp)-2
if sp_i < 1: # if sp < 0: # For post-2020 data (50 spws)
sp_i = 1 # sp_i = 0 # For post-2020 data (50 spws)
if sp_e > 30: # if sp_e > 49: # For post-2020 data (50 spws)
sp_e = 30 # sp_e = 49 # For post-2020 data (50 spws)
sp_=str(sp_i)+'~'+str(sp_e)
try:
tget(tclean)
spw=sp_
print('using spw {0:s} as model'.format(sp_))
tclean()
except:
print 'still not successful. abort...'
break

gaincal(vis=slfcalms, refant=refantenna,antenna=antennas,caltable=slfcal_tb_g,spw=sp, uvrange='',\
gaintable=[],selectdata=True,timerange=trange,solint='inf',gaintype='G',calmode=calmodes[n],\
combine='',minblperant=4,minsnr=2,append=True)
if not os.path.exists(slfcal_tb_g):
print 'No solution found in spw: '+sp
figname=imagedir+'slf_t0_n{:d}.png'.format(n)
plt.subplots_adjust(left=0, bottom=0, right=1, top=1, wspace=0, hspace=0)
plt.savefig(figname)
time.sleep(10)
plt.close()

if os.path.exists(slfcal_tb_g):
slftbs.append(slfcal_tb_g)
slftb=[slfcal_tb_g]
os.chdir(slfcaldir)
if calmodes[n] == 'p':
plotcal(caltable=slfcal_tb_g,antenna='1~12',xaxis='freq',yaxis='phase',\
subplot=431,plotrange=[-1,-1,-180,180],iteration='antenna',figfile=slfcal_tb_g+'.png',showgui=False)
if calmodes[n] == 'a':
plotcal(caltable=slfcal_tb_g,antenna='1~12',xaxis='freq',yaxis='amp',\
subplot=431,plotrange=[-1,-1,0,2.],iteration='antenna',figfile=slfcal_tb_g+'.png',showgui=False)
os.chdir(workdir)
plt.pause(0.5) # Allows viewing of the plot

if doapplycal[n]:
clearcal(slfcalms)
delmod(slfcalms)
applycal(vis=slfcalms,gaintable=slftb,spw=','.join(spws),selectdata=True,\
antenna=antennas,interp='nearest',flagbackup=False,applymode='calonly',calwt=False)

if n < nround-1:
prompt=raw_input('Continuing to selfcal?')
#prompt='y'
if prompt.lower() == 'n':
if os.path.exists(slfcaledms):
os.system('rm -rf '+slfcaledms)
split(slfcalms,slfcaledms,datacolumn='corrected')
print 'Final calibrated ms is {0:s}'.format(slfcaledms)
break
if prompt.lower() == 'y':
slfcalms_=slfcalms+str(n)
if os.path.exists(slfcalms_):
os.system('rm -rf '+slfcalms_)
split(slfcalms,slfcalms_,datacolumn='corrected')
slfcalms=slfcalms_
else:
if os.path.exists(slfcaledms):
os.system('rm -rf '+slfcaledms)
split(slfcalms,slfcaledms,datacolumn='corrected')
print 'Final calibrated ms is {0:s}'.format(slfcaledms)
</pre>

<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
# =========== Step 4: Apply self-calibration tables =========
if doapply:
import glob
os.chdir(workdir)
clearcal(ms_in)
clearcal(slfcalms)
applycal(vis=slfcalms,gaintable=slftbs,spw=','.join(spws),selectdata=True,\
antenna=antennas,interp='linear',flagbackup=False,applymode='calonly',calwt=False)
applycal(vis=ms_in,gaintable=slftbs,spw=','.join(spws),selectdata=True,\
antenna=antennas,interp='linear',flagbackup=False,applymode='calonly',calwt=False)
if os.path.exists(ms_slfcaled):
os.system('rm -rf '+ms_slfcaled)
split(ms_in, ms_slfcaled,datacolumn='corrected')
</pre>
[[file:Slf.G0.png|thumb|center|500px|Figure 1: Phase before self-calibration]]
[[file:Slf.G1.png|thumb|center|500px|Figure 2: Phase after self-calibration]]

<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
# =========== Step 5: Generate final self-calibrated images (optional) =========
if doclean_slfcaled:
import glob
pol='XX'
print('Processing ' + trange)
img_final=imagedir_slfcaled+'/slf_final_{0}_t0'.format(pol)
vis = ms_slfcaled
spws=[str(s+1) for s in range(30)]
tb.open(vis+'/SPECTRAL_WINDOW')
reffreqs=tb.getcol('REF_FREQUENCY')
bdwds=tb.getcol('TOTAL_BANDWIDTH')
cfreqs=reffreqs+bdwds/2.
tb.close()
sbeam=30.
from matplotlib import gridspec as gridspec
from sunpy import map as smap
from matplotlib import pyplot as plt
fitsfiles=[]
for s,sp in enumerate(spws):
cfreq=cfreqs[int(sp)]
bm=max(sbeam*cfreqs[1]/cfreq,4.)
imname=img_final+'_s'+sp.zfill(2)
fitsfile=imname+'.fits'
if not os.path.exists(fitsfile):
print 'cleaning spw {0:s} with beam size {1:.1f}"'.format(sp,bm)
try:
tclean(vis=vis,
antenna=antennas,
imagename=imname,
spw=sp,
specmode='mfs',
timerange=trange,
imsize=[npix],
cell=['1arcsec'],
niter=1000,
gain=0.05,
stokes=pol,
weighting='briggs',
robust=2.0,
restoringbeam=[str(bm)+'arcsec'],
phasecenter=phasecenter,
mask='',
pbcor=True,
interactive=False)
except:
print 'cleaning spw '+sp+' unsuccessful. Proceed to next spw'
continue
if os.path.exists(imname+'.image.pbcor'):
imn = imname+'.image.pbcor'
hf.imreg(vis=vis,imagefile=imn,fitsfile=fitsfile,
timerange=trange,usephacenter=False,toTb=True,verbose=False)
fitsfiles.append(fitsfile)
junks=['.flux','.model','.psf','.residual','.mask','.image','.pb','.image.pbcor','.sumwt'] #Do not run the next 4 lines if needed to view and assess the subset of clean process images
for junk in junks:
if os.path.exists(imname+junk):
os.system('rm -rf '+imname+junk)
else:
print('fits file '+fitsfile+' already exists, skip clean...')
fitsfiles.append(fitsfile)

fig = plt.figure(figsize=(8.4,7.)) # fig = plt.figure(figsize=(14, 7)) # For post-2020 data (50 spws)
gs = gridspec.GridSpec(5, 6) # gs = gridspec.GridSpec(5, 10) # For post-2020 data (50 spws)
for s,sp in enumerate(spws):
cfreq=cfreqs[int(sp)]
ax = fig.add_subplot(gs[s])
eomap=smap.Map(fitsfiles[s])
eomap.plot_settings['cmap'] = plt.get_cmap('jet')
eomap.plot(axes = ax)
eomap.draw_limb()
ax.set_title(' ')
ax.get_xaxis().set_visible(False)
ax.get_yaxis().set_visible(False)
ax.set_xlim(xran)
ax.set_ylim(yran)
plt.text(0.98,0.85,'{0:.1f} GHz'.format(cfreq/1e9),transform=ax.transAxes,ha='right',color='w',fontweight='bold')
plt.subplots_adjust(left=0, bottom=0, right=1, top=1, wspace=0, hspace=0)
plt.show()

</pre>
The .fits files of the self calibrated images at each frequency for the given time are saved at /slfcal/images_slfcaled in your working directory.

[[file:Slf_t0_n0.png|thumb|center|500px|Figure 3: Multi-frequency images before self-calibration]]
[[file:Slf_t0_n1.png|thumb|center|500px|Figure 4: Multi-frequency images after self-calibration]]

====Step 5: Quick-look imaging ==== 
For spectral imaging analysis of the event, follow [http://www.ovsa.njit.edu/wiki/index.php/EOVSA_Data_Analysis_Tutorial#Spectral_Imaging_with_SunCASA this tutorial] using the self-calibrated data obtained from the previous step or use [https://colab.research.google.com/drive/1lSLLxgOG6b8kgu9Sk6kSKvrViyubnXG6?usp=sharing#scrollTo=xbXyyLmCFCGL this link].

<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
from suncasa.utils import qlookplot as ql ## (Optional) Supply the npz file of the dynamic spectrum from previous step.
## If not provided, the program will generate a new one from the visibility.
## set the time interval
from suncasa import dspec as ds
import time
visibility_data = 'IDB20170821202020_slfcaled.ms'
specfile = visibility_data + '.dspec.npz'
d = ds.Dspec(visibility_data, bl='4&9', specfile=specfile)

timerange = '19:02:00~19:02:10' ## select frequency range from 2.5 GHz to 3.5 GHz
spw = '2~5' ## select stokes XX
stokes = 'XX' ## turn off AIA image plotting, default is True
plotaia = False
xycen = [375, 45] ## image center for clean in solar X-Y in arcsec
cell=['2.0arcsec'] ## pixel size
imsize=[128] ## x and y image size in pixels.
fov = [100,100] ## field of view of the zoomed-in panels in unit of arcsec
spw = ['{}'.format(s) for s in range(1,31)]
clevels = [0.5, 1.0] ## contour levels to fill in between.
calpha=0.35 ## now tune down the alpha
restoringbeam=['6arcsec']
ql.qlookplot(vis=msfile, specfile=specfile, timerange=timerange, spw=spw, stokes=stokes, \
restoringbeam=restoringbeam,imsize=imsize,cell=cell, \
xycen=xycen,fov=fov,clevels=clevels,calpha=calpha)
</pre>

Page Last Updated: 06/29/2022

Tohban EOVSA Imaging Tutorial A-Z V2

2022-07-06T10:41:25Z

Schhabra:

This tutorial describes the step-by-step procedure to download EOVSA IDB data, obtain and calibrate the measurement sets (.ms), and, transfer them to the inti server for self-calibration and further analysis.

=== Pre-requisites ===
Accounts on Pipeline and Inti or Baozi servers. Sijie and Bin can help with the account set-up for pipeline, inti and boazi.

==== How to connect to pipeline ====
'''Windows:''' On a windows machine Mobaxterm can be used to connect to Pipeline. For details, see ...

'''Linux/Unix:''' On a linux/Mac the ssh commands can be used directly through the terminal:

<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
<syntaxhighlight lang="py">
ssh -L 5902:helios.solar.pvt:20000 guest@ovsa.njit.edu ### when prompted, enter password to guest@ovsa ###
ssh -XY your_username@pipeline ### add your username and enter password to your pipeline account, when prompted ###
</pre>

Alternatively, to avoid typing the longer commands, proxy commands can be added to the ~/.ssh/config file:

<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
vim ~/.ssh/config ### press 'i' to insert content and esc +':wq' to write and quit ###

Host *
ForwardAgent yes

Host *
ForwardX11 yes

Host ovsa
HostName ovsa.njit.edu
User guest

Host pipeline
Hostname pipeline.solar.pvt
User your_username ### Insert your pipeline username here ###
ProxyCommand ssh -W %h:%p guest@ovsa.njit.edu
</pre>

Upon saving the ~/.ssh/config file and restarting the terminal, you can connect to pipeline directly using:

<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
ssh -XY pipeline
</pre>

==== How to connect to the inti server ====
NOTE: To connect to inti you must have a NJIT UCID. The arcs have granted access to inti for all SRG members. If you encounter any issue with the login. Please let arcs folks (arcs@njit.edu) know.

'''Windows:''' As mentioned above, on a windows machine Mobaxterm can be used to connect to inti. For details, see ...

'''Linux/Unix:''' On a linux/Mac the ssh command can be used:
<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
ssh -XY UCID@inti.hpcnet.campus.njit.edu ### insert UCID here and enter your NJIT UCID password, when prompted ###
</pre>

When logging in for the first time, please add the following lines (demarcated by #) in your ~/.bashrc (on the inti server) file in order to use CASA on inti.

<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
#### setting start ####
if [ $HOSTNAME == "baozi.hpcnet.campus.njit.edu" ]; then
source /srg/.setenv_baozi
fi
if [ $HOSTNAME == "inti.hpcnet.campus.njit.edu" ]; then
source /inti/.setenv_inti
fi
if [ $HOSTNAME == "guko.resource.campus.njit.edu" ]; then
source /data/data/.setenv_guko
fi
#### setting end ####
</pre>
Proxy commands for inti can also be added to the ~/.ssh/config (on local machine) file for convenience:
<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
vim ~/.ssh/config ### press 'i' to insert content and esc +':wq' to write and quit ###

Host inti
Hostname inti.hpcnet.campus.njit.edu
User UCID ### insert NJIT UCID here ###
</pre>
after the proxy is set up, connect to inti by simply typing:
<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
ssh -XY inti ### enter your NJIT UCID password when prompted ###
</pre>

=== Downloading data and Preliminary Calibration ===

==== Step 1: Downloading raw (IDB) data on Pipeline machine====
The initial downloading of data and applying refcal and phasecal to the .ms files is performed on the pipeline since the complete EOVSA SQL database is stored here for now.

If you are not using mobaxterm, directly SSH to Pipeline through your Linux or Mac computer (as shown above) and '''start tcsh to run CASA'''.

The following commands (run within CASA) can be used to import a single IDB file if the given time-range is less than 10 mins.
<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
from astropy.time import Time
import os
trange = Time(['2017-08-21 20:15:00', '2017-08-21 20:25:00']) ###Change accordingly###

#### (Optional) change output path, default current directory "./" #######
outpath = './msdata/' ###Change accordingly###
if not os.path.exists(outpath):
os.makedirs(outpath)

msfiles = importeovsa(idbfiles=trange, ncpu=1, visprefix=outpath)
</pre>

'''NOTE:''' The above step currently gives an error with importeovsa. This is because the data location was changed and the code doesn't know where to look for the IDB files. Sijie is looking into the problem. We will update the page when it is fixed. For now, please use the method below for all time ranges.

OR (for a time range longer than 10 minutes)

<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
from suncasa.tasks import task_calibeovsa as calibeovsa
from suncasa.tasks import task_importeovsa as timporteovsa
from split_cli import split_cli as split
import dump_tsys as dt
from util import Time
import numpy as np
import os
from glob import glob
from eovsapy import util

trange = Time(['2017-08-21 20:15:00', '2017-08-21 20:35:00']) ###Change accordingly###
idbdir = util.get_idbdir(trange[0])

info = dt.rd_fdb(trange[0])
for k, v in info.iteritems():
info[k] = info[k][~(info[k] == '')]

sidx = np.where(
np.logical_and(info['SOURCEID'] == 'Sun', info['PROJECTID'] == 'NormalObserving') & np.logical_and(
info['ST_TS'].astype(np.float) >= trange[0].lv,
info['ST_TS'].astype(np.float) <= trange[
1].lv))
filelist = info['FILE'][sidx]

outpath = './msdata/' ###Change accordingly###
if not os.path.exists(outpath):
os.makedirs(outpath)
inpath = idbdir + '{}/'.format(trange[0].datetime.strftime("%Y%m%d"))
ncpu = 1

msfiles = timporteovsa.importeovsa(idbfiles=[inpath + ll for ll in filelist], ncpu=ncpu, timebin="0s", width=1,
visprefix=outpath,
nocreatms=False, doconcat=False,
modelms="", doscaling=False, keep_nsclms=False, udb_corr=True)
</pre>

If you come across errors with calibeovsa, add the following lines to your ~/.casa/init.py (within pipeline) file.
<pre style="background-color: #FCEBD9">
vim ~/.ssh/config ### press 'i' to insert content and esc +':wq' to write and quit ###
import sys
sys.path.append('/common/python')
sys.path.append('/common/python/packages/pipeline_casa')
execfile('/common/python/suncasa/tasks/mytasks.py')
</pre>

==== Step 2: Concatenate multiple .ms files and Calibrate====
'''Follow this step if there are more than one .ms files for the chosen time-range. If not, move to Step 3.'''

<pre style="background-color: #FCEBD9">
concatvis = os.path.basename(msfiles[0])[:11] + '_concat_cal.ms' ### Set the path/name for the concatenated files ###

vis = calibeovsa.calibeovsa(msfiles, doconcat=True, concatvis=concatvis, caltype=['refpha','phacal'], doimage=True)
### If doimage=True, a quicklook image will be produced (by integrating over the entire time) as shown below ###

vis_str = str(' '.join(vis))
caled_vis=vis_str.replace('.ms','_cal.ms') ### Append '_cal' to the ms filename ###
split(vis=' '.join(vis),outputvis=caled_vis,datacolumn='corrected',timerange='',correlation='XX') ### Split the corrected column to the new 'caled' ms ###
</pre>

[[file:Figure1_imagingtutorial.png|frame|center|800px|'''Figure 1:''' Quick-look full-Sun image after the initial calibration.]]

==== Step 3: Calibration ====
For a single .ms file from Step 1, this will calibrate the input visibility, write out calibration tables under /data1/eovsa/caltable/, and apply the calibration.
<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
vis = calibeovsa.calibeovsa('IDB20170821202020.ms', caltype=['refpha','phacal'], doimage=True) ###Change the vis filename accordingly###

vis_str = str(' '.join(vis))
caled_vis=vis_str.replace('.ms','_cal.ms') ### Append '_cal' to the ms filename ###
split(vis=' '.join(vis),outputvis=caled_vis,datacolumn='corrected',timerange='',correlation='XX') ### Split the corrected column to the new 'caled' ms ###
</pre>

One needs to transfer the created 'caled' ms data files (xxx_concat_cal.ms or xxx_cal.ms) from pipeline to inti server, which has all the casa tasks installed, in order to run the rest of the imaging steps.

=== Connection details to Inti server ===
For Windows, on Mobaxterm,

#With your NJIT VPN connected, connect to one of the afsconnect servers (for example, afsaccess3.njit.edu) using your UCID and password.
#ssh -X UCID@inti.hpcnet.campus.njit.edu

To avoid typing the full inti address each time you attempt for ssh, you may wish to add the following lines with your username to C:\Program Files\Git\etc\ssh\ssh_config on Windows and /.ssh/config on Mac and Linux machines.

<pre style="background-color: #FCEBD9">
Host inti
Hostname inti.hpcnet.campus.njit.edu
User USERNAME ###Insert UCID/inti username here###
</pre>

To have sufficient disk space with EOVSA data analysis over Inti, use your dedicated directory YOURDIRECTORY at the location given below. If you do not have a directory, Please take help from Sijie in creating one.
<pre style="background-color: #FCEBD9">
cd /inti/data/users/YOURDIRECTORY
</pre>

For Linux or Mac machine,
ssh -X UCID@inti.hpcnet.campus.njit.edu

=== Transferring data files between servers ===
For directly transferring your calibrated .ms data between the Pipeline and Inti servers, follow the below given steps.
<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
1. Log into Inti using your username.
ssh -X USERNAME@inti.hpcnet.campus.njit.edu

Then create a tunnel into Pipeline from Inti.
ssh -L 8888:pipeline.solar.pvt:22 guest@ovsa.njit.edu

2. Log into Inti again from a new terminal.

Change to your working directory and give this command to copy your data on Pipeline.
scp -v -C -r -P 8888 USERNAMEofPipeline@localhost:PATHofMSDATA/MSfilename ./
Eg: scp -v -C -r -P 8888 shaheda@localhost:/data1/shaheda/IDB20220118173922_cal.ms ./
where, MSfilename, PATHofMSDATA are your .ms data and its path on Pipeline machine.
</pre>

Alternatively, one can follow the below procedure to do the transfer.

For Windows, on mobaxterm, drag and drop the ms file to your local machine or use scp command as given below.

<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
scp -r -C userid@pipeline:/your/folder/msfile /localmachine/destination/folder/
</pre>

On mobaxterm and on your local terminal, use the following command to finally copy the ms file to Inti.

<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
scp -r -C /localmachine/destination/folder/msfile UCID@inti.hpcnet.campus.njit.edu:/inti/data/users/YOURDIRECTORY
</pre>

For Mac and Linux, SCP can be used in the same way. Add the following lines to your SSH config file, to bypass ovsa.njit.edu from copying.
vi ~/.ssh/config
<pre style="background-color: #FCEBD9">
Host ovsa
HostName ovsa.njit.edu
User guest
Host pipeline
Hostname pipeline.solar.pvt
User userid
ProxyCommand ssh -W %h:%p guest@ovsa.njit.edu
</pre>

=== Software details on the servers ===
On Inti,
when logging in for the first time, please add the following lines to your accounts ~/.bashrc file.

>>vi ~/.bashrc
Insert the text given below and save it.

<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
#### setting start ####
if [ $HOSTNAME == "baozi.hpcnet.campus.njit.edu" ]; then
source /srg/.setenv_baozi
fi
if [ $HOSTNAME == "inti.hpcnet.campus.njit.edu" ]; then
source /inti/.setenv_inti
fi
if [ $HOSTNAME == "guko.resource.campus.njit.edu" ]; then
source /data/data/.setenv_guko
fi
#### setting end ####
</pre>

Both CASA 5 and 6 are available on Inti.

Enter the bash environment on inti, and load the desired casa environment.
To load CASA 5, enter bash environment by giving >>bash
<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
>> loadcasa5
>> suncasa #This should load the software making you ready for the analysis
</pre>

Or to load CASA 6, enter bash environment by giving >>bash
<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
>> loadcasa6
>> ipython #This should load the software making you ready for the analysis
</pre>
Here, for example, to use clean, first start ipython as given above, then type in >>from casatasks import tclean

====Step 4: Self-calibration====
[[file:Figure3_imagingtutorial.png|thumb|center|500px|Figure 2: Cotton-Schwab clean major and minor cycles. [Source: http://www.aoc.nrao.edu/~rurvashi/ImagingAlgorithmsInCasa/node2.html].]]
Follow the below given steps to run the self-calibration of the imaging data and produce the calibrated images in .fits format. https://github.com/binchensun/casa-eovsa/blob/master/slfcal_example.py

<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
from suncasa.utils import helioimage2fits as hf
import os
import numpy as np
import pickle
from matplotlib import gridspec as gridspec
from sunpy import map as smap
from matplotlib import pyplot as plt
from split_cli import split_cli as split
import time

# =========== task handlers =============
dofullsun = 1 # initial full-sun imaging ###Change accordingly###
domasks=1 # get masks ###Change accordingly###
doslfcal=1 # main cycle of doing selfcalibration ###Change accordingly###
doapply=1 # apply the results ###Change accordingly###
doclean_slfcaled=1 # perform clean for self-calibrated data ###Change accordingly###

# ============ declaring the working directories ============
workdir = os.getcwd()+'/' #main working directory. Using current directory in this example
slfcaldir = workdir+'slfcal/' #place to put all selfcalibration products
imagedir = slfcaldir+'images/' #place to put all selfcalibration images
maskdir = slfcaldir+'masks/' #place to put clean masks
imagedir_slfcaled = slfcaldir+'images_slfcaled/' #place to put final self-calibrated images
caltbdir = slfcaldir+'caltbs/' # place to put calibration tables
# make these directories if they do not already exist
dirs = [workdir, slfcaldir, imagedir, maskdir, imagedir_slfcaled, caltbdir]
for d in dirs:
if not os.path.exists(d):
os.makedirs(d)

# ============ Split a short time for self-calibration ===========
# input visibility
ms_in = workdir + 'IDB20170821202020_cal.ms' ###Change the initial calibrated (through calibeovsa) vis accordingly###
# output, selfcaled, visibility
ms_slfcaled = workdir + os.path.basename(ms_in).replace('cal','slfcaled')
# intermediate small visibility for selfcalbration
# selected time range for generating self-calibration solutions
trange='2017/08/21/20:21:10~2017/08/21/20:21:30' ###Change accordingly###
slfcalms = slfcaldir+'slfcalms.XX.slfcal'
slfcaledms = slfcaldir+'slfcalms.XX.slfcaled'
if not os.path.exists(slfcalms):
split(vis=ms_in,outputvis=slfcalms,datacolumn='data',timerange=trange,correlation='XX')

# ============ Prior definitions for spectral windows, antennas, pixel numbers =========
spws=[str(s+1) for s in range(30)] # spws=[str(s+1) for s in range(49)] # For post-2020 data
antennas='0~12&0~12'
npix=512
nround=3 #number of slfcal cycles

# =========== Step 1, doing a full-Sun image to find out phasecenter and appropriate field of view =========
if dofullsun:
#initial mfs clean to find out the image phase center
im_init='fullsun_init'
os.system('rm -rf '+im_init+'*')
tclean(vis=slfcalms,
antenna=antennas,
imagename=im_init,
spw='2~15',
specmode='mfs',
timerange=trange,
imsize=[npix],
cell=['5arcsec'],
niter=1000,
gain=0.05,
stokes='I',
restoringbeam=['30arcsec'],
interactive=False,
pbcor=True)

hf.imreg(vis=slfcalms,imagefile=im_init+'.image.pbcor',fitsfile=im_init+'.fits',
timerange=trange,usephacenter=False,verbose=True)
clnjunks = ['.flux', '.mask', '.model', '.psf', '.residual','.sumwt','.pb','.image'] #Do not run the next 4 lines if needed to view and assess the subset of clean process images
for clnjunk in clnjunks:
if os.path.exists(im_init + clnjunk):
os.system('rm -rf '+im_init + clnjunk)

from sunpy import map as smap
from matplotlib import pyplot as plt
fig = plt.figure(figsize=(6,6))
ax = fig.add_subplot(111)
eomap=smap.Map(im_init+'.fits')
#eomap.data=eomap.data.reshape((npix,npix))
eomap.plot_settings['cmap'] = plt.get_cmap('jet')
eomap.plot(axes = ax)
eomap.draw_limb()
plt.show()
viewer(im_init+'.image.pbcor')
</pre>
[[file:Figure2_imagingtutorial.png|thumb|center|500px|Figure 3: Full-Sun image after initial clean to find the flare location.]]

<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
# parameters specific to the event (found from step 1)
phasecenter='J2000 10h02m59 11d58m07' ###Change accordingly###
xran=[280,480] ###Change accordingly###
yran=[-50,150] ###Change accordingly###

# =========== Step 2 (optional), generate masks =========
# if skipped, will not use any masks
if domasks:
clearcal(slfcalms)
delmod(slfcalms)
antennas=antennas
pol='XX'
imgprefix=maskdir+'slf_t0'

# step 1: set up the clean masks
img_init=imgprefix+'_init_ar_'
os.system('rm -rf '+img_init+'*')
#spwrans_mask=['1~5','6~12','13~20','21~30']
spwrans_mask=['2~12']
#convert to a list of spws
spwrans_mask_list = [[str(i) for i in (np.arange(int(m.split('~')[0]),int(m.split('~')[1])))] for m in spwrans_mask] # Not used?
masks=[]
imnames=[]
for spwran in spwrans_mask:
imname=img_init+spwran.replace('~','-')
try:
tclean(vis=slfcalms,
antenna=antennas,
imagename=imname,
spw=spwran,
specmode='mfs',
timerange=trange,
imsize=[npix],
cell=['2arcsec'],
niter=1000,
gain=0.05,
stokes='XX',
restoringbeam=['20arcsec'],
phasecenter=phasecenter,
weighting='briggs',
robust=1.0,
interactive=True,
datacolumn='data',
pbcor=True,
savemodel='modelcolumn')
imnames.append(imname+'.image')
masks.append(imname+'.mask')
clnjunks = ['.flux', '.model', '.psf', '.residual'] #Do not run the next 4 lines if needed to view and assess the subset of clean process images
for clnjunk in clnjunks:
if os.path.exists(imname + clnjunk):
os.system('rm -rf '+ imname + clnjunk)
except:
print('error in cleaning spw: '+spwran)

pickle.dump(masks,open(slfcaldir+'masks.p','wb'))
</pre>
[[file:Figure4_imagingtutorial.PNG|thumb|center|800px|Figure 4: Interactive clean window to create masks over the source. The white outline surrounding the source is the mask selected by the polygon drawing option.]]

The outlines drawn for masks can be created by any of the icons with the letter 'R' in the Viewer window. The instructions for doing this can be found by hovering over those icons.

<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
# =========== Step 3, main step of selfcalibration =========
if doslfcal:
if os.path.exists(slfcaldir+'masks.p'):
masks=pickle.load(open(slfcaldir+'masks.p','rb'))
if not os.path.exists(slfcaldir+'masks.p'):
print 'masks do not exist. Use default mask'
masks=[]
os.system('rm -rf '+imagedir+'*')
os.system('rm -rf '+caltbdir+'*')
#first step: make a mock caltable for the entire database
print('Processing ' + trange)
slftbs=[]
calprefix=caltbdir+'slf'
imgprefix=imagedir+'slf'
tb.open(slfcalms+'/SPECTRAL_WINDOW')
reffreqs=tb.getcol('REF_FREQUENCY')
bdwds=tb.getcol('TOTAL_BANDWIDTH')
cfreqs=reffreqs+bdwds/2.
tb.close()
# starting beam size at 3.4 GHz in arcsec #Change accordingly
sbeam=40.
strtmp=[m.replace(':','') for m in trange.split('~')]
timestr='t'+strtmp[0]+'-'+strtmp[1]
refantenna='0'
# number of iterations for each round
niters=[100, 300, 500]
# roburst value for weighting the baselines
robusts=[1.0, 0.5, 0.0]
# apply calibration tables? Set to true for most cases
doapplycal=[1,1,1]
# modes for calibration, 'p' for phase-only, 'a' for amplitude only, 'ap' for both
calmodes=['p','p','a']
# setting uvranges for model image (optional, not used here)
uvranges=['','','']
for n in range(nround):
slfcal_tb_g= calprefix+'.G'+str(n)
fig = plt.figure(figsize=(8.4,7.)) # fig = plt.figure(figsize=(14, 7)) # For post-2020 data (50 spws)
gs = gridspec.GridSpec(5, 6) # gs = gridspec.GridSpec(5, 10) # For post-2020 data (50 spws)
for s,sp in enumerate(spws):
print 'processing spw: '+sp
cfreq=cfreqs[int(sp)]
# setting restoring beam size (not very useful for selfcal anyway, but just to see the results)
bm=max(sbeam*cfreqs[1]/cfreq, 6.)
slfcal_img = imgprefix+'.spw'+sp.zfill(2)+'.slfcal'+str(n)
# only the first round uses nearby spws for getting initial model
if n == 0:
spbg=max(int(sp)-2,1) # spbg=max(int(sp)-2,0) # For post-2020 data (50 spws)
sped=min(int(sp)+2,30) # sped=min(int(sp)+2,49) # For post-2020 data (50 spws)
spwran=str(spbg)+'~'+str(sped)
print('using spw {0:s} as model'.format(spwran))
if 'spwrans_mask' in vars():
for m, spwran_mask in enumerate(spwrans_mask):
if sp in spwran_mask:
mask = masks[m]
print('using mask {0:s}'.format(mask))
findmask = True
if not findmask:
print('mask not found. Do use any masks')
else:
spwran = sp
if 'spwrans_mask' in vars():
for m, spwran_mask in enumerate(spwrans_mask):
if sp in spwran_mask:
mask = masks[m]
print 'using mask {0:s}'.format(mask)
findmask = True
if not findmask:
print('mask not found. Do use any masks')
try:
tclean(vis=slfcalms,
antenna=antennas,
imagename=slfcal_img,
uvrange=uvranges[n],
spw=spwran,
specmode='mfs',
timerange=trange,
imsize=[npix],
cell=['2arcsec'],
niter=niters[n],
gain=0.05,
stokes='XX', #use pol XX image as the model
weighting='briggs',
robust=robusts[n],
phasecenter=phasecenter,
mask=mask,
restoringbeam=[str(bm)+'arcsec'],
pbcor=False,
interactive=False,
savemodel='modelcolumn')
if os.path.exists(slfcal_img+'.image'):
fitsfile=slfcal_img+'.fits'
hf.imreg(vis=slfcalms,imagefile=slfcal_img+'.image',fitsfile=fitsfile,
timerange=trange,usephacenter=False,toTb=True,verbose=False,overwrite=True)
clnjunks = ['.mask','.flux', '.model', '.psf', '.residual', '.image','.pb','.image.pbcor','.sumwt'] #Do not run the next 4 lines if needed to view and assess the subset of clean process images
for clnjunk in clnjunks:
if os.path.exists(slfcal_img + clnjunk):
os.system('rm -rf '+ slfcal_img + clnjunk)
ax = fig.add_subplot(gs[s])
eomap=smap.Map(fitsfile)
eomap.plot_settings['cmap'] = plt.get_cmap('jet')
eomap.plot(axes = ax)
eomap.draw_limb()
#eomap.draw_grid()
ax.set_title(' ')
ax.get_xaxis().set_visible(False)
ax.get_yaxis().set_visible(False)
ax.set_xlim(xran)
ax.set_ylim(yran)
plt.pause(0.5) # Allows viewing of each image as it is plotted.
os.system('rm -f '+ fitsfile)

except:
print 'error in cleaning spw: '+sp
print 'using nearby spws for initial model'
sp_e=int(sp)+2
sp_i=int(sp)-2
if sp_i < 1: # if sp < 0: # For post-2020 data (50 spws)
sp_i = 1 # sp_i = 0 # For post-2020 data (50 spws)
if sp_e > 30: # if sp_e > 49: # For post-2020 data (50 spws)
sp_e = 30 # sp_e = 49 # For post-2020 data (50 spws)
sp_=str(sp_i)+'~'+str(sp_e)
try:
tget(tclean)
spw=sp_
print('using spw {0:s} as model'.format(sp_))
tclean()
except:
print 'still not successful. abort...'
break

gaincal(vis=slfcalms, refant=refantenna,antenna=antennas,caltable=slfcal_tb_g,spw=sp, uvrange='',\
gaintable=[],selectdata=True,timerange=trange,solint='inf',gaintype='G',calmode=calmodes[n],\
combine='',minblperant=4,minsnr=2,append=True)
if not os.path.exists(slfcal_tb_g):
print 'No solution found in spw: '+sp
figname=imagedir+'slf_t0_n{:d}.png'.format(n)
plt.subplots_adjust(left=0, bottom=0, right=1, top=1, wspace=0, hspace=0)
plt.savefig(figname)
time.sleep(10)
plt.close()

if os.path.exists(slfcal_tb_g):
slftbs.append(slfcal_tb_g)
slftb=[slfcal_tb_g]
os.chdir(slfcaldir)
if calmodes[n] == 'p':
plotcal(caltable=slfcal_tb_g,antenna='1~12',xaxis='freq',yaxis='phase',\
subplot=431,plotrange=[-1,-1,-180,180],iteration='antenna',figfile=slfcal_tb_g+'.png',showgui=False)
if calmodes[n] == 'a':
plotcal(caltable=slfcal_tb_g,antenna='1~12',xaxis='freq',yaxis='amp',\
subplot=431,plotrange=[-1,-1,0,2.],iteration='antenna',figfile=slfcal_tb_g+'.png',showgui=False)
os.chdir(workdir)
plt.pause(0.5) # Allows viewing of the plot

if doapplycal[n]:
clearcal(slfcalms)
delmod(slfcalms)
applycal(vis=slfcalms,gaintable=slftb,spw=','.join(spws),selectdata=True,\
antenna=antennas,interp='nearest',flagbackup=False,applymode='calonly',calwt=False)

if n < nround-1:
prompt=raw_input('Continuing to selfcal?')
#prompt='y'
if prompt.lower() == 'n':
if os.path.exists(slfcaledms):
os.system('rm -rf '+slfcaledms)
split(slfcalms,slfcaledms,datacolumn='corrected')
print 'Final calibrated ms is {0:s}'.format(slfcaledms)
break
if prompt.lower() == 'y':
slfcalms_=slfcalms+str(n)
if os.path.exists(slfcalms_):
os.system('rm -rf '+slfcalms_)
split(slfcalms,slfcalms_,datacolumn='corrected')
slfcalms=slfcalms_
else:
if os.path.exists(slfcaledms):
os.system('rm -rf '+slfcaledms)
split(slfcalms,slfcaledms,datacolumn='corrected')
print 'Final calibrated ms is {0:s}'.format(slfcaledms)
</pre>

<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
# =========== Step 4: Apply self-calibration tables =========
if doapply:
import glob
os.chdir(workdir)
clearcal(ms_in)
clearcal(slfcalms)
applycal(vis=slfcalms,gaintable=slftbs,spw=','.join(spws),selectdata=True,\
antenna=antennas,interp='linear',flagbackup=False,applymode='calonly',calwt=False)
applycal(vis=ms_in,gaintable=slftbs,spw=','.join(spws),selectdata=True,\
antenna=antennas,interp='linear',flagbackup=False,applymode='calonly',calwt=False)
if os.path.exists(ms_slfcaled):
os.system('rm -rf '+ms_slfcaled)
split(ms_in, ms_slfcaled,datacolumn='corrected')
</pre>
[[file:Slf.G0.png|thumb|center|500px|Figure 1: Phase before self-calibration]]
[[file:Slf.G1.png|thumb|center|500px|Figure 2: Phase after self-calibration]]

<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
# =========== Step 5: Generate final self-calibrated images (optional) =========
if doclean_slfcaled:
import glob
pol='XX'
print('Processing ' + trange)
img_final=imagedir_slfcaled+'/slf_final_{0}_t0'.format(pol)
vis = ms_slfcaled
spws=[str(s+1) for s in range(30)]
tb.open(vis+'/SPECTRAL_WINDOW')
reffreqs=tb.getcol('REF_FREQUENCY')
bdwds=tb.getcol('TOTAL_BANDWIDTH')
cfreqs=reffreqs+bdwds/2.
tb.close()
sbeam=30.
from matplotlib import gridspec as gridspec
from sunpy import map as smap
from matplotlib import pyplot as plt
fitsfiles=[]
for s,sp in enumerate(spws):
cfreq=cfreqs[int(sp)]
bm=max(sbeam*cfreqs[1]/cfreq,4.)
imname=img_final+'_s'+sp.zfill(2)
fitsfile=imname+'.fits'
if not os.path.exists(fitsfile):
print 'cleaning spw {0:s} with beam size {1:.1f}"'.format(sp,bm)
try:
tclean(vis=vis,
antenna=antennas,
imagename=imname,
spw=sp,
specmode='mfs',
timerange=trange,
imsize=[npix],
cell=['1arcsec'],
niter=1000,
gain=0.05,
stokes=pol,
weighting='briggs',
robust=2.0,
restoringbeam=[str(bm)+'arcsec'],
phasecenter=phasecenter,
mask='',
pbcor=True,
interactive=False)
except:
print 'cleaning spw '+sp+' unsuccessful. Proceed to next spw'
continue
if os.path.exists(imname+'.image.pbcor'):
imn = imname+'.image.pbcor'
hf.imreg(vis=vis,imagefile=imn,fitsfile=fitsfile,
timerange=trange,usephacenter=False,toTb=True,verbose=False)
fitsfiles.append(fitsfile)
junks=['.flux','.model','.psf','.residual','.mask','.image','.pb','.image.pbcor','.sumwt'] #Do not run the next 4 lines if needed to view and assess the subset of clean process images
for junk in junks:
if os.path.exists(imname+junk):
os.system('rm -rf '+imname+junk)
else:
print('fits file '+fitsfile+' already exists, skip clean...')
fitsfiles.append(fitsfile)

fig = plt.figure(figsize=(8.4,7.)) # fig = plt.figure(figsize=(14, 7)) # For post-2020 data (50 spws)
gs = gridspec.GridSpec(5, 6) # gs = gridspec.GridSpec(5, 10) # For post-2020 data (50 spws)
for s,sp in enumerate(spws):
cfreq=cfreqs[int(sp)]
ax = fig.add_subplot(gs[s])
eomap=smap.Map(fitsfiles[s])
eomap.plot_settings['cmap'] = plt.get_cmap('jet')
eomap.plot(axes = ax)
eomap.draw_limb()
ax.set_title(' ')
ax.get_xaxis().set_visible(False)
ax.get_yaxis().set_visible(False)
ax.set_xlim(xran)
ax.set_ylim(yran)
plt.text(0.98,0.85,'{0:.1f} GHz'.format(cfreq/1e9),transform=ax.transAxes,ha='right',color='w',fontweight='bold')
plt.subplots_adjust(left=0, bottom=0, right=1, top=1, wspace=0, hspace=0)
plt.show()

</pre>
The .fits files of the self calibrated images at each frequency for the given time are saved at /slfcal/images_slfcaled in your working directory.

[[file:Slf_t0_n0.png|thumb|center|500px|Figure 3: Multi-frequency images before self-calibration]]
[[file:Slf_t0_n1.png|thumb|center|500px|Figure 4: Multi-frequency images after self-calibration]]

====Step 5: Quick-look imaging ==== 
For spectral imaging analysis of the event, follow [http://www.ovsa.njit.edu/wiki/index.php/EOVSA_Data_Analysis_Tutorial#Spectral_Imaging_with_SunCASA this tutorial] using the self-calibrated data obtained from the previous step or use [https://colab.research.google.com/drive/1lSLLxgOG6b8kgu9Sk6kSKvrViyubnXG6?usp=sharing#scrollTo=xbXyyLmCFCGL this link].

<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
from suncasa.utils import qlookplot as ql ## (Optional) Supply the npz file of the dynamic spectrum from previous step.
## If not provided, the program will generate a new one from the visibility.
## set the time interval
from suncasa import dspec as ds
import time
visibility_data = 'IDB20170821202020_slfcaled.ms'
specfile = visibility_data + '.dspec.npz'
d = ds.Dspec(visibility_data, bl='4&9', specfile=specfile)

timerange = '19:02:00~19:02:10' ## select frequency range from 2.5 GHz to 3.5 GHz
spw = '2~5' ## select stokes XX
stokes = 'XX' ## turn off AIA image plotting, default is True
plotaia = False
xycen = [375, 45] ## image center for clean in solar X-Y in arcsec
cell=['2.0arcsec'] ## pixel size
imsize=[128] ## x and y image size in pixels.
fov = [100,100] ## field of view of the zoomed-in panels in unit of arcsec
spw = ['{}'.format(s) for s in range(1,31)]
clevels = [0.5, 1.0] ## contour levels to fill in between.
calpha=0.35 ## now tune down the alpha
restoringbeam=['6arcsec']
ql.qlookplot(vis=msfile, specfile=specfile, timerange=timerange, spw=spw, stokes=stokes, \
restoringbeam=restoringbeam,imsize=imsize,cell=cell, \
xycen=xycen,fov=fov,clevels=clevels,calpha=calpha)
</pre>

Page Last Updated: 06/29/2022

Tohban EOVSA Imaging Tutorial A-Z V2

2022-07-06T10:41:10Z

Schhabra:

This tutorial describes the step-by-step procedure to download EOVSA IDB data, obtain and calibrate the measurement sets (.ms), and, transfer them to the inti server for self-calibration and further analysis.

=== Pre-requisites ===
Accounts on Pipeline and Inti or Baozi servers. Sijie and Bin can help with the account set-up for pipeline, inti and boazi.

==== How to connect to pipeline ====
'''Windows:''' On a windows machine Mobaxterm can be used to connect to Pipeline. For details, see ...

'''Linux/Unix:''' On a linux/Mac the ssh commands can be used directly through the terminal:

<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
<syntaxhighlight lang="pyt">
ssh -L 5902:helios.solar.pvt:20000 guest@ovsa.njit.edu ### when prompted, enter password to guest@ovsa ###
ssh -XY your_username@pipeline ### add your username and enter password to your pipeline account, when prompted ###
</pre>

Alternatively, to avoid typing the longer commands, proxy commands can be added to the ~/.ssh/config file:

<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
vim ~/.ssh/config ### press 'i' to insert content and esc +':wq' to write and quit ###

Host *
ForwardAgent yes

Host *
ForwardX11 yes

Host ovsa
HostName ovsa.njit.edu
User guest

Host pipeline
Hostname pipeline.solar.pvt
User your_username ### Insert your pipeline username here ###
ProxyCommand ssh -W %h:%p guest@ovsa.njit.edu
</pre>

Upon saving the ~/.ssh/config file and restarting the terminal, you can connect to pipeline directly using:

<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
ssh -XY pipeline
</pre>

==== How to connect to the inti server ====
NOTE: To connect to inti you must have a NJIT UCID. The arcs have granted access to inti for all SRG members. If you encounter any issue with the login. Please let arcs folks (arcs@njit.edu) know.

'''Windows:''' As mentioned above, on a windows machine Mobaxterm can be used to connect to inti. For details, see ...

'''Linux/Unix:''' On a linux/Mac the ssh command can be used:
<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
ssh -XY UCID@inti.hpcnet.campus.njit.edu ### insert UCID here and enter your NJIT UCID password, when prompted ###
</pre>

When logging in for the first time, please add the following lines (demarcated by #) in your ~/.bashrc (on the inti server) file in order to use CASA on inti.

<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
#### setting start ####
if [ $HOSTNAME == "baozi.hpcnet.campus.njit.edu" ]; then
source /srg/.setenv_baozi
fi
if [ $HOSTNAME == "inti.hpcnet.campus.njit.edu" ]; then
source /inti/.setenv_inti
fi
if [ $HOSTNAME == "guko.resource.campus.njit.edu" ]; then
source /data/data/.setenv_guko
fi
#### setting end ####
</pre>
Proxy commands for inti can also be added to the ~/.ssh/config (on local machine) file for convenience:
<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
vim ~/.ssh/config ### press 'i' to insert content and esc +':wq' to write and quit ###

Host inti
Hostname inti.hpcnet.campus.njit.edu
User UCID ### insert NJIT UCID here ###
</pre>
after the proxy is set up, connect to inti by simply typing:
<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
ssh -XY inti ### enter your NJIT UCID password when prompted ###
</pre>

=== Downloading data and Preliminary Calibration ===

==== Step 1: Downloading raw (IDB) data on Pipeline machine====
The initial downloading of data and applying refcal and phasecal to the .ms files is performed on the pipeline since the complete EOVSA SQL database is stored here for now.

If you are not using mobaxterm, directly SSH to Pipeline through your Linux or Mac computer (as shown above) and '''start tcsh to run CASA'''.

The following commands (run within CASA) can be used to import a single IDB file if the given time-range is less than 10 mins.
<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
from astropy.time import Time
import os
trange = Time(['2017-08-21 20:15:00', '2017-08-21 20:25:00']) ###Change accordingly###

#### (Optional) change output path, default current directory "./" #######
outpath = './msdata/' ###Change accordingly###
if not os.path.exists(outpath):
os.makedirs(outpath)

msfiles = importeovsa(idbfiles=trange, ncpu=1, visprefix=outpath)
</pre>

'''NOTE:''' The above step currently gives an error with importeovsa. This is because the data location was changed and the code doesn't know where to look for the IDB files. Sijie is looking into the problem. We will update the page when it is fixed. For now, please use the method below for all time ranges.

OR (for a time range longer than 10 minutes)

<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
from suncasa.tasks import task_calibeovsa as calibeovsa
from suncasa.tasks import task_importeovsa as timporteovsa
from split_cli import split_cli as split
import dump_tsys as dt
from util import Time
import numpy as np
import os
from glob import glob
from eovsapy import util

trange = Time(['2017-08-21 20:15:00', '2017-08-21 20:35:00']) ###Change accordingly###
idbdir = util.get_idbdir(trange[0])

info = dt.rd_fdb(trange[0])
for k, v in info.iteritems():
info[k] = info[k][~(info[k] == '')]

sidx = np.where(
np.logical_and(info['SOURCEID'] == 'Sun', info['PROJECTID'] == 'NormalObserving') & np.logical_and(
info['ST_TS'].astype(np.float) >= trange[0].lv,
info['ST_TS'].astype(np.float) <= trange[
1].lv))
filelist = info['FILE'][sidx]

outpath = './msdata/' ###Change accordingly###
if not os.path.exists(outpath):
os.makedirs(outpath)
inpath = idbdir + '{}/'.format(trange[0].datetime.strftime("%Y%m%d"))
ncpu = 1

msfiles = timporteovsa.importeovsa(idbfiles=[inpath + ll for ll in filelist], ncpu=ncpu, timebin="0s", width=1,
visprefix=outpath,
nocreatms=False, doconcat=False,
modelms="", doscaling=False, keep_nsclms=False, udb_corr=True)
</pre>

If you come across errors with calibeovsa, add the following lines to your ~/.casa/init.py (within pipeline) file.
<pre style="background-color: #FCEBD9">
vim ~/.ssh/config ### press 'i' to insert content and esc +':wq' to write and quit ###
import sys
sys.path.append('/common/python')
sys.path.append('/common/python/packages/pipeline_casa')
execfile('/common/python/suncasa/tasks/mytasks.py')
</pre>

==== Step 2: Concatenate multiple .ms files and Calibrate====
'''Follow this step if there are more than one .ms files for the chosen time-range. If not, move to Step 3.'''

<pre style="background-color: #FCEBD9">
concatvis = os.path.basename(msfiles[0])[:11] + '_concat_cal.ms' ### Set the path/name for the concatenated files ###

vis = calibeovsa.calibeovsa(msfiles, doconcat=True, concatvis=concatvis, caltype=['refpha','phacal'], doimage=True)
### If doimage=True, a quicklook image will be produced (by integrating over the entire time) as shown below ###

vis_str = str(' '.join(vis))
caled_vis=vis_str.replace('.ms','_cal.ms') ### Append '_cal' to the ms filename ###
split(vis=' '.join(vis),outputvis=caled_vis,datacolumn='corrected',timerange='',correlation='XX') ### Split the corrected column to the new 'caled' ms ###
</pre>

[[file:Figure1_imagingtutorial.png|frame|center|800px|'''Figure 1:''' Quick-look full-Sun image after the initial calibration.]]

==== Step 3: Calibration ====
For a single .ms file from Step 1, this will calibrate the input visibility, write out calibration tables under /data1/eovsa/caltable/, and apply the calibration.
<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
vis = calibeovsa.calibeovsa('IDB20170821202020.ms', caltype=['refpha','phacal'], doimage=True) ###Change the vis filename accordingly###

vis_str = str(' '.join(vis))
caled_vis=vis_str.replace('.ms','_cal.ms') ### Append '_cal' to the ms filename ###
split(vis=' '.join(vis),outputvis=caled_vis,datacolumn='corrected',timerange='',correlation='XX') ### Split the corrected column to the new 'caled' ms ###
</pre>

One needs to transfer the created 'caled' ms data files (xxx_concat_cal.ms or xxx_cal.ms) from pipeline to inti server, which has all the casa tasks installed, in order to run the rest of the imaging steps.

=== Connection details to Inti server ===
For Windows, on Mobaxterm,

#With your NJIT VPN connected, connect to one of the afsconnect servers (for example, afsaccess3.njit.edu) using your UCID and password.
#ssh -X UCID@inti.hpcnet.campus.njit.edu

To avoid typing the full inti address each time you attempt for ssh, you may wish to add the following lines with your username to C:\Program Files\Git\etc\ssh\ssh_config on Windows and /.ssh/config on Mac and Linux machines.

<pre style="background-color: #FCEBD9">
Host inti
Hostname inti.hpcnet.campus.njit.edu
User USERNAME ###Insert UCID/inti username here###
</pre>

To have sufficient disk space with EOVSA data analysis over Inti, use your dedicated directory YOURDIRECTORY at the location given below. If you do not have a directory, Please take help from Sijie in creating one.
<pre style="background-color: #FCEBD9">
cd /inti/data/users/YOURDIRECTORY
</pre>

For Linux or Mac machine,
ssh -X UCID@inti.hpcnet.campus.njit.edu

=== Transferring data files between servers ===
For directly transferring your calibrated .ms data between the Pipeline and Inti servers, follow the below given steps.
<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
1. Log into Inti using your username.
ssh -X USERNAME@inti.hpcnet.campus.njit.edu

Then create a tunnel into Pipeline from Inti.
ssh -L 8888:pipeline.solar.pvt:22 guest@ovsa.njit.edu

2. Log into Inti again from a new terminal.

Change to your working directory and give this command to copy your data on Pipeline.
scp -v -C -r -P 8888 USERNAMEofPipeline@localhost:PATHofMSDATA/MSfilename ./
Eg: scp -v -C -r -P 8888 shaheda@localhost:/data1/shaheda/IDB20220118173922_cal.ms ./
where, MSfilename, PATHofMSDATA are your .ms data and its path on Pipeline machine.
</pre>

Alternatively, one can follow the below procedure to do the transfer.

For Windows, on mobaxterm, drag and drop the ms file to your local machine or use scp command as given below.

<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
scp -r -C userid@pipeline:/your/folder/msfile /localmachine/destination/folder/
</pre>

On mobaxterm and on your local terminal, use the following command to finally copy the ms file to Inti.

<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
scp -r -C /localmachine/destination/folder/msfile UCID@inti.hpcnet.campus.njit.edu:/inti/data/users/YOURDIRECTORY
</pre>

For Mac and Linux, SCP can be used in the same way. Add the following lines to your SSH config file, to bypass ovsa.njit.edu from copying.
vi ~/.ssh/config
<pre style="background-color: #FCEBD9">
Host ovsa
HostName ovsa.njit.edu
User guest
Host pipeline
Hostname pipeline.solar.pvt
User userid
ProxyCommand ssh -W %h:%p guest@ovsa.njit.edu
</pre>

=== Software details on the servers ===
On Inti,
when logging in for the first time, please add the following lines to your accounts ~/.bashrc file.

>>vi ~/.bashrc
Insert the text given below and save it.

<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
#### setting start ####
if [ $HOSTNAME == "baozi.hpcnet.campus.njit.edu" ]; then
source /srg/.setenv_baozi
fi
if [ $HOSTNAME == "inti.hpcnet.campus.njit.edu" ]; then
source /inti/.setenv_inti
fi
if [ $HOSTNAME == "guko.resource.campus.njit.edu" ]; then
source /data/data/.setenv_guko
fi
#### setting end ####
</pre>

Both CASA 5 and 6 are available on Inti.

Enter the bash environment on inti, and load the desired casa environment.
To load CASA 5, enter bash environment by giving >>bash
<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
>> loadcasa5
>> suncasa #This should load the software making you ready for the analysis
</pre>

Or to load CASA 6, enter bash environment by giving >>bash
<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
>> loadcasa6
>> ipython #This should load the software making you ready for the analysis
</pre>
Here, for example, to use clean, first start ipython as given above, then type in >>from casatasks import tclean

====Step 4: Self-calibration====
[[file:Figure3_imagingtutorial.png|thumb|center|500px|Figure 2: Cotton-Schwab clean major and minor cycles. [Source: http://www.aoc.nrao.edu/~rurvashi/ImagingAlgorithmsInCasa/node2.html].]]
Follow the below given steps to run the self-calibration of the imaging data and produce the calibrated images in .fits format. https://github.com/binchensun/casa-eovsa/blob/master/slfcal_example.py

<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
from suncasa.utils import helioimage2fits as hf
import os
import numpy as np
import pickle
from matplotlib import gridspec as gridspec
from sunpy import map as smap
from matplotlib import pyplot as plt
from split_cli import split_cli as split
import time

# =========== task handlers =============
dofullsun = 1 # initial full-sun imaging ###Change accordingly###
domasks=1 # get masks ###Change accordingly###
doslfcal=1 # main cycle of doing selfcalibration ###Change accordingly###
doapply=1 # apply the results ###Change accordingly###
doclean_slfcaled=1 # perform clean for self-calibrated data ###Change accordingly###

# ============ declaring the working directories ============
workdir = os.getcwd()+'/' #main working directory. Using current directory in this example
slfcaldir = workdir+'slfcal/' #place to put all selfcalibration products
imagedir = slfcaldir+'images/' #place to put all selfcalibration images
maskdir = slfcaldir+'masks/' #place to put clean masks
imagedir_slfcaled = slfcaldir+'images_slfcaled/' #place to put final self-calibrated images
caltbdir = slfcaldir+'caltbs/' # place to put calibration tables
# make these directories if they do not already exist
dirs = [workdir, slfcaldir, imagedir, maskdir, imagedir_slfcaled, caltbdir]
for d in dirs:
if not os.path.exists(d):
os.makedirs(d)

# ============ Split a short time for self-calibration ===========
# input visibility
ms_in = workdir + 'IDB20170821202020_cal.ms' ###Change the initial calibrated (through calibeovsa) vis accordingly###
# output, selfcaled, visibility
ms_slfcaled = workdir + os.path.basename(ms_in).replace('cal','slfcaled')
# intermediate small visibility for selfcalbration
# selected time range for generating self-calibration solutions
trange='2017/08/21/20:21:10~2017/08/21/20:21:30' ###Change accordingly###
slfcalms = slfcaldir+'slfcalms.XX.slfcal'
slfcaledms = slfcaldir+'slfcalms.XX.slfcaled'
if not os.path.exists(slfcalms):
split(vis=ms_in,outputvis=slfcalms,datacolumn='data',timerange=trange,correlation='XX')

# ============ Prior definitions for spectral windows, antennas, pixel numbers =========
spws=[str(s+1) for s in range(30)] # spws=[str(s+1) for s in range(49)] # For post-2020 data
antennas='0~12&0~12'
npix=512
nround=3 #number of slfcal cycles

# =========== Step 1, doing a full-Sun image to find out phasecenter and appropriate field of view =========
if dofullsun:
#initial mfs clean to find out the image phase center
im_init='fullsun_init'
os.system('rm -rf '+im_init+'*')
tclean(vis=slfcalms,
antenna=antennas,
imagename=im_init,
spw='2~15',
specmode='mfs',
timerange=trange,
imsize=[npix],
cell=['5arcsec'],
niter=1000,
gain=0.05,
stokes='I',
restoringbeam=['30arcsec'],
interactive=False,
pbcor=True)

hf.imreg(vis=slfcalms,imagefile=im_init+'.image.pbcor',fitsfile=im_init+'.fits',
timerange=trange,usephacenter=False,verbose=True)
clnjunks = ['.flux', '.mask', '.model', '.psf', '.residual','.sumwt','.pb','.image'] #Do not run the next 4 lines if needed to view and assess the subset of clean process images
for clnjunk in clnjunks:
if os.path.exists(im_init + clnjunk):
os.system('rm -rf '+im_init + clnjunk)

from sunpy import map as smap
from matplotlib import pyplot as plt
fig = plt.figure(figsize=(6,6))
ax = fig.add_subplot(111)
eomap=smap.Map(im_init+'.fits')
#eomap.data=eomap.data.reshape((npix,npix))
eomap.plot_settings['cmap'] = plt.get_cmap('jet')
eomap.plot(axes = ax)
eomap.draw_limb()
plt.show()
viewer(im_init+'.image.pbcor')
</pre>
[[file:Figure2_imagingtutorial.png|thumb|center|500px|Figure 3: Full-Sun image after initial clean to find the flare location.]]

<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
# parameters specific to the event (found from step 1)
phasecenter='J2000 10h02m59 11d58m07' ###Change accordingly###
xran=[280,480] ###Change accordingly###
yran=[-50,150] ###Change accordingly###

# =========== Step 2 (optional), generate masks =========
# if skipped, will not use any masks
if domasks:
clearcal(slfcalms)
delmod(slfcalms)
antennas=antennas
pol='XX'
imgprefix=maskdir+'slf_t0'

# step 1: set up the clean masks
img_init=imgprefix+'_init_ar_'
os.system('rm -rf '+img_init+'*')
#spwrans_mask=['1~5','6~12','13~20','21~30']
spwrans_mask=['2~12']
#convert to a list of spws
spwrans_mask_list = [[str(i) for i in (np.arange(int(m.split('~')[0]),int(m.split('~')[1])))] for m in spwrans_mask] # Not used?
masks=[]
imnames=[]
for spwran in spwrans_mask:
imname=img_init+spwran.replace('~','-')
try:
tclean(vis=slfcalms,
antenna=antennas,
imagename=imname,
spw=spwran,
specmode='mfs',
timerange=trange,
imsize=[npix],
cell=['2arcsec'],
niter=1000,
gain=0.05,
stokes='XX',
restoringbeam=['20arcsec'],
phasecenter=phasecenter,
weighting='briggs',
robust=1.0,
interactive=True,
datacolumn='data',
pbcor=True,
savemodel='modelcolumn')
imnames.append(imname+'.image')
masks.append(imname+'.mask')
clnjunks = ['.flux', '.model', '.psf', '.residual'] #Do not run the next 4 lines if needed to view and assess the subset of clean process images
for clnjunk in clnjunks:
if os.path.exists(imname + clnjunk):
os.system('rm -rf '+ imname + clnjunk)
except:
print('error in cleaning spw: '+spwran)

pickle.dump(masks,open(slfcaldir+'masks.p','wb'))
</pre>
[[file:Figure4_imagingtutorial.PNG|thumb|center|800px|Figure 4: Interactive clean window to create masks over the source. The white outline surrounding the source is the mask selected by the polygon drawing option.]]

The outlines drawn for masks can be created by any of the icons with the letter 'R' in the Viewer window. The instructions for doing this can be found by hovering over those icons.

<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
# =========== Step 3, main step of selfcalibration =========
if doslfcal:
if os.path.exists(slfcaldir+'masks.p'):
masks=pickle.load(open(slfcaldir+'masks.p','rb'))
if not os.path.exists(slfcaldir+'masks.p'):
print 'masks do not exist. Use default mask'
masks=[]
os.system('rm -rf '+imagedir+'*')
os.system('rm -rf '+caltbdir+'*')
#first step: make a mock caltable for the entire database
print('Processing ' + trange)
slftbs=[]
calprefix=caltbdir+'slf'
imgprefix=imagedir+'slf'
tb.open(slfcalms+'/SPECTRAL_WINDOW')
reffreqs=tb.getcol('REF_FREQUENCY')
bdwds=tb.getcol('TOTAL_BANDWIDTH')
cfreqs=reffreqs+bdwds/2.
tb.close()
# starting beam size at 3.4 GHz in arcsec #Change accordingly
sbeam=40.
strtmp=[m.replace(':','') for m in trange.split('~')]
timestr='t'+strtmp[0]+'-'+strtmp[1]
refantenna='0'
# number of iterations for each round
niters=[100, 300, 500]
# roburst value for weighting the baselines
robusts=[1.0, 0.5, 0.0]
# apply calibration tables? Set to true for most cases
doapplycal=[1,1,1]
# modes for calibration, 'p' for phase-only, 'a' for amplitude only, 'ap' for both
calmodes=['p','p','a']
# setting uvranges for model image (optional, not used here)
uvranges=['','','']
for n in range(nround):
slfcal_tb_g= calprefix+'.G'+str(n)
fig = plt.figure(figsize=(8.4,7.)) # fig = plt.figure(figsize=(14, 7)) # For post-2020 data (50 spws)
gs = gridspec.GridSpec(5, 6) # gs = gridspec.GridSpec(5, 10) # For post-2020 data (50 spws)
for s,sp in enumerate(spws):
print 'processing spw: '+sp
cfreq=cfreqs[int(sp)]
# setting restoring beam size (not very useful for selfcal anyway, but just to see the results)
bm=max(sbeam*cfreqs[1]/cfreq, 6.)
slfcal_img = imgprefix+'.spw'+sp.zfill(2)+'.slfcal'+str(n)
# only the first round uses nearby spws for getting initial model
if n == 0:
spbg=max(int(sp)-2,1) # spbg=max(int(sp)-2,0) # For post-2020 data (50 spws)
sped=min(int(sp)+2,30) # sped=min(int(sp)+2,49) # For post-2020 data (50 spws)
spwran=str(spbg)+'~'+str(sped)
print('using spw {0:s} as model'.format(spwran))
if 'spwrans_mask' in vars():
for m, spwran_mask in enumerate(spwrans_mask):
if sp in spwran_mask:
mask = masks[m]
print('using mask {0:s}'.format(mask))
findmask = True
if not findmask:
print('mask not found. Do use any masks')
else:
spwran = sp
if 'spwrans_mask' in vars():
for m, spwran_mask in enumerate(spwrans_mask):
if sp in spwran_mask:
mask = masks[m]
print 'using mask {0:s}'.format(mask)
findmask = True
if not findmask:
print('mask not found. Do use any masks')
try:
tclean(vis=slfcalms,
antenna=antennas,
imagename=slfcal_img,
uvrange=uvranges[n],
spw=spwran,
specmode='mfs',
timerange=trange,
imsize=[npix],
cell=['2arcsec'],
niter=niters[n],
gain=0.05,
stokes='XX', #use pol XX image as the model
weighting='briggs',
robust=robusts[n],
phasecenter=phasecenter,
mask=mask,
restoringbeam=[str(bm)+'arcsec'],
pbcor=False,
interactive=False,
savemodel='modelcolumn')
if os.path.exists(slfcal_img+'.image'):
fitsfile=slfcal_img+'.fits'
hf.imreg(vis=slfcalms,imagefile=slfcal_img+'.image',fitsfile=fitsfile,
timerange=trange,usephacenter=False,toTb=True,verbose=False,overwrite=True)
clnjunks = ['.mask','.flux', '.model', '.psf', '.residual', '.image','.pb','.image.pbcor','.sumwt'] #Do not run the next 4 lines if needed to view and assess the subset of clean process images
for clnjunk in clnjunks:
if os.path.exists(slfcal_img + clnjunk):
os.system('rm -rf '+ slfcal_img + clnjunk)
ax = fig.add_subplot(gs[s])
eomap=smap.Map(fitsfile)
eomap.plot_settings['cmap'] = plt.get_cmap('jet')
eomap.plot(axes = ax)
eomap.draw_limb()
#eomap.draw_grid()
ax.set_title(' ')
ax.get_xaxis().set_visible(False)
ax.get_yaxis().set_visible(False)
ax.set_xlim(xran)
ax.set_ylim(yran)
plt.pause(0.5) # Allows viewing of each image as it is plotted.
os.system('rm -f '+ fitsfile)

except:
print 'error in cleaning spw: '+sp
print 'using nearby spws for initial model'
sp_e=int(sp)+2
sp_i=int(sp)-2
if sp_i < 1: # if sp < 0: # For post-2020 data (50 spws)
sp_i = 1 # sp_i = 0 # For post-2020 data (50 spws)
if sp_e > 30: # if sp_e > 49: # For post-2020 data (50 spws)
sp_e = 30 # sp_e = 49 # For post-2020 data (50 spws)
sp_=str(sp_i)+'~'+str(sp_e)
try:
tget(tclean)
spw=sp_
print('using spw {0:s} as model'.format(sp_))
tclean()
except:
print 'still not successful. abort...'
break

gaincal(vis=slfcalms, refant=refantenna,antenna=antennas,caltable=slfcal_tb_g,spw=sp, uvrange='',\
gaintable=[],selectdata=True,timerange=trange,solint='inf',gaintype='G',calmode=calmodes[n],\
combine='',minblperant=4,minsnr=2,append=True)
if not os.path.exists(slfcal_tb_g):
print 'No solution found in spw: '+sp
figname=imagedir+'slf_t0_n{:d}.png'.format(n)
plt.subplots_adjust(left=0, bottom=0, right=1, top=1, wspace=0, hspace=0)
plt.savefig(figname)
time.sleep(10)
plt.close()

if os.path.exists(slfcal_tb_g):
slftbs.append(slfcal_tb_g)
slftb=[slfcal_tb_g]
os.chdir(slfcaldir)
if calmodes[n] == 'p':
plotcal(caltable=slfcal_tb_g,antenna='1~12',xaxis='freq',yaxis='phase',\
subplot=431,plotrange=[-1,-1,-180,180],iteration='antenna',figfile=slfcal_tb_g+'.png',showgui=False)
if calmodes[n] == 'a':
plotcal(caltable=slfcal_tb_g,antenna='1~12',xaxis='freq',yaxis='amp',\
subplot=431,plotrange=[-1,-1,0,2.],iteration='antenna',figfile=slfcal_tb_g+'.png',showgui=False)
os.chdir(workdir)
plt.pause(0.5) # Allows viewing of the plot

if doapplycal[n]:
clearcal(slfcalms)
delmod(slfcalms)
applycal(vis=slfcalms,gaintable=slftb,spw=','.join(spws),selectdata=True,\
antenna=antennas,interp='nearest',flagbackup=False,applymode='calonly',calwt=False)

if n < nround-1:
prompt=raw_input('Continuing to selfcal?')
#prompt='y'
if prompt.lower() == 'n':
if os.path.exists(slfcaledms):
os.system('rm -rf '+slfcaledms)
split(slfcalms,slfcaledms,datacolumn='corrected')
print 'Final calibrated ms is {0:s}'.format(slfcaledms)
break
if prompt.lower() == 'y':
slfcalms_=slfcalms+str(n)
if os.path.exists(slfcalms_):
os.system('rm -rf '+slfcalms_)
split(slfcalms,slfcalms_,datacolumn='corrected')
slfcalms=slfcalms_
else:
if os.path.exists(slfcaledms):
os.system('rm -rf '+slfcaledms)
split(slfcalms,slfcaledms,datacolumn='corrected')
print 'Final calibrated ms is {0:s}'.format(slfcaledms)
</pre>

<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
# =========== Step 4: Apply self-calibration tables =========
if doapply:
import glob
os.chdir(workdir)
clearcal(ms_in)
clearcal(slfcalms)
applycal(vis=slfcalms,gaintable=slftbs,spw=','.join(spws),selectdata=True,\
antenna=antennas,interp='linear',flagbackup=False,applymode='calonly',calwt=False)
applycal(vis=ms_in,gaintable=slftbs,spw=','.join(spws),selectdata=True,\
antenna=antennas,interp='linear',flagbackup=False,applymode='calonly',calwt=False)
if os.path.exists(ms_slfcaled):
os.system('rm -rf '+ms_slfcaled)
split(ms_in, ms_slfcaled,datacolumn='corrected')
</pre>
[[file:Slf.G0.png|thumb|center|500px|Figure 1: Phase before self-calibration]]
[[file:Slf.G1.png|thumb|center|500px|Figure 2: Phase after self-calibration]]

<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
# =========== Step 5: Generate final self-calibrated images (optional) =========
if doclean_slfcaled:
import glob
pol='XX'
print('Processing ' + trange)
img_final=imagedir_slfcaled+'/slf_final_{0}_t0'.format(pol)
vis = ms_slfcaled
spws=[str(s+1) for s in range(30)]
tb.open(vis+'/SPECTRAL_WINDOW')
reffreqs=tb.getcol('REF_FREQUENCY')
bdwds=tb.getcol('TOTAL_BANDWIDTH')
cfreqs=reffreqs+bdwds/2.
tb.close()
sbeam=30.
from matplotlib import gridspec as gridspec
from sunpy import map as smap
from matplotlib import pyplot as plt
fitsfiles=[]
for s,sp in enumerate(spws):
cfreq=cfreqs[int(sp)]
bm=max(sbeam*cfreqs[1]/cfreq,4.)
imname=img_final+'_s'+sp.zfill(2)
fitsfile=imname+'.fits'
if not os.path.exists(fitsfile):
print 'cleaning spw {0:s} with beam size {1:.1f}"'.format(sp,bm)
try:
tclean(vis=vis,
antenna=antennas,
imagename=imname,
spw=sp,
specmode='mfs',
timerange=trange,
imsize=[npix],
cell=['1arcsec'],
niter=1000,
gain=0.05,
stokes=pol,
weighting='briggs',
robust=2.0,
restoringbeam=[str(bm)+'arcsec'],
phasecenter=phasecenter,
mask='',
pbcor=True,
interactive=False)
except:
print 'cleaning spw '+sp+' unsuccessful. Proceed to next spw'
continue
if os.path.exists(imname+'.image.pbcor'):
imn = imname+'.image.pbcor'
hf.imreg(vis=vis,imagefile=imn,fitsfile=fitsfile,
timerange=trange,usephacenter=False,toTb=True,verbose=False)
fitsfiles.append(fitsfile)
junks=['.flux','.model','.psf','.residual','.mask','.image','.pb','.image.pbcor','.sumwt'] #Do not run the next 4 lines if needed to view and assess the subset of clean process images
for junk in junks:
if os.path.exists(imname+junk):
os.system('rm -rf '+imname+junk)
else:
print('fits file '+fitsfile+' already exists, skip clean...')
fitsfiles.append(fitsfile)

fig = plt.figure(figsize=(8.4,7.)) # fig = plt.figure(figsize=(14, 7)) # For post-2020 data (50 spws)
gs = gridspec.GridSpec(5, 6) # gs = gridspec.GridSpec(5, 10) # For post-2020 data (50 spws)
for s,sp in enumerate(spws):
cfreq=cfreqs[int(sp)]
ax = fig.add_subplot(gs[s])
eomap=smap.Map(fitsfiles[s])
eomap.plot_settings['cmap'] = plt.get_cmap('jet')
eomap.plot(axes = ax)
eomap.draw_limb()
ax.set_title(' ')
ax.get_xaxis().set_visible(False)
ax.get_yaxis().set_visible(False)
ax.set_xlim(xran)
ax.set_ylim(yran)
plt.text(0.98,0.85,'{0:.1f} GHz'.format(cfreq/1e9),transform=ax.transAxes,ha='right',color='w',fontweight='bold')
plt.subplots_adjust(left=0, bottom=0, right=1, top=1, wspace=0, hspace=0)
plt.show()

</pre>
The .fits files of the self calibrated images at each frequency for the given time are saved at /slfcal/images_slfcaled in your working directory.

[[file:Slf_t0_n0.png|thumb|center|500px|Figure 3: Multi-frequency images before self-calibration]]
[[file:Slf_t0_n1.png|thumb|center|500px|Figure 4: Multi-frequency images after self-calibration]]

====Step 5: Quick-look imaging ==== 
For spectral imaging analysis of the event, follow [http://www.ovsa.njit.edu/wiki/index.php/EOVSA_Data_Analysis_Tutorial#Spectral_Imaging_with_SunCASA this tutorial] using the self-calibrated data obtained from the previous step or use [https://colab.research.google.com/drive/1lSLLxgOG6b8kgu9Sk6kSKvrViyubnXG6?usp=sharing#scrollTo=xbXyyLmCFCGL this link].

<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
from suncasa.utils import qlookplot as ql ## (Optional) Supply the npz file of the dynamic spectrum from previous step.
## If not provided, the program will generate a new one from the visibility.
## set the time interval
from suncasa import dspec as ds
import time
visibility_data = 'IDB20170821202020_slfcaled.ms'
specfile = visibility_data + '.dspec.npz'
d = ds.Dspec(visibility_data, bl='4&9', specfile=specfile)

timerange = '19:02:00~19:02:10' ## select frequency range from 2.5 GHz to 3.5 GHz
spw = '2~5' ## select stokes XX
stokes = 'XX' ## turn off AIA image plotting, default is True
plotaia = False
xycen = [375, 45] ## image center for clean in solar X-Y in arcsec
cell=['2.0arcsec'] ## pixel size
imsize=[128] ## x and y image size in pixels.
fov = [100,100] ## field of view of the zoomed-in panels in unit of arcsec
spw = ['{}'.format(s) for s in range(1,31)]
clevels = [0.5, 1.0] ## contour levels to fill in between.
calpha=0.35 ## now tune down the alpha
restoringbeam=['6arcsec']
ql.qlookplot(vis=msfile, specfile=specfile, timerange=timerange, spw=spw, stokes=stokes, \
restoringbeam=restoringbeam,imsize=imsize,cell=cell, \
xycen=xycen,fov=fov,clevels=clevels,calpha=calpha)
</pre>

Page Last Updated: 06/29/2022

Tohban EOVSA Imaging Tutorial A-Z V2

2022-07-06T10:02:57Z

Schhabra:

This tutorial describes the step-by-step procedure to download EOVSA IDB data, obtain and calibrate the measurement sets (.ms), and, transfer them to the inti server for self-calibration and further analysis.

=== Pre-requisites ===
Accounts on Pipeline and Inti or Baozi servers. Sijie and Bin can help with the account set-up for pipeline, inti and boazi.

==== How to connect to pipeline ====
'''Windows:''' On a windows machine Mobaxterm can be used to connect to Pipeline. For details, see ...

'''Linux/Unix:''' On a linux/Mac the ssh commands can be used directly through the terminal:

<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
ssh -L 5902:helios.solar.pvt:20000 guest@ovsa.njit.edu {{font color|red|This text is different}} ### when prompted, enter password to guest@ovsa ###
ssh -XY your_username@pipeline ### add your username and enter password to your pipeline account, when prompted ###
</pre>

Alternatively, to avoid typing the longer commands, proxy commands can be added to the ~/.ssh/config file:

<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
vim ~/.ssh/config ### press 'i' to insert content and esc +':wq' to write and quit ###

Host *
ForwardAgent yes

Host *
ForwardX11 yes

Host ovsa
HostName ovsa.njit.edu
User guest

Host pipeline
Hostname pipeline.solar.pvt
User your_username ### Insert your pipeline username here ###
ProxyCommand ssh -W %h:%p guest@ovsa.njit.edu
</pre>

Upon saving the ~/.ssh/config file and restarting the terminal, you can connect to pipeline directly using:

<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
ssh -XY pipeline
</pre>

==== How to connect to the inti server ====
NOTE: To connect to inti you must have a NJIT UCID. The arcs have granted access to inti for all SRG members. If you encounter any issue with the login. Please let arcs folks (arcs@njit.edu) know.

'''Windows:''' As mentioned above, on a windows machine Mobaxterm can be used to connect to inti. For details, see ...

'''Linux/Unix:''' On a linux/Mac the ssh command can be used:
<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
ssh -XY UCID@inti.hpcnet.campus.njit.edu ### insert UCID here and enter your NJIT UCID password, when prompted ###
</pre>

When logging in for the first time, please add the following lines (demarcated by #) in your ~/.bashrc (on the inti server) file in order to use CASA on inti.

<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
#### setting start ####
if [ $HOSTNAME == "baozi.hpcnet.campus.njit.edu" ]; then
source /srg/.setenv_baozi
fi
if [ $HOSTNAME == "inti.hpcnet.campus.njit.edu" ]; then
source /inti/.setenv_inti
fi
if [ $HOSTNAME == "guko.resource.campus.njit.edu" ]; then
source /data/data/.setenv_guko
fi
#### setting end ####
</pre>
Proxy commands for inti can also be added to the ~/.ssh/config (on local machine) file for convenience:
<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
vim ~/.ssh/config ### press 'i' to insert content and esc +':wq' to write and quit ###

Host inti
Hostname inti.hpcnet.campus.njit.edu
User UCID ### insert NJIT UCID here ###
</pre>
after the proxy is set up, connect to inti by simply typing:
<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
ssh -XY inti ### enter your NJIT UCID password when prompted ###
</pre>

=== Downloading data and Preliminary Calibration ===

==== Step 1: Downloading raw (IDB) data on Pipeline machine====
The initial downloading of data and applying refcal and phasecal to the .ms files is performed on the pipeline since the complete EOVSA SQL database is stored here for now.

If you are not using mobaxterm, directly SSH to Pipeline through your Linux or Mac computer (as shown above) and '''start tcsh to run CASA'''.

The following commands (run within CASA) can be used to import a single IDB file if the given time-range is less than 10 mins.
<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
from astropy.time import Time
import os
trange = Time(['2017-08-21 20:15:00', '2017-08-21 20:25:00']) ###Change accordingly###

#### (Optional) change output path, default current directory "./" #######
outpath = './msdata/' ###Change accordingly###
if not os.path.exists(outpath):
os.makedirs(outpath)

msfiles = importeovsa(idbfiles=trange, ncpu=1, visprefix=outpath)
</pre>

'''NOTE:''' The above step currently gives an error with importeovsa. This is because the data location was changed and the code doesn't know where to look for the IDB files. Sijie is looking into the problem. We will update the page when it is fixed. For now, please use the method below for all time ranges.

OR (for a time range longer than 10 minutes)

<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
from suncasa.tasks import task_calibeovsa as calibeovsa
from suncasa.tasks import task_importeovsa as timporteovsa
from split_cli import split_cli as split
import dump_tsys as dt
from util import Time
import numpy as np
import os
from glob import glob
from eovsapy import util

trange = Time(['2017-08-21 20:15:00', '2017-08-21 20:35:00']) ###Change accordingly###
idbdir = util.get_idbdir(trange[0])

info = dt.rd_fdb(trange[0])
for k, v in info.iteritems():
info[k] = info[k][~(info[k] == '')]

sidx = np.where(
np.logical_and(info['SOURCEID'] == 'Sun', info['PROJECTID'] == 'NormalObserving') & np.logical_and(
info['ST_TS'].astype(np.float) >= trange[0].lv,
info['ST_TS'].astype(np.float) <= trange[
1].lv))
filelist = info['FILE'][sidx]

outpath = './msdata/' ###Change accordingly###
if not os.path.exists(outpath):
os.makedirs(outpath)
inpath = idbdir + '{}/'.format(trange[0].datetime.strftime("%Y%m%d"))
ncpu = 1

msfiles = timporteovsa.importeovsa(idbfiles=[inpath + ll for ll in filelist], ncpu=ncpu, timebin="0s", width=1,
visprefix=outpath,
nocreatms=False, doconcat=False,
modelms="", doscaling=False, keep_nsclms=False, udb_corr=True)
</pre>

If you come across errors with calibeovsa, add the following lines to your ~/.casa/init.py (within pipeline) file.
<pre style="background-color: #FCEBD9">
vim ~/.ssh/config ### press 'i' to insert content and esc +':wq' to write and quit ###
import sys
sys.path.append('/common/python')
sys.path.append('/common/python/packages/pipeline_casa')
execfile('/common/python/suncasa/tasks/mytasks.py')
</pre>

==== Step 2: Concatenate multiple .ms files and Calibrate====
'''Follow this step if there are more than one .ms files for the chosen time-range. If not, move to Step 3.'''

<pre style="background-color: #FCEBD9">
concatvis = os.path.basename(msfiles[0])[:11] + '_concat_cal.ms' ### Set the path/name for the concatenated files ###

vis = calibeovsa.calibeovsa(msfiles, doconcat=True, concatvis=concatvis, caltype=['refpha','phacal'], doimage=True)
### If doimage=True, a quicklook image will be produced (by integrating over the entire time) as shown below ###

vis_str = str(' '.join(vis))
caled_vis=vis_str.replace('.ms','_cal.ms') ### Append '_cal' to the ms filename ###
split(vis=' '.join(vis),outputvis=caled_vis,datacolumn='corrected',timerange='',correlation='XX') ### Split the corrected column to the new 'caled' ms ###
</pre>

[[file:Figure1_imagingtutorial.png|frame|center|800px|'''Figure 1:''' Quick-look full-Sun image after the initial calibration.]]

==== Step 3: Calibration ====
For a single .ms file from Step 1, this will calibrate the input visibility, write out calibration tables under /data1/eovsa/caltable/, and apply the calibration.
<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
vis = calibeovsa.calibeovsa('IDB20170821202020.ms', caltype=['refpha','phacal'], doimage=True) ###Change the vis filename accordingly###

vis_str = str(' '.join(vis))
caled_vis=vis_str.replace('.ms','_cal.ms') ### Append '_cal' to the ms filename ###
split(vis=' '.join(vis),outputvis=caled_vis,datacolumn='corrected',timerange='',correlation='XX') ### Split the corrected column to the new 'caled' ms ###
</pre>

One needs to transfer the created 'caled' ms data files (xxx_concat_cal.ms or xxx_cal.ms) from pipeline to inti server, which has all the casa tasks installed, in order to run the rest of the imaging steps.

=== Connection details to Inti server ===
For Windows, on Mobaxterm,

#With your NJIT VPN connected, connect to one of the afsconnect servers (for example, afsaccess3.njit.edu) using your UCID and password.
#ssh -X UCID@inti.hpcnet.campus.njit.edu

To avoid typing the full inti address each time you attempt for ssh, you may wish to add the following lines with your username to C:\Program Files\Git\etc\ssh\ssh_config on Windows and /.ssh/config on Mac and Linux machines.

<pre style="background-color: #FCEBD9">
Host inti
Hostname inti.hpcnet.campus.njit.edu
User USERNAME ###Insert UCID/inti username here###
</pre>

To have sufficient disk space with EOVSA data analysis over Inti, use your dedicated directory YOURDIRECTORY at the location given below. If you do not have a directory, Please take help from Sijie in creating one.
<pre style="background-color: #FCEBD9">
cd /inti/data/users/YOURDIRECTORY
</pre>

For Linux or Mac machine,
ssh -X UCID@inti.hpcnet.campus.njit.edu

=== Transferring data files between servers ===
For directly transferring your calibrated .ms data between the Pipeline and Inti servers, follow the below given steps.
<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
1. Log into Inti using your username.
ssh -X USERNAME@inti.hpcnet.campus.njit.edu

Then create a tunnel into Pipeline from Inti.
ssh -L 8888:pipeline.solar.pvt:22 guest@ovsa.njit.edu

2. Log into Inti again from a new terminal.

Change to your working directory and give this command to copy your data on Pipeline.
scp -v -C -r -P 8888 USERNAMEofPipeline@localhost:PATHofMSDATA/MSfilename ./
Eg: scp -v -C -r -P 8888 shaheda@localhost:/data1/shaheda/IDB20220118173922_cal.ms ./
where, MSfilename, PATHofMSDATA are your .ms data and its path on Pipeline machine.
</pre>

Alternatively, one can follow the below procedure to do the transfer.

For Windows, on mobaxterm, drag and drop the ms file to your local machine or use scp command as given below.

<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
scp -r -C userid@pipeline:/your/folder/msfile /localmachine/destination/folder/
</pre>

On mobaxterm and on your local terminal, use the following command to finally copy the ms file to Inti.

<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
scp -r -C /localmachine/destination/folder/msfile UCID@inti.hpcnet.campus.njit.edu:/inti/data/users/YOURDIRECTORY
</pre>

For Mac and Linux, SCP can be used in the same way. Add the following lines to your SSH config file, to bypass ovsa.njit.edu from copying.
vi ~/.ssh/config
<pre style="background-color: #FCEBD9">
Host ovsa
HostName ovsa.njit.edu
User guest
Host pipeline
Hostname pipeline.solar.pvt
User userid
ProxyCommand ssh -W %h:%p guest@ovsa.njit.edu
</pre>

=== Software details on the servers ===
On Inti,
when logging in for the first time, please add the following lines to your accounts ~/.bashrc file.

>>vi ~/.bashrc
Insert the text given below and save it.

<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
#### setting start ####
if [ $HOSTNAME == "baozi.hpcnet.campus.njit.edu" ]; then
source /srg/.setenv_baozi
fi
if [ $HOSTNAME == "inti.hpcnet.campus.njit.edu" ]; then
source /inti/.setenv_inti
fi
if [ $HOSTNAME == "guko.resource.campus.njit.edu" ]; then
source /data/data/.setenv_guko
fi
#### setting end ####
</pre>

Both CASA 5 and 6 are available on Inti.

Enter the bash environment on inti, and load the desired casa environment.
To load CASA 5, enter bash environment by giving >>bash
<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
>> loadcasa5
>> suncasa #This should load the software making you ready for the analysis
</pre>

Or to load CASA 6, enter bash environment by giving >>bash
<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
>> loadcasa6
>> ipython #This should load the software making you ready for the analysis
</pre>
Here, for example, to use clean, first start ipython as given above, then type in >>from casatasks import tclean

====Step 4: Self-calibration====
[[file:Figure3_imagingtutorial.png|thumb|center|500px|Figure 2: Cotton-Schwab clean major and minor cycles. [Source: http://www.aoc.nrao.edu/~rurvashi/ImagingAlgorithmsInCasa/node2.html].]]
Follow the below given steps to run the self-calibration of the imaging data and produce the calibrated images in .fits format. https://github.com/binchensun/casa-eovsa/blob/master/slfcal_example.py

<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
from suncasa.utils import helioimage2fits as hf
import os
import numpy as np
import pickle
from matplotlib import gridspec as gridspec
from sunpy import map as smap
from matplotlib import pyplot as plt
from split_cli import split_cli as split
import time

# =========== task handlers =============
dofullsun = 1 # initial full-sun imaging ###Change accordingly###
domasks=1 # get masks ###Change accordingly###
doslfcal=1 # main cycle of doing selfcalibration ###Change accordingly###
doapply=1 # apply the results ###Change accordingly###
doclean_slfcaled=1 # perform clean for self-calibrated data ###Change accordingly###

# ============ declaring the working directories ============
workdir = os.getcwd()+'/' #main working directory. Using current directory in this example
slfcaldir = workdir+'slfcal/' #place to put all selfcalibration products
imagedir = slfcaldir+'images/' #place to put all selfcalibration images
maskdir = slfcaldir+'masks/' #place to put clean masks
imagedir_slfcaled = slfcaldir+'images_slfcaled/' #place to put final self-calibrated images
caltbdir = slfcaldir+'caltbs/' # place to put calibration tables
# make these directories if they do not already exist
dirs = [workdir, slfcaldir, imagedir, maskdir, imagedir_slfcaled, caltbdir]
for d in dirs:
if not os.path.exists(d):
os.makedirs(d)

# ============ Split a short time for self-calibration ===========
# input visibility
ms_in = workdir + 'IDB20170821202020_cal.ms' ###Change the initial calibrated (through calibeovsa) vis accordingly###
# output, selfcaled, visibility
ms_slfcaled = workdir + os.path.basename(ms_in).replace('cal','slfcaled')
# intermediate small visibility for selfcalbration
# selected time range for generating self-calibration solutions
trange='2017/08/21/20:21:10~2017/08/21/20:21:30' ###Change accordingly###
slfcalms = slfcaldir+'slfcalms.XX.slfcal'
slfcaledms = slfcaldir+'slfcalms.XX.slfcaled'
if not os.path.exists(slfcalms):
split(vis=ms_in,outputvis=slfcalms,datacolumn='data',timerange=trange,correlation='XX')

# ============ Prior definitions for spectral windows, antennas, pixel numbers =========
spws=[str(s+1) for s in range(30)] # spws=[str(s+1) for s in range(49)] # For post-2020 data
antennas='0~12&0~12'
npix=512
nround=3 #number of slfcal cycles

# =========== Step 1, doing a full-Sun image to find out phasecenter and appropriate field of view =========
if dofullsun:
#initial mfs clean to find out the image phase center
im_init='fullsun_init'
os.system('rm -rf '+im_init+'*')
tclean(vis=slfcalms,
antenna=antennas,
imagename=im_init,
spw='2~15',
specmode='mfs',
timerange=trange,
imsize=[npix],
cell=['5arcsec'],
niter=1000,
gain=0.05,
stokes='I',
restoringbeam=['30arcsec'],
interactive=False,
pbcor=True)

hf.imreg(vis=slfcalms,imagefile=im_init+'.image.pbcor',fitsfile=im_init+'.fits',
timerange=trange,usephacenter=False,verbose=True)
clnjunks = ['.flux', '.mask', '.model', '.psf', '.residual','.sumwt','.pb','.image'] #Do not run the next 4 lines if needed to view and assess the subset of clean process images
for clnjunk in clnjunks:
if os.path.exists(im_init + clnjunk):
os.system('rm -rf '+im_init + clnjunk)

from sunpy import map as smap
from matplotlib import pyplot as plt
fig = plt.figure(figsize=(6,6))
ax = fig.add_subplot(111)
eomap=smap.Map(im_init+'.fits')
#eomap.data=eomap.data.reshape((npix,npix))
eomap.plot_settings['cmap'] = plt.get_cmap('jet')
eomap.plot(axes = ax)
eomap.draw_limb()
plt.show()
viewer(im_init+'.image.pbcor')
</pre>
[[file:Figure2_imagingtutorial.png|thumb|center|500px|Figure 3: Full-Sun image after initial clean to find the flare location.]]

<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
# parameters specific to the event (found from step 1)
phasecenter='J2000 10h02m59 11d58m07' ###Change accordingly###
xran=[280,480] ###Change accordingly###
yran=[-50,150] ###Change accordingly###

# =========== Step 2 (optional), generate masks =========
# if skipped, will not use any masks
if domasks:
clearcal(slfcalms)
delmod(slfcalms)
antennas=antennas
pol='XX'
imgprefix=maskdir+'slf_t0'

# step 1: set up the clean masks
img_init=imgprefix+'_init_ar_'
os.system('rm -rf '+img_init+'*')
#spwrans_mask=['1~5','6~12','13~20','21~30']
spwrans_mask=['2~12']
#convert to a list of spws
spwrans_mask_list = [[str(i) for i in (np.arange(int(m.split('~')[0]),int(m.split('~')[1])))] for m in spwrans_mask] # Not used?
masks=[]
imnames=[]
for spwran in spwrans_mask:
imname=img_init+spwran.replace('~','-')
try:
tclean(vis=slfcalms,
antenna=antennas,
imagename=imname,
spw=spwran,
specmode='mfs',
timerange=trange,
imsize=[npix],
cell=['2arcsec'],
niter=1000,
gain=0.05,
stokes='XX',
restoringbeam=['20arcsec'],
phasecenter=phasecenter,
weighting='briggs',
robust=1.0,
interactive=True,
datacolumn='data',
pbcor=True,
savemodel='modelcolumn')
imnames.append(imname+'.image')
masks.append(imname+'.mask')
clnjunks = ['.flux', '.model', '.psf', '.residual'] #Do not run the next 4 lines if needed to view and assess the subset of clean process images
for clnjunk in clnjunks:
if os.path.exists(imname + clnjunk):
os.system('rm -rf '+ imname + clnjunk)
except:
print('error in cleaning spw: '+spwran)

pickle.dump(masks,open(slfcaldir+'masks.p','wb'))
</pre>
[[file:Figure4_imagingtutorial.PNG|thumb|center|800px|Figure 4: Interactive clean window to create masks over the source. The white outline surrounding the source is the mask selected by the polygon drawing option.]]

The outlines drawn for masks can be created by any of the icons with the letter 'R' in the Viewer window. The instructions for doing this can be found by hovering over those icons.

<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
# =========== Step 3, main step of selfcalibration =========
if doslfcal:
if os.path.exists(slfcaldir+'masks.p'):
masks=pickle.load(open(slfcaldir+'masks.p','rb'))
if not os.path.exists(slfcaldir+'masks.p'):
print 'masks do not exist. Use default mask'
masks=[]
os.system('rm -rf '+imagedir+'*')
os.system('rm -rf '+caltbdir+'*')
#first step: make a mock caltable for the entire database
print('Processing ' + trange)
slftbs=[]
calprefix=caltbdir+'slf'
imgprefix=imagedir+'slf'
tb.open(slfcalms+'/SPECTRAL_WINDOW')
reffreqs=tb.getcol('REF_FREQUENCY')
bdwds=tb.getcol('TOTAL_BANDWIDTH')
cfreqs=reffreqs+bdwds/2.
tb.close()
# starting beam size at 3.4 GHz in arcsec #Change accordingly
sbeam=40.
strtmp=[m.replace(':','') for m in trange.split('~')]
timestr='t'+strtmp[0]+'-'+strtmp[1]
refantenna='0'
# number of iterations for each round
niters=[100, 300, 500]
# roburst value for weighting the baselines
robusts=[1.0, 0.5, 0.0]
# apply calibration tables? Set to true for most cases
doapplycal=[1,1,1]
# modes for calibration, 'p' for phase-only, 'a' for amplitude only, 'ap' for both
calmodes=['p','p','a']
# setting uvranges for model image (optional, not used here)
uvranges=['','','']
for n in range(nround):
slfcal_tb_g= calprefix+'.G'+str(n)
fig = plt.figure(figsize=(8.4,7.)) # fig = plt.figure(figsize=(14, 7)) # For post-2020 data (50 spws)
gs = gridspec.GridSpec(5, 6) # gs = gridspec.GridSpec(5, 10) # For post-2020 data (50 spws)
for s,sp in enumerate(spws):
print 'processing spw: '+sp
cfreq=cfreqs[int(sp)]
# setting restoring beam size (not very useful for selfcal anyway, but just to see the results)
bm=max(sbeam*cfreqs[1]/cfreq, 6.)
slfcal_img = imgprefix+'.spw'+sp.zfill(2)+'.slfcal'+str(n)
# only the first round uses nearby spws for getting initial model
if n == 0:
spbg=max(int(sp)-2,1) # spbg=max(int(sp)-2,0) # For post-2020 data (50 spws)
sped=min(int(sp)+2,30) # sped=min(int(sp)+2,49) # For post-2020 data (50 spws)
spwran=str(spbg)+'~'+str(sped)
print('using spw {0:s} as model'.format(spwran))
if 'spwrans_mask' in vars():
for m, spwran_mask in enumerate(spwrans_mask):
if sp in spwran_mask:
mask = masks[m]
print('using mask {0:s}'.format(mask))
findmask = True
if not findmask:
print('mask not found. Do use any masks')
else:
spwran = sp
if 'spwrans_mask' in vars():
for m, spwran_mask in enumerate(spwrans_mask):
if sp in spwran_mask:
mask = masks[m]
print 'using mask {0:s}'.format(mask)
findmask = True
if not findmask:
print('mask not found. Do use any masks')
try:
tclean(vis=slfcalms,
antenna=antennas,
imagename=slfcal_img,
uvrange=uvranges[n],
spw=spwran,
specmode='mfs',
timerange=trange,
imsize=[npix],
cell=['2arcsec'],
niter=niters[n],
gain=0.05,
stokes='XX', #use pol XX image as the model
weighting='briggs',
robust=robusts[n],
phasecenter=phasecenter,
mask=mask,
restoringbeam=[str(bm)+'arcsec'],
pbcor=False,
interactive=False,
savemodel='modelcolumn')
if os.path.exists(slfcal_img+'.image'):
fitsfile=slfcal_img+'.fits'
hf.imreg(vis=slfcalms,imagefile=slfcal_img+'.image',fitsfile=fitsfile,
timerange=trange,usephacenter=False,toTb=True,verbose=False,overwrite=True)
clnjunks = ['.mask','.flux', '.model', '.psf', '.residual', '.image','.pb','.image.pbcor','.sumwt'] #Do not run the next 4 lines if needed to view and assess the subset of clean process images
for clnjunk in clnjunks:
if os.path.exists(slfcal_img + clnjunk):
os.system('rm -rf '+ slfcal_img + clnjunk)
ax = fig.add_subplot(gs[s])
eomap=smap.Map(fitsfile)
eomap.plot_settings['cmap'] = plt.get_cmap('jet')
eomap.plot(axes = ax)
eomap.draw_limb()
#eomap.draw_grid()
ax.set_title(' ')
ax.get_xaxis().set_visible(False)
ax.get_yaxis().set_visible(False)
ax.set_xlim(xran)
ax.set_ylim(yran)
plt.pause(0.5) # Allows viewing of each image as it is plotted.
os.system('rm -f '+ fitsfile)

except:
print 'error in cleaning spw: '+sp
print 'using nearby spws for initial model'
sp_e=int(sp)+2
sp_i=int(sp)-2
if sp_i < 1: # if sp < 0: # For post-2020 data (50 spws)
sp_i = 1 # sp_i = 0 # For post-2020 data (50 spws)
if sp_e > 30: # if sp_e > 49: # For post-2020 data (50 spws)
sp_e = 30 # sp_e = 49 # For post-2020 data (50 spws)
sp_=str(sp_i)+'~'+str(sp_e)
try:
tget(tclean)
spw=sp_
print('using spw {0:s} as model'.format(sp_))
tclean()
except:
print 'still not successful. abort...'
break

gaincal(vis=slfcalms, refant=refantenna,antenna=antennas,caltable=slfcal_tb_g,spw=sp, uvrange='',\
gaintable=[],selectdata=True,timerange=trange,solint='inf',gaintype='G',calmode=calmodes[n],\
combine='',minblperant=4,minsnr=2,append=True)
if not os.path.exists(slfcal_tb_g):
print 'No solution found in spw: '+sp
figname=imagedir+'slf_t0_n{:d}.png'.format(n)
plt.subplots_adjust(left=0, bottom=0, right=1, top=1, wspace=0, hspace=0)
plt.savefig(figname)
time.sleep(10)
plt.close()

if os.path.exists(slfcal_tb_g):
slftbs.append(slfcal_tb_g)
slftb=[slfcal_tb_g]
os.chdir(slfcaldir)
if calmodes[n] == 'p':
plotcal(caltable=slfcal_tb_g,antenna='1~12',xaxis='freq',yaxis='phase',\
subplot=431,plotrange=[-1,-1,-180,180],iteration='antenna',figfile=slfcal_tb_g+'.png',showgui=False)
if calmodes[n] == 'a':
plotcal(caltable=slfcal_tb_g,antenna='1~12',xaxis='freq',yaxis='amp',\
subplot=431,plotrange=[-1,-1,0,2.],iteration='antenna',figfile=slfcal_tb_g+'.png',showgui=False)
os.chdir(workdir)
plt.pause(0.5) # Allows viewing of the plot

if doapplycal[n]:
clearcal(slfcalms)
delmod(slfcalms)
applycal(vis=slfcalms,gaintable=slftb,spw=','.join(spws),selectdata=True,\
antenna=antennas,interp='nearest',flagbackup=False,applymode='calonly',calwt=False)

if n < nround-1:
prompt=raw_input('Continuing to selfcal?')
#prompt='y'
if prompt.lower() == 'n':
if os.path.exists(slfcaledms):
os.system('rm -rf '+slfcaledms)
split(slfcalms,slfcaledms,datacolumn='corrected')
print 'Final calibrated ms is {0:s}'.format(slfcaledms)
break
if prompt.lower() == 'y':
slfcalms_=slfcalms+str(n)
if os.path.exists(slfcalms_):
os.system('rm -rf '+slfcalms_)
split(slfcalms,slfcalms_,datacolumn='corrected')
slfcalms=slfcalms_
else:
if os.path.exists(slfcaledms):
os.system('rm -rf '+slfcaledms)
split(slfcalms,slfcaledms,datacolumn='corrected')
print 'Final calibrated ms is {0:s}'.format(slfcaledms)
</pre>

<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
# =========== Step 4: Apply self-calibration tables =========
if doapply:
import glob
os.chdir(workdir)
clearcal(ms_in)
clearcal(slfcalms)
applycal(vis=slfcalms,gaintable=slftbs,spw=','.join(spws),selectdata=True,\
antenna=antennas,interp='linear',flagbackup=False,applymode='calonly',calwt=False)
applycal(vis=ms_in,gaintable=slftbs,spw=','.join(spws),selectdata=True,\
antenna=antennas,interp='linear',flagbackup=False,applymode='calonly',calwt=False)
if os.path.exists(ms_slfcaled):
os.system('rm -rf '+ms_slfcaled)
split(ms_in, ms_slfcaled,datacolumn='corrected')
</pre>
[[file:Slf.G0.png|thumb|center|500px|Figure 1: Phase before self-calibration]]
[[file:Slf.G1.png|thumb|center|500px|Figure 2: Phase after self-calibration]]

<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
# =========== Step 5: Generate final self-calibrated images (optional) =========
if doclean_slfcaled:
import glob
pol='XX'
print('Processing ' + trange)
img_final=imagedir_slfcaled+'/slf_final_{0}_t0'.format(pol)
vis = ms_slfcaled
spws=[str(s+1) for s in range(30)]
tb.open(vis+'/SPECTRAL_WINDOW')
reffreqs=tb.getcol('REF_FREQUENCY')
bdwds=tb.getcol('TOTAL_BANDWIDTH')
cfreqs=reffreqs+bdwds/2.
tb.close()
sbeam=30.
from matplotlib import gridspec as gridspec
from sunpy import map as smap
from matplotlib import pyplot as plt
fitsfiles=[]
for s,sp in enumerate(spws):
cfreq=cfreqs[int(sp)]
bm=max(sbeam*cfreqs[1]/cfreq,4.)
imname=img_final+'_s'+sp.zfill(2)
fitsfile=imname+'.fits'
if not os.path.exists(fitsfile):
print 'cleaning spw {0:s} with beam size {1:.1f}"'.format(sp,bm)
try:
tclean(vis=vis,
antenna=antennas,
imagename=imname,
spw=sp,
specmode='mfs',
timerange=trange,
imsize=[npix],
cell=['1arcsec'],
niter=1000,
gain=0.05,
stokes=pol,
weighting='briggs',
robust=2.0,
restoringbeam=[str(bm)+'arcsec'],
phasecenter=phasecenter,
mask='',
pbcor=True,
interactive=False)
except:
print 'cleaning spw '+sp+' unsuccessful. Proceed to next spw'
continue
if os.path.exists(imname+'.image.pbcor'):
imn = imname+'.image.pbcor'
hf.imreg(vis=vis,imagefile=imn,fitsfile=fitsfile,
timerange=trange,usephacenter=False,toTb=True,verbose=False)
fitsfiles.append(fitsfile)
junks=['.flux','.model','.psf','.residual','.mask','.image','.pb','.image.pbcor','.sumwt'] #Do not run the next 4 lines if needed to view and assess the subset of clean process images
for junk in junks:
if os.path.exists(imname+junk):
os.system('rm -rf '+imname+junk)
else:
print('fits file '+fitsfile+' already exists, skip clean...')
fitsfiles.append(fitsfile)

fig = plt.figure(figsize=(8.4,7.)) # fig = plt.figure(figsize=(14, 7)) # For post-2020 data (50 spws)
gs = gridspec.GridSpec(5, 6) # gs = gridspec.GridSpec(5, 10) # For post-2020 data (50 spws)
for s,sp in enumerate(spws):
cfreq=cfreqs[int(sp)]
ax = fig.add_subplot(gs[s])
eomap=smap.Map(fitsfiles[s])
eomap.plot_settings['cmap'] = plt.get_cmap('jet')
eomap.plot(axes = ax)
eomap.draw_limb()
ax.set_title(' ')
ax.get_xaxis().set_visible(False)
ax.get_yaxis().set_visible(False)
ax.set_xlim(xran)
ax.set_ylim(yran)
plt.text(0.98,0.85,'{0:.1f} GHz'.format(cfreq/1e9),transform=ax.transAxes,ha='right',color='w',fontweight='bold')
plt.subplots_adjust(left=0, bottom=0, right=1, top=1, wspace=0, hspace=0)
plt.show()

</pre>
The .fits files of the self calibrated images at each frequency for the given time are saved at /slfcal/images_slfcaled in your working directory.

[[file:Slf_t0_n0.png|thumb|center|500px|Figure 3: Multi-frequency images before self-calibration]]
[[file:Slf_t0_n1.png|thumb|center|500px|Figure 4: Multi-frequency images after self-calibration]]

====Step 5: Quick-look imaging ==== 
For spectral imaging analysis of the event, follow [http://www.ovsa.njit.edu/wiki/index.php/EOVSA_Data_Analysis_Tutorial#Spectral_Imaging_with_SunCASA this tutorial] using the self-calibrated data obtained from the previous step or use [https://colab.research.google.com/drive/1lSLLxgOG6b8kgu9Sk6kSKvrViyubnXG6?usp=sharing#scrollTo=xbXyyLmCFCGL this link].

<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
from suncasa.utils import qlookplot as ql ## (Optional) Supply the npz file of the dynamic spectrum from previous step.
## If not provided, the program will generate a new one from the visibility.
## set the time interval
from suncasa import dspec as ds
import time
visibility_data = 'IDB20170821202020_slfcaled.ms'
specfile = visibility_data + '.dspec.npz'
d = ds.Dspec(visibility_data, bl='4&9', specfile=specfile)

timerange = '19:02:00~19:02:10' ## select frequency range from 2.5 GHz to 3.5 GHz
spw = '2~5' ## select stokes XX
stokes = 'XX' ## turn off AIA image plotting, default is True
plotaia = False
xycen = [375, 45] ## image center for clean in solar X-Y in arcsec
cell=['2.0arcsec'] ## pixel size
imsize=[128] ## x and y image size in pixels.
fov = [100,100] ## field of view of the zoomed-in panels in unit of arcsec
spw = ['{}'.format(s) for s in range(1,31)]
clevels = [0.5, 1.0] ## contour levels to fill in between.
calpha=0.35 ## now tune down the alpha
restoringbeam=['6arcsec']
ql.qlookplot(vis=msfile, specfile=specfile, timerange=timerange, spw=spw, stokes=stokes, \
restoringbeam=restoringbeam,imsize=imsize,cell=cell, \
xycen=xycen,fov=fov,clevels=clevels,calpha=calpha)
</pre>

Page Last Updated: 06/29/2022

Guide to Upgrade SolarSoft(SSW)

2022-06-07T18:57:28Z

Schhabra:

This guide compiles all the (scattered) steps to upgrade SSW taken from different sources.
SSW upgrade using passive_ftp was stopped in August 2019. The current way to upgrade SSW is to use the WGET executable. If you have done an update on SSW since August 2019 and your system already has WGET, any further upgrade can be made by directly following the steps underlined [https://hesperia.gsfc.nasa.gov/rhessi3/docs/ssw_update_help.html here]. For Linux and MacOS you can check if you already have WGET by typing the command "which wget" in the bash shell.

If your system was not updated since August 2019, follow the steps below:

=== MacOS ===
In order to perform the upgrade, you must be the owner of the SSW tree. If you do not have admin privileges, consider installing ssw in $HOME directory.
If you already have wget installed in your system, then '''skip to point 3'''.
'''Using homebrew'''

To install homebrew, type the following command in the terminal:
<pre style="background-color: #FCEBD9">
/usr/bin/ruby -e "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/install)"
</pre>

Once homebrew is installed, you can simply install wget by using the following command:
<pre style="background-color: #FCEBD9">
brew install wget
</pre>

'''Using MacPorts'''

Alternatively, wget can also be installed using MacPorts. Directions to install MacPorts for specific OS versions are given [https://www.macports.org/install.php here]
After installation, the following command will install wget:

<pre style="background-color: #FCEBD9">
sudo port install wget
</pre>

==== 2. Prepare SSW to use wget for the first time ====
* Go to [http://www.lmsal.com/solarsoft/] and scroll all the way down to the SSW installation form.
* For "Installation Type", select "Upgrade Existing".
* For "Transfer Protocol", select "curl".
* Specify the existing path to SSW on your computer. (You can find it in your ~/.bashrc or ~/.cshrc)
* Click on "Generate Installation Script" and download it.
* If you are in the bash shell, please change to csh or tcsh, open the directory where the installation script was downloaded and run it
<pre style="background-color: #FCEBD9">
% tcsh
Execute the script using the C-Shell
% csh -f scriptname
</pre>

==== 3. Upgrade SSW ====
* While in SSWIDL type:
<pre style="background-color: #FCEBD9">
ssw_upgrade, /loud , /spawn
</pre>
This will update all the SSW branches listed in the SSW_INSTR environment variable (as well as their dependencies, and always includes gen and packages/binaries). This most likely will not update every branch in your SSW tree since you probably run SSW with a subset of branches activated.
* To upgrade specific branches, type for example
<pre style="background-color: #FCEBD9">
ssw_upgrade, /lasco, /spawn
</pre>

===== Possible Errors =====
* '''Error: bash:wget command not found''': Indicates that wget was installed using bash and csh/tcsh is not able to locate it. To resolve, simply give the path to wget in the upgrade command
<pre style="background-color: #FCEBD9">
ssw_upgrade, /loud, /spawn, wget = 'path/to/wget'
</pre>

* '''RD_TFILE: No file: /usr/local/ssw/site/mirror/ssw_upgrade.mirror''': check if the "'''/spawn'''" keyword has been missed

* '''Other errors with WGET ''': Sometimes using an older version of wget gives errors during SSW_upgrade, in this case add the keyword "'''/lmsal'''" to the upgrade command. This allows use of older wget versions.

Guide to Upgrade SolarSoft(SSW)

2022-04-05T22:07:00Z

Schhabra:

This guide compiles all the (scattered) steps to upgrade SSW taken from different sources.
SSW upgrade using passive_ftp was stopped in August 2019. The current way to upgrade SSW is to use the WGET executable. If you have done an update on SSW since August 2019 and your system already has WGET, any further upgrade can be made by directly following the steps underlined [https://hesperia.gsfc.nasa.gov/rhessi3/docs/ssw_update_help.html here]. For Linux and MacOS you can check if you already have WGET by typing the command "which wget" in the bash shell.

If your system was not updated since August 2019, follow the steps below:

=== MacOS ===
In order to perform the upgrade, you must be the owner of the SSW tree.
If you already have wget installed in your system, then '''skip to point 3'''.
'''Using homebrew'''

To install homebrew, type the following command in the terminal:
<pre style="background-color: #FCEBD9">
/usr/bin/ruby -e "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/install)"
</pre>

Once homebrew is installed, you can simply install wget by using the following command:
<pre style="background-color: #FCEBD9">
brew install wget
</pre>

'''Using MacPorts'''

Alternatively, wget can also be installed using MacPorts. Directions to install MacPorts for specific OS versions are given [https://www.macports.org/install.php here]
After installation, the following command will install wget:

<pre style="background-color: #FCEBD9">
sudo port install wget
</pre>

==== 2. Prepare SSW to use wget for the first time ====
* Go to [http://www.lmsal.com/solarsoft/] and scroll all the way down to the SSW installation form.
* For "Installation Type", select "Upgrade Existing".
* For "Transfer Protocol", select "curl".
* Specify the existing path to SSW on your computer. (You can find it in your ~/.bashrc or ~/.cshrc)
* Click on "Generate Installation Script" and download it.
* If you are in the bash shell, please change to csh or tcsh,
<pre style="background-color: #FCEBD9">
% tcsh
Execute the script using the C-Shell
% csh -f scriptname
</pre>

==== 3. Upgrade SSW ====
* While in SSWIDL type:
<pre style="background-color: #FCEBD9">
ssw_upgrade, /loud , /spawn
</pre>
This will update all the SSW branches listed in the SSW_INSTR environment variable (as well as their dependencies, and always includes gen and packages/binaries). This most likely will not update every branch in your SSW tree since you probably run SSW with a subset of branches activated.
* To upgrade specific branches, type for example
<pre style="background-color: #FCEBD9">
ssw_upgrade, /lasco, /spawn
</pre>

===== Possible Errors =====
* '''Error: bash:wget command not found''': Indicates that wget was installed using bash and csh/tcsh is not able to locate it. To resolve, simply give the path to wget in the upgrade command
<pre style="background-color: #FCEBD9">
ssw_upgrade, /loud, /spawn, wget = 'path/to/wget'
</pre>

* '''RD_TFILE: No file: /usr/local/ssw/site/mirror/ssw_upgrade.mirror''': check if the "'''/spawn'''" keyword has been missed

* '''Other errors with WGET ''': Sometimes using an older version of wget gives errors during SSW_upgrade, in this case add the keyword "'''/lmsal'''" to the upgrade command. This allows use of older wget versions.

Guide to Upgrade SolarSoft(SSW)

2022-04-05T22:02:21Z

Schhabra:

This guide compiles all the (scattered) steps to upgrade SSW taken from different sources.
SSW upgrade using passive_ftp was stopped in August 2019. The current way to upgrade SSW is to use the WGET executable. If you have done an update on SSW since August 2019 and your system already has WGET, any further upgrade can be made by directly following the steps underlined [https://hesperia.gsfc.nasa.gov/rhessi3/docs/ssw_update_help.html here]. For Linux and MacOS you can check if you already have WGET by typing the command "which wget" in the bash shell.

If your system was not updated since August 2019, follow the steps below:

=== MacOS ===
In order to perform the upgrade, you must be the owner of the SSW tree.
==== 1. Installing wget ====
'''USING HOMEBREW'''

To install homebrew, type the following command in the terminal:
<pre style="background-color: #FCEBD9">
/usr/bin/ruby -e "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/install)"
</pre>

Once homebrew is installed, you can simply install wget by using the following command:
<pre style="background-color: #FCEBD9">
brew install wget
</pre>

'''USING MacPorts'''

Alternatively, wget can also be installed using MacPorts. Directions to install MacPorts for specific OS versions are given [https://www.macports.org/install.php here]
After installation, the following command will install wget:

<pre style="background-color: #FCEBD9">
sudo port install wget
</pre>

==== 2. Prepare SSW to use wget for the first time ====
* Go to [http://www.lmsal.com/solarsoft/] and scroll all the way down to the SSW installation form.
* For "Installation Type", select "Upgrade Existing".
* For "Transfer Protocol", select "curl".
* Specify the existing path to SSW on your computer. (You can find it in your ~/.bashrc or ~/.cshrc)
* Click on "Generate Installation Script" and download it.
* If you are in the bash shell, please change to csh or tcsh,
<pre style="background-color: #FCEBD9">
% tcsh
Execute the script using the C-Shell
% csh -f scriptname
</pre>

==== 3. Upgrade SSW ====
* While in SSWIDL type:
<pre style="background-color: #FCEBD9">
ssw_upgrade, /loud , /spawn
</pre>
This will update all the SSW branches listed in the SSW_INSTR environment variable (as well as their dependencies, and always includes gen and packages/binaries). This most likely will not update every branch in your SSW tree since you probably run SSW with a subset of branches activated.
* To upgrade specific branches, type for example
<pre style="background-color: #FCEBD9">
ssw_upgrade, /lasco, /spawn
</pre>

===== Possible Errors =====
* '''Error: bash:wget command not found''': Indicates that wget was installed using bash and csh/tcsh is not able to locate it. To resolve, simply give the path to wget in the upgrade command
<pre style="background-color: #FCEBD9">
ssw_upgrade, /loud, /spawn, wget = 'path/to/wget'
</pre>

* '''RD_TFILE: No file: /usr/local/ssw/site/mirror/ssw_upgrade.mirror''': check if the "'''/spawn'''" keyword has been missed

* '''Other errors with WGET ''': Sometimes using an older version of wget gives errors during SSW_upgrade, in this case add the keyword "'''/lmsal'''" to the upgrade command. This allows use of older wget versions.

Guide to Upgrade SolarSoft(SSW)

2022-04-05T22:01:35Z

Schhabra:

This guide compiles all the (scattered) steps to upgrade SSW taken from different sources.
SSW upgrade using passive_ftp was stopped in August 2019. The current way to upgrade SSW is to use the WGET executable. If you have done an update on SSW since August 2019 and your system already has WGET, any further upgrade can be made by directly following the steps underlined [https://hesperia.gsfc.nasa.gov/rhessi3/docs/ssw_update_help.html here]. For Linux and MacOS you can check if you already have WGET by typing the command "which wget" in the bash shell.

If your system was not updated since August 2019, follow the steps below:

=== MacOS ===
In order to perform the upgrade, you must be the owner of the SSW tree.
==== 1. Installing wget ====
'''USING HOMEBREW'''

To install homebrew, type the following command in the terminal:
<pre style="background-color: #FCEBD9">
/usr/bin/ruby -e "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/install)"
</pre>

Once homebrew is installed, you can simply install wget by using the following command:
<pre style="background-color: #FCEBD9">
brew install wget
</pre>

'''USING MacPorts'''

Alternatively, wget can also be installed using MacPorts. Directions to install MacPorts for specific OS versions are given [https://www.macports.org/install.php here]
After installation, the following command will install wget:

<pre style="background-color: #FCEBD9">
sudo port install wget
</pre>

==== 2. Prepare SSW to use wget for the first time ====
* Go to [http://www.lmsal.com/solarsoft/] and scroll all the way down to the SSW installation form.
* For "Installation Type", select "Upgrade Existing".
* For "Transfer Protocol", select "curl".
* Specify the existing path to SSW on your computer. (You can find it in your ~/.bashrc or ~/.cshrc)
* Click on "Generate Installation Script" and download it.
* If you are in the bash shell, please change to csh or tcsh,
<pre style="background-color: #FCEBD9">
% tcsh
Execute the script using the C-Shell
% csh -f scriptname
</pre>

==== 3. Upgrade SSW ====
* While in SSWIDL type:
<pre style="background-color: #FCEBD9">
ssw_upgrade, /loud , /spawn
</pre>
This will update all the SSW branches listed in the SSW_INSTR environment variable (as well as their dependencies, and always includes gen and packages/binaries). This most likely will not update every branch in your SSW tree since you probably run SSW with a subset of branches activated.
* To upgrade specific branches, type for example
<pre style="background-color: #FCEBD9">
ssw_upgrade, /lasco, /spawn
</pre>

===== Possible Errors =====
* '''Error: bash:wget command not found''': Indicates that wget was installed using bash and csh/tcsh is not able to locate it. To resolve, simply give the path to wget in the upgrade command
<pre style="background-color: #FCEBD9">
ssw_upgrade, /loud, /spawn, wget = 'path/to/wget'
</pre>

* '''RD_TFILE: No file: /usr/local/ssw/site/mirror/ssw_upgrade.mirror''': check if the "''/spawn''" keyword has been missed

* '''Other errors with WGET ''': Sometimes using an older version of wget gives errors during SSW_upgrade, in this case add the keyword "''/lmsal''" to the upgrade command. This allows use of older wget versions.

Tohban EOVSA Imaging Tutorial A-Z V2

2022-04-04T13:49:42Z

Schhabra:

This tutorial describes the step-by-step procedure to download EOVSA IDB data, obtain and calibrate the measurement sets (.ms), and, transfer them to the inti server for self-calibration and further analysis.

=== Pre-requisites ===
Accounts on Pipeline and Inti or Baozi servers. Sijie and Bin can help with the account set-up for pipeline, inti and boazi.

==== How to connect to pipeline ====
'''Windows:''' On a windows machine Mobaxterm can be used to connect to Pipeline. For details, see ...

'''Linux/Unix:''' On a linux/Mac the ssh commands can be used:

<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
ssh -L 5902:helios.solar.pvt:20000 guest@ovsa.njit.edu ### when prompted, enter password to guest@ovsa ###
ssh -XY your_username@pipeline ### add your username and enter password to your pipeline account, when prompted ###
</pre>

Alternatively, to avoid typing the longer commands, proxy commands can be added to the ~/.ssh/config file:

<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
vim ~/.ssh/config ### press 'i' to insert content and esc +':wq' to write and quit ###

Host *
ForwardAgent yes

Host *
ForwardX11 yes

Host ovsa
HostName ovsa.njit.edu
User guest

Host pipeline
Hostname pipeline.solar.pvt
User your_username ### Insert your pipeline username here ###
ProxyCommand ssh -W %h:%p guest@ovsa.njit.edu
</pre>

Upon saving the .ssh/config file and restarting the terminal, you can connect to pipeline directly using:

<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
ssh -XY pipeline
</pre>

==== How to connect to the inti server ====
NOTE: To connect to inti you must have a NJIT UCID. The arcs have granted access to inti for all SRG members. If you encounter any issue with the login. Please let arcs folks (arcs@njit.edu) know.

'''Windows:''' As mentioned above, on a windows machine Mobaxterm can be used to connect to inti. For details, see ...

'''Linux/Unix:''' On a linux/Mac the ssh command can be used:
<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
ssh -XY UCID@inti.hpcnet.campus.njit.edu ### insert UCID here and enter your NJIT UCID password, when prompted ###
</pre>

When logging in for the first time, please add the following lines (demarcated by #) in your ~/.bashrc file in order to use CASA on inti.

<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
#### setting start ####
if [ $HOSTNAME == "baozi.hpcnet.campus.njit.edu" ]; then
source /srg/.setenv_baozi
fi
if [ $HOSTNAME == "inti.hpcnet.campus.njit.edu" ]; then
source /inti/.setenv_inti
fi
if [ $HOSTNAME == "guko.resource.campus.njit.edu" ]; then
source /data/data/.setenv_guko
fi
#### setting end ####
</pre>
Proxy commands for inti can also be added to the ~/.ssh/config file for convenience:
<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
vim ~/.ssh/config ### press 'i' to insert content and esc +':wq' to write and quit ###

Host inti
Hostname inti.hpcnet.campus.njit.edu
User UCID ### insert NJIT UCID here ###
</pre>
after the proxy is set up, connect to inti by simply typing:
<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
ssh -XY inti ### enter your NJIT UCID password when prompted ###
</pre>

=== Downloading data and Preliminary Calibration ===

==== Step 1: Downloading raw (IDB) data on Pipeline machine====
The initial downloading of data and applying refcal and phasecal to the .ms files is performed on the pipeline since the complete EOVSA SQL database is stored here for now.

If you are not using mobaxterm, directly SSH to Pipeline through your Linux or Mac computer (as shown above) and '''start tcsh to run CASA'''.

The following commands (run within CASA) can be used to import a single IDB file if the given time-range is less than 10 mins.
<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
from astropy.time import Time
import os
trange = Time(['2017-08-21 20:15:00', '2017-08-21 20:25:00']) ###Change accordingly###

#### (Optional) change output path, default current directory "./" #######
outpath = './msdata/' ###Change accordingly###
if not os.path.exists(outpath):
os.makedirs(outpath)
##########################################################################

msfiles = importeovsa(idbfiles=trange, ncpu=1, visprefix=outpath)
</pre>

'''NOTE:''' The above step currently gives an error with importeovsa. This is because the data location was changed and the code doesn't know where to look for the IDB files. Sijie is looking into the problem. We will update the page when it is fixed. For now, please use the method below for all time ranges.

OR (for a time range longer than 10 minutes)

<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
from suncasa.tasks import task_calibeovsa as calibeovsa
from suncasa.tasks import task_importeovsa as timporteovsa
from split_cli import split_cli as split
import dump_tsys as dt
from util import Time
import numpy as np
import os
from glob import glob
from eovsapy import util

trange = Time(['2017-08-21 20:15:00', '2017-08-21 20:35:00']) ###Change accordingly###
idbdir = util.get_idbdir(trange[0])

info = dt.rd_fdb(trange[0])
for k, v in info.iteritems():
info[k] = info[k][~(info[k] == '')]

sidx = np.where(
np.logical_and(info['SOURCEID'] == 'Sun', info['PROJECTID'] == 'NormalObserving') & np.logical_and(
info['ST_TS'].astype(np.float) >= trange[0].lv,
info['ST_TS'].astype(np.float) <= trange[
1].lv))
filelist = info['FILE'][sidx]

outpath = './msdata/' ###Change accordingly###
if not os.path.exists(outpath):
os.makedirs(outpath)
inpath = idbdir + '{}/'.format(trange[0].datetime.strftime("%Y%m%d"))
ncpu = 1

msfiles = timporteovsa.importeovsa(idbfiles=[inpath + ll for ll in filelist], ncpu=ncpu, timebin="0s", width=1,
visprefix=outpath,
nocreatms=False, doconcat=False,
modelms="", doscaling=False, keep_nsclms=False, udb_corr=True)
</pre>

If you come across errors with calibeovsa, add the following lines to your ~/.casa/init.py file.
<pre style="background-color: #FCEBD9">
vim ~/.ssh/config ### press 'i' to insert content and esc +':wq' to write and quit ###
import sys
sys.path.append('/common/python')
sys.path.append('/common/python/packages/pipeline_casa')
execfile('/common/python/suncasa/tasks/mytasks.py')
</pre>

==== Step 2: Concatenate multiple .ms files and Calibrate====
'''Follow this step if there are more than one .ms files for the chosen time-range. If not, move to step 3.'''

<pre style="background-color: #FCEBD9">
# This is to set the path/name for the concatenated files
concatvis = os.path.basename(msfiles[0])[:11] + '_concat_cal.ms'
vis = calibeovsa.calibeovsa(msfiles, doconcat=True, concatvis=concatvis, caltype=['refpha','phacal'], doimage=True)
### If doimage=True, a quicklook image will be produced (by integrating over the entire time) as shown below ###

# Split the concatenated .ms and replace data column by the 'corrected' data. Append '_cal' to the concatenated .ms
vis_str = str(' '.join(vis))
caled_vis=vis_str.replace('.ms','_cal.ms')
split(vis=' '.join(vis),outputvis=caled_vis,datacolumn='corrected',timerange='',correlation='XX')
</pre>

[[file:Figure1_imagingtutorial.png|frame|center|800px|'''Figure 1:''' Quick-look full-Sun image after the initial calibration.]]

==== Step 3: Calibration ====
This will calibrate the input visibility, write out calibration tables under /data1/eovsa/caltable/, and apply the calibration.
<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
vis = calibeovsa.calibeovsa('IDB20170821202020.ms', caltype=['refpha','phacal'], doimage=True) ###Change the vis filename accordingly###
# Append '_cal' to the ms filename and split the corrected column to the new caled ms
vis_str = str(' '.join(vis))
caled_vis=vis_str.replace('.ms','_cal.ms')
split(vis=' '.join(vis),outputvis=caled_vis,datacolumn='corrected',timerange='',correlation='XX')
</pre>

One needs to transfer the created caled ms data files (xxx_concat_cal.ms or xxx_cal.ms) from pipeline to inti server, which has all the casatasks installed, in order to run the rest of the imaging steps.

=== Connection details to Inti server ===
For Windows, on Mobaxterm,

#With your NJIT VPN connected, connect to one of the afsconnect servers (for example, afsaccess3.njit.edu) using your UCID and password.
#ssh -X UCID@inti.hpcnet.campus.njit.edu

To avoid typing the full inti address each time you attempt for ssh, you may wish to add the following lines with your username to C:\Program Files\Git\etc\ssh\ssh_config on Windows and /.ssh/config on Mac and Linux machines.

<pre style="background-color: #FCEBD9">
Host inti
Hostname inti.hpcnet.campus.njit.edu
User USERNAME ###Insert UCID/inti username here###
</pre>

To have sufficient disk space with EOVSA data analysis over Inti, use your dedicated directory YOURDIRECTORY at the location given below. If you do not have a directory, Please take help from Sijie in creating one.
<pre style="background-color: #FCEBD9">
cd /inti/data/users/YOURDIRECTORY
</pre>

For Linux or Mac machine,
ssh -X UCID@inti.hpcnet.campus.njit.edu

=== Transferring data files between servers ===
For directly transferring your calibrated .ms data between the Pipeline and Inti servers, follow the below given steps.
<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
1. Log into Inti using your username.
ssh -X USERNAME@inti.hpcnet.campus.njit.edu

Then create a tunnel into Pipeline from Inti.
ssh -L 8888:pipeline.solar.pvt:22 guest@ovsa.njit.edu

2. Log into Inti again from a new terminal.

Change to your working directory and give this command to copy your data on Pipeline.
scp -v -C -r -P 8888 USERNAMEofPipeline@localhost:PATHofMSDATA/MSfilename ./
Eg: scp -v -C -r -P 8888 shaheda@localhost:/data1/shaheda/IDB20220118173922_cal.ms ./
where, MSfilename, PATHofMSDATA are your .ms data and its path on Pipeline machine.
</pre>

Alternatively, one can follow the below procedure to do the transfer.

For Windows, on mobaxterm, drag and drop the ms file to your local machine or use scp command as given below.

<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
scp -r -C userid@pipeline:/your/folder/msfile /localmachine/destination/folder/
</pre>

On mobaxterm and on your local terminal, use the following command to finally copy the ms file to Inti.

<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
scp -r -C /localmachine/destination/folder/msfile UCID@inti.hpcnet.campus.njit.edu:/inti/data/users/YOURDIRECTORY
</pre>

For Mac and Linux, SCP can be used in the same way. Add the following lines to your SSH config file, to bypass ovsa.njit.edu from copying.
vi ~/.ssh/config
<pre style="background-color: #FCEBD9">
Host ovsa
HostName ovsa.njit.edu
User guest
Host pipeline
Hostname pipeline.solar.pvt
User userid
ProxyCommand ssh -W %h:%p guest@ovsa.njit.edu
</pre>

=== Software details on the servers ===
On Inti,
when logging in for the first time, please add the following lines to your accounts ~/.bashrc file.

>>vi ~/.bashrc
Insert the text given below and save it.

<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
#### setting start ####
if [ $HOSTNAME == "baozi.hpcnet.campus.njit.edu" ]; then
source /srg/.setenv_baozi
fi
if [ $HOSTNAME == "inti.hpcnet.campus.njit.edu" ]; then
source /inti/.setenv_inti
fi
if [ $HOSTNAME == "guko.resource.campus.njit.edu" ]; then
source /data/data/.setenv_guko
fi
#### setting end ####
</pre>

Both CASA 5 and 6 are available on Inti.

Enter the bash environment on inti, and load the desired casa environment.
To load CASA 5, enter bash environment by giving >>bash
<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
>> loadcasa5
>> suncasa #This should load the software making you ready for the analysis
</pre>

Or to load CASA 6, enter bash environment by giving >>bash
<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
>> loadcasa6
>> ipython #This should load the software making you ready for the analysis
</pre>
Here, for example, to use clean, first start ipython as given above, then type in >>from casatasks import tclean

====Step 4: Self-calibration====
[[file:Figure3_imagingtutorial.png|thumb|center|500px|Figure 2: Cotton-Schwab clean major and minor cycles. [Source: http://www.aoc.nrao.edu/~rurvashi/ImagingAlgorithmsInCasa/node2.html].]]
Follow the below given steps to run the self-calibration of the imaging data and produce the calibrated images in .fits format. https://github.com/binchensun/casa-eovsa/blob/master/slfcal_example.py

<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
from suncasa.utils import helioimage2fits as hf
import os
import numpy as np
import pickle
from matplotlib import gridspec as gridspec
from sunpy import map as smap
from matplotlib import pyplot as plt
from split_cli import split_cli as split
import time

# =========== task handlers =============
dofullsun = 1 # initial full-sun imaging ###Change accordingly###
domasks=1 # get masks ###Change accordingly###
doslfcal=1 # main cycle of doing selfcalibration ###Change accordingly###
doapply=1 # apply the results ###Change accordingly###
doclean_slfcaled=1 # perform clean for self-calibrated data ###Change accordingly###

# ============ declaring the working directories ============
workdir = os.getcwd()+'/' #main working directory. Using current directory in this example
slfcaldir = workdir+'slfcal/' #place to put all selfcalibration products
imagedir = slfcaldir+'images/' #place to put all selfcalibration images
maskdir = slfcaldir+'masks/' #place to put clean masks
imagedir_slfcaled = slfcaldir+'images_slfcaled/' #place to put final self-calibrated images
caltbdir = slfcaldir+'caltbs/' # place to put calibration tables
# make these directories if they do not already exist
dirs = [workdir, slfcaldir, imagedir, maskdir, imagedir_slfcaled, caltbdir]
for d in dirs:
if not os.path.exists(d):
os.makedirs(d)

# ============ Split a short time for self-calibration ===========
# input visibility
ms_in = workdir + 'IDB20170821202020_cal.ms' ###Change the initial calibrated (through calibeovsa) vis accordingly###
# output, selfcaled, visibility
ms_slfcaled = workdir + os.path.basename(ms_in).replace('cal','slfcaled')
# intermediate small visibility for selfcalbration
# selected time range for generating self-calibration solutions
trange='2017/08/21/20:21:10~2017/08/21/20:21:30' ###Change accordingly###
slfcalms = slfcaldir+'slfcalms.XX.slfcal'
slfcaledms = slfcaldir+'slfcalms.XX.slfcaled'
if not os.path.exists(slfcalms):
split(vis=ms_in,outputvis=slfcalms,datacolumn='data',timerange=trange,correlation='XX')

# ============ Prior definitions for spectral windows, antennas, pixel numbers =========
spws=[str(s+1) for s in range(30)]
antennas='0~12'
npix=512
nround=3 #number of slfcal cycles

# =========== Step 1, doing a full-Sun image to find out phasecenter and appropriate field of view =========
if dofullsun:
#initial mfs clean to find out the image phase center
im_init='fullsun_init'
os.system('rm -rf '+im_init+'*')
tclean(vis=slfcalms,
antenna='0~12',
imagename=im_init,
spw='1~15',
specmode='mfs',
timerange=trange,
imsize=[npix],
cell=['5arcsec'],
niter=1000,
gain=0.05,
stokes='I',
restoringbeam=['30arcsec'],
interactive=False,
pbcor=True)

hf.imreg(vis=slfcalms,imagefile=im_init+'.image.pbcor',fitsfile=im_init+'.fits',
timerange=trange,usephacenter=False,verbose=True)
clnjunks = ['.flux', '.mask', '.model', '.psf', '.residual','.sumwt','.pb','.image'] #Do not run the next 4 lines if needed to view and assess the subset of clean process images
for clnjunk in clnjunks:
if os.path.exists(im_init + clnjunk):
os.system('rm -rf '+im_init + clnjunk)

from sunpy import map as smap
from matplotlib import pyplot as plt
fig = plt.figure(figsize=(6,6))
ax = fig.add_subplot(111)
eomap=smap.Map(im_init+'.fits')
#eomap.data=eomap.data.reshape((npix,npix))
eomap.plot_settings['cmap'] = plt.get_cmap('jet')
eomap.plot(axes = ax)
eomap.draw_limb()
plt.show()
viewer(im_init+'.image.pbcor')
</pre>
[[file:Figure2_imagingtutorial.png|thumb|center|500px|Figure 3: Full-Sun image after initial clean to find the flare location.]]

<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
# parameters specific to the event (found from step 1)
phasecenter='J2000 10h02m59 11d58m07' ###Change accordingly###
xran=[280,480] ###Change accordingly###
yran=[-50,150] ###Change accordingly###

# =========== Step 2 (optional), generate masks =========
# if skipped, will not use any masks
if domasks:
clearcal(slfcalms)
delmod(slfcalms)
antennas='0~12'
pol='XX'
imgprefix=maskdir+'slf_t0'

# step 1: set up the clean masks
img_init=imgprefix+'_init_ar_'
os.system('rm -rf '+img_init+'*')
#spwrans_mask=['1~5','6~12','13~20','21~30']
spwrans_mask=['1~12']
#convert to a list of spws
spwrans_mask_list = [[str(i) for i in (np.arange(int(m.split('~')[0]),int(m.split('~')[1])))] for m in spwrans_mask]
masks=[]
imnames=[]
for spwran in spwrans_mask:
imname=img_init+spwran.replace('~','-')
try:
tclean(vis=slfcalms,
antenna='0~12',
imagename=imname,
spw=spwran,
specmode='mfs',
timerange=trange,
imsize=[npix],
cell=['2arcsec'],
niter=1000,
gain=0.05,
stokes='XX',
restoringbeam=['20arcsec'],
phasecenter=phasecenter,
weighting='briggs',
robust=1.0,
interactive=True,
datacolumn='data',
pbcor=True,
savemodel='modelcolumn')
imnames.append(imname+'.image')
masks.append(imname+'.mask')
clnjunks = ['.flux', '.model', '.psf', '.residual'] #Do not run the next 4 lines if needed to view and assess the subset of clean process images
for clnjunk in clnjunks:
if os.path.exists(imname + clnjunk):
os.system('rm -rf '+ imname + clnjunk)
except:
print('error in cleaning spw: '+spwran)

pickle.dump(masks,open(slfcaldir+'masks.p','wb'))
</pre>
[[file:Figure4_imagingtutorial.PNG|thumb|center|800px|Figure 4: Interactive clean window to create masks over the source. The white outline surrounding the source is the mask selected by the polygon drawing option.]]

The outlines drawn for masks can be created by any of the icons with the letter 'R' in the Viewer window. The instructions for doing this can be found by hovering over those icons.

<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
# =========== Step 3, main step of selfcalibration =========
if doslfcal:
if os.path.exists(slfcaldir+'masks.p'):
masks=pickle.load(open(slfcaldir+'masks.p','rb'))
if not os.path.exists(slfcaldir+'masks.p'):
print 'masks do not exist. Use default mask'
masks=[]
os.system('rm -rf '+imagedir+'*')
os.system('rm -rf '+caltbdir+'*')
#first step: make a mock caltable for the entire database
print('Processing ' + trange)
slftbs=[]
calprefix=caltbdir+'slf'
imgprefix=imagedir+'slf'
tb.open(slfcalms+'/SPECTRAL_WINDOW')
reffreqs=tb.getcol('REF_FREQUENCY')
bdwds=tb.getcol('TOTAL_BANDWIDTH')
cfreqs=reffreqs+bdwds/2.
tb.close()
# starting beam size at 3.4 GHz in arcsec #Change accordingly
sbeam=40.
strtmp=[m.replace(':','') for m in trange.split('~')]
timestr='t'+strtmp[0]+'-'+strtmp[1]
refantenna='0'
# number of iterations for each round
niters=[100, 300, 500]
# roburst value for weighting the baselines
robusts=[1.0, 0.5, 0.0]
# apply calibration tables? Set to true for most cases
doapplycal=[1,1,1]
# modes for calibration, 'p' for phase-only, 'a' for amplitude only, 'ap' for both
calmodes=['p','p','a']
# setting uvranges for model image (optional, not used here)
uvranges=['','','']
for n in range(nround):
slfcal_tb_g= calprefix+'.G'+str(n)
fig = plt.figure(figsize=(8.4,7.))
gs = gridspec.GridSpec(5, 6)
for s,sp in enumerate(spws):
print 'processing spw: '+sp
cfreq=cfreqs[int(sp)]
# setting restoring beam size (not very useful for selfcal anyway, but just to see the results)
bm=max(sbeam*cfreqs[1]/cfreq, 6.)
slfcal_img = imgprefix+'.spw'+sp.zfill(2)+'.slfcal'+str(n)
# only the first round uses nearby spws for getting initial model
if n == 0:
spbg=max(int(sp)-2,1)
sped=min(int(sp)+2,30)
spwran=str(spbg)+'~'+str(sped)
print('using spw {0:s} as model'.format(spwran))
if 'spwrans_mask' in vars():
for m, spwran_mask in enumerate(spwrans_mask):
if sp in spwran_mask:
mask = masks[m]
print('using mask {0:s}'.format(mask))
findmask = True
if not findmask:
print('mask not found. Do use any masks')
else:
spwran = sp
if 'spwrans_mask' in vars():
for m, spwran_mask in enumerate(spwrans_mask):
if sp in spwran_mask:
mask = masks[m]
print 'using mask {0:s}'.format(mask)
findmask = True
if not findmask:
print('mask not found. Do use any masks')
try:
tclean(vis=slfcalms,
antenna=antennas,
imagename=slfcal_img,
uvrange=uvranges[n],
spw=spwran,
specmode='mfs',
timerange=trange,
imsize=[npix],
cell=['2arcsec'],
niter=niters[n],
gain=0.05,
stokes='XX', #use pol XX image as the model
weighting='briggs',
robust=robusts[n],
phasecenter=phasecenter,
mask=mask,
restoringbeam=[str(bm)+'arcsec'],
pbcor=False,
interactive=False,
savemodel='modelcolumn')
if os.path.exists(slfcal_img+'.image'):
fitsfile=slfcal_img+'.fits'
hf.imreg(vis=slfcalms,imagefile=slfcal_img+'.image',fitsfile=fitsfile,
timerange=trange,usephacenter=False,toTb=True,verbose=False,overwrite=True)
clnjunks = ['.mask','.flux', '.model', '.psf', '.residual', '.image','.pb','.image.pbcor','.sumwt'] #Do not run the next 4 lines if needed to view and assess the subset of clean process images
for clnjunk in clnjunks:
if os.path.exists(slfcal_img + clnjunk):
os.system('rm -rf '+ slfcal_img + clnjunk)
ax = fig.add_subplot(gs[s])
eomap=smap.Map(fitsfile)
eomap.plot_settings['cmap'] = plt.get_cmap('jet')
eomap.plot(axes = ax)
eomap.draw_limb()
#eomap.draw_grid()
ax.set_title(' ')
ax.get_xaxis().set_visible(False)
ax.get_yaxis().set_visible(False)
ax.set_xlim(xran)
ax.set_ylim(yran)
os.system('rm -f '+ fitsfile)

except:
print 'error in cleaning spw: '+sp
print 'using nearby spws for initial model'
sp_e=int(sp)+2
sp_i=int(sp)-2
if sp_i < 1:
sp_i = 1
if sp_e > 30:
sp_e = 30
sp_=str(sp_i)+'~'+str(sp_e)
try:
tget(tclean)
spw=sp_
print('using spw {0:s} as model'.format(sp_))
tclean()
except:
print 'still not successful. abort...'
break

gaincal(vis=slfcalms, refant=refantenna,antenna=antennas,caltable=slfcal_tb_g,spw=sp, uvrange='',\
gaintable=[],selectdata=True,timerange=trange,solint='inf',gaintype='G',calmode=calmodes[n],\
combine='',minblperant=4,minsnr=2,append=True)
if not os.path.exists(slfcal_tb_g):
print 'No solution found in spw: '+sp
figname=imagedir+'slf_t0_n{:d}.png'.format(n)
plt.subplots_adjust(left=0, bottom=0, right=1, top=1, wspace=0, hspace=0)
plt.savefig(figname)
time.sleep(10)
plt.close()

if os.path.exists(slfcal_tb_g):
slftbs.append(slfcal_tb_g)
slftb=[slfcal_tb_g]
os.chdir(slfcaldir)
if calmodes[n] == 'p':
plotcal(caltable=slfcal_tb_g,antenna='1~12',xaxis='freq',yaxis='phase',\
subplot=431,plotrange=[-1,-1,-180,180],iteration='antenna',figfile=slfcal_tb_g+'.png',showgui=False)
if calmodes[n] == 'a':
plotcal(caltable=slfcal_tb_g,antenna='1~12',xaxis='freq',yaxis='amp',\
subplot=431,plotrange=[-1,-1,0,2.],iteration='antenna',figfile=slfcal_tb_g+'.png',showgui=False)
os.chdir(workdir)

if doapplycal[n]:
clearcal(slfcalms)
delmod(slfcalms)
applycal(vis=slfcalms,gaintable=slftb,spw=','.join(spws),selectdata=True,\
antenna=antennas,interp='nearest',flagbackup=False,applymode='calonly',calwt=False)

if n < nround-1:
prompt=raw_input('Continuing to selfcal?')
#prompt='y'
if prompt.lower() == 'n':
if os.path.exists(slfcaledms):
os.system('rm -rf '+slfcaledms)
split(slfcalms,slfcaledms,datacolumn='corrected')
print 'Final calibrated ms is {0:s}'.format(slfcaledms)
break
if prompt.lower() == 'y':
slfcalms_=slfcalms+str(n)
if os.path.exists(slfcalms_):
os.system('rm -rf '+slfcalms_)
split(slfcalms,slfcalms_,datacolumn='corrected')
slfcalms=slfcalms_
else:
if os.path.exists(slfcaledms):
os.system('rm -rf '+slfcaledms)
split(slfcalms,slfcaledms,datacolumn='corrected')
print 'Final calibrated ms is {0:s}'.format(slfcaledms)
</pre>

<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
# =========== Step 4: Apply self-calibration tables =========
if doapply:
import glob
os.chdir(workdir)
clearcal(ms_in)
clearcal(slfcalms)
applycal(vis=slfcalms,gaintable=slftbs,spw=','.join(spws),selectdata=True,\
antenna=antennas,interp='linear',flagbackup=False,applymode='calonly',calwt=False)
applycal(vis=ms_in,gaintable=slftbs,spw=','.join(spws),selectdata=True,\
antenna=antennas,interp='linear',flagbackup=False,applymode='calonly',calwt=False)
if os.path.exists(ms_slfcaled):
os.system('rm -rf '+ms_slfcaled)
split(ms_in, ms_slfcaled,datacolumn='corrected')
</pre>
[[file:Slf.G0.png|thumb|center|500px|Figure 1: Phase before self-calibration]]
[[file:Slf.G1.png|thumb|center|500px|Figure 2: Phase after self-calibration]]

<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
# =========== Step 5: Generate final self-calibrated images (optional) =========
if doclean_slfcaled:
import glob
pol='XX'
print('Processing ' + trange)
img_final=imagedir_slfcaled+'/slf_final_{0}_t0'.format(pol)
vis = ms_slfcaled
spws=[str(s+1) for s in range(30)]
tb.open(vis+'/SPECTRAL_WINDOW')
reffreqs=tb.getcol('REF_FREQUENCY')
bdwds=tb.getcol('TOTAL_BANDWIDTH')
cfreqs=reffreqs+bdwds/2.
tb.close()
sbeam=30.
from matplotlib import gridspec as gridspec
from sunpy import map as smap
from matplotlib import pyplot as plt
fitsfiles=[]
for s,sp in enumerate(spws):
cfreq=cfreqs[int(sp)]
bm=max(sbeam*cfreqs[1]/cfreq,4.)
imname=img_final+'_s'+sp.zfill(2)
fitsfile=imname+'.fits'
if not os.path.exists(fitsfile):
print 'cleaning spw {0:s} with beam size {1:.1f}"'.format(sp,bm)
try:
tclean(vis=vis,
antenna=antennas,
imagename=imname,
spw=sp,
specmode='mfs',
timerange=trange,
imsize=[npix],
cell=['1arcsec'],
niter=1000,
gain=0.05,
stokes=pol,
weighting='briggs',
robust=2.0,
restoringbeam=[str(bm)+'arcsec'],
phasecenter=phasecenter,
mask='',
pbcor=True,
interactive=False)
except:
print 'cleaning spw '+sp+' unsuccessful. Proceed to next spw'
continue
if os.path.exists(imname+'.image.pbcor'):
imn = imname+'.image.pbcor'
hf.imreg(vis=vis,imagefile=imn,fitsfile=fitsfile,
timerange=trange,usephacenter=False,toTb=True,verbose=False)
fitsfiles.append(fitsfile)
junks=['.flux','.model','.psf','.residual','.mask','.image','.pb','.image.pbcor','.sumwt'] #Do not run the next 4 lines if needed to view and assess the subset of clean process images
for junk in junks:
if os.path.exists(imname+junk):
os.system('rm -rf '+imname+junk)
else:
print('fits file '+fitsfile+' already exists, skip clean...')
fitsfiles.append(fitsfile)

fig = plt.figure(figsize=(8.4,7.))
gs = gridspec.GridSpec(5, 6)
for s,sp in enumerate(spws):
cfreq=cfreqs[int(sp)]
ax = fig.add_subplot(gs[s])
eomap=smap.Map(fitsfiles[s])
eomap.plot_settings['cmap'] = plt.get_cmap('jet')
eomap.plot(axes = ax)
eomap.draw_limb()
ax.set_title(' ')
ax.get_xaxis().set_visible(False)
ax.get_yaxis().set_visible(False)
ax.set_xlim(xran)
ax.set_ylim(yran)
plt.text(0.98,0.85,'{0:.1f} GHz'.format(cfreq/1e9),transform=ax.transAxes,ha='right',color='w',fontweight='bold')
plt.subplots_adjust(left=0, bottom=0, right=1, top=1, wspace=0, hspace=0)
plt.show()

</pre>
The .fits files of the self calibrated images at each frequency for the given time are saved at /slfcal/images_slfcaled in your working directory.

[[file:Slf_t0_n0.png|thumb|center|500px|Figure 3: Multi-frequency images before self-calibration]]
[[file:Slf_t0_n1.png|thumb|center|500px|Figure 4: Multi-frequency images after self-calibration]]

====Step 5: Quick-look imaging ==== 
For spectral imaging analysis of the event, follow [http://www.ovsa.njit.edu/wiki/index.php/EOVSA_Data_Analysis_Tutorial#Spectral_Imaging_with_SunCASA this tutorial] using the self-calibrated data obtained from the previous step or use [https://colab.research.google.com/drive/1lSLLxgOG6b8kgu9Sk6kSKvrViyubnXG6?usp=sharing#scrollTo=xbXyyLmCFCGL this link].

<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
from suncasa.utils import qlookplot as ql ## (Optional) Supply the npz file of the dynamic spectrum from previous step.
## If not provided, the program will generate a new one from the visibility.
## set the time interval
from suncasa import dspec as ds
import time
visibility_data = 'IDB20170821202020_slfcaled.ms'
specfile = visibility_data + '.dspec.npz'
d = ds.Dspec(visibility_data, bl='4&9', specfile=specfile)

timerange = '19:02:00~19:02:10' ## select frequency range from 2.5 GHz to 3.5 GHz
spw = '2~5' ## select stokes XX
stokes = 'XX' ## turn off AIA image plotting, default is True
plotaia = False
xycen = [375, 45] ## image center for clean in solar X-Y in arcsec
cell=['2.0arcsec'] ## pixel size
imsize=[128] ## x and y image size in pixels.
fov = [100,100] ## field of view of the zoomed-in panels in unit of arcsec
spw = ['{}'.format(s) for s in range(1,31)]
clevels = [0.5, 1.0] ## contour levels to fill in between.
calpha=0.35 ## now tune down the alpha
restoringbeam=['6arcsec']
ql.qlookplot(vis=msfile, specfile=specfile, timerange=timerange, spw=spw, stokes=stokes, \
restoringbeam=restoringbeam,imsize=imsize,cell=cell, \
xycen=xycen,fov=fov,clevels=clevels,calpha=calpha)
</pre>

Page Last Updated: 03/29/2022

Tohban EOVSA Imaging Tutorial A-Z V2

2022-03-29T22:58:23Z

Schhabra:

This tutorial describes the step-by-step procedure to download EOVSA IDB data, obtain and calibrate the measurement sets (.ms), and, transfer them to the inti server for self-calibration and further analysis.

=== Pre-requisites ===
Accounts on Pipeline and Inti or Baozi servers. Sijie and Bin can help with the account set-up for pipeline, inti and boazi.

==== How to connect to pipeline ====
'''Windows:''' On a windows machine Mobaxterm can be used to connect to Pipeline. For details, see ...

'''Linux/Unix:''' On a linux/Mac the ssh commands can be used:

<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
ssh -L 5902:helios.solar.pvt:20000 guest@ovsa.njit.edu ### when prompted, enter password to guest@ovsa ###
ssh -XY your_username@pipeline ### add your username and enter password to your pipeline account, when prompted ###
</pre>

Alternatively, to avoid typing the longer commands, proxy commands can be added to the ~/.ssh/config file:

<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
vim ~/.ssh/config ### press 'i' to insert content and esc +':wq' to write and quit ###

Host *
ForwardAgent yes

Host *
ForwardX11 yes

Host ovsa
HostName ovsa.njit.edu
User guest

Host pipeline
Hostname pipeline.solar.pvt
User your_username ### Insert your pipeline username here ###
ProxyCommand ssh -W %h:%p guest@ovsa.njit.edu
</pre>

Upon saving the .ssh/config file and restarting the terminal, you can connect to pipeline directly using:

<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
ssh -XY pipeline
</pre>

==== How to connect to the inti server ====
NOTE: To connect to inti you must have a NJIT UCID. The arcs have granted access to inti for all SRG members. If you encounter any issue with the login. Please let arcs folks (arcs@njit.edu) know.

'''Windows:''' As mentioned above, on a windows machine Mobaxterm can be used to connect to inti. For details, see ...

'''Linux/Unix:''' On a linux/Mac the ssh command can be used:
<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
ssh -XY UCID@inti.hpcnet.campus.njit.edu ### insert UCID here and enter your NJIT UCID password, when prompted ###
</pre>

When logging in for the first time, please add the following lines (demarcated by #) in your ~/.bashrc file in order to use the casa on inti.

<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
#### setting start ####
if [ $HOSTNAME == "baozi.hpcnet.campus.njit.edu" ]; then
source /srg/.setenv_baozi
fi
if [ $HOSTNAME == "inti.hpcnet.campus.njit.edu" ]; then
source /inti/.setenv_inti
fi
if [ $HOSTNAME == "guko.resource.campus.njit.edu" ]; then
source /data/data/.setenv_guko
fi
#### setting end ####
</pre>
Proxy commands for inti can also be added to the ~/.ssh/config file for convenience:
<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
vim ~/.ssh/config ### press 'i' to insert content and esc +':wq' to write and quit ###

Host inti
Hostname inti.hpcnet.campus.njit.edu
User UCID ### insert NJIT UCID here ###
</pre>
after the proxy is set up, connect to inti by simply typing:
<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
ssh -XY inti ### enter your NJIT UCID password when prompted ###
</pre>

=== Downloading data and Preliminary Calibration ===

==== Step 1: Downloading raw data (IDB) on Pipeline machine====
The initial downloading of data and applying refcal and phasecal to the .ms files is performed on the pipeline since the complete EOVSA SQL database is stored here for now.

If you are not using mobaxterm, directly SSH to Pipeline through your Linux or Mac computer (as shown above) and '''start tcsh to run CASA'''.

The following commands (run within CASA) can be used to import a single IDB file if the given time-range is less than 10 mins.
<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
from astropy.time import Time
import os
trange = Time(['2017-08-21 20:15:00', '2017-08-21 20:25:00']) ###Change accordingly###
#### (Optional) change output path, default current directory "./" #####
outpath = './msdata/' ###Change accordingly###
if not os.path.exists(outpath):
os.makedirs(outpath)
######################################################
msfiles = importeovsa(idbfiles=trange, ncpu=1, visprefix=outpath)
</pre>

'''NOTE:''' The above step currently gives an error with importeovsa. This is because the data location was changed and the code doesn't know where to look for the IDB files. Sijie is looking into the problem. We will update the page when it is fixed. For now, please use the method below for all time ranges.

OR (for a time range longer than 10 minutes)

<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
from suncasa.tasks import task_calibeovsa as calibeovsa
from suncasa.tasks import task_importeovsa as timporteovsa
from split_cli import split_cli as split
import dump_tsys as dt
from util import Time
import numpy as np
import os
from glob import glob
from eovsapy import util

trange = Time(['2017-08-21 20:15:00', '2017-08-21 20:35:00']) ###Change accordingly###
idbdir = util.get_idbdir(trange[0])

info = dt.rd_fdb(trange[0])
for k, v in info.iteritems():
info[k] = info[k][~(info[k] == '')]

sidx = np.where(
np.logical_and(info['SOURCEID'] == 'Sun', info['PROJECTID'] == 'NormalObserving') & np.logical_and(
info['ST_TS'].astype(np.float) >= trange[0].lv,
info['ST_TS'].astype(np.float) <= trange[
1].lv))
filelist = info['FILE'][sidx]

outpath = './msdata/' ###Change accordingly###
if not os.path.exists(outpath):
os.makedirs(outpath)
inpath = idbdir + '{}/'.format(trange[0].datetime.strftime("%Y%m%d"))
ncpu = 1

msfiles = timporteovsa.importeovsa(idbfiles=[inpath + ll for ll in filelist], ncpu=ncpu, timebin="0s", width=1,
visprefix=outpath,
nocreatms=False, doconcat=False,
modelms="", doscaling=False, keep_nsclms=False, udb_corr=True)
</pre>

If you come across errors with calibeovsa, add the following lines to your ~/.casa/init.py file.
<pre style="background-color: #FCEBD9">
vim ~/.ssh/config ### press 'i' to insert content and esc +':wq' to write and quit ###
import sys
sys.path.append('/common/python')
sys.path.append('/common/python/packages/pipeline_casa')
execfile('/common/python/suncasa/tasks/mytasks.py')
</pre>

==== Step 2: Concatenate multiple .ms files and Calibrate====
'''Follow this step if there are more than one .ms files for the chosen time-range. If not, move to step 3.'''

<pre style="background-color: #FCEBD9">
# This is to set the path/name for the concatenated files
concatvis = os.path.basename(msfiles[0])[:11] + '_concat_cal.ms'
vis = calibeovsa.calibeovsa(msfiles, doconcat=True, concatvis=concatvis, caltype=['refpha','phacal'], doimage=True)
### If doimage=True, a quicklook image will be produced (by integrating over the entire time) as shown below ###
</pre>

### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ###
[[file:Figure1_imagingtutorial.png|frame|center|800px|'''Figure 1:''' Quick-look full-Sun image after the initial calibration.]]

==== Step 3: Calibration ====
This will calibrate the input visibility, write out calibration tables under /data1/eovsa/caltable/, and apply the calibration.
<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
vis = calibeovsa.calibeovsa('IDB20170821202020.ms', caltype=['refpha','phacal'], doimage=True) ###Change the vis filename accordingly###
# Append '_cal' to the ms filename and split the corrected column to the new caled ms
vis_str = str(' '.join(vis))
caled_vis=vis_str.replace('.ms','_cal.ms')
split(vis=' '.join(vis),outputvis=caled_vis,datacolumn='corrected',timerange='',correlation='XX')
</pre>

One needs to transfer the created caled ms data files (xxx_concat_cal.ms or xxx_cal.ms) from pipeline to inti server, which has all the casatasks installed, in order to run the rest of the imaging steps.

=== Connection details to Inti server ===
For Windows, on Mobaxterm,

#With your NJIT VPN connected, connect to one of the afsconnect servers (for example, afsaccess3.njit.edu) using your UCID and password.
#ssh -X UCID@inti.hpcnet.campus.njit.edu

To avoid typing the full inti address each time you attempt for ssh, you may wish to add the following lines with your username to C:\Program Files\Git\etc\ssh\ssh_config on Windows and /.ssh/config on Mac and Linux machines.

<pre style="background-color: #FCEBD9">
Host inti
Hostname inti.hpcnet.campus.njit.edu
User USERNAME ###Insert UCID/inti username here###
</pre>

To have sufficient disk space with EOVSA data analysis over Inti, use your dedicated directory YOURDIRECTORY at the location given below. If you do not have a directory, Please take help from Sijie in creating one.
<pre style="background-color: #FCEBD9">
cd /inti/data/users/YOURDIRECTORY
</pre>

For Linux or Mac machine,
ssh -X UCID@inti.hpcnet.campus.njit.edu

=== Transferring data files between servers ===
For directly transferring your calibrated .ms data between the Pipeline and Inti servers, follow the below given steps.
<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
1. Log into Inti using your username.
ssh -X USERNAME@inti.hpcnet.campus.njit.edu

Then create a tunnel into Pipeline from Inti.
ssh -L 8888:pipeline.solar.pvt:22 guest@ovsa.njit.edu

2. Log into Inti again from a new terminal.

Change to your working directory and give this command to copy your data on Pipeline.
scp -v -C -r -P 8888 USERNAMEofPipeline@localhost:PATHofMSDATA/MSfilename ./
Eg: scp -v -C -r -P 8888 shaheda@localhost:/data1/shaheda/IDB20220118173922_cal.ms ./
where, MSfilename, PATHofMSDATA are your .ms data and its path on Pipeline machine.
</pre>

Alternatively, one can follow the below procedure to do the transfer.

For Windows, on mobaxterm, drag and drop the ms file to your local machine or use scp command as given below.

<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
scp -r -C userid@pipeline:/your/folder/msfile /localmachine/destination/folder/
</pre>

On mobaxterm and on your local terminal, use the following command to finally copy the ms file to Inti.

<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
scp -r -C /localmachine/destination/folder/msfile UCID@inti.hpcnet.campus.njit.edu:/inti/data/users/YOURDIRECTORY
</pre>

For Mac and Linux, SCP can be used in the same way. Add the following lines to your SSH config file, to bypass ovsa.njit.edu from copying.
vi ~/.ssh/config
<pre style="background-color: #FCEBD9">
Host ovsa
HostName ovsa.njit.edu
User guest
Host pipeline
Hostname pipeline.solar.pvt
User userid
ProxyCommand ssh -W %h:%p guest@ovsa.njit.edu
</pre>

=== Software details on the servers ===
On Inti,
when logging in for the first time, please add the following lines to your accounts ~/.bashrc file.

>>vi ~/.bashrc
Insert the text given below and save it.

<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
#### setting start ####
if [ $HOSTNAME == "baozi.hpcnet.campus.njit.edu" ]; then
source /srg/.setenv_baozi
fi
if [ $HOSTNAME == "inti.hpcnet.campus.njit.edu" ]; then
source /inti/.setenv_inti
fi
if [ $HOSTNAME == "guko.resource.campus.njit.edu" ]; then
source /data/data/.setenv_guko
fi
#### setting end ####
</pre>

Both CASA 5 and 6 are available on Inti.

Enter the bash environment on inti, and load the desired casa environment.
To load CASA 5, enter bash environment by giving >>bash
<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
>> loadcasa5
>> suncasa #This should load the software making you ready for the analysis
</pre>

Or to load CASA 6, enter bash environment by giving >>bash
<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
>> loadcasa6
>> ipython #This should load the software making you ready for the analysis
</pre>
Here, for example, to use clean, first start ipython as given above, then type in >>from casatasks import tclean

====Step 4: Self-calibration====
[[file:Figure3_imagingtutorial.png|thumb|center|500px|Figure 2: Cotton-Schwab clean major and minor cycles. [Source: http://www.aoc.nrao.edu/~rurvashi/ImagingAlgorithmsInCasa/node2.html].]]
Follow the below given steps to run the self-calibration of the imaging data and produce the calibrated images in .fits format. https://github.com/binchensun/casa-eovsa/blob/master/slfcal_example.py

<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
from suncasa.utils import helioimage2fits as hf
import os
import numpy as np
import pickle
from matplotlib import gridspec as gridspec
from sunpy import map as smap
from matplotlib import pyplot as plt
from split_cli import split_cli as split
import time

# =========== task handlers =============
dofullsun = 1 # initial full-sun imaging ###Change accordingly###
domasks=1 # get masks ###Change accordingly###
doslfcal=1 # main cycle of doing selfcalibration ###Change accordingly###
doapply=1 # apply the results ###Change accordingly###
doclean_slfcaled=1 # perform clean for self-calibrated data ###Change accordingly###

# ============ declaring the working directories ============
workdir = os.getcwd()+'/' #main working directory. Using current directory in this example
slfcaldir = workdir+'slfcal/' #place to put all selfcalibration products
imagedir = slfcaldir+'images/' #place to put all selfcalibration images
maskdir = slfcaldir+'masks/' #place to put clean masks
imagedir_slfcaled = slfcaldir+'images_slfcaled/' #place to put final self-calibrated images
caltbdir = slfcaldir+'caltbs/' # place to put calibration tables
# make these directories if they do not already exist
dirs = [workdir, slfcaldir, imagedir, maskdir, imagedir_slfcaled, caltbdir]
for d in dirs:
if not os.path.exists(d):
os.makedirs(d)

# ============ Split a short time for self-calibration ===========
# input visibility
ms_in = workdir + 'IDB20170821202020_cal.ms' ###Change the initial calibrated (through calibeovsa) vis accordingly###
# output, selfcaled, visibility
ms_slfcaled = workdir + os.path.basename(ms_in).replace('cal','slfcaled')
# intermediate small visibility for selfcalbration
# selected time range for generating self-calibration solutions
trange='2017/08/21/20:21:10~2017/08/21/20:21:30' ###Change accordingly###
slfcalms = slfcaldir+'slfcalms.XX.slfcal'
slfcaledms = slfcaldir+'slfcalms.XX.slfcaled'
if not os.path.exists(slfcalms):
split(vis=ms_in,outputvis=slfcalms,datacolumn='data',timerange=trange,correlation='XX')

# ============ Prior definitions for spectral windows, antennas, pixel numbers =========
spws=[str(s+1) for s in range(30)]
antennas='0~12'
npix=512
nround=3 #number of slfcal cycles

# =========== Step 1, doing a full-Sun image to find out phasecenter and appropriate field of view =========
if dofullsun:
#initial mfs clean to find out the image phase center
im_init='fullsun_init'
os.system('rm -rf '+im_init+'*')
tclean(vis=slfcalms,
antenna='0~12',
imagename=im_init,
spw='1~15',
specmode='mfs',
timerange=trange,
imsize=[npix],
cell=['5arcsec'],
niter=1000,
gain=0.05,
stokes='I',
restoringbeam=['30arcsec'],
interactive=False,
pbcor=True)

hf.imreg(vis=slfcalms,imagefile=im_init+'.image.pbcor',fitsfile=im_init+'.fits',
timerange=trange,usephacenter=False,verbose=True)
clnjunks = ['.flux', '.mask', '.model', '.psf', '.residual','.sumwt','.pb','.image'] #Do not run the next 4 lines if needed to view and assess the subset of clean process images
for clnjunk in clnjunks:
if os.path.exists(im_init + clnjunk):
os.system('rm -rf '+im_init + clnjunk)

from sunpy import map as smap
from matplotlib import pyplot as plt
fig = plt.figure(figsize=(6,6))
ax = fig.add_subplot(111)
eomap=smap.Map(im_init+'.fits')
#eomap.data=eomap.data.reshape((npix,npix))
eomap.plot_settings['cmap'] = plt.get_cmap('jet')
eomap.plot(axes = ax)
eomap.draw_limb()
plt.show()
viewer(im_init+'.image.pbcor')
</pre>
[[file:Figure2_imagingtutorial.png|thumb|center|500px|Figure 3: Full-Sun image after initial clean to find the flare location.]]

<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
# parameters specific to the event (found from step 1)
phasecenter='J2000 10h02m59 11d58m07' ###Change accordingly###
xran=[280,480] ###Change accordingly###
yran=[-50,150] ###Change accordingly###

# =========== Step 2 (optional), generate masks =========
# if skipped, will not use any masks
if domasks:
clearcal(slfcalms)
delmod(slfcalms)
antennas='0~12'
pol='XX'
imgprefix=maskdir+'slf_t0'

# step 1: set up the clean masks
img_init=imgprefix+'_init_ar_'
os.system('rm -rf '+img_init+'*')
#spwrans_mask=['1~5','6~12','13~20','21~30']
spwrans_mask=['1~12']
#convert to a list of spws
spwrans_mask_list = [[str(i) for i in (np.arange(int(m.split('~')[0]),int(m.split('~')[1])))] for m in spwrans_mask]
masks=[]
imnames=[]
for spwran in spwrans_mask:
imname=img_init+spwran.replace('~','-')
try:
tclean(vis=slfcalms,
antenna='0~12',
imagename=imname,
spw=spwran,
specmode='mfs',
timerange=trange,
imsize=[npix],
cell=['2arcsec'],
niter=1000,
gain=0.05,
stokes='XX',
restoringbeam=['20arcsec'],
phasecenter=phasecenter,
weighting='briggs',
robust=1.0,
interactive=True,
datacolumn='data',
pbcor=True,
savemodel='modelcolumn')
imnames.append(imname+'.image')
masks.append(imname+'.mask')
clnjunks = ['.flux', '.model', '.psf', '.residual'] #Do not run the next 4 lines if needed to view and assess the subset of clean process images
for clnjunk in clnjunks:
if os.path.exists(imname + clnjunk):
os.system('rm -rf '+ imname + clnjunk)
except:
print('error in cleaning spw: '+spwran)

pickle.dump(masks,open(slfcaldir+'masks.p','wb'))
</pre>
[[file:Figure4_imagingtutorial.PNG|thumb|center|800px|Figure 4: Interactive clean window to create masks over the source. The white outline surrounding the source is the mask selected by the polygon drawing option.]]

The outlines drawn for masks can be created by any of the icons with the letter 'R' in the Viewer window. The instructions for doing this can be found by hovering over those icons.

<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
# =========== Step 3, main step of selfcalibration =========
if doslfcal:
if os.path.exists(slfcaldir+'masks.p'):
masks=pickle.load(open(slfcaldir+'masks.p','rb'))
if not os.path.exists(slfcaldir+'masks.p'):
print 'masks do not exist. Use default mask'
masks=[]
os.system('rm -rf '+imagedir+'*')
os.system('rm -rf '+caltbdir+'*')
#first step: make a mock caltable for the entire database
print('Processing ' + trange)
slftbs=[]
calprefix=caltbdir+'slf'
imgprefix=imagedir+'slf'
tb.open(slfcalms+'/SPECTRAL_WINDOW')
reffreqs=tb.getcol('REF_FREQUENCY')
bdwds=tb.getcol('TOTAL_BANDWIDTH')
cfreqs=reffreqs+bdwds/2.
tb.close()
# starting beam size at 3.4 GHz in arcsec #Change accordingly
sbeam=40.
strtmp=[m.replace(':','') for m in trange.split('~')]
timestr='t'+strtmp[0]+'-'+strtmp[1]
refantenna='0'
# number of iterations for each round
niters=[100, 300, 500]
# roburst value for weighting the baselines
robusts=[1.0, 0.5, 0.0]
# apply calibration tables? Set to true for most cases
doapplycal=[1,1,1]
# modes for calibration, 'p' for phase-only, 'a' for amplitude only, 'ap' for both
calmodes=['p','p','a']
# setting uvranges for model image (optional, not used here)
uvranges=['','','']
for n in range(nround):
slfcal_tb_g= calprefix+'.G'+str(n)
fig = plt.figure(figsize=(8.4,7.))
gs = gridspec.GridSpec(5, 6)
for s,sp in enumerate(spws):
print 'processing spw: '+sp
cfreq=cfreqs[int(sp)]
# setting restoring beam size (not very useful for selfcal anyway, but just to see the results)
bm=max(sbeam*cfreqs[1]/cfreq, 6.)
slfcal_img = imgprefix+'.spw'+sp.zfill(2)+'.slfcal'+str(n)
# only the first round uses nearby spws for getting initial model
if n == 0:
spbg=max(int(sp)-2,1)
sped=min(int(sp)+2,30)
spwran=str(spbg)+'~'+str(sped)
print('using spw {0:s} as model'.format(spwran))
if 'spwrans_mask' in vars():
for m, spwran_mask in enumerate(spwrans_mask):
if sp in spwran_mask:
mask = masks[m]
print('using mask {0:s}'.format(mask))
findmask = True
if not findmask:
print('mask not found. Do use any masks')
else:
spwran = sp
if 'spwrans_mask' in vars():
for m, spwran_mask in enumerate(spwrans_mask):
if sp in spwran_mask:
mask = masks[m]
print 'using mask {0:s}'.format(mask)
findmask = True
if not findmask:
print('mask not found. Do use any masks')
try:
tclean(vis=slfcalms,
antenna=antennas,
imagename=slfcal_img,
uvrange=uvranges[n],
spw=spwran,
specmode='mfs',
timerange=trange,
imsize=[npix],
cell=['2arcsec'],
niter=niters[n],
gain=0.05,
stokes='XX', #use pol XX image as the model
weighting='briggs',
robust=robusts[n],
phasecenter=phasecenter,
mask=mask,
restoringbeam=[str(bm)+'arcsec'],
pbcor=False,
interactive=False,
savemodel='modelcolumn')
if os.path.exists(slfcal_img+'.image'):
fitsfile=slfcal_img+'.fits'
hf.imreg(vis=slfcalms,imagefile=slfcal_img+'.image',fitsfile=fitsfile,
timerange=trange,usephacenter=False,toTb=True,verbose=False,overwrite=True)
clnjunks = ['.mask','.flux', '.model', '.psf', '.residual', '.image','.pb','.image.pbcor','.sumwt'] #Do not run the next 4 lines if needed to view and assess the subset of clean process images
for clnjunk in clnjunks:
if os.path.exists(slfcal_img + clnjunk):
os.system('rm -rf '+ slfcal_img + clnjunk)
ax = fig.add_subplot(gs[s])
eomap=smap.Map(fitsfile)
eomap.plot_settings['cmap'] = plt.get_cmap('jet')
eomap.plot(axes = ax)
eomap.draw_limb()
#eomap.draw_grid()
ax.set_title(' ')
ax.get_xaxis().set_visible(False)
ax.get_yaxis().set_visible(False)
ax.set_xlim(xran)
ax.set_ylim(yran)
os.system('rm -f '+ fitsfile)

except:
print 'error in cleaning spw: '+sp
print 'using nearby spws for initial model'
sp_e=int(sp)+2
sp_i=int(sp)-2
if sp_i < 1:
sp_i = 1
if sp_e > 30:
sp_e = 30
sp_=str(sp_i)+'~'+str(sp_e)
try:
tget(tclean)
spw=sp_
print('using spw {0:s} as model'.format(sp_))
tclean()
except:
print 'still not successful. abort...'
break

gaincal(vis=slfcalms, refant=refantenna,antenna=antennas,caltable=slfcal_tb_g,spw=sp, uvrange='',\
gaintable=[],selectdata=True,timerange=trange,solint='inf',gaintype='G',calmode=calmodes[n],\
combine='',minblperant=4,minsnr=2,append=True)
if not os.path.exists(slfcal_tb_g):
print 'No solution found in spw: '+sp
figname=imagedir+'slf_t0_n{:d}.png'.format(n)
plt.subplots_adjust(left=0, bottom=0, right=1, top=1, wspace=0, hspace=0)
plt.savefig(figname)
time.sleep(10)
plt.close()

if os.path.exists(slfcal_tb_g):
slftbs.append(slfcal_tb_g)
slftb=[slfcal_tb_g]
os.chdir(slfcaldir)
if calmodes[n] == 'p':
plotcal(caltable=slfcal_tb_g,antenna='1~12',xaxis='freq',yaxis='phase',\
subplot=431,plotrange=[-1,-1,-180,180],iteration='antenna',figfile=slfcal_tb_g+'.png',showgui=False)
if calmodes[n] == 'a':
plotcal(caltable=slfcal_tb_g,antenna='1~12',xaxis='freq',yaxis='amp',\
subplot=431,plotrange=[-1,-1,0,2.],iteration='antenna',figfile=slfcal_tb_g+'.png',showgui=False)
os.chdir(workdir)

if doapplycal[n]:
clearcal(slfcalms)
delmod(slfcalms)
applycal(vis=slfcalms,gaintable=slftb,spw=','.join(spws),selectdata=True,\
antenna=antennas,interp='nearest',flagbackup=False,applymode='calonly',calwt=False)

if n < nround-1:
prompt=raw_input('Continuing to selfcal?')
#prompt='y'
if prompt.lower() == 'n':
if os.path.exists(slfcaledms):
os.system('rm -rf '+slfcaledms)
split(slfcalms,slfcaledms,datacolumn='corrected')
print 'Final calibrated ms is {0:s}'.format(slfcaledms)
break
if prompt.lower() == 'y':
slfcalms_=slfcalms+str(n)
if os.path.exists(slfcalms_):
os.system('rm -rf '+slfcalms_)
split(slfcalms,slfcalms_,datacolumn='corrected')
slfcalms=slfcalms_
else:
if os.path.exists(slfcaledms):
os.system('rm -rf '+slfcaledms)
split(slfcalms,slfcaledms,datacolumn='corrected')
print 'Final calibrated ms is {0:s}'.format(slfcaledms)
</pre>

<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
# =========== Step 4: Apply self-calibration tables =========
if doapply:
import glob
os.chdir(workdir)
clearcal(ms_in)
clearcal(slfcalms)
applycal(vis=slfcalms,gaintable=slftbs,spw=','.join(spws),selectdata=True,\
antenna=antennas,interp='linear',flagbackup=False,applymode='calonly',calwt=False)
applycal(vis=ms_in,gaintable=slftbs,spw=','.join(spws),selectdata=True,\
antenna=antennas,interp='linear',flagbackup=False,applymode='calonly',calwt=False)
if os.path.exists(ms_slfcaled):
os.system('rm -rf '+ms_slfcaled)
split(ms_in, ms_slfcaled,datacolumn='corrected')
</pre>
[[file:Slf.G0.png|thumb|center|500px|Figure 1: Phase before self-calibration]]
[[file:Slf.G1.png|thumb|center|500px|Figure 2: Phase after self-calibration]]

<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
# =========== Step 5: Generate final self-calibrated images (optional) =========
if doclean_slfcaled:
import glob
pol='XX'
print('Processing ' + trange)
img_final=imagedir_slfcaled+'/slf_final_{0}_t0'.format(pol)
vis = ms_slfcaled
spws=[str(s+1) for s in range(30)]
tb.open(vis+'/SPECTRAL_WINDOW')
reffreqs=tb.getcol('REF_FREQUENCY')
bdwds=tb.getcol('TOTAL_BANDWIDTH')
cfreqs=reffreqs+bdwds/2.
tb.close()
sbeam=30.
from matplotlib import gridspec as gridspec
from sunpy import map as smap
from matplotlib import pyplot as plt
fitsfiles=[]
for s,sp in enumerate(spws):
cfreq=cfreqs[int(sp)]
bm=max(sbeam*cfreqs[1]/cfreq,4.)
imname=img_final+'_s'+sp.zfill(2)
fitsfile=imname+'.fits'
if not os.path.exists(fitsfile):
print 'cleaning spw {0:s} with beam size {1:.1f}"'.format(sp,bm)
try:
tclean(vis=vis,
antenna=antennas,
imagename=imname,
spw=sp,
specmode='mfs',
timerange=trange,
imsize=[npix],
cell=['1arcsec'],
niter=1000,
gain=0.05,
stokes=pol,
weighting='briggs',
robust=2.0,
restoringbeam=[str(bm)+'arcsec'],
phasecenter=phasecenter,
mask='',
pbcor=True,
interactive=False)
except:
print 'cleaning spw '+sp+' unsuccessful. Proceed to next spw'
continue
if os.path.exists(imname+'.image.pbcor'):
imn = imname+'.image.pbcor'
hf.imreg(vis=vis,imagefile=imn,fitsfile=fitsfile,
timerange=trange,usephacenter=False,toTb=True,verbose=False)
fitsfiles.append(fitsfile)
junks=['.flux','.model','.psf','.residual','.mask','.image','.pb','.image.pbcor','.sumwt'] #Do not run the next 4 lines if needed to view and assess the subset of clean process images
for junk in junks:
if os.path.exists(imname+junk):
os.system('rm -rf '+imname+junk)
else:
print('fits file '+fitsfile+' already exists, skip clean...')
fitsfiles.append(fitsfile)

fig = plt.figure(figsize=(8.4,7.))
gs = gridspec.GridSpec(5, 6)
for s,sp in enumerate(spws):
cfreq=cfreqs[int(sp)]
ax = fig.add_subplot(gs[s])
eomap=smap.Map(fitsfiles[s])
eomap.plot_settings['cmap'] = plt.get_cmap('jet')
eomap.plot(axes = ax)
eomap.draw_limb()
ax.set_title(' ')
ax.get_xaxis().set_visible(False)
ax.get_yaxis().set_visible(False)
ax.set_xlim(xran)
ax.set_ylim(yran)
plt.text(0.98,0.85,'{0:.1f} GHz'.format(cfreq/1e9),transform=ax.transAxes,ha='right',color='w',fontweight='bold')
plt.subplots_adjust(left=0, bottom=0, right=1, top=1, wspace=0, hspace=0)
plt.show()

</pre>
The .fits files of the self calibrated images at each frequency for the given time are saved at /slfcal/images_slfcaled in your working directory.

[[file:Slf_t0_n0.png|thumb|center|500px|Figure 3: Multi-frequency images before self-calibration]]
[[file:Slf_t0_n1.png|thumb|center|500px|Figure 4: Multi-frequency images after self-calibration]]

====Step 5: Quick-look imaging ==== 
For spectral imaging analysis of the event, follow [http://www.ovsa.njit.edu/wiki/index.php/EOVSA_Data_Analysis_Tutorial#Spectral_Imaging_with_SunCASA this tutorial] using the self-calibrated data obtained from the previous step or use [https://colab.research.google.com/drive/1lSLLxgOG6b8kgu9Sk6kSKvrViyubnXG6?usp=sharing#scrollTo=xbXyyLmCFCGL this link].

<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
from suncasa.utils import qlookplot as ql ## (Optional) Supply the npz file of the dynamic spectrum from previous step.
## If not provided, the program will generate a new one from the visibility.
## set the time interval
from suncasa import dspec as ds
import time
visibility_data = 'IDB20170821202020_slfcaled.ms'
specfile = visibility_data + '.dspec.npz'
d = ds.Dspec(visibility_data, bl='4&9', specfile=specfile)

timerange = '19:02:00~19:02:10' ## select frequency range from 2.5 GHz to 3.5 GHz
spw = '2~5' ## select stokes XX
stokes = 'XX' ## turn off AIA image plotting, default is True
plotaia = False
xycen = [375, 45] ## image center for clean in solar X-Y in arcsec
cell=['2.0arcsec'] ## pixel size
imsize=[128] ## x and y image size in pixels.
fov = [100,100] ## field of view of the zoomed-in panels in unit of arcsec
spw = ['{}'.format(s) for s in range(1,31)]
clevels = [0.5, 1.0] ## contour levels to fill in between.
calpha=0.35 ## now tune down the alpha
restoringbeam=['6arcsec']
ql.qlookplot(vis=msfile, specfile=specfile, timerange=timerange, spw=spw, stokes=stokes, \
restoringbeam=restoringbeam,imsize=imsize,cell=cell, \
xycen=xycen,fov=fov,clevels=clevels,calpha=calpha)
</pre>

Page Last Updated: 03/29/2022

Tohban EOVSA Imaging Tutorial A-Z V2

2022-03-29T22:42:11Z

This tutorial describes the step-by-step procedure to download EOVSA IDB data, obtain and calibrate the measurement sets (.ms), and, transfer them to the inti server for self-calibration and further analysis.

=== Pre-requisites ===
Accounts on Pipeline and Inti or Baozi servers. Sijie and Bin can help with the account set-up for pipeline, inti and boazi.

==== How to connect to pipeline ====
'''Windows:''' On a windows machine Mobaxterm can be used to connect to Pipeline. For details, see ...

'''Linux/Unix:''' On a linux/Mac the ssh commands can be used:

<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
ssh -L 5902:helios.solar.pvt:20000 guest@ovsa.njit.edu ### when prompted, enter password to guest@ovsa ###
ssh -XY your_username@pipeline ### add your username and enter password to your pipeline account, when prompted ###
</pre>

Alternatively, to avoid typing the longer commands, proxy commands can be added to the ssh config file:

<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
vim ~/.ssh/config ### press 'i' to insert content and esc +':wq' to write and quit ###

Host *
ForwardAgent yes

Host *
ForwardX11 yes

Host ovsa
HostName ovsa.njit.edu
User guest

Host pipeline
Hostname pipeline.solar.pvt
User your_username ### Insert your pipeline username here ###
ProxyCommand ssh -W %h:%p guest@ovsa.njit.edu
</pre>

Upon saving the .ssh/config file and restarting the terminal, you can connect to pipeline directly using:

<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
ssh -XY pipeline
</pre>

==== How to connect to the inti server ====
NOTE: To connect to inti you must have a NJIT UCID. The arcs have granted access to inti for all SRG members. If you encounter any issue with the login. Please let arcs folks (arcs@njit.edu) know.

'''Windows:''' As mentioned above, on a windows machine Mobaxterm can be used to connect to inti. For details, see ...

'''Linux/Unix:''' On a linux/Mac the ssh command can be used:
<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
ssh -XY UCID@inti.hpcnet.campus.njit.edu ### insert UCID here and enter your NJIT UCID password, when prompted ###
</pre>

When logging in for the first time, please add the following lines (demarcated by #) in your ~/.bashrc file in order to use the casa on inti.

<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
#### setting start ####
if [ $HOSTNAME == "baozi.hpcnet.campus.njit.edu" ]; then
source /srg/.setenv_baozi
fi
if [ $HOSTNAME == "inti.hpcnet.campus.njit.edu" ]; then
source /inti/.setenv_inti
fi
if [ $HOSTNAME == "guko.resource.campus.njit.edu" ]; then
source /data/data/.setenv_guko
fi
#### setting end ####
</pre>
Proxy commands for inti can also be added to the .ssh/config file for convenience:
<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
vim ~/.ssh/config ### press 'i' to insert content and esc +':wq' to write and quit ###

Host inti
Hostname inti.hpcnet.campus.njit.edu
User UCID ### insert NJIT UCID here ###
</pre>
after the proxy is set up, connect to inti by simply typing:
<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
ssh -XY inti ### enter your NJIT UCID password when prompted ###
</pre>

=== Downloading data and Preliminary Calibration ===

==== Step 1: Downloading raw data (IDB) on Pipeline machine====
The initial downloading of data and applying refcal and phasecal to the .ms files is performed on the pipeline since the complete EOVSA SQL database is stored here for now.

If you are not using mobaxterm, directly SSH to Pipeline through your Linux or Mac computer and start tcsh to run CASA.

<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
from astropy.time import Time
import os
trange = Time(['2017-08-21 20:15:00', '2017-08-21 20:25:00']) ###Change accordingly###
#### (Optional) change output path, default current directory "./" #####
outpath = './msdata/' ###Change accordingly###
if not os.path.exists(outpath):
os.makedirs(outpath)
######################################################
msfiles = importeovsa(idbfiles=trange, ncpu=1, visprefix=outpath)
</pre>

NOTE: The above step currently gives an error with importeovsa. This is because the data location was changed and the code doesn't know where to look for the IDB files. Sijie is looking into the problem. We will update the page when it is fixed. For now, please use the method below for all time ranges.

OR (for a time range longer than 10 minutes)

<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
from suncasa.tasks import task_calibeovsa as calibeovsa
from suncasa.tasks import task_importeovsa as timporteovsa
from split_cli import split_cli as split
import dump_tsys as dt
from util import Time
import numpy as np
import os
from glob import glob
from eovsapy import util

trange = Time(['2017-08-21 20:15:00', '2017-08-21 20:35:00']) ###Change accordingly###
idbdir = util.get_idbdir(trange[0])

info = dt.rd_fdb(trange[0])
for k, v in info.iteritems():
info[k] = info[k][~(info[k] == '')]

sidx = np.where(
np.logical_and(info['SOURCEID'] == 'Sun', info['PROJECTID'] == 'NormalObserving') & np.logical_and(
info['ST_TS'].astype(np.float) >= trange[0].lv,
info['ST_TS'].astype(np.float) <= trange[
1].lv))
filelist = info['FILE'][sidx]

outpath = './msdata/' ###Change accordingly###
if not os.path.exists(outpath):
os.makedirs(outpath)
inpath = idbdir + '{}/'.format(trange[0].datetime.strftime("%Y%m%d"))
ncpu = 1

msfiles = timporteovsa.importeovsa(idbfiles=[inpath + ll for ll in filelist], ncpu=ncpu, timebin="0s", width=1,
visprefix=outpath,
nocreatms=False, doconcat=False,
modelms="", doscaling=False, keep_nsclms=False, udb_corr=True)
</pre>

If you come across errors with calibeovsa, add following lines to your ~/.casa/init.py file.
<pre style="background-color: #FCEBD9">
import sys
sys.path.append('/common/python')
sys.path.append('/common/python/packages/pipeline_casa')
execfile('/common/python/suncasa/tasks/mytasks.py')
</pre>

==== Step 2: Concatenate all the 10 mins data, if there are any====
Follow this step if there are more than one .ms files, if not run step 3 directly. If doimage=True, a quicklook image will be produced (by integrating over the entire time) as shown below.
<pre style="background-color: #FCEBD9">
# This is to set the path/name for the concatenated files
concatvis = os.path.basename(msfiles[0])[:11] + '_concat_cal.ms'
vis = calibeovsa.calibeovsa(msfiles, doconcat=True, concatvis=concatvis, caltype=['refpha','phacal'], doimage=True)
</pre>

[[file:Figure1_imagingtutorial.png|frame|center|800px|'''Figure 1:''' Quick-look full-Sun image after the initial calibration.]]

==== Step 3: Calibration ====
This will calibrate the input visibility, write out calibration tables under /data1/eovsa/caltable/, and apply the calibration.
<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
vis = calibeovsa.calibeovsa('IDB20170821202020.ms', caltype=['refpha','phacal'], doimage=True) ###Change the vis filename accordingly###
# Append '_cal' to the ms filename and split the corrected column to the new caled ms
vis_str = str(' '.join(vis))
caled_vis=vis_str.replace('.ms','_cal.ms')
split(vis=' '.join(vis),outputvis=caled_vis,datacolumn='corrected',timerange='',correlation='XX')
</pre>

One needs to transfer the created caled ms data files (xxx_concat_cal.ms or xxx_cal.ms) from pipeline to inti server, which has all the casatasks installed, in order to run the rest of the imaging steps.

=== Connection details to Inti server ===
For Windows, on Mobaxterm,

#With your NJIT VPN connected, connect to one of the afsconnect servers (for example, afsaccess3.njit.edu) using your UCID and password.
#ssh -X UCID@inti.hpcnet.campus.njit.edu

To avoid typing the full inti address each time you attempt for ssh, you may wish to add the following lines with your username to C:\Program Files\Git\etc\ssh\ssh_config on Windows and /.ssh/config on Mac and Linux machines.

<pre style="background-color: #FCEBD9">
Host inti
Hostname inti.hpcnet.campus.njit.edu
User USERNAME ###Insert UCID/inti username here###
</pre>

To have sufficient disk space with EOVSA data analysis over Inti, use your dedicated directory YOURDIRECTORY at the location given below. If you do not have a directory, Please take help from Sijie in creating one.
<pre style="background-color: #FCEBD9">
cd /inti/data/users/YOURDIRECTORY
</pre>

For Linux or Mac machine,
ssh -X UCID@inti.hpcnet.campus.njit.edu

=== Transferring data files between servers ===
For directly transferring your calibrated .ms data between the Pipeline and Inti servers, follow the below given steps.
<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
1. Log into Inti using your username.
ssh -X USERNAME@inti.hpcnet.campus.njit.edu

Then create a tunnel into Pipeline from Inti.
ssh -L 8888:pipeline.solar.pvt:22 guest@ovsa.njit.edu

2. Log into Inti again from a new terminal.

Change to your working directory and give this command to copy your data on Pipeline.
scp -v -C -r -P 8888 USERNAMEofPipeline@localhost:PATHofMSDATA/MSfilename ./
Eg: scp -v -C -r -P 8888 shaheda@localhost:/data1/shaheda/IDB20220118173922_cal.ms ./
where, MSfilename, PATHofMSDATA are your .ms data and its path on Pipeline machine.
</pre>

Alternatively, one can follow the below procedure to do the transfer.

For Windows, on mobaxterm, drag and drop the ms file to your local machine or use scp command as given below.

<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
scp -r -C userid@pipeline:/your/folder/msfile /localmachine/destination/folder/
</pre>

On mobaxterm and on your local terminal, use the following command to finally copy the ms file to Inti.

<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
scp -r -C /localmachine/destination/folder/msfile UCID@inti.hpcnet.campus.njit.edu:/inti/data/users/YOURDIRECTORY
</pre>

For Mac and Linux, SCP can be used in the same way. Add the following lines to your SSH config file, to bypass ovsa.njit.edu from copying.
vi ~/.ssh/config
<pre style="background-color: #FCEBD9">
Host ovsa
HostName ovsa.njit.edu
User guest
Host pipeline
Hostname pipeline.solar.pvt
User userid
ProxyCommand ssh -W %h:%p guest@ovsa.njit.edu
</pre>

=== Software details on the servers ===
On Inti,
when logging in for the first time, please add the following lines to your accounts ~/.bashrc file.

>>vi ~/.bashrc
Insert the text given below and save it.

<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
#### setting start ####
if [ $HOSTNAME == "baozi.hpcnet.campus.njit.edu" ]; then
source /srg/.setenv_baozi
fi
if [ $HOSTNAME == "inti.hpcnet.campus.njit.edu" ]; then
source /inti/.setenv_inti
fi
if [ $HOSTNAME == "guko.resource.campus.njit.edu" ]; then
source /data/data/.setenv_guko
fi
#### setting end ####
</pre>

Both CASA 5 and 6 are available on Inti.

Enter the bash environment on inti, and load the desired casa environment.
To load CASA 5, enter bash environment by giving >>bash
<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
>> loadcasa5
>> suncasa #This should load the software making you ready for the analysis
</pre>

Or to load CASA 6, enter bash environment by giving >>bash
<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
>> loadcasa6
>> ipython #This should load the software making you ready for the analysis
</pre>
Here, for example, to use clean, first start ipython as given above, then type in >>from casatasks import tclean

====Step 4: Self-calibration====
[[file:Figure3_imagingtutorial.png|thumb|center|500px|Figure 2: Cotton-Schwab clean major and minor cycles. [Source: http://www.aoc.nrao.edu/~rurvashi/ImagingAlgorithmsInCasa/node2.html].]]
Follow the below given steps to run the self-calibration of the imaging data and produce the calibrated images in .fits format. https://github.com/binchensun/casa-eovsa/blob/master/slfcal_example.py

<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
from suncasa.utils import helioimage2fits as hf
import os
import numpy as np
import pickle
from matplotlib import gridspec as gridspec
from sunpy import map as smap
from matplotlib import pyplot as plt
from split_cli import split_cli as split
import time

# =========== task handlers =============
dofullsun = 1 # initial full-sun imaging ###Change accordingly###
domasks=1 # get masks ###Change accordingly###
doslfcal=1 # main cycle of doing selfcalibration ###Change accordingly###
doapply=1 # apply the results ###Change accordingly###
doclean_slfcaled=1 # perform clean for self-calibrated data ###Change accordingly###

# ============ declaring the working directories ============
workdir = os.getcwd()+'/' #main working directory. Using current directory in this example
slfcaldir = workdir+'slfcal/' #place to put all selfcalibration products
imagedir = slfcaldir+'images/' #place to put all selfcalibration images
maskdir = slfcaldir+'masks/' #place to put clean masks
imagedir_slfcaled = slfcaldir+'images_slfcaled/' #place to put final self-calibrated images
caltbdir = slfcaldir+'caltbs/' # place to put calibration tables
# make these directories if they do not already exist
dirs = [workdir, slfcaldir, imagedir, maskdir, imagedir_slfcaled, caltbdir]
for d in dirs:
if not os.path.exists(d):
os.makedirs(d)

# ============ Split a short time for self-calibration ===========
# input visibility
ms_in = workdir + 'IDB20170821202020_cal.ms' ###Change the initial calibrated (through calibeovsa) vis accordingly###
# output, selfcaled, visibility
ms_slfcaled = workdir + os.path.basename(ms_in).replace('cal','slfcaled')
# intermediate small visibility for selfcalbration
# selected time range for generating self-calibration solutions
trange='2017/08/21/20:21:10~2017/08/21/20:21:30' ###Change accordingly###
slfcalms = slfcaldir+'slfcalms.XX.slfcal'
slfcaledms = slfcaldir+'slfcalms.XX.slfcaled'
if not os.path.exists(slfcalms):
split(vis=ms_in,outputvis=slfcalms,datacolumn='data',timerange=trange,correlation='XX')

# ============ Prior definitions for spectral windows, antennas, pixel numbers =========
spws=[str(s+1) for s in range(30)]
antennas='0~12'
npix=512
nround=3 #number of slfcal cycles

# =========== Step 1, doing a full-Sun image to find out phasecenter and appropriate field of view =========
if dofullsun:
#initial mfs clean to find out the image phase center
im_init='fullsun_init'
os.system('rm -rf '+im_init+'*')
tclean(vis=slfcalms,
antenna='0~12',
imagename=im_init,
spw='1~15',
specmode='mfs',
timerange=trange,
imsize=[npix],
cell=['5arcsec'],
niter=1000,
gain=0.05,
stokes='I',
restoringbeam=['30arcsec'],
interactive=False,
pbcor=True)

hf.imreg(vis=slfcalms,imagefile=im_init+'.image.pbcor',fitsfile=im_init+'.fits',
timerange=trange,usephacenter=False,verbose=True)
clnjunks = ['.flux', '.mask', '.model', '.psf', '.residual','.sumwt','.pb','.image'] #Do not run the next 4 lines if needed to view and assess the subset of clean process images
for clnjunk in clnjunks:
if os.path.exists(im_init + clnjunk):
os.system('rm -rf '+im_init + clnjunk)

from sunpy import map as smap
from matplotlib import pyplot as plt
fig = plt.figure(figsize=(6,6))
ax = fig.add_subplot(111)
eomap=smap.Map(im_init+'.fits')
#eomap.data=eomap.data.reshape((npix,npix))
eomap.plot_settings['cmap'] = plt.get_cmap('jet')
eomap.plot(axes = ax)
eomap.draw_limb()
plt.show()
viewer(im_init+'.image.pbcor')
</pre>
[[file:Figure2_imagingtutorial.png|thumb|center|500px|Figure 3: Full-Sun image after initial clean to find the flare location.]]

<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
# parameters specific to the event (found from step 1)
phasecenter='J2000 10h02m59 11d58m07' ###Change accordingly###
xran=[280,480] ###Change accordingly###
yran=[-50,150] ###Change accordingly###

# =========== Step 2 (optional), generate masks =========
# if skipped, will not use any masks
if domasks:
clearcal(slfcalms)
delmod(slfcalms)
antennas='0~12'
pol='XX'
imgprefix=maskdir+'slf_t0'

# step 1: set up the clean masks
img_init=imgprefix+'_init_ar_'
os.system('rm -rf '+img_init+'*')
#spwrans_mask=['1~5','6~12','13~20','21~30']
spwrans_mask=['1~12']
#convert to a list of spws
spwrans_mask_list = [[str(i) for i in (np.arange(int(m.split('~')[0]),int(m.split('~')[1])))] for m in spwrans_mask]
masks=[]
imnames=[]
for spwran in spwrans_mask:
imname=img_init+spwran.replace('~','-')
try:
tclean(vis=slfcalms,
antenna='0~12',
imagename=imname,
spw=spwran,
specmode='mfs',
timerange=trange,
imsize=[npix],
cell=['2arcsec'],
niter=1000,
gain=0.05,
stokes='XX',
restoringbeam=['20arcsec'],
phasecenter=phasecenter,
weighting='briggs',
robust=1.0,
interactive=True,
datacolumn='data',
pbcor=True,
savemodel='modelcolumn')
imnames.append(imname+'.image')
masks.append(imname+'.mask')
clnjunks = ['.flux', '.model', '.psf', '.residual'] #Do not run the next 4 lines if needed to view and assess the subset of clean process images
for clnjunk in clnjunks:
if os.path.exists(imname + clnjunk):
os.system('rm -rf '+ imname + clnjunk)
except:
print('error in cleaning spw: '+spwran)

pickle.dump(masks,open(slfcaldir+'masks.p','wb'))
</pre>
[[file:Figure4_imagingtutorial.PNG|thumb|center|800px|Figure 4: Interactive clean window to create masks over the source. The white outline surrounding the source is the mask selected by the polygon drawing option.]]

The outlines drawn for masks can be created by any of the icons with the letter 'R' in the Viewer window. The instructions for doing this can be found by hovering over those icons.

<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
# =========== Step 3, main step of selfcalibration =========
if doslfcal:
if os.path.exists(slfcaldir+'masks.p'):
masks=pickle.load(open(slfcaldir+'masks.p','rb'))
if not os.path.exists(slfcaldir+'masks.p'):
print 'masks do not exist. Use default mask'
masks=[]
os.system('rm -rf '+imagedir+'*')
os.system('rm -rf '+caltbdir+'*')
#first step: make a mock caltable for the entire database
print('Processing ' + trange)
slftbs=[]
calprefix=caltbdir+'slf'
imgprefix=imagedir+'slf'
tb.open(slfcalms+'/SPECTRAL_WINDOW')
reffreqs=tb.getcol('REF_FREQUENCY')
bdwds=tb.getcol('TOTAL_BANDWIDTH')
cfreqs=reffreqs+bdwds/2.
tb.close()
# starting beam size at 3.4 GHz in arcsec #Change accordingly
sbeam=40.
strtmp=[m.replace(':','') for m in trange.split('~')]
timestr='t'+strtmp[0]+'-'+strtmp[1]
refantenna='0'
# number of iterations for each round
niters=[100, 300, 500]
# roburst value for weighting the baselines
robusts=[1.0, 0.5, 0.0]
# apply calibration tables? Set to true for most cases
doapplycal=[1,1,1]
# modes for calibration, 'p' for phase-only, 'a' for amplitude only, 'ap' for both
calmodes=['p','p','a']
# setting uvranges for model image (optional, not used here)
uvranges=['','','']
for n in range(nround):
slfcal_tb_g= calprefix+'.G'+str(n)
fig = plt.figure(figsize=(8.4,7.))
gs = gridspec.GridSpec(5, 6)
for s,sp in enumerate(spws):
print 'processing spw: '+sp
cfreq=cfreqs[int(sp)]
# setting restoring beam size (not very useful for selfcal anyway, but just to see the results)
bm=max(sbeam*cfreqs[1]/cfreq, 6.)
slfcal_img = imgprefix+'.spw'+sp.zfill(2)+'.slfcal'+str(n)
# only the first round uses nearby spws for getting initial model
if n == 0:
spbg=max(int(sp)-2,1)
sped=min(int(sp)+2,30)
spwran=str(spbg)+'~'+str(sped)
print('using spw {0:s} as model'.format(spwran))
if 'spwrans_mask' in vars():
for m, spwran_mask in enumerate(spwrans_mask):
if sp in spwran_mask:
mask = masks[m]
print('using mask {0:s}'.format(mask))
findmask = True
if not findmask:
print('mask not found. Do use any masks')
else:
spwran = sp
if 'spwrans_mask' in vars():
for m, spwran_mask in enumerate(spwrans_mask):
if sp in spwran_mask:
mask = masks[m]
print 'using mask {0:s}'.format(mask)
findmask = True
if not findmask:
print('mask not found. Do use any masks')
try:
tclean(vis=slfcalms,
antenna=antennas,
imagename=slfcal_img,
uvrange=uvranges[n],
spw=spwran,
specmode='mfs',
timerange=trange,
imsize=[npix],
cell=['2arcsec'],
niter=niters[n],
gain=0.05,
stokes='XX', #use pol XX image as the model
weighting='briggs',
robust=robusts[n],
phasecenter=phasecenter,
mask=mask,
restoringbeam=[str(bm)+'arcsec'],
pbcor=False,
interactive=False,
savemodel='modelcolumn')
if os.path.exists(slfcal_img+'.image'):
fitsfile=slfcal_img+'.fits'
hf.imreg(vis=slfcalms,imagefile=slfcal_img+'.image',fitsfile=fitsfile,
timerange=trange,usephacenter=False,toTb=True,verbose=False,overwrite=True)
clnjunks = ['.mask','.flux', '.model', '.psf', '.residual', '.image','.pb','.image.pbcor','.sumwt'] #Do not run the next 4 lines if needed to view and assess the subset of clean process images
for clnjunk in clnjunks:
if os.path.exists(slfcal_img + clnjunk):
os.system('rm -rf '+ slfcal_img + clnjunk)
ax = fig.add_subplot(gs[s])
eomap=smap.Map(fitsfile)
eomap.plot_settings['cmap'] = plt.get_cmap('jet')
eomap.plot(axes = ax)
eomap.draw_limb()
#eomap.draw_grid()
ax.set_title(' ')
ax.get_xaxis().set_visible(False)
ax.get_yaxis().set_visible(False)
ax.set_xlim(xran)
ax.set_ylim(yran)
os.system('rm -f '+ fitsfile)

except:
print 'error in cleaning spw: '+sp
print 'using nearby spws for initial model'
sp_e=int(sp)+2
sp_i=int(sp)-2
if sp_i < 1:
sp_i = 1
if sp_e > 30:
sp_e = 30
sp_=str(sp_i)+'~'+str(sp_e)
try:
tget(tclean)
spw=sp_
print('using spw {0:s} as model'.format(sp_))
tclean()
except:
print 'still not successful. abort...'
break

gaincal(vis=slfcalms, refant=refantenna,antenna=antennas,caltable=slfcal_tb_g,spw=sp, uvrange='',\
gaintable=[],selectdata=True,timerange=trange,solint='inf',gaintype='G',calmode=calmodes[n],\
combine='',minblperant=4,minsnr=2,append=True)
if not os.path.exists(slfcal_tb_g):
print 'No solution found in spw: '+sp
figname=imagedir+'slf_t0_n{:d}.png'.format(n)
plt.subplots_adjust(left=0, bottom=0, right=1, top=1, wspace=0, hspace=0)
plt.savefig(figname)
time.sleep(10)
plt.close()

if os.path.exists(slfcal_tb_g):
slftbs.append(slfcal_tb_g)
slftb=[slfcal_tb_g]
os.chdir(slfcaldir)
if calmodes[n] == 'p':
plotcal(caltable=slfcal_tb_g,antenna='1~12',xaxis='freq',yaxis='phase',\
subplot=431,plotrange=[-1,-1,-180,180],iteration='antenna',figfile=slfcal_tb_g+'.png',showgui=False)
if calmodes[n] == 'a':
plotcal(caltable=slfcal_tb_g,antenna='1~12',xaxis='freq',yaxis='amp',\
subplot=431,plotrange=[-1,-1,0,2.],iteration='antenna',figfile=slfcal_tb_g+'.png',showgui=False)
os.chdir(workdir)

if doapplycal[n]:
clearcal(slfcalms)
delmod(slfcalms)
applycal(vis=slfcalms,gaintable=slftb,spw=','.join(spws),selectdata=True,\
antenna=antennas,interp='nearest',flagbackup=False,applymode='calonly',calwt=False)

if n < nround-1:
prompt=raw_input('Continuing to selfcal?')
#prompt='y'
if prompt.lower() == 'n':
if os.path.exists(slfcaledms):
os.system('rm -rf '+slfcaledms)
split(slfcalms,slfcaledms,datacolumn='corrected')
print 'Final calibrated ms is {0:s}'.format(slfcaledms)
break
if prompt.lower() == 'y':
slfcalms_=slfcalms+str(n)
if os.path.exists(slfcalms_):
os.system('rm -rf '+slfcalms_)
split(slfcalms,slfcalms_,datacolumn='corrected')
slfcalms=slfcalms_
else:
if os.path.exists(slfcaledms):
os.system('rm -rf '+slfcaledms)
split(slfcalms,slfcaledms,datacolumn='corrected')
print 'Final calibrated ms is {0:s}'.format(slfcaledms)
</pre>

<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
# =========== Step 4: Apply self-calibration tables =========
if doapply:
import glob
os.chdir(workdir)
clearcal(ms_in)
clearcal(slfcalms)
applycal(vis=slfcalms,gaintable=slftbs,spw=','.join(spws),selectdata=True,\
antenna=antennas,interp='linear',flagbackup=False,applymode='calonly',calwt=False)
applycal(vis=ms_in,gaintable=slftbs,spw=','.join(spws),selectdata=True,\
antenna=antennas,interp='linear',flagbackup=False,applymode='calonly',calwt=False)
if os.path.exists(ms_slfcaled):
os.system('rm -rf '+ms_slfcaled)
split(ms_in, ms_slfcaled,datacolumn='corrected')
</pre>
[[file:Slf.G0.png|thumb|center|500px|Figure 1: Phase before self-calibration]]
[[file:Slf.G1.png|thumb|center|500px|Figure 2: Phase after self-calibration]]

<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
# =========== Step 5: Generate final self-calibrated images (optional) =========
if doclean_slfcaled:
import glob
pol='XX'
print('Processing ' + trange)
img_final=imagedir_slfcaled+'/slf_final_{0}_t0'.format(pol)
vis = ms_slfcaled
spws=[str(s+1) for s in range(30)]
tb.open(vis+'/SPECTRAL_WINDOW')
reffreqs=tb.getcol('REF_FREQUENCY')
bdwds=tb.getcol('TOTAL_BANDWIDTH')
cfreqs=reffreqs+bdwds/2.
tb.close()
sbeam=30.
from matplotlib import gridspec as gridspec
from sunpy import map as smap
from matplotlib import pyplot as plt
fitsfiles=[]
for s,sp in enumerate(spws):
cfreq=cfreqs[int(sp)]
bm=max(sbeam*cfreqs[1]/cfreq,4.)
imname=img_final+'_s'+sp.zfill(2)
fitsfile=imname+'.fits'
if not os.path.exists(fitsfile):
print 'cleaning spw {0:s} with beam size {1:.1f}"'.format(sp,bm)
try:
tclean(vis=vis,
antenna=antennas,
imagename=imname,
spw=sp,
specmode='mfs',
timerange=trange,
imsize=[npix],
cell=['1arcsec'],
niter=1000,
gain=0.05,
stokes=pol,
weighting='briggs',
robust=2.0,
restoringbeam=[str(bm)+'arcsec'],
phasecenter=phasecenter,
mask='',
pbcor=True,
interactive=False)
except:
print 'cleaning spw '+sp+' unsuccessful. Proceed to next spw'
continue
if os.path.exists(imname+'.image.pbcor'):
imn = imname+'.image.pbcor'
hf.imreg(vis=vis,imagefile=imn,fitsfile=fitsfile,
timerange=trange,usephacenter=False,toTb=True,verbose=False)
fitsfiles.append(fitsfile)
junks=['.flux','.model','.psf','.residual','.mask','.image','.pb','.image.pbcor','.sumwt'] #Do not run the next 4 lines if needed to view and assess the subset of clean process images
for junk in junks:
if os.path.exists(imname+junk):
os.system('rm -rf '+imname+junk)
else:
print('fits file '+fitsfile+' already exists, skip clean...')
fitsfiles.append(fitsfile)

fig = plt.figure(figsize=(8.4,7.))
gs = gridspec.GridSpec(5, 6)
for s,sp in enumerate(spws):
cfreq=cfreqs[int(sp)]
ax = fig.add_subplot(gs[s])
eomap=smap.Map(fitsfiles[s])
eomap.plot_settings['cmap'] = plt.get_cmap('jet')
eomap.plot(axes = ax)
eomap.draw_limb()
ax.set_title(' ')
ax.get_xaxis().set_visible(False)
ax.get_yaxis().set_visible(False)
ax.set_xlim(xran)
ax.set_ylim(yran)
plt.text(0.98,0.85,'{0:.1f} GHz'.format(cfreq/1e9),transform=ax.transAxes,ha='right',color='w',fontweight='bold')
plt.subplots_adjust(left=0, bottom=0, right=1, top=1, wspace=0, hspace=0)
plt.show()

</pre>
The .fits files of the self calibrated images at each frequency for the given time are saved at /slfcal/images_slfcaled in your working directory.

[[file:Slf_t0_n0.png|thumb|center|500px|Figure 3: Multi-frequency images before self-calibration]]
[[file:Slf_t0_n1.png|thumb|center|500px|Figure 4: Multi-frequency images after self-calibration]]

====Step 5: Quick-look imaging ==== 
For spectral imaging analysis of the event, follow [http://www.ovsa.njit.edu/wiki/index.php/EOVSA_Data_Analysis_Tutorial#Spectral_Imaging_with_SunCASA this tutorial] using the self-calibrated data obtained from the previous step or use [https://colab.research.google.com/drive/1lSLLxgOG6b8kgu9Sk6kSKvrViyubnXG6?usp=sharing#scrollTo=xbXyyLmCFCGL this link].

<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
from suncasa.utils import qlookplot as ql ## (Optional) Supply the npz file of the dynamic spectrum from previous step.
## If not provided, the program will generate a new one from the visibility.
## set the time interval
from suncasa import dspec as ds
import time
visibility_data = 'IDB20170821202020_slfcaled.ms'
specfile = visibility_data + '.dspec.npz'
d = ds.Dspec(visibility_data, bl='4&9', specfile=specfile)

timerange = '19:02:00~19:02:10' ## select frequency range from 2.5 GHz to 3.5 GHz
spw = '2~5' ## select stokes XX
stokes = 'XX' ## turn off AIA image plotting, default is True
plotaia = False
xycen = [375, 45] ## image center for clean in solar X-Y in arcsec
cell=['2.0arcsec'] ## pixel size
imsize=[128] ## x and y image size in pixels.
fov = [100,100] ## field of view of the zoomed-in panels in unit of arcsec
spw = ['{}'.format(s) for s in range(1,31)]
clevels = [0.5, 1.0] ## contour levels to fill in between.
calpha=0.35 ## now tune down the alpha
restoringbeam=['6arcsec']
ql.qlookplot(vis=msfile, specfile=specfile, timerange=timerange, spw=spw, stokes=stokes, \
restoringbeam=restoringbeam,imsize=imsize,cell=cell, \
xycen=xycen,fov=fov,clevels=clevels,calpha=calpha)
</pre>

Page Last Updated: 03/29/2022

Owens Valley Solar Arrays

2022-03-29T21:47:32Z

Schhabra: /* Tohbans */

[[File:Eovsa1.png|border|text-top|800px]]

<big>[http://ovsa.njit.edu/ EOVSA] (Expanded Owens Valley Solar Array) is a new solar-dedicated radio interferometer operated by the New Jersey Institute of Technology. This wiki serves as the site for EOVSA documentation. </big>

== EOVSA Documentation ==

* <big>General</big>
** [[Downconversion and Frequency Tuning]]
** [[Dealing with Radio Frequency Interference]]
** [[Switching between 200 MHz and 300 MHz Correlator]]

* <big>Computer-Network</big>
** [[Computing Systems]]
** [[Network]]

* <big>Control System</big>
** [[27-m Antenna Commands]]
** [[Schedule Commands]]
** [[Control Commands]]
** [[Attenuation and Level Control]]

* <big>Hardware</big>
** [[Hardware Overview]]
** [[2.1-m Antennas]]
** [[27-m Antennas]]

* <big>System Software</big>
** [[Calibration Database]]
** [[Stateframe Database]]
** [[Create CASA measurement sets]]

* <big>Calibration</big>
**[[Calibration Overview]]
**[[Pointing Calibration]]
**[[Total Power Calibration]]
**[[System Gain Calibration]]
**[[Antenna Position]] (Baseline Calibration)
**[[Reference Gain Calibration]]
**[[Daily Gain Calibration]]
**[[Delay Calibration]]
**[[Bandpass Calibration]]
**[[Polarization Calibration]]
**[[Calibrator Survey]]
**[[Practical Calibration Tutorial]]

* <big>[[Starburst]]</big>

== Using EOVSA Data ==
* <big>[[EOVSA Data products]]</big>
* <big>Analysis Software</big>
** [https://github.com/suncasa/suncasa SunCASA] A wrapper around [https://casa.nrao.edu/ CASA (the Common Astronomy Software Applications package)] for synthesis imaging and visualizing solar spectral imaging data. CASA is one of the leading software tool for "supporting the data post-processing needs of the next generation of radio astronomical telescopes such as ALMA and VLA", an international effort led by the [https://public.nrao.edu/ National Radio Astronomy Observatory]. The current version of CASA uses Python (2.7) interface. More information about CASA can be found on [https://casa.nrao.edu/ NRAO's CASA website ]. Note, CASA is available ONLY on UNIX-BASED PLATFORMS (and therefore, so is SunCASA).
** [https://github.com/Gelu-Nita/GSFIT GSFIT] A IDL-widget(GUI)-based spectral fitting package called gsfit, which provides a user-friendly display of EOVSA image cubes and an interface to fast fitting codes (via platform-dependent shared-object libraries).
** [[Spectrogram Software]]
** [[Mapping Software]]
* <big>Data Analysis Guides</big>
** <big>[https://colab.research.google.com/drive/1lSLLxgOG6b8kgu9Sk6kSKvrViyubnXG6?usp=sharing#scrollTo=xbXyyLmCFCGL EOVSA Data Analysis Tutorial at RHESSI 19 Workshop]</big>
** <big>[[EOVSA Data Analysis Tutorial]]</big> at [http://rhessi18.umn.edu/ RHESSI XVIII Workshop]
** [[Self-Calibrating Flare Data]] Example script and guides for self-calibrating EOVSA flare data (to be completed)


**[[IDB flare pipeline]] Tutorial to run the flare pipeline for quicklook images



* <big>EOVSA Modeling Guide</big>
**[[GX Simulator]]

* Other helpful links
** [https://casaguides.nrao.edu CASA Guides]
** [http://www.lmsal.com/solarsoft/ SolarSoft IDL]
** [http://www.atnf.csiro.au/computing/software/miriad/userguide/userhtml.html Miriad Guides]
** [https://sites.google.com/site/fgscodes/ Fast Gyrosynchrotron Codes (Alexey Kuznetsov's website)]
** [[Basic GitHub Tutorial]]


* <big>[[Full Disk Simulations]]</big>
* <big>[[All-Day Synthesis Issues]]</big>
* <big>[[Analyzing Pre-2017 Data]]</big>
* <big>[[Fixing Pipeline Problems pre-2021-Feb-07]]</big>

== System Software ==

* LabVIEW software
* Python code [https://github.com/dgary50/eovsa Github repository]

== Observing Log ==
[[2016 November]]; [[2016 December| December]]

[[2017 January]]; [[2017 February | February]]; [[2017 March | March]]; [[2017 April | April]]; [[2017 May | May]]; [[2017 June | June]];
[[2017 July | July]]; [[2017 August | August]]; [[2017 September | September]]; [[2017 October | October]]; [[2017 November | November]]; [[2017 December | December]]

[[2018 January]]; [[2018 February | February]]; [[2018 March | March]]; [[2018 April | April]]; [[2018 May | May]]; [[2018 June | June]];
[[2018 July | July]]; [[2018 August | August]]; [[2018 September | September]]; [[2018 October | October]]; [[2018 November | November]]; [[2018 December | December]]

[[2019 January]]; [[2019 February | February]]; [[2019 March | March]]; [[2019 April | April]]; [[2019 May | May]]; [[2019 June | June]];
[[2019 July | July]]; [[2019 August | August]]; [[2019 September | September]]; [[2019 October | October]]; [[2019 November | November]]; [[2019 December | December]]

[[2020 January]]; [[2020 February | February]]; [[2020 March | March]]; [[2020 April | April]]; [[2020 May | May]]; [[2020 June | June]];
[[2020 July | July]]; [[2020 August | August]]; [[2020 September | September]]; [[2020 October | October]]; [[2020 November | November]]; [[2020 December | December]]

[[2021 January]]; [[2021 February | February]]; [[2021 March | March]]; [[2021 April | April]]; [[2021 May | May]]; [[2021 June | June]];
[[2021 July | July]]; [[2021 August | August]]; [[2021 September | September]]; [[2021 October | October]]; [[2021 November | November]]; [[2021 December | December]]

[[2022 SQL Outage]]

== Tohbans ==

[[Trouble Shooting Guide]]

[[Tohban Records]]

[[Owen's Notes]]

[[Tohban EOVSA Imaging Tutorial A-Z]]

[[Tohban EOVSA Imaging Tutorial A-Z V2]]

[[Guide to Upgrade SolarSoft(SSW)]]

== EOVSA Flare List ==

See [https://docs.google.com/spreadsheets/d/1P8jHuDRF93dMflU6RMQcsJqVepD9vFkPkofV8Imj4xA/edit?usp=sharing this link] for a list of EOVSA flares as a Google Spreadsheet.

[[Recent Flare List (2021-)]]

[http://ovsa.njit.edu/jay/rd_db.php An older link] is available at the EOVSA website.

== EOVSA Publications ==
Here is a (partial) list of publications that utilize EOVSA data. See also the collection of EOVSA publications at [https://ui.adsabs.harvard.edu/public-libraries/eQ7HfPkySqydu-B8BCt6QQ this NASA/ADS Library].
; 2021
: [https://ui.adsabs.harvard.edu/abs/2021arXiv211015751L/abstract Lopez et al. (2021), arXiv 2110.15751] ''A solar flare driven by thermal conduction observed in mid-infrared''
: [https://ui.adsabs.harvard.edu/abs/2021ApJ...923..213W/abstract Wei et al. (2021), ApJ, 923, 213] ''Coronal Magnetic Field Measurements along a Partially Erupting Filament in a Solar Flare''
: [https://ui.adsabs.harvard.edu/abs/2021ApJ...919...44S/abstract Shaik & Gary (2021), ApJ, 919, 44] ''Implications of Flat Optically Thick Microwave Spectra in Solar Flares for Source Size and Morphology''
: [https://ui.adsabs.harvard.edu/abs/2021ApJ...915...12K/abstract Kocharov et al. (2021), ApJ, 915, 12] ''Multiple Sources of Solar High-energy Protons''
: [https://ui.adsabs.harvard.edu/abs/2021ApJ...908L..55C/abstract Chen et al. (2021), ApJL, 908, L55] ''Energetic Electron Distribution of the Coronal Acceleration Region: First results from Joint Microwave and Hard X-ray Imaging Spectroscopy''
: [https://ui.adsabs.harvard.edu/abs/2021ApJ...906..132C/abstract Chhabra et al. (2021), ApJ, 906, 132] ''Imaging Spectroscopy of CME-Associated Solar Radio Bursts''
; 2020
: [https://ui.adsabs.harvard.edu/abs/2020ApJ...905..165R/abstract Reeves et al. (2020), ApJ, 905, 165] ''Hot Plasma Flows and Oscillations in the Loop-top Region During the September 10 2017 X8.2 Solar Flare''
: [https://ui.adsabs.harvard.edu/abs/2020ApJ...900...17Y/abstract Yu et al. (2020), ApJ, 900, 17] ''Magnetic Reconnection During the Post Impulsive Phase of the X8.2 Solar Flare: Bi-Directional Outflows as a Cause of Microwave and X-ray Bursts''
: [https://ui.adsabs.harvard.edu/abs/2020NatAs...4.1140C/abstract Chen et al. (2020b), Nature Astronomy, 4, 1140] ''Measurement of magnetic field and relativistic electrons along a solar flare current sheet''
: [https://ui.adsabs.harvard.edu/abs/2020ApJ...895L..50C/abstract Chen et al. (2020a), ApJL, 895, 50] ''Microwave Spectral Imaging of an Erupting Magnetic Flux Rope: Implications for the Standard Solar Flare Model in Three Dimensions''
: [https://ui.adsabs.harvard.edu/abs/2020FrASS...7...22K/abstract Kuroda et al. (2020), Frontiers, 7, 22] ''Evolution of Flare-accelerated Electrons Quantified by Spatially Resolved Analysis''
: [https://ui.adsabs.harvard.edu/abs/2020ApJ...891L..34G/abstract Glesener et al. (2020), ApJL, 891, 34] ''Accelerated Electrons Observed Down to <7 keV in a NuSTAR Solar Microflare''
: [https://ui.adsabs.harvard.edu/abs/2020ApJ...889...72K/abstract Karlicky at al. (2020), ApJ, 889, 72] ''Drifting Pulsation Structure at the Very Beginning of the 2017 September 10 Limb Flare''
: [https://ui.adsabs.harvard.edu/abs/2020Sci...367..278F/abstract Fleishman et al. (2020), Science, 367, 278] ''Decay of the coronal magnetic field can release sufficient energy to power a solar flare''
: [https://ui.adsabs.harvard.edu/abs/2020AAS...23538501G/abstract Gary et al. (2020), BAAS 52, 385.01] [https://aas235-aas.ipostersessions.com/default.aspx?s=97-69-9E-4B-34-19-68-53-1B-C6-21-0C-16-1C-5C-82&guestview=true Direct link to AAS iPoster] ''A new view of the solar atmosphere: daily full-disk multifrequency radio images from EOVSA''
; 2018
: [https://ui.adsabs.harvard.edu/#abs/2018ApJ...864...63P/abstract Polito et al. (2018), ApJ, 864, 63] ''Broad Non-Gaussian Fe XXIV Line Profiles in the Impulsive Phase of the 2017 September 10 X8.3-class Flare Observed by Hinode/EIS''
: [https://ui.adsabs.harvard.edu/#abs/2018ApJ...863...83G/abstract Gary et al. (2018), ApJ, 863, 83] ''Microwave and Hard X-Ray Observations of the 2017 September 10 Solar Limb Flare''
: [https://ui.adsabs.harvard.edu/#abs/2018ApJ...852...32K/abstract Kuroda et al. (2018), ApJ, 852, 32] ''Three-dimensional Forward-fit Modeling of the Hard X-ray and the Microwave Emissions of the 2015 June 22 M6.5 flare''
; 2017
: [https://ui.adsabs.harvard.edu/abs/2017NatAs...1E..85W/abstract Wang et al. (2017), Nature Astronomy, 1, 85] ''High-resolution observations of flare precursors in the low solar atmosphere''
; 2016
: [https://ui.adsabs.harvard.edu/abs/2016JAI.....541009N/abstract Nita et al. (2016), J. Astron. Instr., 5, 1641009-7366] ''EOVSA Implementation of a Spectral Kurtosis Correlator for Transient Detection and Classification''

== VLA Flare List and Publications ==
See [http://www.ovsa.njit.edu/wiki/index.php/VLA_Data_Survey#List_of_Jansky_VLA_Solar_Observations this link] for a list of flare observations made by the [https://science.nrao.edu/facilities/vla/ Karl G. Jansky Very Large Array] (VLA). Below is a partial list of publications that utilize VLA solar data (see also [https://ui.adsabs.harvard.edu/public-libraries/ZwbjpLo9RS-viufWEoQ95Q this NASA/ADS Library]).
* [https://ui.adsabs.harvard.edu/abs/2021ApJ...911....4L/abstract Luo et al. (2021), ApJ, 911, 4] ''Radio Spectral Imaging of an M8.4 Eruptive Solar Flare: Possible Evidence of a Termination Shock''
* [https://ui.adsabs.harvard.edu/abs/2021ApJ...910...40Z/abstract Zhang et al. (2021), ApJ, 910, 40] ''Multiwavelength Observations of the Formation and Eruption of a Complex Filament''
* [https://ui.adsabs.harvard.edu/abs/2020ApJ...904...94S/abstract Sharma et al. (2020), ApJ, 904, 94] ''Radio and X-Ray Observations of Short-lived Episodes of Electron Acceleration in a Solar Microflare''
* [https://ui.adsabs.harvard.edu/abs/2019ApJ...884...63C/abstract Chen et al. (2019), ApJ, 884, 63] ''Radio Spectroscopic Imaging of a Solar Flare Termination Shock: Split-band Feature as Evidence for Shock Compression''
* [https://ui.adsabs.harvard.edu/abs/2019ApJ...872...71Y/abstract Yu & Chen (2019), ApJ, 872, 71] ''Possible Detection of Subsecond-period Propagating Magnetohydrodynamics Waves in Post-reconnection Magnetic Loops during a Two-ribbon Solar Flare''
* [https://ui.adsabs.harvard.edu/abs/2018ApJ...866...62C/abstract Chen et al. (2018), ApJ, 866, 62] ''Magnetic Reconnection Null Points as the Origin of Semirelativistic Electron Beams in a Solar Jet
''
* [https://ui.adsabs.harvard.edu/abs/2017ApJ...848...77W/abstract Wang et al. (2016), ApJ, 848, 77] ''Dynamic Spectral Imaging of Decimetric Fiber Bursts in an Eruptive Solar Flare''
* [https://ui.adsabs.harvard.edu/abs/2015Sci...350.1238C/abstract Chen et al. (2015), Science, 350, 1238] ''Particle acceleration by a solar flare termination shock''
* [https://ui.adsabs.harvard.edu/abs/2014ApJ...794..149C/abstract Chen et al. (2014), ApJ, 794, 149] ''Direct Evidence of an Eruptive, Filament-hosting Magnetic Flux Rope Leading to a Fast Solar Coronal Mass Ejection''
* [https://ui.adsabs.harvard.edu/abs/2013ApJ...763L..21C/abstract Chen et al. (2013), ApJL, 763, 21] ''Tracing Electron Beams in the Sun's Corona with Radio Dynamic Imaging Spectroscopy''

Tohban EOVSA Imaging Tutorial A-Z

2022-03-29T21:46:28Z

Schhabra:

This tutorial describes the step-by-step procedure to download EOVSA IDB data, obtain and calibrate the measurement sets (.ms), and, transfer them to the inti server for self-calibration and further analysis.

'''Pre-requisites:''' Accounts on Pipeline and Inti or Baozi servers.

=== Connection details to pipeline server ===
One can use the Mobaxterm platform to connect to the Pipeline server through a Windows machine or use SSH through a Mac machine.

==== Step 1: Downloading raw data (IDB) on Pipeline machine====
On CASA of the Pipeline machine, which has the complete EOVSA SQL database.

If you are not using mobaxterm, directly SSH to Pipeline through your Linux or Mac computer and start tcsh to run CASA.

<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
from astropy.time import Time
import os
trange = Time(['2017-08-21 20:15:00', '2017-08-21 20:25:00']) ###Change accordingly###
#### (Optional) change output path, default current directory "./" #####
outpath = './msdata/' ###Change accordingly###
if not os.path.exists(outpath):
os.makedirs(outpath)
######################################################
msfiles = importeovsa(idbfiles=trange, ncpu=1, visprefix=outpath)
</pre>

NOTE: The above step currently gives an error with importeovsa. This is because the data location was changed and the code doesn't know where to look for the IDB files. Sijie is looking into the problem. We will update the page when it is fixed. For now, please use the method below for all time ranges.

OR (for a time range longer than 10 minutes)

<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
from suncasa.tasks import task_calibeovsa as calibeovsa
from suncasa.tasks import task_importeovsa as timporteovsa
from split_cli import split_cli as split
import dump_tsys as dt
from util import Time
import numpy as np
import os
from glob import glob
from eovsapy import util

trange = Time(['2017-08-21 20:15:00', '2017-08-21 20:35:00']) ###Change accordingly###
idbdir = util.get_idbdir(trange[0])

info = dt.rd_fdb(trange[0])
for k, v in info.iteritems():
info[k] = info[k][~(info[k] == '')]

sidx = np.where(
np.logical_and(info['SOURCEID'] == 'Sun', info['PROJECTID'] == 'NormalObserving') & np.logical_and(
info['ST_TS'].astype(np.float) >= trange[0].lv,
info['ST_TS'].astype(np.float) <= trange[
1].lv))
filelist = info['FILE'][sidx]

outpath = './msdata/' ###Change accordingly###
if not os.path.exists(outpath):
os.makedirs(outpath)
inpath = idbdir + '{}/'.format(trange[0].datetime.strftime("%Y%m%d"))
ncpu = 1

msfiles = timporteovsa.importeovsa(idbfiles=[inpath + ll for ll in filelist], ncpu=ncpu, timebin="0s", width=1,
visprefix=outpath,
nocreatms=False, doconcat=False,
modelms="", doscaling=False, keep_nsclms=False, udb_corr=True)
</pre>

If you come across errors with calibeovsa, add following lines to your ~/.casa/init.py file.
<pre style="background-color: #FCEBD9">
import sys
sys.path.append('/common/python')
sys.path.append('/common/python/packages/pipeline_casa')
execfile('/common/python/suncasa/tasks/mytasks.py')
</pre>

==== Step 2: Concatenate all the 10 mins data, if there are any====
Follow this step if there are more than one .ms files, if not run step 3 directly. If doimage=True, a quicklook image will be produced (by integrating over the entire time) as shown below.
<pre style="background-color: #FCEBD9">
# This is to set the path/name for the concatenated files
concatvis = os.path.basename(msfiles[0])[:11] + '_concat_cal.ms'
vis = calibeovsa.calibeovsa(msfiles, doconcat=True, concatvis=concatvis, caltype=['refpha','phacal'], doimage=True)
</pre>

[[file:Figure1_imagingtutorial.png|frame|center|800px|'''Figure 1:''' Quick-look full-Sun image after the initial calibration.]]

==== Step 3: Calibration ====
This will calibrate the input visibility, write out calibration tables under /data1/eovsa/caltable/, and apply the calibration.
<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
vis = calibeovsa.calibeovsa('IDB20170821202020.ms', caltype=['refpha','phacal'], doimage=True) ###Change the vis filename accordingly###
# Append '_cal' to the ms filename and split the corrected column to the new caled ms
vis_str = str(' '.join(vis))
caled_vis=vis_str.replace('.ms','_cal.ms')
split(vis=' '.join(vis),outputvis=caled_vis,datacolumn='corrected',timerange='',correlation='XX')
</pre>

One needs to transfer the created caled ms data files (xxx_concat_cal.ms or xxx_cal.ms) from pipeline to inti server, which has all the casatasks installed, in order to run the rest of the imaging steps.

=== Connection details to Inti server ===
For Windows, on Mobaxterm,

#With your NJIT VPN connected, connect to one of the afsconnect servers (for example, afsaccess3.njit.edu) using your UCID and password.
#ssh -X UCID@inti.hpcnet.campus.njit.edu

To avoid typing the full inti address each time you attempt for ssh, you may wish to add the following lines with your username to C:\Program Files\Git\etc\ssh\ssh_config on Windows and /.ssh/config on Mac and Linux machines.

<pre style="background-color: #FCEBD9">
Host inti
Hostname inti.hpcnet.campus.njit.edu
User USERNAME ###Insert UCID/inti username here###
</pre>

To have sufficient disk space with EOVSA data analysis over Inti, use your dedicated directory YOURDIRECTORY at the location given below. If you do not have a directory, Please take help from Sijie in creating one.
<pre style="background-color: #FCEBD9">
cd /inti/data/users/YOURDIRECTORY
</pre>

For Linux or Mac machine,
ssh -X UCID@inti.hpcnet.campus.njit.edu

=== Transferring data files between servers ===
For directly transferring your calibrated .ms data between the Pipeline and Inti servers, follow the below given steps.
<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
1. Log into Inti using your username.
ssh -X USERNAME@inti.hpcnet.campus.njit.edu

Then create a tunnel into Pipeline from Inti.
ssh -L 8888:pipeline.solar.pvt:22 guest@ovsa.njit.edu

2. Log into Inti again from a new terminal.

Change to your working directory and give this command to copy your data on Pipeline.
scp -v -C -r -P 8888 USERNAMEofPipeline@localhost:PATHofMSDATA/MSfilename ./
Eg: scp -v -C -r -P 8888 shaheda@localhost:/data1/shaheda/IDB20220118173922_cal.ms ./
where, MSfilename, PATHofMSDATA are your .ms data and its path on Pipeline machine.
</pre>

Alternatively, one can follow the below procedure to do the transfer.

For Windows, on mobaxterm, drag and drop the ms file to your local machine or use scp command as given below.

<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
scp -r -C userid@pipeline:/your/folder/msfile /localmachine/destination/folder/
</pre>

On mobaxterm and on your local terminal, use the following command to finally copy the ms file to Inti.

<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
scp -r -C /localmachine/destination/folder/msfile UCID@inti.hpcnet.campus.njit.edu:/inti/data/users/YOURDIRECTORY
</pre>

For Mac and Linux, SCP can be used in the same way. Add the following lines to your SSH config file, to bypass ovsa.njit.edu from copying.
vi ~/.ssh/config
<pre style="background-color: #FCEBD9">
Host ovsa
HostName ovsa.njit.edu
User guest
Host pipeline
Hostname pipeline.solar.pvt
User userid
ProxyCommand ssh -W %h:%p guest@ovsa.njit.edu
</pre>

=== Software details on the servers ===
On Inti,
when logging in for the first time, please add the following lines to your accounts ~/.bashrc file.

>>vi ~/.bashrc
Insert the text given below and save it.

<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
#### setting start ####
if [ $HOSTNAME == "baozi.hpcnet.campus.njit.edu" ]; then
source /srg/.setenv_baozi
fi
if [ $HOSTNAME == "inti.hpcnet.campus.njit.edu" ]; then
source /inti/.setenv_inti
fi
if [ $HOSTNAME == "guko.resource.campus.njit.edu" ]; then
source /data/data/.setenv_guko
fi
#### setting end ####
</pre>

Both CASA 5 and 6 are available on Inti.

Enter the bash environment on inti, and load the desired casa environment.
To load CASA 5, enter bash environment by giving >>bash
<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
>> loadcasa5
>> suncasa #This should load the software making you ready for the analysis
</pre>

Or to load CASA 6, enter bash environment by giving >>bash
<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
>> loadcasa6
>> ipython #This should load the software making you ready for the analysis
</pre>
Here, for example, to use clean, first start ipython as given above, then type in >>from casatasks import tclean

====Step 4: Self-calibration====
[[file:Figure3_imagingtutorial.png|thumb|center|500px|Figure 2: Cotton-Schwab clean major and minor cycles. [Source: http://www.aoc.nrao.edu/~rurvashi/ImagingAlgorithmsInCasa/node2.html].]]
Follow the below given steps to run the self-calibration of the imaging data and produce the calibrated images in .fits format. https://github.com/binchensun/casa-eovsa/blob/master/slfcal_example.py

<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
from suncasa.utils import helioimage2fits as hf
import os
import numpy as np
import pickle
from matplotlib import gridspec as gridspec
from sunpy import map as smap
from matplotlib import pyplot as plt
from split_cli import split_cli as split
import time

# =========== task handlers =============
dofullsun = 1 # initial full-sun imaging ###Change accordingly###
domasks=1 # get masks ###Change accordingly###
doslfcal=1 # main cycle of doing selfcalibration ###Change accordingly###
doapply=1 # apply the results ###Change accordingly###
doclean_slfcaled=1 # perform clean for self-calibrated data ###Change accordingly###

# ============ declaring the working directories ============
workdir = os.getcwd()+'/' #main working directory. Using current directory in this example
slfcaldir = workdir+'slfcal/' #place to put all selfcalibration products
imagedir = slfcaldir+'images/' #place to put all selfcalibration images
maskdir = slfcaldir+'masks/' #place to put clean masks
imagedir_slfcaled = slfcaldir+'images_slfcaled/' #place to put final self-calibrated images
caltbdir = slfcaldir+'caltbs/' # place to put calibration tables
# make these directories if they do not already exist
dirs = [workdir, slfcaldir, imagedir, maskdir, imagedir_slfcaled, caltbdir]
for d in dirs:
if not os.path.exists(d):
os.makedirs(d)

# ============ Split a short time for self-calibration ===========
# input visibility
ms_in = workdir + 'IDB20170821202020_cal.ms' ###Change the initial calibrated (through calibeovsa) vis accordingly###
# output, selfcaled, visibility
ms_slfcaled = workdir + os.path.basename(ms_in).replace('cal','slfcaled')
# intermediate small visibility for selfcalbration
# selected time range for generating self-calibration solutions
trange='2017/08/21/20:21:10~2017/08/21/20:21:30' ###Change accordingly###
slfcalms = slfcaldir+'slfcalms.XX.slfcal'
slfcaledms = slfcaldir+'slfcalms.XX.slfcaled'
if not os.path.exists(slfcalms):
split(vis=ms_in,outputvis=slfcalms,datacolumn='data',timerange=trange,correlation='XX')

# ============ Prior definitions for spectral windows, antennas, pixel numbers =========
spws=[str(s+1) for s in range(30)]
antennas='0~12'
npix=512
nround=3 #number of slfcal cycles

# =========== Step 1, doing a full-Sun image to find out phasecenter and appropriate field of view =========
if dofullsun:
#initial mfs clean to find out the image phase center
im_init='fullsun_init'
os.system('rm -rf '+im_init+'*')
tclean(vis=slfcalms,
antenna='0~12',
imagename=im_init,
spw='1~15',
specmode='mfs',
timerange=trange,
imsize=[npix],
cell=['5arcsec'],
niter=1000,
gain=0.05,
stokes='I',
restoringbeam=['30arcsec'],
interactive=False,
pbcor=True)

hf.imreg(vis=slfcalms,imagefile=im_init+'.image.pbcor',fitsfile=im_init+'.fits',
timerange=trange,usephacenter=False,verbose=True)
clnjunks = ['.flux', '.mask', '.model', '.psf', '.residual','.sumwt','.pb','.image'] #Do not run the next 4 lines if needed to view and assess the subset of clean process images
for clnjunk in clnjunks:
if os.path.exists(im_init + clnjunk):
os.system('rm -rf '+im_init + clnjunk)

from sunpy import map as smap
from matplotlib import pyplot as plt
fig = plt.figure(figsize=(6,6))
ax = fig.add_subplot(111)
eomap=smap.Map(im_init+'.fits')
#eomap.data=eomap.data.reshape((npix,npix))
eomap.plot_settings['cmap'] = plt.get_cmap('jet')
eomap.plot(axes = ax)
eomap.draw_limb()
plt.show()
viewer(im_init+'.image.pbcor')
</pre>
[[file:Figure2_imagingtutorial.png|thumb|center|500px|Figure 3: Full-Sun image after initial clean to find the flare location.]]

<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
# parameters specific to the event (found from step 1)
phasecenter='J2000 10h02m59 11d58m07' ###Change accordingly###
xran=[280,480] ###Change accordingly###
yran=[-50,150] ###Change accordingly###

# =========== Step 2 (optional), generate masks =========
# if skipped, will not use any masks
if domasks:
clearcal(slfcalms)
delmod(slfcalms)
antennas='0~12'
pol='XX'
imgprefix=maskdir+'slf_t0'

# step 1: set up the clean masks
img_init=imgprefix+'_init_ar_'
os.system('rm -rf '+img_init+'*')
#spwrans_mask=['1~5','6~12','13~20','21~30']
spwrans_mask=['1~12']
#convert to a list of spws
spwrans_mask_list = [[str(i) for i in (np.arange(int(m.split('~')[0]),int(m.split('~')[1])))] for m in spwrans_mask]
masks=[]
imnames=[]
for spwran in spwrans_mask:
imname=img_init+spwran.replace('~','-')
try:
tclean(vis=slfcalms,
antenna='0~12',
imagename=imname,
spw=spwran,
specmode='mfs',
timerange=trange,
imsize=[npix],
cell=['2arcsec'],
niter=1000,
gain=0.05,
stokes='XX',
restoringbeam=['20arcsec'],
phasecenter=phasecenter,
weighting='briggs',
robust=1.0,
interactive=True,
datacolumn='data',
pbcor=True,
savemodel='modelcolumn')
imnames.append(imname+'.image')
masks.append(imname+'.mask')
clnjunks = ['.flux', '.model', '.psf', '.residual'] #Do not run the next 4 lines if needed to view and assess the subset of clean process images
for clnjunk in clnjunks:
if os.path.exists(imname + clnjunk):
os.system('rm -rf '+ imname + clnjunk)
except:
print('error in cleaning spw: '+spwran)

pickle.dump(masks,open(slfcaldir+'masks.p','wb'))
</pre>
[[file:Figure4_imagingtutorial.PNG|thumb|center|800px|Figure 4: Interactive clean window to create masks over the source. The white outline surrounding the source is the mask selected by the polygon drawing option.]]

The outlines drawn for masks can be created by any of the icons with the letter 'R' in the Viewer window. The instructions for doing this can be found by hovering over those icons.

<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
# =========== Step 3, main step of selfcalibration =========
if doslfcal:
if os.path.exists(slfcaldir+'masks.p'):
masks=pickle.load(open(slfcaldir+'masks.p','rb'))
if not os.path.exists(slfcaldir+'masks.p'):
print 'masks do not exist. Use default mask'
masks=[]
os.system('rm -rf '+imagedir+'*')
os.system('rm -rf '+caltbdir+'*')
#first step: make a mock caltable for the entire database
print('Processing ' + trange)
slftbs=[]
calprefix=caltbdir+'slf'
imgprefix=imagedir+'slf'
tb.open(slfcalms+'/SPECTRAL_WINDOW')
reffreqs=tb.getcol('REF_FREQUENCY')
bdwds=tb.getcol('TOTAL_BANDWIDTH')
cfreqs=reffreqs+bdwds/2.
tb.close()
# starting beam size at 3.4 GHz in arcsec #Change accordingly
sbeam=40.
strtmp=[m.replace(':','') for m in trange.split('~')]
timestr='t'+strtmp[0]+'-'+strtmp[1]
refantenna='0'
# number of iterations for each round
niters=[100, 300, 500]
# roburst value for weighting the baselines
robusts=[1.0, 0.5, 0.0]
# apply calibration tables? Set to true for most cases
doapplycal=[1,1,1]
# modes for calibration, 'p' for phase-only, 'a' for amplitude only, 'ap' for both
calmodes=['p','p','a']
# setting uvranges for model image (optional, not used here)
uvranges=['','','']
for n in range(nround):
slfcal_tb_g= calprefix+'.G'+str(n)
fig = plt.figure(figsize=(8.4,7.))
gs = gridspec.GridSpec(5, 6)
for s,sp in enumerate(spws):
print 'processing spw: '+sp
cfreq=cfreqs[int(sp)]
# setting restoring beam size (not very useful for selfcal anyway, but just to see the results)
bm=max(sbeam*cfreqs[1]/cfreq, 6.)
slfcal_img = imgprefix+'.spw'+sp.zfill(2)+'.slfcal'+str(n)
# only the first round uses nearby spws for getting initial model
if n == 0:
spbg=max(int(sp)-2,1)
sped=min(int(sp)+2,30)
spwran=str(spbg)+'~'+str(sped)
print('using spw {0:s} as model'.format(spwran))
if 'spwrans_mask' in vars():
for m, spwran_mask in enumerate(spwrans_mask):
if sp in spwran_mask:
mask = masks[m]
print('using mask {0:s}'.format(mask))
findmask = True
if not findmask:
print('mask not found. Do use any masks')
else:
spwran = sp
if 'spwrans_mask' in vars():
for m, spwran_mask in enumerate(spwrans_mask):
if sp in spwran_mask:
mask = masks[m]
print 'using mask {0:s}'.format(mask)
findmask = True
if not findmask:
print('mask not found. Do use any masks')
try:
tclean(vis=slfcalms,
antenna=antennas,
imagename=slfcal_img,
uvrange=uvranges[n],
spw=spwran,
specmode='mfs',
timerange=trange,
imsize=[npix],
cell=['2arcsec'],
niter=niters[n],
gain=0.05,
stokes='XX', #use pol XX image as the model
weighting='briggs',
robust=robusts[n],
phasecenter=phasecenter,
mask=mask,
restoringbeam=[str(bm)+'arcsec'],
pbcor=False,
interactive=False,
savemodel='modelcolumn')
if os.path.exists(slfcal_img+'.image'):
fitsfile=slfcal_img+'.fits'
hf.imreg(vis=slfcalms,imagefile=slfcal_img+'.image',fitsfile=fitsfile,
timerange=trange,usephacenter=False,toTb=True,verbose=False,overwrite=True)
clnjunks = ['.mask','.flux', '.model', '.psf', '.residual', '.image','.pb','.image.pbcor','.sumwt'] #Do not run the next 4 lines if needed to view and assess the subset of clean process images
for clnjunk in clnjunks:
if os.path.exists(slfcal_img + clnjunk):
os.system('rm -rf '+ slfcal_img + clnjunk)
ax = fig.add_subplot(gs[s])
eomap=smap.Map(fitsfile)
eomap.plot_settings['cmap'] = plt.get_cmap('jet')
eomap.plot(axes = ax)
eomap.draw_limb()
#eomap.draw_grid()
ax.set_title(' ')
ax.get_xaxis().set_visible(False)
ax.get_yaxis().set_visible(False)
ax.set_xlim(xran)
ax.set_ylim(yran)
os.system('rm -f '+ fitsfile)

except:
print 'error in cleaning spw: '+sp
print 'using nearby spws for initial model'
sp_e=int(sp)+2
sp_i=int(sp)-2
if sp_i < 1:
sp_i = 1
if sp_e > 30:
sp_e = 30
sp_=str(sp_i)+'~'+str(sp_e)
try:
tget(tclean)
spw=sp_
print('using spw {0:s} as model'.format(sp_))
tclean()
except:
print 'still not successful. abort...'
break

gaincal(vis=slfcalms, refant=refantenna,antenna=antennas,caltable=slfcal_tb_g,spw=sp, uvrange='',\
gaintable=[],selectdata=True,timerange=trange,solint='inf',gaintype='G',calmode=calmodes[n],\
combine='',minblperant=4,minsnr=2,append=True)
if not os.path.exists(slfcal_tb_g):
print 'No solution found in spw: '+sp
figname=imagedir+'slf_t0_n{:d}.png'.format(n)
plt.subplots_adjust(left=0, bottom=0, right=1, top=1, wspace=0, hspace=0)
plt.savefig(figname)
time.sleep(10)
plt.close()

if os.path.exists(slfcal_tb_g):
slftbs.append(slfcal_tb_g)
slftb=[slfcal_tb_g]
os.chdir(slfcaldir)
if calmodes[n] == 'p':
plotcal(caltable=slfcal_tb_g,antenna='1~12',xaxis='freq',yaxis='phase',\
subplot=431,plotrange=[-1,-1,-180,180],iteration='antenna',figfile=slfcal_tb_g+'.png',showgui=False)
if calmodes[n] == 'a':
plotcal(caltable=slfcal_tb_g,antenna='1~12',xaxis='freq',yaxis='amp',\
subplot=431,plotrange=[-1,-1,0,2.],iteration='antenna',figfile=slfcal_tb_g+'.png',showgui=False)
os.chdir(workdir)

if doapplycal[n]:
clearcal(slfcalms)
delmod(slfcalms)
applycal(vis=slfcalms,gaintable=slftb,spw=','.join(spws),selectdata=True,\
antenna=antennas,interp='nearest',flagbackup=False,applymode='calonly',calwt=False)

if n < nround-1:
prompt=raw_input('Continuing to selfcal?')
#prompt='y'
if prompt.lower() == 'n':
if os.path.exists(slfcaledms):
os.system('rm -rf '+slfcaledms)
split(slfcalms,slfcaledms,datacolumn='corrected')
print 'Final calibrated ms is {0:s}'.format(slfcaledms)
break
if prompt.lower() == 'y':
slfcalms_=slfcalms+str(n)
if os.path.exists(slfcalms_):
os.system('rm -rf '+slfcalms_)
split(slfcalms,slfcalms_,datacolumn='corrected')
slfcalms=slfcalms_
else:
if os.path.exists(slfcaledms):
os.system('rm -rf '+slfcaledms)
split(slfcalms,slfcaledms,datacolumn='corrected')
print 'Final calibrated ms is {0:s}'.format(slfcaledms)
</pre>

<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
# =========== Step 4: Apply self-calibration tables =========
if doapply:
import glob
os.chdir(workdir)
clearcal(ms_in)
clearcal(slfcalms)
applycal(vis=slfcalms,gaintable=slftbs,spw=','.join(spws),selectdata=True,\
antenna=antennas,interp='linear',flagbackup=False,applymode='calonly',calwt=False)
applycal(vis=ms_in,gaintable=slftbs,spw=','.join(spws),selectdata=True,\
antenna=antennas,interp='linear',flagbackup=False,applymode='calonly',calwt=False)
if os.path.exists(ms_slfcaled):
os.system('rm -rf '+ms_slfcaled)
split(ms_in, ms_slfcaled,datacolumn='corrected')
</pre>
[[file:Slf.G0.png|thumb|center|500px|Figure 1: Phase before self-calibration]]
[[file:Slf.G1.png|thumb|center|500px|Figure 2: Phase after self-calibration]]

<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
# =========== Step 5: Generate final self-calibrated images (optional) =========
if doclean_slfcaled:
import glob
pol='XX'
print('Processing ' + trange)
img_final=imagedir_slfcaled+'/slf_final_{0}_t0'.format(pol)
vis = ms_slfcaled
spws=[str(s+1) for s in range(30)]
tb.open(vis+'/SPECTRAL_WINDOW')
reffreqs=tb.getcol('REF_FREQUENCY')
bdwds=tb.getcol('TOTAL_BANDWIDTH')
cfreqs=reffreqs+bdwds/2.
tb.close()
sbeam=30.
from matplotlib import gridspec as gridspec
from sunpy import map as smap
from matplotlib import pyplot as plt
fitsfiles=[]
for s,sp in enumerate(spws):
cfreq=cfreqs[int(sp)]
bm=max(sbeam*cfreqs[1]/cfreq,4.)
imname=img_final+'_s'+sp.zfill(2)
fitsfile=imname+'.fits'
if not os.path.exists(fitsfile):
print 'cleaning spw {0:s} with beam size {1:.1f}"'.format(sp,bm)
try:
tclean(vis=vis,
antenna=antennas,
imagename=imname,
spw=sp,
specmode='mfs',
timerange=trange,
imsize=[npix],
cell=['1arcsec'],
niter=1000,
gain=0.05,
stokes=pol,
weighting='briggs',
robust=2.0,
restoringbeam=[str(bm)+'arcsec'],
phasecenter=phasecenter,
mask='',
pbcor=True,
interactive=False)
except:
print 'cleaning spw '+sp+' unsuccessful. Proceed to next spw'
continue
if os.path.exists(imname+'.image.pbcor'):
imn = imname+'.image.pbcor'
hf.imreg(vis=vis,imagefile=imn,fitsfile=fitsfile,
timerange=trange,usephacenter=False,toTb=True,verbose=False)
fitsfiles.append(fitsfile)
junks=['.flux','.model','.psf','.residual','.mask','.image','.pb','.image.pbcor','.sumwt'] #Do not run the next 4 lines if needed to view and assess the subset of clean process images
for junk in junks:
if os.path.exists(imname+junk):
os.system('rm -rf '+imname+junk)
else:
print('fits file '+fitsfile+' already exists, skip clean...')
fitsfiles.append(fitsfile)

fig = plt.figure(figsize=(8.4,7.))
gs = gridspec.GridSpec(5, 6)
for s,sp in enumerate(spws):
cfreq=cfreqs[int(sp)]
ax = fig.add_subplot(gs[s])
eomap=smap.Map(fitsfiles[s])
eomap.plot_settings['cmap'] = plt.get_cmap('jet')
eomap.plot(axes = ax)
eomap.draw_limb()
ax.set_title(' ')
ax.get_xaxis().set_visible(False)
ax.get_yaxis().set_visible(False)
ax.set_xlim(xran)
ax.set_ylim(yran)
plt.text(0.98,0.85,'{0:.1f} GHz'.format(cfreq/1e9),transform=ax.transAxes,ha='right',color='w',fontweight='bold')
plt.subplots_adjust(left=0, bottom=0, right=1, top=1, wspace=0, hspace=0)
plt.show()

</pre>
The .fits files of the self calibrated images at each frequency for the given time are saved at /slfcal/images_slfcaled in your working directory.

[[file:Slf_t0_n0.png|thumb|center|500px|Figure 3: Multi-frequency images before self-calibration]]
[[file:Slf_t0_n1.png|thumb|center|500px|Figure 4: Multi-frequency images after self-calibration]]

====Step 5: Quick-look imaging ==== 
For spectral imaging analysis of the event, follow [http://www.ovsa.njit.edu/wiki/index.php/EOVSA_Data_Analysis_Tutorial#Spectral_Imaging_with_SunCASA this tutorial] using the self-calibrated data obtained from the previous step or use [https://colab.research.google.com/drive/1lSLLxgOG6b8kgu9Sk6kSKvrViyubnXG6?usp=sharing#scrollTo=xbXyyLmCFCGL this link].

<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
from suncasa.utils import qlookplot as ql ## (Optional) Supply the npz file of the dynamic spectrum from previous step.
## If not provided, the program will generate a new one from the visibility.
## set the time interval
from suncasa import dspec as ds
import time
visibility_data = 'IDB20170821202020_slfcaled.ms'
specfile = visibility_data + '.dspec.npz'
d = ds.Dspec(visibility_data, bl='4&9', specfile=specfile)

timerange = '19:02:00~19:02:10' ## select frequency range from 2.5 GHz to 3.5 GHz
spw = '2~5' ## select stokes XX
stokes = 'XX' ## turn off AIA image plotting, default is True
plotaia = False
xycen = [375, 45] ## image center for clean in solar X-Y in arcsec
cell=['2.0arcsec'] ## pixel size
imsize=[128] ## x and y image size in pixels.
fov = [100,100] ## field of view of the zoomed-in panels in unit of arcsec
spw = ['{}'.format(s) for s in range(1,31)]
clevels = [0.5, 1.0] ## contour levels to fill in between.
calpha=0.35 ## now tune down the alpha
restoringbeam=['6arcsec']
ql.qlookplot(vis=msfile, specfile=specfile, timerange=timerange, spw=spw, stokes=stokes, \
restoringbeam=restoringbeam,imsize=imsize,cell=cell, \
xycen=xycen,fov=fov,clevels=clevels,calpha=calpha)
</pre>

Guide to Upgrade SolarSoft(SSW)

2022-03-29T19:33:21Z

Schhabra:

SSW upgrade using passive_ftp was stopped in August 2019. The current way to upgrade SSW is to use the WGET executable. If you have done and update on SSW since August 2019 and your system already has WGET, any further upgrade can be made by directly following the steps underlined [https://hesperia.gsfc.nasa.gov/rhessi3/docs/ssw_update_help.html here]. For Linux and MacOS you can check if you already have WGET by typing the command "which wget" in the bash shell.

If your system was not updated since August 2019, follow the steps below:

=== MacOS ===
In order to perform the upgrade, you must be the owner of the SSW tree.
==== 1. Installing wget ====
'''USING HOMEBREW'''

To install homebrew, type the following command in the bash shell:
<pre style="background-color: #FCEBD9">
/usr/bin/ruby -e "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/install)"
</pre>

Once homebrew is installed, you can simply install wget by using the following command:
<pre style="background-color: #FCEBD9">
brew install wget
</pre>

'''USING MacPorts'''

Alternatively, wget can also be installed using MacPorts. Directions to install MacPorts for specific OS versions are given [https://www.macports.org/install.php here]
After installation, the following command will install wget:

<pre style="background-color: #FCEBD9">
sudo port install wget
</pre>

==== 2. Update SSW to use wget ====

Tohban EOVSA Imaging Tutorial A-Z

2022-03-29T19:16:06Z

Schhabra:

This tutorial describes the step-by-step procedure to calibrate EOVSA IDB data to obtain measurement sets (.ms), transfer it to the inti server and self-calibrate for further analysis.

'''Pre-requisites:''' Accounts on Pipeline and Inti or Baozi servers.

=== Connection details to pipeline server ===
One can use the Mobaxterm platform to connect to the Pipeline server through a Windows machine or use SSH through a Mac machine.

==== Step 1: Downloading raw data (IDB) on Pipeline machine====
On CASA of the Pipeline machine, which has the complete EOVSA SQL database.

If you are not using mobaxterm, directly SSH to Pipeline through your Linux or Mac computer and start tcsh to run CASA.

<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
from astropy.time import Time
import os
trange = Time(['2017-08-21 20:15:00', '2017-08-21 20:25:00']) ###Change accordingly###
#### (Optional) change output path, default current directory "./" #####
outpath = './msdata/' ###Change accordingly###
if not os.path.exists(outpath):
os.makedirs(outpath)
######################################################
msfiles = importeovsa(idbfiles=trange, ncpu=1, visprefix=outpath)
</pre>

NOTE: The above step currently gives an error with importeovsa. This is because the data location was changed and the code doesn't know where to look for the IDB files. Sijie is looking into the problem. We will update the page when it is fixed. For now, please use the method below for all time ranges.

OR (for a time range longer than 10 minutes)

<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
from suncasa.tasks import task_calibeovsa as calibeovsa
from suncasa.tasks import task_importeovsa as timporteovsa
from split_cli import split_cli as split
import dump_tsys as dt
from util import Time
import numpy as np
import os
from glob import glob
from eovsapy import util

trange = Time(['2017-08-21 20:15:00', '2017-08-21 20:35:00']) ###Change accordingly###
idbdir = util.get_idbdir(trange[0])

info = dt.rd_fdb(trange[0])
for k, v in info.iteritems():
info[k] = info[k][~(info[k] == '')]

sidx = np.where(
np.logical_and(info['SOURCEID'] == 'Sun', info['PROJECTID'] == 'NormalObserving') & np.logical_and(
info['ST_TS'].astype(np.float) >= trange[0].lv,
info['ST_TS'].astype(np.float) <= trange[
1].lv))
filelist = info['FILE'][sidx]

outpath = './msdata/' ###Change accordingly###
if not os.path.exists(outpath):
os.makedirs(outpath)
inpath = idbdir + '{}/'.format(trange[0].datetime.strftime("%Y%m%d"))
ncpu = 1

msfiles = timporteovsa.importeovsa(idbfiles=[inpath + ll for ll in filelist], ncpu=ncpu, timebin="0s", width=1,
visprefix=outpath,
nocreatms=False, doconcat=False,
modelms="", doscaling=False, keep_nsclms=False, udb_corr=True)
</pre>

If you come across errors with calibeovsa, add following lines to your ~/.casa/init.py file.
<pre style="background-color: #FCEBD9">
import sys
sys.path.append('/common/python')
sys.path.append('/common/python/packages/pipeline_casa')
execfile('/common/python/suncasa/tasks/mytasks.py')
</pre>

==== Step 2: Concatenate all the 10 mins data, if there are any====
Follow this step if there are more than one .ms files, if not run step 3 directly. If doimage=True, a quicklook image will be produced (by integrating over the entire time) as shown below.
<pre style="background-color: #FCEBD9">
# This is to set the path/name for the concatenated files
concatvis = os.path.basename(msfiles[0])[:11] + '_concat_cal.ms'
vis = calibeovsa.calibeovsa(msfiles, doconcat=True, concatvis=concatvis, caltype=['refpha','phacal'], doimage=True)
</pre>

[[file:Figure1_imagingtutorial.png|frame|center|800px|'''Figure 1:''' Quick-look full-Sun image after the initial calibration.]]

==== Step 3: Calibration ====
This will calibrate the input visibility, write out calibration tables under /data1/eovsa/caltable/, and apply the calibration.
<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
vis = calibeovsa.calibeovsa('IDB20170821202020.ms', caltype=['refpha','phacal'], doimage=True) ###Change the vis filename accordingly###
# Append '_cal' to the ms filename and split the corrected column to the new caled ms
vis_str = str(' '.join(vis))
caled_vis=vis_str.replace('.ms','_cal.ms')
split(vis=' '.join(vis),outputvis=caled_vis,datacolumn='corrected',timerange='',correlation='XX')
</pre>

One needs to transfer the created caled ms data files (xxx_concat_cal.ms or xxx_cal.ms) from pipeline to inti server, which has all the casatasks installed, in order to run the rest of the imaging steps.

=== Connection details to Inti server ===
For Windows, on Mobaxterm,

#With your NJIT VPN connected, connect to one of the afsconnect servers (for example, afsaccess3.njit.edu) using your UCID and password.
#ssh -X UCID@inti.hpcnet.campus.njit.edu

To avoid typing the full inti address each time you attempt for ssh, you may wish to add the following lines with your username to C:\Program Files\Git\etc\ssh\ssh_config on Windows and /.ssh/config on Mac and Linux machines.

<pre style="background-color: #FCEBD9">
Host inti
Hostname inti.hpcnet.campus.njit.edu
User USERNAME ###Insert UCID/inti username here###
</pre>

To have sufficient disk space with EOVSA data analysis over Inti, use your dedicated directory YOURDIRECTORY at the location given below. If you do not have a directory, Please take help from Sijie in creating one.
<pre style="background-color: #FCEBD9">
cd /inti/data/users/YOURDIRECTORY
</pre>

For Linux or Mac machine,
ssh -X UCID@inti.hpcnet.campus.njit.edu

=== Transferring data files between servers ===
For directly transferring your calibrated .ms data between the Pipeline and Inti servers, follow the below given steps.
<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
1. Log into Inti using your username.
ssh -X USERNAME@inti.hpcnet.campus.njit.edu

Then create a tunnel into Pipeline from Inti.
ssh -L 8888:pipeline.solar.pvt:22 guest@ovsa.njit.edu

2. Log into Inti again from a new terminal.

Change to your working directory and give this command to copy your data on Pipeline.
scp -v -C -r -P 8888 USERNAMEofPipeline@localhost:PATHofMSDATA/MSfilename ./
Eg: scp -v -C -r -P 8888 shaheda@localhost:/data1/shaheda/IDB20220118173922_cal.ms ./
where, MSfilename, PATHofMSDATA are your .ms data and its path on Pipeline machine.
</pre>

Alternatively, one can follow the below procedure to do the transfer.

For Windows, on mobaxterm, drag and drop the ms file to your local machine or use scp command as given below.

<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
scp -r -C userid@pipeline:/your/folder/msfile /localmachine/destination/folder/
</pre>

On mobaxterm and on your local terminal, use the following command to finally copy the ms file to Inti.

<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
scp -r -C /localmachine/destination/folder/msfile UCID@inti.hpcnet.campus.njit.edu:/inti/data/users/YOURDIRECTORY
</pre>

For Mac and Linux, SCP can be used in the same way. Add the following lines to your SSH config file, to bypass ovsa.njit.edu from copying.
vi ~/.ssh/config
<pre style="background-color: #FCEBD9">
Host ovsa
HostName ovsa.njit.edu
User guest
Host pipeline
Hostname pipeline.solar.pvt
User userid
ProxyCommand ssh -W %h:%p guest@ovsa.njit.edu
</pre>

=== Software details on the servers ===
On Inti,
when logging in for the first time, please add the following lines to your accounts ~/.bashrc file.

>>vi ~/.bashrc
Insert the text given below and save it.

<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
#### setting start ####
if [ $HOSTNAME == "baozi.hpcnet.campus.njit.edu" ]; then
source /srg/.setenv_baozi
fi
if [ $HOSTNAME == "inti.hpcnet.campus.njit.edu" ]; then
source /inti/.setenv_inti
fi
if [ $HOSTNAME == "guko.resource.campus.njit.edu" ]; then
source /data/data/.setenv_guko
fi
#### setting end ####
</pre>

Both CASA 5 and 6 are available on Inti.

Enter the bash environment on inti, and load the desired casa environment.
To load CASA 5, enter bash environment by giving >>bash
<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
>> loadcasa5
>> suncasa #This should load the software making you ready for the analysis
</pre>

Or to load CASA 6, enter bash environment by giving >>bash
<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
>> loadcasa6
>> ipython #This should load the software making you ready for the analysis
</pre>
Here, for example, to use clean, first start ipython as given above, then type in >>from casatasks import tclean

====Step 4: Self-calibration====
[[file:Figure3_imagingtutorial.png|thumb|center|500px|Figure 2: Cotton-Schwab clean major and minor cycles. [Source: http://www.aoc.nrao.edu/~rurvashi/ImagingAlgorithmsInCasa/node2.html].]]
https://github.com/binchensun/casa-eovsa/blob/master/slfcal_example.py

<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
from suncasa.utils import helioimage2fits as hf
import os
import numpy as np
import pickle
from matplotlib import gridspec as gridspec
from sunpy import map as smap
from matplotlib import pyplot as plt
from split_cli import split_cli as split
import time

# =========== task handlers =============
dofullsun = 1 # initial full-sun imaging ###Change accordingly###
domasks=1 # get masks ###Change accordingly###
doslfcal=1 # main cycle of doing selfcalibration ###Change accordingly###
doapply=1 # apply the results ###Change accordingly###
doclean_slfcaled=1 # perform clean for self-calibrated data ###Change accordingly###

# ============ declaring the working directories ============
workdir = os.getcwd()+'/' #main working directory. Using current directory in this example
slfcaldir = workdir+'slfcal/' #place to put all selfcalibration products
imagedir = slfcaldir+'images/' #place to put all selfcalibration images
maskdir = slfcaldir+'masks/' #place to put clean masks
imagedir_slfcaled = slfcaldir+'images_slfcaled/' #place to put final self-calibrated images
caltbdir = slfcaldir+'caltbs/' # place to put calibration tables
# make these directories if they do not already exist
dirs = [workdir, slfcaldir, imagedir, maskdir, imagedir_slfcaled, caltbdir]
for d in dirs:
if not os.path.exists(d):
os.makedirs(d)

# ============ Split a short time for self-calibration ===========
# input visibility
ms_in = workdir + 'IDB20170821202020_cal.ms' ###Change the initial calibrated (through calibeovsa) vis accordingly###
# output, selfcaled, visibility
ms_slfcaled = workdir + os.path.basename(ms_in).replace('cal','slfcaled')
# intermediate small visibility for selfcalbration
# selected time range for generating self-calibration solutions
trange='2017/08/21/20:21:10~2017/08/21/20:21:30' ###Change accordingly###
slfcalms = slfcaldir+'slfcalms.XX.slfcal'
slfcaledms = slfcaldir+'slfcalms.XX.slfcaled'
if not os.path.exists(slfcalms):
split(vis=ms_in,outputvis=slfcalms,datacolumn='data',timerange=trange,correlation='XX')

# ============ Prior definitions for spectral windows, antennas, pixel numbers =========
spws=[str(s+1) for s in range(30)]
antennas='0~12'
npix=512
nround=3 #number of slfcal cycles

# =========== Step 1, doing a full-Sun image to find out phasecenter and appropriate field of view =========
if dofullsun:
#initial mfs clean to find out the image phase center
im_init='fullsun_init'
os.system('rm -rf '+im_init+'*')
tclean(vis=slfcalms,
antenna='0~12',
imagename=im_init,
spw='1~15',
specmode='mfs',
timerange=trange,
imsize=[npix],
cell=['5arcsec'],
niter=1000,
gain=0.05,
stokes='I',
restoringbeam=['30arcsec'],
interactive=False,
pbcor=True)

hf.imreg(vis=slfcalms,imagefile=im_init+'.image.pbcor',fitsfile=im_init+'.fits',
timerange=trange,usephacenter=False,verbose=True)
clnjunks = ['.flux', '.mask', '.model', '.psf', '.residual','.sumwt','.pb','.image'] #Do not run the next 4 lines if needed to view and assess the subset of clean process images
for clnjunk in clnjunks:
if os.path.exists(im_init + clnjunk):
os.system('rm -rf '+im_init + clnjunk)

from sunpy import map as smap
from matplotlib import pyplot as plt
fig = plt.figure(figsize=(6,6))
ax = fig.add_subplot(111)
eomap=smap.Map(im_init+'.fits')
#eomap.data=eomap.data.reshape((npix,npix))
eomap.plot_settings['cmap'] = plt.get_cmap('jet')
eomap.plot(axes = ax)
eomap.draw_limb()
plt.show()
viewer(im_init+'.image.pbcor')
</pre>
[[file:Figure2_imagingtutorial.png|thumb|center|500px|Figure 3: Full-Sun image after initial clean to find the flare location.]]

<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
# parameters specific to the event (found from step 1)
phasecenter='J2000 10h02m59 11d58m07' ###Change accordingly###
xran=[280,480] ###Change accordingly###
yran=[-50,150] ###Change accordingly###

# =========== Step 2 (optional), generate masks =========
# if skipped, will not use any masks
if domasks:
clearcal(slfcalms)
delmod(slfcalms)
antennas='0~12'
pol='XX'
imgprefix=maskdir+'slf_t0'

# step 1: set up the clean masks
img_init=imgprefix+'_init_ar_'
os.system('rm -rf '+img_init+'*')
#spwrans_mask=['1~5','6~12','13~20','21~30']
spwrans_mask=['1~12']
#convert to a list of spws
spwrans_mask_list = [[str(i) for i in (np.arange(int(m.split('~')[0]),int(m.split('~')[1])))] for m in spwrans_mask]
masks=[]
imnames=[]
for spwran in spwrans_mask:
imname=img_init+spwran.replace('~','-')
try:
tclean(vis=slfcalms,
antenna='0~12',
imagename=imname,
spw=spwran,
specmode='mfs',
timerange=trange,
imsize=[npix],
cell=['2arcsec'],
niter=1000,
gain=0.05,
stokes='XX',
restoringbeam=['20arcsec'],
phasecenter=phasecenter,
weighting='briggs',
robust=1.0,
interactive=True,
datacolumn='data',
pbcor=True,
savemodel='modelcolumn')
imnames.append(imname+'.image')
masks.append(imname+'.mask')
clnjunks = ['.flux', '.model', '.psf', '.residual'] #Do not run the next 4 lines if needed to view and assess the subset of clean process images
for clnjunk in clnjunks:
if os.path.exists(imname + clnjunk):
os.system('rm -rf '+ imname + clnjunk)
except:
print('error in cleaning spw: '+spwran)

pickle.dump(masks,open(slfcaldir+'masks.p','wb'))
</pre>
[[file:Figure4_imagingtutorial.PNG|thumb|center|800px|Figure 4: Interactive clean window to create masks over the source. The white outline surrounding the source is the mask selected by the polygon drawing option.]]

The outlines drawn for masks can be created by any of the icons with the letter 'R' in the Viewer window. The instructions for doing this can be found by hovering over those icons.

<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
# =========== Step 3, main step of selfcalibration =========
if doslfcal:
if os.path.exists(slfcaldir+'masks.p'):
masks=pickle.load(open(slfcaldir+'masks.p','rb'))
if not os.path.exists(slfcaldir+'masks.p'):
print 'masks do not exist. Use default mask'
masks=[]
os.system('rm -rf '+imagedir+'*')
os.system('rm -rf '+caltbdir+'*')
#first step: make a mock caltable for the entire database
print('Processing ' + trange)
slftbs=[]
calprefix=caltbdir+'slf'
imgprefix=imagedir+'slf'
tb.open(slfcalms+'/SPECTRAL_WINDOW')
reffreqs=tb.getcol('REF_FREQUENCY')
bdwds=tb.getcol('TOTAL_BANDWIDTH')
cfreqs=reffreqs+bdwds/2.
tb.close()
# starting beam size at 3.4 GHz in arcsec
sbeam=40.
strtmp=[m.replace(':','') for m in trange.split('~')]
timestr='t'+strtmp[0]+'-'+strtmp[1]
refantenna='0'
# number of iterations for each round
niters=[100, 300, 500]
# roburst value for weighting the baselines
robusts=[1.0, 0.5, 0.0]
# apply calibration tables? Set to true for most cases
doapplycal=[1,1,1]
# modes for calibration, 'p' for phase-only, 'a' for amplitude only, 'ap' for both
calmodes=['p','p','a']
# setting uvranges for model image (optional, not used here)
uvranges=['','','']
for n in range(nround):
slfcal_tb_g= calprefix+'.G'+str(n)
fig = plt.figure(figsize=(8.4,7.))
gs = gridspec.GridSpec(5, 6)
for s,sp in enumerate(spws):
print 'processing spw: '+sp
cfreq=cfreqs[int(sp)]
# setting restoring beam size (not very useful for selfcal anyway, but just to see the results)
bm=max(sbeam*cfreqs[1]/cfreq, 6.)
slfcal_img = imgprefix+'.spw'+sp.zfill(2)+'.slfcal'+str(n)
# only the first round uses nearby spws for getting initial model
if n == 0:
spbg=max(int(sp)-2,1)
sped=min(int(sp)+2,30)
spwran=str(spbg)+'~'+str(sped)
print('using spw {0:s} as model'.format(spwran))
if 'spwrans_mask' in vars():
for m, spwran_mask in enumerate(spwrans_mask):
if sp in spwran_mask:
mask = masks[m]
print('using mask {0:s}'.format(mask))
findmask = True
if not findmask:
print('mask not found. Do use any masks')
else:
spwran = sp
if 'spwrans_mask' in vars():
for m, spwran_mask in enumerate(spwrans_mask):
if sp in spwran_mask:
mask = masks[m]
print 'using mask {0:s}'.format(mask)
findmask = True
if not findmask:
print('mask not found. Do use any masks')
try:
tclean(vis=slfcalms,
antenna=antennas,
imagename=slfcal_img,
uvrange=uvranges[n],
spw=spwran,
specmode='mfs',
timerange=trange,
imsize=[npix],
cell=['2arcsec'],
niter=niters[n],
gain=0.05,
stokes='XX', #use pol XX image as the model
weighting='briggs',
robust=robusts[n],
phasecenter=phasecenter,
mask=mask,
restoringbeam=[str(bm)+'arcsec'],
pbcor=False,
interactive=False,
savemodel='modelcolumn')
if os.path.exists(slfcal_img+'.image'):
fitsfile=slfcal_img+'.fits'
hf.imreg(vis=slfcalms,imagefile=slfcal_img+'.image',fitsfile=fitsfile,
timerange=trange,usephacenter=False,toTb=True,verbose=False,overwrite=True)
clnjunks = ['.mask','.flux', '.model', '.psf', '.residual', '.image','.pb','.image.pbcor','.sumwt'] #Do not run the next 4 lines if needed to view and assess the subset of clean process images
for clnjunk in clnjunks:
if os.path.exists(slfcal_img + clnjunk):
os.system('rm -rf '+ slfcal_img + clnjunk)
ax = fig.add_subplot(gs[s])
eomap=smap.Map(fitsfile)
eomap.plot_settings['cmap'] = plt.get_cmap('jet')
eomap.plot(axes = ax)
eomap.draw_limb()
#eomap.draw_grid()
ax.set_title(' ')
ax.get_xaxis().set_visible(False)
ax.get_yaxis().set_visible(False)
ax.set_xlim(xran)
ax.set_ylim(yran)
os.system('rm -f '+ fitsfile)

except:
print 'error in cleaning spw: '+sp
print 'using nearby spws for initial model'
sp_e=int(sp)+2
sp_i=int(sp)-2
if sp_i < 1:
sp_i = 1
if sp_e > 30:
sp_e = 30
sp_=str(sp_i)+'~'+str(sp_e)
try:
tget(tclean)
spw=sp_
print('using spw {0:s} as model'.format(sp_))
tclean()
except:
print 'still not successful. abort...'
break

gaincal(vis=slfcalms, refant=refantenna,antenna=antennas,caltable=slfcal_tb_g,spw=sp, uvrange='',\
gaintable=[],selectdata=True,timerange=trange,solint='inf',gaintype='G',calmode=calmodes[n],\
combine='',minblperant=4,minsnr=2,append=True)
if not os.path.exists(slfcal_tb_g):
print 'No solution found in spw: '+sp
figname=imagedir+'slf_t0_n{:d}.png'.format(n)
plt.subplots_adjust(left=0, bottom=0, right=1, top=1, wspace=0, hspace=0)
plt.savefig(figname)
time.sleep(10)
plt.close()

if os.path.exists(slfcal_tb_g):
slftbs.append(slfcal_tb_g)
slftb=[slfcal_tb_g]
os.chdir(slfcaldir)
if calmodes[n] == 'p':
plotcal(caltable=slfcal_tb_g,antenna='1~12',xaxis='freq',yaxis='phase',\
subplot=431,plotrange=[-1,-1,-180,180],iteration='antenna',figfile=slfcal_tb_g+'.png',showgui=False)
if calmodes[n] == 'a':
plotcal(caltable=slfcal_tb_g,antenna='1~12',xaxis='freq',yaxis='amp',\
subplot=431,plotrange=[-1,-1,0,2.],iteration='antenna',figfile=slfcal_tb_g+'.png',showgui=False)
os.chdir(workdir)

if doapplycal[n]:
clearcal(slfcalms)
delmod(slfcalms)
applycal(vis=slfcalms,gaintable=slftb,spw=','.join(spws),selectdata=True,\
antenna=antennas,interp='nearest',flagbackup=False,applymode='calonly',calwt=False)

if n < nround-1:
prompt=raw_input('Continuing to selfcal?')
#prompt='y'
if prompt.lower() == 'n':
if os.path.exists(slfcaledms):
os.system('rm -rf '+slfcaledms)
split(slfcalms,slfcaledms,datacolumn='corrected')
print 'Final calibrated ms is {0:s}'.format(slfcaledms)
break
if prompt.lower() == 'y':
slfcalms_=slfcalms+str(n)
if os.path.exists(slfcalms_):
os.system('rm -rf '+slfcalms_)
split(slfcalms,slfcalms_,datacolumn='corrected')
slfcalms=slfcalms_
else:
if os.path.exists(slfcaledms):
os.system('rm -rf '+slfcaledms)
split(slfcalms,slfcaledms,datacolumn='corrected')
print 'Final calibrated ms is {0:s}'.format(slfcaledms)
</pre>

<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
# =========== Step 4: Apply self-calibration tables =========
if doapply:
import glob
os.chdir(workdir)
clearcal(ms_in)
clearcal(slfcalms)
applycal(vis=slfcalms,gaintable=slftbs,spw=','.join(spws),selectdata=True,\
antenna=antennas,interp='linear',flagbackup=False,applymode='calonly',calwt=False)
applycal(vis=ms_in,gaintable=slftbs,spw=','.join(spws),selectdata=True,\
antenna=antennas,interp='linear',flagbackup=False,applymode='calonly',calwt=False)
if os.path.exists(ms_slfcaled):
os.system('rm -rf '+ms_slfcaled)
split(ms_in, ms_slfcaled,datacolumn='corrected')
</pre>
[[file:Slf.G0.png|thumb|center|500px|Figure 1: Phase before self-calibration]]
[[file:Slf.G1.png|thumb|center|500px|Figure 2: Phase after self-calibration]]

<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
# =========== Step 5: Generate final self-calibrated images (optional) =========
if doclean_slfcaled:
import glob
pol='XX'
print('Processing ' + trange)
img_final=imagedir_slfcaled+'/slf_final_{0}_t0'.format(pol)
vis = ms_slfcaled
spws=[str(s+1) for s in range(30)]
tb.open(vis+'/SPECTRAL_WINDOW')
reffreqs=tb.getcol('REF_FREQUENCY')
bdwds=tb.getcol('TOTAL_BANDWIDTH')
cfreqs=reffreqs+bdwds/2.
tb.close()
sbeam=30.
from matplotlib import gridspec as gridspec
from sunpy import map as smap
from matplotlib import pyplot as plt
fitsfiles=[]
for s,sp in enumerate(spws):
cfreq=cfreqs[int(sp)]
bm=max(sbeam*cfreqs[1]/cfreq,4.)
imname=img_final+'_s'+sp.zfill(2)
fitsfile=imname+'.fits'
if not os.path.exists(fitsfile):
print 'cleaning spw {0:s} with beam size {1:.1f}"'.format(sp,bm)
try:
tclean(vis=vis,
antenna=antennas,
imagename=imname,
spw=sp,
specmode='mfs',
timerange=trange,
imsize=[npix],
cell=['1arcsec'],
niter=1000,
gain=0.05,
stokes=pol,
weighting='briggs',
robust=2.0,
restoringbeam=[str(bm)+'arcsec'],
phasecenter=phasecenter,
mask='',
pbcor=True,
interactive=False)
except:
print 'cleaning spw '+sp+' unsuccessful. Proceed to next spw'
continue
if os.path.exists(imname+'.image.pbcor'):
imn = imname+'.image.pbcor'
hf.imreg(vis=vis,imagefile=imn,fitsfile=fitsfile,
timerange=trange,usephacenter=False,toTb=True,verbose=False)
fitsfiles.append(fitsfile)
junks=['.flux','.model','.psf','.residual','.mask','.image','.pb','.image.pbcor','.sumwt'] #Do not run the next 4 lines if needed to view and assess the subset of clean process images
for junk in junks:
if os.path.exists(imname+junk):
os.system('rm -rf '+imname+junk)
else:
print('fits file '+fitsfile+' already exists, skip clean...')
fitsfiles.append(fitsfile)

fig = plt.figure(figsize=(8.4,7.))
gs = gridspec.GridSpec(5, 6)
for s,sp in enumerate(spws):
cfreq=cfreqs[int(sp)]
ax = fig.add_subplot(gs[s])
eomap=smap.Map(fitsfiles[s])
eomap.plot_settings['cmap'] = plt.get_cmap('jet')
eomap.plot(axes = ax)
eomap.draw_limb()
ax.set_title(' ')
ax.get_xaxis().set_visible(False)
ax.get_yaxis().set_visible(False)
ax.set_xlim(xran)
ax.set_ylim(yran)
plt.text(0.98,0.85,'{0:.1f} GHz'.format(cfreq/1e9),transform=ax.transAxes,ha='right',color='w',fontweight='bold')
plt.subplots_adjust(left=0, bottom=0, right=1, top=1, wspace=0, hspace=0)
plt.show()

</pre>
The .fits files of the self calibrated images at each frequency for the given time are saved at /slfcal/images_slfcaled in your working directory.

[[file:Slf_t0_n0.png|thumb|center|500px|Figure 3: Images before self-calibration]]
[[file:Slf_t0_n1.png|thumb|center|500px|Figure 4: Images after self-calibration]]

====Step 5: Quick-look imaging ==== 
For spectral imaging analysis of the event, follow [http://www.ovsa.njit.edu/wiki/index.php/EOVSA_Data_Analysis_Tutorial#Spectral_Imaging_with_SunCASA this tutorial] using the self-calibrated data obtained from the previous step or use [https://colab.research.google.com/drive/1lSLLxgOG6b8kgu9Sk6kSKvrViyubnXG6?usp=sharing#scrollTo=xbXyyLmCFCGL this link].

<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
from suncasa.utils import qlookplot as ql ## (Optional) Supply the npz file of the dynamic spectrum from previous step.
## If not provided, the program will generate a new one from the visibility.
## set the time interval
from suncasa import dspec as ds
import time
visibility_data = 'IDB20170821202020_slfcaled.ms'
specfile = visibility_data + '.dspec.npz'
d = ds.Dspec(visibility_data, bl='4&9', specfile=specfile)

timerange = '19:02:00~19:02:10' ## select frequency range from 2.5 GHz to 3.5 GHz
spw = '2~5' ## select stokes XX
stokes = 'XX' ## turn off AIA image plotting, default is True
plotaia = False
xycen = [375, 45] ## image center for clean in solar X-Y in arcsec
cell=['2.0arcsec'] ## pixel size
imsize=[128] ## x and y image size in pixels.
fov = [100,100] ## field of view of the zoomed-in panels in unit of arcsec
spw = ['{}'.format(s) for s in range(1,31)]
clevels = [0.5, 1.0] ## contour levels to fill in between.
calpha=0.35 ## now tune down the alpha
restoringbeam=['6arcsec']
ql.qlookplot(vis=msfile, specfile=specfile, timerange=timerange, spw=spw, stokes=stokes, \
restoringbeam=restoringbeam,imsize=imsize,cell=cell, \
xycen=xycen,fov=fov,clevels=clevels,calpha=calpha)
</pre>

Tohban EOVSA Imaging Tutorial A-Z

2022-03-29T19:15:46Z

Schhabra:

This tutorial describes the step-by-step procedure to calibrate EOVSA IDB data to obtain measurement sets (.ms), transfer it to the inti server and self-calibrate for further analysis.

Before starting to work on this tutorial one needs accounts and setup on Pipeline and Inti or Baozi servers.
'''Pre-requisites:''' Accounts on Pipeline and Inti or Baozi servers.

=== Connection details to pipeline server ===
One can use the Mobaxterm platform to connect to the Pipeline server through a Windows machine or use SSH through a Mac machine.

==== Step 1: Downloading raw data (IDB) on Pipeline machine====
On CASA of the Pipeline machine, which has the complete EOVSA SQL database.

If you are not using mobaxterm, directly SSH to Pipeline through your Linux or Mac computer and start tcsh to run CASA.

<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
from astropy.time import Time
import os
trange = Time(['2017-08-21 20:15:00', '2017-08-21 20:25:00']) ###Change accordingly###
#### (Optional) change output path, default current directory "./" #####
outpath = './msdata/' ###Change accordingly###
if not os.path.exists(outpath):
os.makedirs(outpath)
######################################################
msfiles = importeovsa(idbfiles=trange, ncpu=1, visprefix=outpath)
</pre>

NOTE: The above step currently gives an error with importeovsa. This is because the data location was changed and the code doesn't know where to look for the IDB files. Sijie is looking into the problem. We will update the page when it is fixed. For now, please use the method below for all time ranges.

OR (for a time range longer than 10 minutes)

<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
from suncasa.tasks import task_calibeovsa as calibeovsa
from suncasa.tasks import task_importeovsa as timporteovsa
from split_cli import split_cli as split
import dump_tsys as dt
from util import Time
import numpy as np
import os
from glob import glob
from eovsapy import util

trange = Time(['2017-08-21 20:15:00', '2017-08-21 20:35:00']) ###Change accordingly###
idbdir = util.get_idbdir(trange[0])

info = dt.rd_fdb(trange[0])
for k, v in info.iteritems():
info[k] = info[k][~(info[k] == '')]

sidx = np.where(
np.logical_and(info['SOURCEID'] == 'Sun', info['PROJECTID'] == 'NormalObserving') & np.logical_and(
info['ST_TS'].astype(np.float) >= trange[0].lv,
info['ST_TS'].astype(np.float) <= trange[
1].lv))
filelist = info['FILE'][sidx]

outpath = './msdata/' ###Change accordingly###
if not os.path.exists(outpath):
os.makedirs(outpath)
inpath = idbdir + '{}/'.format(trange[0].datetime.strftime("%Y%m%d"))
ncpu = 1

msfiles = timporteovsa.importeovsa(idbfiles=[inpath + ll for ll in filelist], ncpu=ncpu, timebin="0s", width=1,
visprefix=outpath,
nocreatms=False, doconcat=False,
modelms="", doscaling=False, keep_nsclms=False, udb_corr=True)
</pre>

If you come across errors with calibeovsa, add following lines to your ~/.casa/init.py file.
<pre style="background-color: #FCEBD9">
import sys
sys.path.append('/common/python')
sys.path.append('/common/python/packages/pipeline_casa')
execfile('/common/python/suncasa/tasks/mytasks.py')
</pre>

==== Step 2: Concatenate all the 10 mins data, if there are any====
Follow this step if there are more than one .ms files, if not run step 3 directly. If doimage=True, a quicklook image will be produced (by integrating over the entire time) as shown below.
<pre style="background-color: #FCEBD9">
# This is to set the path/name for the concatenated files
concatvis = os.path.basename(msfiles[0])[:11] + '_concat_cal.ms'
vis = calibeovsa.calibeovsa(msfiles, doconcat=True, concatvis=concatvis, caltype=['refpha','phacal'], doimage=True)
</pre>

[[file:Figure1_imagingtutorial.png|frame|center|800px|'''Figure 1:''' Quick-look full-Sun image after the initial calibration.]]

==== Step 3: Calibration ====
This will calibrate the input visibility, write out calibration tables under /data1/eovsa/caltable/, and apply the calibration.
<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
vis = calibeovsa.calibeovsa('IDB20170821202020.ms', caltype=['refpha','phacal'], doimage=True) ###Change the vis filename accordingly###
# Append '_cal' to the ms filename and split the corrected column to the new caled ms
vis_str = str(' '.join(vis))
caled_vis=vis_str.replace('.ms','_cal.ms')
split(vis=' '.join(vis),outputvis=caled_vis,datacolumn='corrected',timerange='',correlation='XX')
</pre>

One needs to transfer the created caled ms data files (xxx_concat_cal.ms or xxx_cal.ms) from pipeline to inti server, which has all the casatasks installed, in order to run the rest of the imaging steps.

=== Connection details to Inti server ===
For Windows, on Mobaxterm,

#With your NJIT VPN connected, connect to one of the afsconnect servers (for example, afsaccess3.njit.edu) using your UCID and password.
#ssh -X UCID@inti.hpcnet.campus.njit.edu

To avoid typing the full inti address each time you attempt for ssh, you may wish to add the following lines with your username to C:\Program Files\Git\etc\ssh\ssh_config on Windows and /.ssh/config on Mac and Linux machines.

<pre style="background-color: #FCEBD9">
Host inti
Hostname inti.hpcnet.campus.njit.edu
User USERNAME ###Insert UCID/inti username here###
</pre>

To have sufficient disk space with EOVSA data analysis over Inti, use your dedicated directory YOURDIRECTORY at the location given below. If you do not have a directory, Please take help from Sijie in creating one.
<pre style="background-color: #FCEBD9">
cd /inti/data/users/YOURDIRECTORY
</pre>

For Linux or Mac machine,
ssh -X UCID@inti.hpcnet.campus.njit.edu

=== Transferring data files between servers ===
For directly transferring your calibrated .ms data between the Pipeline and Inti servers, follow the below given steps.
<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
1. Log into Inti using your username.
ssh -X USERNAME@inti.hpcnet.campus.njit.edu

Then create a tunnel into Pipeline from Inti.
ssh -L 8888:pipeline.solar.pvt:22 guest@ovsa.njit.edu

2. Log into Inti again from a new terminal.

Change to your working directory and give this command to copy your data on Pipeline.
scp -v -C -r -P 8888 USERNAMEofPipeline@localhost:PATHofMSDATA/MSfilename ./
Eg: scp -v -C -r -P 8888 shaheda@localhost:/data1/shaheda/IDB20220118173922_cal.ms ./
where, MSfilename, PATHofMSDATA are your .ms data and its path on Pipeline machine.
</pre>

Alternatively, one can follow the below procedure to do the transfer.

For Windows, on mobaxterm, drag and drop the ms file to your local machine or use scp command as given below.

<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
scp -r -C userid@pipeline:/your/folder/msfile /localmachine/destination/folder/
</pre>

On mobaxterm and on your local terminal, use the following command to finally copy the ms file to Inti.

<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
scp -r -C /localmachine/destination/folder/msfile UCID@inti.hpcnet.campus.njit.edu:/inti/data/users/YOURDIRECTORY
</pre>

For Mac and Linux, SCP can be used in the same way. Add the following lines to your SSH config file, to bypass ovsa.njit.edu from copying.
vi ~/.ssh/config
<pre style="background-color: #FCEBD9">
Host ovsa
HostName ovsa.njit.edu
User guest
Host pipeline
Hostname pipeline.solar.pvt
User userid
ProxyCommand ssh -W %h:%p guest@ovsa.njit.edu
</pre>

=== Software details on the servers ===
On Inti,
when logging in for the first time, please add the following lines to your accounts ~/.bashrc file.

>>vi ~/.bashrc
Insert the text given below and save it.

<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
#### setting start ####
if [ $HOSTNAME == "baozi.hpcnet.campus.njit.edu" ]; then
source /srg/.setenv_baozi
fi
if [ $HOSTNAME == "inti.hpcnet.campus.njit.edu" ]; then
source /inti/.setenv_inti
fi
if [ $HOSTNAME == "guko.resource.campus.njit.edu" ]; then
source /data/data/.setenv_guko
fi
#### setting end ####
</pre>

Both CASA 5 and 6 are available on Inti.

Enter the bash environment on inti, and load the desired casa environment.
To load CASA 5, enter bash environment by giving >>bash
<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
>> loadcasa5
>> suncasa #This should load the software making you ready for the analysis
</pre>

Or to load CASA 6, enter bash environment by giving >>bash
<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
>> loadcasa6
>> ipython #This should load the software making you ready for the analysis
</pre>
Here, for example, to use clean, first start ipython as given above, then type in >>from casatasks import tclean

====Step 4: Self-calibration====
[[file:Figure3_imagingtutorial.png|thumb|center|500px|Figure 2: Cotton-Schwab clean major and minor cycles. [Source: http://www.aoc.nrao.edu/~rurvashi/ImagingAlgorithmsInCasa/node2.html].]]
https://github.com/binchensun/casa-eovsa/blob/master/slfcal_example.py

<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
from suncasa.utils import helioimage2fits as hf
import os
import numpy as np
import pickle
from matplotlib import gridspec as gridspec
from sunpy import map as smap
from matplotlib import pyplot as plt
from split_cli import split_cli as split
import time

# =========== task handlers =============
dofullsun = 1 # initial full-sun imaging ###Change accordingly###
domasks=1 # get masks ###Change accordingly###
doslfcal=1 # main cycle of doing selfcalibration ###Change accordingly###
doapply=1 # apply the results ###Change accordingly###
doclean_slfcaled=1 # perform clean for self-calibrated data ###Change accordingly###

# ============ declaring the working directories ============
workdir = os.getcwd()+'/' #main working directory. Using current directory in this example
slfcaldir = workdir+'slfcal/' #place to put all selfcalibration products
imagedir = slfcaldir+'images/' #place to put all selfcalibration images
maskdir = slfcaldir+'masks/' #place to put clean masks
imagedir_slfcaled = slfcaldir+'images_slfcaled/' #place to put final self-calibrated images
caltbdir = slfcaldir+'caltbs/' # place to put calibration tables
# make these directories if they do not already exist
dirs = [workdir, slfcaldir, imagedir, maskdir, imagedir_slfcaled, caltbdir]
for d in dirs:
if not os.path.exists(d):
os.makedirs(d)

# ============ Split a short time for self-calibration ===========
# input visibility
ms_in = workdir + 'IDB20170821202020_cal.ms' ###Change the initial calibrated (through calibeovsa) vis accordingly###
# output, selfcaled, visibility
ms_slfcaled = workdir + os.path.basename(ms_in).replace('cal','slfcaled')
# intermediate small visibility for selfcalbration
# selected time range for generating self-calibration solutions
trange='2017/08/21/20:21:10~2017/08/21/20:21:30' ###Change accordingly###
slfcalms = slfcaldir+'slfcalms.XX.slfcal'
slfcaledms = slfcaldir+'slfcalms.XX.slfcaled'
if not os.path.exists(slfcalms):
split(vis=ms_in,outputvis=slfcalms,datacolumn='data',timerange=trange,correlation='XX')

# ============ Prior definitions for spectral windows, antennas, pixel numbers =========
spws=[str(s+1) for s in range(30)]
antennas='0~12'
npix=512
nround=3 #number of slfcal cycles

# =========== Step 1, doing a full-Sun image to find out phasecenter and appropriate field of view =========
if dofullsun:
#initial mfs clean to find out the image phase center
im_init='fullsun_init'
os.system('rm -rf '+im_init+'*')
tclean(vis=slfcalms,
antenna='0~12',
imagename=im_init,
spw='1~15',
specmode='mfs',
timerange=trange,
imsize=[npix],
cell=['5arcsec'],
niter=1000,
gain=0.05,
stokes='I',
restoringbeam=['30arcsec'],
interactive=False,
pbcor=True)

hf.imreg(vis=slfcalms,imagefile=im_init+'.image.pbcor',fitsfile=im_init+'.fits',
timerange=trange,usephacenter=False,verbose=True)
clnjunks = ['.flux', '.mask', '.model', '.psf', '.residual','.sumwt','.pb','.image'] #Do not run the next 4 lines if needed to view and assess the subset of clean process images
for clnjunk in clnjunks:
if os.path.exists(im_init + clnjunk):
os.system('rm -rf '+im_init + clnjunk)

from sunpy import map as smap
from matplotlib import pyplot as plt
fig = plt.figure(figsize=(6,6))
ax = fig.add_subplot(111)
eomap=smap.Map(im_init+'.fits')
#eomap.data=eomap.data.reshape((npix,npix))
eomap.plot_settings['cmap'] = plt.get_cmap('jet')
eomap.plot(axes = ax)
eomap.draw_limb()
plt.show()
viewer(im_init+'.image.pbcor')
</pre>
[[file:Figure2_imagingtutorial.png|thumb|center|500px|Figure 3: Full-Sun image after initial clean to find the flare location.]]

<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
# parameters specific to the event (found from step 1)
phasecenter='J2000 10h02m59 11d58m07' ###Change accordingly###
xran=[280,480] ###Change accordingly###
yran=[-50,150] ###Change accordingly###

# =========== Step 2 (optional), generate masks =========
# if skipped, will not use any masks
if domasks:
clearcal(slfcalms)
delmod(slfcalms)
antennas='0~12'
pol='XX'
imgprefix=maskdir+'slf_t0'

# step 1: set up the clean masks
img_init=imgprefix+'_init_ar_'
os.system('rm -rf '+img_init+'*')
#spwrans_mask=['1~5','6~12','13~20','21~30']
spwrans_mask=['1~12']
#convert to a list of spws
spwrans_mask_list = [[str(i) for i in (np.arange(int(m.split('~')[0]),int(m.split('~')[1])))] for m in spwrans_mask]
masks=[]
imnames=[]
for spwran in spwrans_mask:
imname=img_init+spwran.replace('~','-')
try:
tclean(vis=slfcalms,
antenna='0~12',
imagename=imname,
spw=spwran,
specmode='mfs',
timerange=trange,
imsize=[npix],
cell=['2arcsec'],
niter=1000,
gain=0.05,
stokes='XX',
restoringbeam=['20arcsec'],
phasecenter=phasecenter,
weighting='briggs',
robust=1.0,
interactive=True,
datacolumn='data',
pbcor=True,
savemodel='modelcolumn')
imnames.append(imname+'.image')
masks.append(imname+'.mask')
clnjunks = ['.flux', '.model', '.psf', '.residual'] #Do not run the next 4 lines if needed to view and assess the subset of clean process images
for clnjunk in clnjunks:
if os.path.exists(imname + clnjunk):
os.system('rm -rf '+ imname + clnjunk)
except:
print('error in cleaning spw: '+spwran)

pickle.dump(masks,open(slfcaldir+'masks.p','wb'))
</pre>
[[file:Figure4_imagingtutorial.PNG|thumb|center|800px|Figure 4: Interactive clean window to create masks over the source. The white outline surrounding the source is the mask selected by the polygon drawing option.]]

The outlines drawn for masks can be created by any of the icons with the letter 'R' in the Viewer window. The instructions for doing this can be found by hovering over those icons.

<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
# =========== Step 3, main step of selfcalibration =========
if doslfcal:
if os.path.exists(slfcaldir+'masks.p'):
masks=pickle.load(open(slfcaldir+'masks.p','rb'))
if not os.path.exists(slfcaldir+'masks.p'):
print 'masks do not exist. Use default mask'
masks=[]
os.system('rm -rf '+imagedir+'*')
os.system('rm -rf '+caltbdir+'*')
#first step: make a mock caltable for the entire database
print('Processing ' + trange)
slftbs=[]
calprefix=caltbdir+'slf'
imgprefix=imagedir+'slf'
tb.open(slfcalms+'/SPECTRAL_WINDOW')
reffreqs=tb.getcol('REF_FREQUENCY')
bdwds=tb.getcol('TOTAL_BANDWIDTH')
cfreqs=reffreqs+bdwds/2.
tb.close()
# starting beam size at 3.4 GHz in arcsec
sbeam=40.
strtmp=[m.replace(':','') for m in trange.split('~')]
timestr='t'+strtmp[0]+'-'+strtmp[1]
refantenna='0'
# number of iterations for each round
niters=[100, 300, 500]
# roburst value for weighting the baselines
robusts=[1.0, 0.5, 0.0]
# apply calibration tables? Set to true for most cases
doapplycal=[1,1,1]
# modes for calibration, 'p' for phase-only, 'a' for amplitude only, 'ap' for both
calmodes=['p','p','a']
# setting uvranges for model image (optional, not used here)
uvranges=['','','']
for n in range(nround):
slfcal_tb_g= calprefix+'.G'+str(n)
fig = plt.figure(figsize=(8.4,7.))
gs = gridspec.GridSpec(5, 6)
for s,sp in enumerate(spws):
print 'processing spw: '+sp
cfreq=cfreqs[int(sp)]
# setting restoring beam size (not very useful for selfcal anyway, but just to see the results)
bm=max(sbeam*cfreqs[1]/cfreq, 6.)
slfcal_img = imgprefix+'.spw'+sp.zfill(2)+'.slfcal'+str(n)
# only the first round uses nearby spws for getting initial model
if n == 0:
spbg=max(int(sp)-2,1)
sped=min(int(sp)+2,30)
spwran=str(spbg)+'~'+str(sped)
print('using spw {0:s} as model'.format(spwran))
if 'spwrans_mask' in vars():
for m, spwran_mask in enumerate(spwrans_mask):
if sp in spwran_mask:
mask = masks[m]
print('using mask {0:s}'.format(mask))
findmask = True
if not findmask:
print('mask not found. Do use any masks')
else:
spwran = sp
if 'spwrans_mask' in vars():
for m, spwran_mask in enumerate(spwrans_mask):
if sp in spwran_mask:
mask = masks[m]
print 'using mask {0:s}'.format(mask)
findmask = True
if not findmask:
print('mask not found. Do use any masks')
try:
tclean(vis=slfcalms,
antenna=antennas,
imagename=slfcal_img,
uvrange=uvranges[n],
spw=spwran,
specmode='mfs',
timerange=trange,
imsize=[npix],
cell=['2arcsec'],
niter=niters[n],
gain=0.05,
stokes='XX', #use pol XX image as the model
weighting='briggs',
robust=robusts[n],
phasecenter=phasecenter,
mask=mask,
restoringbeam=[str(bm)+'arcsec'],
pbcor=False,
interactive=False,
savemodel='modelcolumn')
if os.path.exists(slfcal_img+'.image'):
fitsfile=slfcal_img+'.fits'
hf.imreg(vis=slfcalms,imagefile=slfcal_img+'.image',fitsfile=fitsfile,
timerange=trange,usephacenter=False,toTb=True,verbose=False,overwrite=True)
clnjunks = ['.mask','.flux', '.model', '.psf', '.residual', '.image','.pb','.image.pbcor','.sumwt'] #Do not run the next 4 lines if needed to view and assess the subset of clean process images
for clnjunk in clnjunks:
if os.path.exists(slfcal_img + clnjunk):
os.system('rm -rf '+ slfcal_img + clnjunk)
ax = fig.add_subplot(gs[s])
eomap=smap.Map(fitsfile)
eomap.plot_settings['cmap'] = plt.get_cmap('jet')
eomap.plot(axes = ax)
eomap.draw_limb()
#eomap.draw_grid()
ax.set_title(' ')
ax.get_xaxis().set_visible(False)
ax.get_yaxis().set_visible(False)
ax.set_xlim(xran)
ax.set_ylim(yran)
os.system('rm -f '+ fitsfile)

except:
print 'error in cleaning spw: '+sp
print 'using nearby spws for initial model'
sp_e=int(sp)+2
sp_i=int(sp)-2
if sp_i < 1:
sp_i = 1
if sp_e > 30:
sp_e = 30
sp_=str(sp_i)+'~'+str(sp_e)
try:
tget(tclean)
spw=sp_
print('using spw {0:s} as model'.format(sp_))
tclean()
except:
print 'still not successful. abort...'
break

gaincal(vis=slfcalms, refant=refantenna,antenna=antennas,caltable=slfcal_tb_g,spw=sp, uvrange='',\
gaintable=[],selectdata=True,timerange=trange,solint='inf',gaintype='G',calmode=calmodes[n],\
combine='',minblperant=4,minsnr=2,append=True)
if not os.path.exists(slfcal_tb_g):
print 'No solution found in spw: '+sp
figname=imagedir+'slf_t0_n{:d}.png'.format(n)
plt.subplots_adjust(left=0, bottom=0, right=1, top=1, wspace=0, hspace=0)
plt.savefig(figname)
time.sleep(10)
plt.close()

if os.path.exists(slfcal_tb_g):
slftbs.append(slfcal_tb_g)
slftb=[slfcal_tb_g]
os.chdir(slfcaldir)
if calmodes[n] == 'p':
plotcal(caltable=slfcal_tb_g,antenna='1~12',xaxis='freq',yaxis='phase',\
subplot=431,plotrange=[-1,-1,-180,180],iteration='antenna',figfile=slfcal_tb_g+'.png',showgui=False)
if calmodes[n] == 'a':
plotcal(caltable=slfcal_tb_g,antenna='1~12',xaxis='freq',yaxis='amp',\
subplot=431,plotrange=[-1,-1,0,2.],iteration='antenna',figfile=slfcal_tb_g+'.png',showgui=False)
os.chdir(workdir)

if doapplycal[n]:
clearcal(slfcalms)
delmod(slfcalms)
applycal(vis=slfcalms,gaintable=slftb,spw=','.join(spws),selectdata=True,\
antenna=antennas,interp='nearest',flagbackup=False,applymode='calonly',calwt=False)

if n < nround-1:
prompt=raw_input('Continuing to selfcal?')
#prompt='y'
if prompt.lower() == 'n':
if os.path.exists(slfcaledms):
os.system('rm -rf '+slfcaledms)
split(slfcalms,slfcaledms,datacolumn='corrected')
print 'Final calibrated ms is {0:s}'.format(slfcaledms)
break
if prompt.lower() == 'y':
slfcalms_=slfcalms+str(n)
if os.path.exists(slfcalms_):
os.system('rm -rf '+slfcalms_)
split(slfcalms,slfcalms_,datacolumn='corrected')
slfcalms=slfcalms_
else:
if os.path.exists(slfcaledms):
os.system('rm -rf '+slfcaledms)
split(slfcalms,slfcaledms,datacolumn='corrected')
print 'Final calibrated ms is {0:s}'.format(slfcaledms)
</pre>

<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
# =========== Step 4: Apply self-calibration tables =========
if doapply:
import glob
os.chdir(workdir)
clearcal(ms_in)
clearcal(slfcalms)
applycal(vis=slfcalms,gaintable=slftbs,spw=','.join(spws),selectdata=True,\
antenna=antennas,interp='linear',flagbackup=False,applymode='calonly',calwt=False)
applycal(vis=ms_in,gaintable=slftbs,spw=','.join(spws),selectdata=True,\
antenna=antennas,interp='linear',flagbackup=False,applymode='calonly',calwt=False)
if os.path.exists(ms_slfcaled):
os.system('rm -rf '+ms_slfcaled)
split(ms_in, ms_slfcaled,datacolumn='corrected')
</pre>
[[file:Slf.G0.png|thumb|center|500px|Figure 1: Phase before self-calibration]]
[[file:Slf.G1.png|thumb|center|500px|Figure 2: Phase after self-calibration]]

<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
# =========== Step 5: Generate final self-calibrated images (optional) =========
if doclean_slfcaled:
import glob
pol='XX'
print('Processing ' + trange)
img_final=imagedir_slfcaled+'/slf_final_{0}_t0'.format(pol)
vis = ms_slfcaled
spws=[str(s+1) for s in range(30)]
tb.open(vis+'/SPECTRAL_WINDOW')
reffreqs=tb.getcol('REF_FREQUENCY')
bdwds=tb.getcol('TOTAL_BANDWIDTH')
cfreqs=reffreqs+bdwds/2.
tb.close()
sbeam=30.
from matplotlib import gridspec as gridspec
from sunpy import map as smap
from matplotlib import pyplot as plt
fitsfiles=[]
for s,sp in enumerate(spws):
cfreq=cfreqs[int(sp)]
bm=max(sbeam*cfreqs[1]/cfreq,4.)
imname=img_final+'_s'+sp.zfill(2)
fitsfile=imname+'.fits'
if not os.path.exists(fitsfile):
print 'cleaning spw {0:s} with beam size {1:.1f}"'.format(sp,bm)
try:
tclean(vis=vis,
antenna=antennas,
imagename=imname,
spw=sp,
specmode='mfs',
timerange=trange,
imsize=[npix],
cell=['1arcsec'],
niter=1000,
gain=0.05,
stokes=pol,
weighting='briggs',
robust=2.0,
restoringbeam=[str(bm)+'arcsec'],
phasecenter=phasecenter,
mask='',
pbcor=True,
interactive=False)
except:
print 'cleaning spw '+sp+' unsuccessful. Proceed to next spw'
continue
if os.path.exists(imname+'.image.pbcor'):
imn = imname+'.image.pbcor'
hf.imreg(vis=vis,imagefile=imn,fitsfile=fitsfile,
timerange=trange,usephacenter=False,toTb=True,verbose=False)
fitsfiles.append(fitsfile)
junks=['.flux','.model','.psf','.residual','.mask','.image','.pb','.image.pbcor','.sumwt'] #Do not run the next 4 lines if needed to view and assess the subset of clean process images
for junk in junks:
if os.path.exists(imname+junk):
os.system('rm -rf '+imname+junk)
else:
print('fits file '+fitsfile+' already exists, skip clean...')
fitsfiles.append(fitsfile)

fig = plt.figure(figsize=(8.4,7.))
gs = gridspec.GridSpec(5, 6)
for s,sp in enumerate(spws):
cfreq=cfreqs[int(sp)]
ax = fig.add_subplot(gs[s])
eomap=smap.Map(fitsfiles[s])
eomap.plot_settings['cmap'] = plt.get_cmap('jet')
eomap.plot(axes = ax)
eomap.draw_limb()
ax.set_title(' ')
ax.get_xaxis().set_visible(False)
ax.get_yaxis().set_visible(False)
ax.set_xlim(xran)
ax.set_ylim(yran)
plt.text(0.98,0.85,'{0:.1f} GHz'.format(cfreq/1e9),transform=ax.transAxes,ha='right',color='w',fontweight='bold')
plt.subplots_adjust(left=0, bottom=0, right=1, top=1, wspace=0, hspace=0)
plt.show()

</pre>
The .fits files of the self calibrated images at each frequency for the given time are saved at /slfcal/images_slfcaled in your working directory.

[[file:Slf_t0_n0.png|thumb|center|500px|Figure 3: Images before self-calibration]]
[[file:Slf_t0_n1.png|thumb|center|500px|Figure 4: Images after self-calibration]]

====Step 5: Quick-look imaging ==== 
For spectral imaging analysis of the event, follow [http://www.ovsa.njit.edu/wiki/index.php/EOVSA_Data_Analysis_Tutorial#Spectral_Imaging_with_SunCASA this tutorial] using the self-calibrated data obtained from the previous step or use [https://colab.research.google.com/drive/1lSLLxgOG6b8kgu9Sk6kSKvrViyubnXG6?usp=sharing#scrollTo=xbXyyLmCFCGL this link].

<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
from suncasa.utils import qlookplot as ql ## (Optional) Supply the npz file of the dynamic spectrum from previous step.
## If not provided, the program will generate a new one from the visibility.
## set the time interval
from suncasa import dspec as ds
import time
visibility_data = 'IDB20170821202020_slfcaled.ms'
specfile = visibility_data + '.dspec.npz'
d = ds.Dspec(visibility_data, bl='4&9', specfile=specfile)

timerange = '19:02:00~19:02:10' ## select frequency range from 2.5 GHz to 3.5 GHz
spw = '2~5' ## select stokes XX
stokes = 'XX' ## turn off AIA image plotting, default is True
plotaia = False
xycen = [375, 45] ## image center for clean in solar X-Y in arcsec
cell=['2.0arcsec'] ## pixel size
imsize=[128] ## x and y image size in pixels.
fov = [100,100] ## field of view of the zoomed-in panels in unit of arcsec
spw = ['{}'.format(s) for s in range(1,31)]
clevels = [0.5, 1.0] ## contour levels to fill in between.
calpha=0.35 ## now tune down the alpha
restoringbeam=['6arcsec']
ql.qlookplot(vis=msfile, specfile=specfile, timerange=timerange, spw=spw, stokes=stokes, \
restoringbeam=restoringbeam,imsize=imsize,cell=cell, \
xycen=xycen,fov=fov,clevels=clevels,calpha=calpha)
</pre>

Guide to Upgrade SolarSoft(SSW)

2022-02-16T20:02:29Z

Schhabra: Created page with " SSW upgrade using passive_ftp was stopped in August 2019. The current way to upgrade SSW is to use the WGET executable. If you have done and update on SSW since August 2019 a..."

SSW upgrade using passive_ftp was stopped in August 2019. The current way to upgrade SSW is to use the WGET executable. If you have done and update on SSW since August 2019 and your system already has WGET, any further upgrade can be made by directly following the steps underlined [https://hesperia.gsfc.nasa.gov/rhessi3/docs/ssw_update_help.html here]. For Linux and MacOS you can check if you already have WGET by typing the command "which wget" in the bash shell.

If your system was not updated since August 2019, follow the steps below:

=== MacOS ===
==== 1. Installing wget ====
'''USING HOMEBREW'''

To install homebrew, type the following command in the bash shell:
<pre style="background-color: #FCEBD9">
/usr/bin/ruby -e "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/install)"
</pre>

Once homebrew is installed, you can simply install wget by using the following command:
<pre style="background-color: #FCEBD9">
brew install wget
</pre>

'''USING MacPorts'''

Alternatively, wget can also be installed using MacPorts. Directions to install MacPorts for specific OS versions are given [https://www.macports.org/install.php here]

Owens Valley Solar Arrays

2022-02-16T19:20:41Z

Schhabra: /* Tohbans */

[[File:Eovsa1.png|border|text-top|800px]]

<big>[http://ovsa.njit.edu/ EOVSA] (Expanded Owens Valley Solar Array) is a new solar-dedicated radio interferometer operated by the New Jersey Institute of Technology. This wiki serves as the site for EOVSA documentation. </big>

== EOVSA Documentation ==

* <big>General</big>
** [[Downconversion and Frequency Tuning]]
** [[Dealing with Radio Frequency Interference]]
** [[Switching between 200 MHz and 300 MHz Correlator]]

* <big>Computer-Network</big>
** [[Computing Systems]]
** [[Network]]

* <big>Control System</big>
** [[27-m Antenna Commands]]
** [[Schedule Commands]]
** [[Control Commands]]
** [[Attenuation and Level Control]]

* <big>Hardware</big>
** [[Hardware Overview]]
** [[2.1-m Antennas]]
** [[27-m Antennas]]

* <big>System Software</big>
** [[Calibration Database]]
** [[Stateframe Database]]
** [[Create CASA measurement sets]]

* <big>Calibration</big>
**[[Calibration Overview]]
**[[Pointing Calibration]]
**[[Total Power Calibration]]
**[[System Gain Calibration]]
**[[Antenna Position]] (Baseline Calibration)
**[[Reference Gain Calibration]]
**[[Daily Gain Calibration]]
**[[Delay Calibration]]
**[[Bandpass Calibration]]
**[[Polarization Calibration]]
**[[Calibrator Survey]]
**[[Practical Calibration Tutorial]]

* <big>[[Starburst]]</big>

== Using EOVSA Data ==
* <big>[[EOVSA Data products]]</big>
* <big>Analysis Software</big>
** [https://github.com/suncasa/suncasa SunCASA] A wrapper around [https://casa.nrao.edu/ CASA (the Common Astronomy Software Applications package)] for synthesis imaging and visualizing solar spectral imaging data. CASA is one of the leading software tool for "supporting the data post-processing needs of the next generation of radio astronomical telescopes such as ALMA and VLA", an international effort led by the [https://public.nrao.edu/ National Radio Astronomy Observatory]. The current version of CASA uses Python (2.7) interface. More information about CASA can be found on [https://casa.nrao.edu/ NRAO's CASA website ]. Note, CASA is available ONLY on UNIX-BASED PLATFORMS (and therefore, so is SunCASA).
** [https://github.com/Gelu-Nita/GSFIT GSFIT] A IDL-widget(GUI)-based spectral fitting package called gsfit, which provides a user-friendly display of EOVSA image cubes and an interface to fast fitting codes (via platform-dependent shared-object libraries).
** [[Spectrogram Software]]
** [[Mapping Software]]
* <big>Data Analysis Guides</big>
** <big>[https://colab.research.google.com/drive/1lSLLxgOG6b8kgu9Sk6kSKvrViyubnXG6?usp=sharing#scrollTo=xbXyyLmCFCGL EOVSA Data Analysis Tutorial at RHESSI 19 Workshop]</big>
** <big>[[EOVSA Data Analysis Tutorial]]</big> at [http://rhessi18.umn.edu/ RHESSI XVIII Workshop]
** [[Self-Calibrating Flare Data]] Example script and guides for self-calibrating EOVSA flare data (to be completed)


**[[IDB flare pipeline]] Tutorial to run the flare pipeline for quicklook images



* <big>EOVSA Modeling Guide</big>
**[[GX Simulator]]

* Other helpful links
** [https://casaguides.nrao.edu CASA Guides]
** [http://www.lmsal.com/solarsoft/ SolarSoft IDL]
** [http://www.atnf.csiro.au/computing/software/miriad/userguide/userhtml.html Miriad Guides]
** [https://sites.google.com/site/fgscodes/ Fast Gyrosynchrotron Codes (Alexey Kuznetsov's website)]
** [[Basic GitHub Tutorial]]


* <big>[[Full Disk Simulations]]</big>
* <big>[[All-Day Synthesis Issues]]</big>
* <big>[[Analyzing Pre-2017 Data]]</big>
* <big>[[Fixing Pipeline Problems pre-2021-Feb-07]]</big>

== System Software ==

* LabVIEW software
* Python code [https://github.com/dgary50/eovsa Github repository]

== Observing Log ==
[[2016 November]]; [[2016 December| December]]

[[2017 January]]; [[2017 February | February]]; [[2017 March | March]]; [[2017 April | April]]; [[2017 May | May]]; [[2017 June | June]];
[[2017 July | July]]; [[2017 August | August]]; [[2017 September | September]]; [[2017 October | October]]; [[2017 November | November]]; [[2017 December | December]]

[[2018 January]]; [[2018 February | February]]; [[2018 March | March]]; [[2018 April | April]]; [[2018 May | May]]; [[2018 June | June]];
[[2018 July | July]]; [[2018 August | August]]; [[2018 September | September]]; [[2018 October | October]]; [[2018 November | November]]; [[2018 December | December]]

[[2019 January]]; [[2019 February | February]]; [[2019 March | March]]; [[2019 April | April]]; [[2019 May | May]]; [[2019 June | June]];
[[2019 July | July]]; [[2019 August | August]]; [[2019 September | September]]; [[2019 October | October]]; [[2019 November | November]]; [[2019 December | December]]

[[2020 January]]; [[2020 February | February]]; [[2020 March | March]]; [[2020 April | April]]; [[2020 May | May]]; [[2020 June | June]];
[[2020 July | July]]; [[2020 August | August]]; [[2020 September | September]]; [[2020 October | October]]; [[2020 November | November]]; [[2020 December | December]]

[[2021 January]]; [[2021 February | February]]; [[2021 March | March]]; [[2021 April | April]]; [[2021 May | May]]; [[2021 June | June]];
[[2021 July | July]]; [[2021 August | August]]; [[2021 September | September]]; [[2021 October | October]]; [[2021 November | November]]; [[2021 December | December]]

== Tohbans ==

[[Trouble Shooting Guide]]

[[Tohban Records]]

[[Owen's Notes]]

[[Tohban EOVSA Imaging Tutorial A-Z]]

[[Guide to Upgrade SolarSoft(SSW)]]

== EOVSA Flare List ==

See [https://docs.google.com/spreadsheets/d/1P8jHuDRF93dMflU6RMQcsJqVepD9vFkPkofV8Imj4xA/edit?usp=sharing this link] for a list of EOVSA flares as a Google Spreadsheet.

[[Recent Flare List (2021-)]]

[http://ovsa.njit.edu/jay/rd_db.php An older link] is available at the EOVSA website.

== EOVSA Publications ==
Here is a (partial) list of publications that utilize EOVSA data. See also the collection of EOVSA publications at [https://ui.adsabs.harvard.edu/public-libraries/eQ7HfPkySqydu-B8BCt6QQ this NASA/ADS Library].
; 2021
: [https://ui.adsabs.harvard.edu/abs/2021arXiv211015751L/abstract Lopez et al. (2021), arXiv 2110.15751] ''A solar flare driven by thermal conduction observed in mid-infrared''
: [https://ui.adsabs.harvard.edu/abs/2021ApJ...923..213W/abstract Wei et al. (2021), ApJ, 923, 213] ''Coronal Magnetic Field Measurements along a Partially Erupting Filament in a Solar Flare''
: [https://ui.adsabs.harvard.edu/abs/2021ApJ...919...44S/abstract Shaik & Gary (2021), ApJ, 919, 44] ''Implications of Flat Optically Thick Microwave Spectra in Solar Flares for Source Size and Morphology''
: [https://ui.adsabs.harvard.edu/abs/2021ApJ...915...12K/abstract Kocharov et al. (2021), ApJ, 915, 12] ''Multiple Sources of Solar High-energy Protons''
: [https://ui.adsabs.harvard.edu/abs/2021ApJ...908L..55C/abstract Chen et al. (2021), ApJL, 908, L55] ''Energetic Electron Distribution of the Coronal Acceleration Region: First results from Joint Microwave and Hard X-ray Imaging Spectroscopy''
: [https://ui.adsabs.harvard.edu/abs/2021ApJ...906..132C/abstract Chhabra et al. (2021), ApJ, 906, 132] ''Imaging Spectroscopy of CME-Associated Solar Radio Bursts''
; 2020
: [https://ui.adsabs.harvard.edu/abs/2020ApJ...905..165R/abstract Reeves et al. (2020), ApJ, 905, 165] ''Hot Plasma Flows and Oscillations in the Loop-top Region During the September 10 2017 X8.2 Solar Flare''
: [https://ui.adsabs.harvard.edu/abs/2020ApJ...900...17Y/abstract Yu et al. (2020), ApJ, 900, 17] ''Magnetic Reconnection During the Post Impulsive Phase of the X8.2 Solar Flare: Bi-Directional Outflows as a Cause of Microwave and X-ray Bursts''
: [https://ui.adsabs.harvard.edu/abs/2020NatAs...4.1140C/abstract Chen et al. (2020b), Nature Astronomy, 4, 1140] ''Measurement of magnetic field and relativistic electrons along a solar flare current sheet''
: [https://ui.adsabs.harvard.edu/abs/2020ApJ...895L..50C/abstract Chen et al. (2020a), ApJL, 895, 50] ''Microwave Spectral Imaging of an Erupting Magnetic Flux Rope: Implications for the Standard Solar Flare Model in Three Dimensions''
: [https://ui.adsabs.harvard.edu/abs/2020FrASS...7...22K/abstract Kuroda et al. (2020), Frontiers, 7, 22] ''Evolution of Flare-accelerated Electrons Quantified by Spatially Resolved Analysis''
: [https://ui.adsabs.harvard.edu/abs/2020ApJ...891L..34G/abstract Glesener et al. (2020), ApJL, 891, 34] ''Accelerated Electrons Observed Down to <7 keV in a NuSTAR Solar Microflare''
: [https://ui.adsabs.harvard.edu/abs/2020ApJ...889...72K/abstract Karlicky at al. (2020), ApJ, 889, 72] ''Drifting Pulsation Structure at the Very Beginning of the 2017 September 10 Limb Flare''
: [https://ui.adsabs.harvard.edu/abs/2020Sci...367..278F/abstract Fleishman et al. (2020), Science, 367, 278] ''Decay of the coronal magnetic field can release sufficient energy to power a solar flare''
: [https://ui.adsabs.harvard.edu/abs/2020AAS...23538501G/abstract Gary et al. (2020), BAAS 52, 385.01] [https://aas235-aas.ipostersessions.com/default.aspx?s=97-69-9E-4B-34-19-68-53-1B-C6-21-0C-16-1C-5C-82&guestview=true Direct link to AAS iPoster] ''A new view of the solar atmosphere: daily full-disk multifrequency radio images from EOVSA''
; 2018
: [https://ui.adsabs.harvard.edu/#abs/2018ApJ...864...63P/abstract Polito et al. (2018), ApJ, 864, 63] ''Broad Non-Gaussian Fe XXIV Line Profiles in the Impulsive Phase of the 2017 September 10 X8.3-class Flare Observed by Hinode/EIS''
: [https://ui.adsabs.harvard.edu/#abs/2018ApJ...863...83G/abstract Gary et al. (2018), ApJ, 863, 83] ''Microwave and Hard X-Ray Observations of the 2017 September 10 Solar Limb Flare''
: [https://ui.adsabs.harvard.edu/#abs/2018ApJ...852...32K/abstract Kuroda et al. (2018), ApJ, 852, 32] ''Three-dimensional Forward-fit Modeling of the Hard X-ray and the Microwave Emissions of the 2015 June 22 M6.5 flare''
; 2017
: [https://ui.adsabs.harvard.edu/abs/2017NatAs...1E..85W/abstract Wang et al. (2017), Nature Astronomy, 1, 85] ''High-resolution observations of flare precursors in the low solar atmosphere''
; 2016
: [https://ui.adsabs.harvard.edu/abs/2016JAI.....541009N/abstract Nita et al. (2016), J. Astron. Instr., 5, 1641009-7366] ''EOVSA Implementation of a Spectral Kurtosis Correlator for Transient Detection and Classification''

== VLA Flare List and Publications ==
See [http://www.ovsa.njit.edu/wiki/index.php/VLA_Data_Survey#List_of_Jansky_VLA_Solar_Observations this link] for a list of flare observations made by the [https://science.nrao.edu/facilities/vla/ Karl G. Jansky Very Large Array] (VLA). Below is a partial list of publications that utilize VLA solar data (see also [https://ui.adsabs.harvard.edu/public-libraries/ZwbjpLo9RS-viufWEoQ95Q this NASA/ADS Library]).
* [https://ui.adsabs.harvard.edu/abs/2021ApJ...911....4L/abstract Luo et al. (2021), ApJ, 911, 4] ''Radio Spectral Imaging of an M8.4 Eruptive Solar Flare: Possible Evidence of a Termination Shock''
* [https://ui.adsabs.harvard.edu/abs/2021ApJ...910...40Z/abstract Zhang et al. (2021), ApJ, 910, 40] ''Multiwavelength Observations of the Formation and Eruption of a Complex Filament''
* [https://ui.adsabs.harvard.edu/abs/2020ApJ...904...94S/abstract Sharma et al. (2020), ApJ, 904, 94] ''Radio and X-Ray Observations of Short-lived Episodes of Electron Acceleration in a Solar Microflare''
* [https://ui.adsabs.harvard.edu/abs/2019ApJ...884...63C/abstract Chen et al. (2019), ApJ, 884, 63] ''Radio Spectroscopic Imaging of a Solar Flare Termination Shock: Split-band Feature as Evidence for Shock Compression''
* [https://ui.adsabs.harvard.edu/abs/2019ApJ...872...71Y/abstract Yu & Chen (2019), ApJ, 872, 71] ''Possible Detection of Subsecond-period Propagating Magnetohydrodynamics Waves in Post-reconnection Magnetic Loops during a Two-ribbon Solar Flare''
* [https://ui.adsabs.harvard.edu/abs/2018ApJ...866...62C/abstract Chen et al. (2018), ApJ, 866, 62] ''Magnetic Reconnection Null Points as the Origin of Semirelativistic Electron Beams in a Solar Jet
''
* [https://ui.adsabs.harvard.edu/abs/2017ApJ...848...77W/abstract Wang et al. (2016), ApJ, 848, 77] ''Dynamic Spectral Imaging of Decimetric Fiber Bursts in an Eruptive Solar Flare''
* [https://ui.adsabs.harvard.edu/abs/2015Sci...350.1238C/abstract Chen et al. (2015), Science, 350, 1238] ''Particle acceleration by a solar flare termination shock''
* [https://ui.adsabs.harvard.edu/abs/2014ApJ...794..149C/abstract Chen et al. (2014), ApJ, 794, 149] ''Direct Evidence of an Eruptive, Filament-hosting Magnetic Flux Rope Leading to a Fast Solar Coronal Mass Ejection''
* [https://ui.adsabs.harvard.edu/abs/2013ApJ...763L..21C/abstract Chen et al. (2013), ApJL, 763, 21] ''Tracing Electron Beams in the Sun's Corona with Radio Dynamic Imaging Spectroscopy''