Expanded Owens Valley Solar Array

EOVSA (Expanded Owens Valley Solar Array) is a solar-dedicated radio interferometer operated by the New Jersey Institute of Technology and serving as a National Science Foundation Geospace Facility.

Operation of EOVSA is supported by the National Science Foundation under Grant No. AGS-2130832. Any opinions, findings, and conclusions or  recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the National Science  Foundation. 

This wiki serves as the site for EOVSA documentation.

OVRO-LWA (Owens Valley Radio Observatory Long Wavelength Array) is an all-sky imager that has a new solar-dedicated spectroscopic imaging mode. OVRO-LWA is a multi-institutional collaboration led by Caltech. NJIT Solar Radio Group is leading its solar-mode development and science. At the bottom of this page are new links for that facility.

EOVSA Flare List

• Query EOVSA Flare list
• List of EOVSA flares in separate years: 2024, 2023, 2022, 2021, 2020, 2019, 2017

Using EOVSA Data

• EOVSA Data products: An introduction to standard EOVSA spectrogram and spectral image products with example scripts for reading and plotting.
• EOVSA Data Policy: Policy for using EOVSA data products.
• Analysis Software: These are for in-depth use of EOVSA data (from calibrated visibilities) and tools for quantitative analysis.
  • SunCASA A wrapper around CASA (the Common Astronomy Software Applications package) for synthesis imaging and visualizing solar spectral imaging data. CASA is one of the leading software tool for "supporting the data post-processing needs of the next generation of radio astronomical telescopes such as ALMA and VLA", an international effort led by the National Radio Astronomy Observatory. The current version of CASA uses Python (2.7) interface. More information about CASA can be found on NRAO's CASA website . Note, CASA is available ONLY on UNIX-BASED PLATFORMS (and therefore, so is SunCASA).
  • GSFIT A IDL-widget(GUI)-based spectral fitting package called gsfit, which provides a user-friendly display of EOVSA image cubes and an interface to fast fitting codes (via platform-dependent shared-object libraries).
  • pyGSFIT A Python-widget(pyQT)-based spectral fitting package, which provides a user-friendly display of EOVSA image cubes, spatially resolved spectra, and an interface to scipy-based fitting codes.
  • Spectrogram Software
  • Mapping Software
• Data Analysis Guides
  • EOVSA Data Analysis Tutorial 2022 and EOVSA Workspace at SPHERE 2022 Workshop
  • EOVSA Data Analysis Tutorial at RHESSI 19 Workshop
  • EOVSA Data Analysis Tutorial at RHESSI XVIII Workshop
  • Self-Calibrating Flare Data Example script and guides for self-calibrating EOVSA flare data (to be completed)
  • IDB flare pipeline Tutorial to run the flare pipeline for quicklook images

• EOVSA Modeling Guide
  • GX Simulator

• Other helpful links
  • CASA Guides
  • SolarSoft IDL
  • Miriad Guides
  • Fast Gyrosynchrotron Codes (Alexey Kuznetsov's website)
  • Basic GitHub Tutorial

• Full Disk Simulations
• All-Day Synthesis Issues
• Analyzing Pre-2017 Data
• Fixing Pipeline Problems pre-2021-Feb-07

EOVSA Documentation

• General
  • Downconversion and Frequency Tuning
  • Dealing with Radio Frequency Interference
  • Switching between 200 MHz and 300 MHz Correlator
  • Observing in "Fast" Mode

• Computer-Network
  • Computing Systems
  • Network

• Control System
  • 27-m Antenna Commands
  • Schedule Commands
  • Control Commands
  • Attenuation and Level Control

• Hardware
  • Hardware Overview
  • 2.1-m Antennas
  • 27-m Antennas

• System Software
  • Calibration Database
  • Stateframe Database
  • Database Maintenance
  • Create CASA measurement sets

• Calibration
  • Calibration Overview
  • Pointing Calibration
  • Total Power Calibration
  • System Gain Calibration
  • Antenna Position (Baseline Calibration)
  • Reference Gain Calibration
  • Daily Gain Calibration
  • Delay Calibration
  • Bandpass Calibration
  • Polarization Calibration
  • Calibrator Survey
  • Practical Calibration Tutorial

• Starburst

System Software

• LabVIEW software
• Python code Github repository
• Python3 Code Installation

EOVSA Observing Log

2016 November; December

2017 January; February; March; April; May; June; July; August; September; October; November; December

2018 January; February; March; April; May; June; July; August; September; October; November; December

2019 January; February; March; April; May; June; July; August; September; October; November; December

2020 January; February; March; April; May; June; July; August; September; October; November; December

2021 January; February; March; April; May; June; July; August; September; October; November; December

2022 SQL Outage

2023 January; February; March; April; May; June; July; August; September; October; November; December

2024 January; February; March; April;May; June; July; August; September

SoD Observing Logs

• See SoD Routines for detailed instructions for Scientist-on-Duty routines.
• 2024 May (and before that), June, July, August, September, October, November, December

Tohbans

Trouble Shooting Guide

Tohban Records

Owen's Notes

Caius' Notes

Tohban EOVSA Imaging Tutorial A-Z

Tohban OVRO-LWA Imaging Tutorial

Tohban Guide to Self Calibration and Imaging for EOVSA

Guide to Upgrade SolarSoft(SSW)

EOVSA Publications

Here is a (partial) list of publications that utilize EOVSA data. See also the collection of EOVSA publications at this NASA/ADS Library.

2024

Collier, H., Hayes, L. A., Yu, S., Battaglia, A. F., Ashfield, W., Polito, V., Harra, L. K., & Krucker, S. (2024), arXiv e-prints, arXiv:2402.10546. “Localising pulsations in the hard X-ray and microwave emission of an X-class flare”

Saqri, J., Veronig, A. M., Battaglia, A. F., Dickson, E. C. M., Gary, D. E., & Krucker, S. (2024), Astronomy and Astrophysics, 683, A41. "Efficiency of solar microflares in accelerating electrons when rooted in a sunspot"

2023

Tan, B., Yan, Y., Huang, J., Zhang, Y., Tan, C., & Zhu, X. (2023), Advances in Space Research, 72, 5563. "The physics of solar spectral imaging observations in dm-cm wavelengths and the application on space weather"

Li, D., Li, Z., Shi, F., Su, Y., Chen, W., Yu, F., Li, C., Qiu, Y., Huang, Y., & Ning, Z. (2023), Astronomy and Astrophysics, 680, L15. "Observational signature of continuously operating drivers of decayless kink oscillation"

Wang, M., Chen, B., Yu, S., Gary, D. E., Lee, J., Wang, H., & Cohen, C. (2023), The Astrophysical Journal, 954, 32. "The Solar Origin of an In Situ Type III Radio Burst Event"

Gary, D. E. (2023), Annual Review of Astronomy and Astrophysics, 61, 427. "New Insights from Imaging Spectroscopy of Solar Radio Emission"

Nita, G. M., Fleishman, G. D., Kuznetsov, A. A., Anfinogentov, S. A., Stupishin, A. G., Kontar, E. P., Schonfeld, S. J., Klimchuk, J. A., & Gary, D. E. (2023), The Astrophysical Journal Supplement Series, 267, 6. "Data-constrained Solar Modeling with GX Simulator"

Song, D.-C., Tian, J., Li, Y., Ding, M. D., Su, Y., Yu, S., Hong, J., Qiu, Y., Rao, S., Liu, X., Li, Q., Chen, X., Li, C., & Fang, C. (2023), The Astrophysical Journal, 952, L6. "Spectral Observations and Modeling of a Solar White-light Flare Observed by CHASE"

Mondal, S., Chen, B., & Yu, S. (2023), The Astrophysical Journal, 949, 56. "Multifrequency Microwave Imaging of Weak Transients from the Quiet Solar Corona"

Kontar, E. P., Emslie, A. G., Motorina, G. G., & Dennis, B. R. (2023), The Astrophysical Journal, 947, L13. "The Efficiency of Electron Acceleration during the Impulsive Phase of a Solar Flare"

Saqri, J., Veronig, A. M., Dickson, E. C. M., Podladchikova, T., Warmuth, A., Xiao, H., Gary, D. E., Battaglia, A. F., & Krucker, S. (2023), Astronomy and Astrophysics, 672, A23. "Multi-point study of the energy release and impulsive CME dynamics in an eruptive C7 flare"

2022

Kou, Y., Cheng, X., Wang, Y., Yu, S., Chen, B., Kontar, E. P., & Ding, M. (2022), Nature Communications, 13, 7680. "Microwave imaging of quasi-periodic pulsations at flare current sheet"

Chertok, I. M. (2022), Monthly Notices of the Royal Astronomical Society, 517, 2709. "On some features of the solar proton event on 2021 October 28 - GLE73"

Lörinčík, J., Polito, V., De Pontieu, B., Yu, S., & Freij, N. (2022), Frontiers in Astronomy and Space Sciences, 9, 334. "Rapid variations of Si IV spectra in a flare observed by interface region imaging spectrograph at a sub-second cadence"

Klein, K.-L., Musset, S., Vilmer, N., Briand, C., Krucker, S., Francesco Battaglia, A., Dresing, N., Palmroos, C., & Gary, D. E. (2022), Astronomy and Astrophysics, 663, A173. "The relativistic solar particle event on 28 October 2021: Evidence of particle acceleration within and escape from the solar corona"

Fleishman, G. D., Nita, G. M., Chen, B., Yu, S., & Gary, D. E. (2022), Nature, 606, 674. "Solar flare accelerates nearly all electrons in a large coronal volume"

Li, X., Guo, F., Chen, B., Shen, C., & Glesener, L. (2022), The Astrophysical Journal, 932, 92. "Modeling Electron Acceleration and Transport in the Early Impulsive Phase of the 2017 September 10th Solar Flare"

Zhang, J., Chen, B., Yu, S., Tian, H., Wei, Y., Chen, H., Tan, G., Luo, Y., & Chen, X. (2022), The Astrophysical Journal, 932, 53. "Implications for Additional Plasma Heating Driving the Extreme-ultraviolet Late Phase of a Solar Flare with Microwave Imaging Spectroscopy"

Liu, N., Jing, J., Xu, Y., & Wang, H. (2022), The Astrophysical Journal, 930, 154. "Multi-instrument Comparative Study of Temperature, Number Density, and Emission Measure during the Precursor Phase of a Solar Flare"

López, F. M., Giménez de Castro, C. G., Mandrini, C. H., Simões, P. J. A., Cristiani, G. D., Gary, D. E., Francile, C., & Démoulin, P. (2022), Astronomy and Astrophysics, 657, A51. "A solar flare driven by thermal conduction observed in mid-infrared"

Unverferth, J., & Longcope, D. (2021), The Astrophysical Journal, 923, 248. "Examining Flux Tube Interactions as a Cause of Sub-alfvénic Outflow"

2021

Wei, Y., Chen, B., Yu, S., Wang, H., Jing, J., & Gary, D. E. (2021), The Astrophysical Journal, 923, 213. "Coronal Magnetic Field Measurements along a Partially Erupting Filament in a Solar Flare"

Jing, J., Inoue, S., Lee, J., Li, Q., Nita, G. M., Xu, Y., Liu, C., Gary, D. E., & Wang, H. (2021), The Astrophysical Journal, 922, 108. "Understanding the Initiation of the M2.4 Flare on 2017 July 14"

Battaglia, A. F., Saqri, J., Massa, P., Perracchione, E., Dickson, E. C. M., Xiao, H., Veronig, A. M., Warmuth, A., Battaglia, M., Hurford, G. J., Meuris, A., Limousin, O., Etesi, L., Maloney, S. A., Schwartz, R. A., Kuhar, M., Schuller, F., Senthamizh Pavai, V., Musset, S., Ryan, D. F., Kleint, L., Piana, M., Massone, A. M., Benvenuto, F., Sylwester, J., Litwicka, M., Stȩślicki, M., Mrozek, T., Vilmer, N., Fárník, F., Kašparová, J., Mann, G., Gallagher, P. T., Dennis, B. R., Csillaghy, A., Benz, A. O., & Krucker, S. (2021), Astronomy and Astrophysics, 656, A4. "STIX X-ray microflare observations during the Solar Orbiter commissioning phase"

Shaik, S. B., & Gary, D. E. (2021), The Astrophysical Journal, 919, 44. "Implications of Flat Optically Thick Microwave Spectra in Solar Flares for Source Size and Morphology"

Kocharov, L., Omodei, N., Mishev, A., Pesce-Rollins, M., Longo, F., Yu, S., Gary, D. E., Vainio, R., & Usoskin, I. (2021), The Astrophysical Journal, 915, 12. "Multiple Sources of Solar High-energy Protons"

Chen, B., Battaglia, M., Krucker, S., Reeves, K. K., & Glesener, L. (2021), The Astrophysical Journal, 908, L55. "Energetic Electron Distribution of the Coronal Acceleration Region: First Results from Joint Microwave and Hard X-Ray Imaging Spectroscopy"

Chhabra, S., Gary, D. E., Hallinan, G., Anderson, M. M., Chen, B., Greenhill, L. J., & Price, D. C. (2021), The Astrophysical Journal, 906, 132. "Imaging Spectroscopy of CME-associated Solar Radio Bursts using OVRO-LWA"

2020 and earlier

Reeves, K. K., Polito, V., Chen, B., Galan, G., Yu, S., Liu, W., & Li, G. (2020), The Astrophysical Journal, 905, 165. "Hot Plasma Flows and Oscillations in the Loop-top Region During the 2017 September 10 X8.2 Solar Flare"

Nindos, A. (2020), Frontiers in Astronomy and Space Sciences, 7, 57. "Incoherent Solar Radio Emission"

Yu, S., Chen, B., Reeves, K. K., Gary, D. E., Musset, S., Fleishman, G. D., Nita, G. M., & Glesener, L. (2020), The Astrophysical Journal, 900, 17. "Magnetic Reconnection during the Post-impulsive Phase of a Long-duration Solar Flare: Bidirectional Outflows as a Cause of Microwave and X-Ray Bursts"

Chen, B., Yu, S., Reeves, K. K., & Gary, D. E. (2020), The Astrophysical Journal, 895, L50. "Microwave Spectral Imaging of an Erupting Magnetic Flux Rope: Implications for the Standard Solar Flare Model in Three Dimensions"

Kuroda, N., Fleishman, G. D., Gary, D. E., Nita, G. M., Chen, B., & Yu, S. (2020), Frontiers in Astronomy and Space Sciences, 7, 22. "Evolution of Flare-accelerated Electrons Quantified by Spatially Resolved Analysis"

Glesener, L., Krucker, S., Duncan, J., Hannah, I. G., Grefenstette, B. W., Chen, B., Smith, D. M., White, S. M., & Hudson, H. (2020), The Astrophysical Journal, 891, L34. "Accelerated Electrons Observed Down to <7 keV in a NuSTAR Solar Microflare"

Karlický, M., Chen, B., Gary, D. E., Kašparová, J., & Rybák, J. (2020), The Astrophysical Journal, 889, 72. "Drifting Pulsation Structure at the Very Beginning of the 2017 September 10 Limb Flare"

Fleishman, G. D., Gary, D. E., Chen, B., Kuroda, N., Yu, S., & Nita, G. M. (2020), Science, 367, 278. "Decay of the coronal magnetic field can release sufficient energy to power a solar flare"

Chen, B., Shen, C., Gary, D. E., Reeves, K. K., Fleishman, G. D., Yu, S., Guo, F., Krucker, S., Lin, J., Nita, G. M., & Kong, X. (2020), Nature Astronomy, 4, 1140. "Measurement of magnetic field and relativistic electrons along a solar flare current sheet"

Lee, J. (2018), Journal of Astronomy and Space Sciences, 35, 211. "Analysis of Solar Microwave Burst Spectrum, I. Nonuniform Magnetic Field"

Gary, D. E., Bastian, T. S., Chen, B., Fleishman, G. D., & Glesener, L. (2018), Science with a Next Generation Very Large Array, 517, 99. "Radio Observations of Solar Flares"

Polito, V., Dudík, J., Kašparová, J., Dzifčáková, E., Reeves, K. K., Testa, P., & Chen, B. (2018), The Astrophysical Journal, 864, 63. "Broad Non-Gaussian Fe XXIV Line Profiles in the Impulsive Phase of the 2017 September 10 X8.3-class Flare Observed by Hinode/EIS"

Gary, D. E., Chen, B., Dennis, B. R., Fleishman, G. D., Hurford, G. J., Krucker, S., McTiernan, J. M., Nita, G. M., Shih, A. Y., White, S. M., & Yu, S. (2018), The Astrophysical Journal, 863, 83. "Microwave and Hard X-Ray Observations of the 2017 September 10 Solar Limb Flare"

Fleishman, G. D., Nita, G. M., Kuroda, N., Jia, S., Tong, K., Wen, R. R., & Zhizhuo, Z. (2018), The Astrophysical Journal, 859, 17. "Revealing the Evolution of Non-thermal Electrons in Solar Flares Using 3D Modeling"

Kuroda, N., Gary, D. E., Wang, H., Fleishman, G. D., Nita, G. M., & Jing, J. (2018), The Astrophysical Journal, 852, 32. "Three-dimensional Forward-fit Modeling of the Hard X-Ray and Microwave Emissions of the 2015 June 22 M6.5 Flare"

Wang, H., Liu, C., Ahn, K., Xu, Y., Jing, J., Deng, N., Huang, N., Liu, R., Kusano, K., Fleishman, G. D., Gary, D. E., & Cao, W. (2017), Nature Astronomy, 1, 0085. "High-resolution observations of flare precursors in the low solar atmosphere"

Nita, G. M., Hickish, J., MacMahon, D., & Gary, D. E. (2016), Journal of Astronomical Instrumentation, 5, 1641009-7366. "EOVSA Implementation of a Spectral Kurtosis Correlator for Transient Detection and Classification"

Nita, G. M., & Gary, D. E. (2016), Journal of Geophysical Research (Space Physics), 121, 7353. "Measurement of duration and signal-to-noise ratio of astronomical transients using a Spectral Kurtosis spectrometer"

Wang, Z., Gary, D. E., Fleishman, G. D., & White, S. M. (2015), The Astrophysical Journal, 805, 93. "Coronal Magnetography of a Simulated Solar Active Region from Microwave Imaging Spectropolarimetry"

Gary, D. E., Fleishman, G. D., & Nita, G. M. (2013), Solar Physics, 288, 549. "Magnetography of Solar Flaring Loops with Microwave Imaging Spectropolarimetry"

VLA Flare List and Publications

See this link for a list of flare observations made by the Karl G. Jansky Very Large Array (VLA). Below is a partial list of publications that utilize VLA solar data (see also this NASA/ADS Library).

• Luo et al. (2022), ApJ, 940, 137 Multiple Regions of Nonthermal Quasiperiodic Pulsations during the Impulsive Phase of a Solar Flare
• Battaglia et al. (2021), ApJ, 922, 134 Multiple Electron Acceleration Instances during a Series of Solar Microflares Observed Simultaneously at X-Rays and Microwaves
• Luo et al. (2021), ApJ, 911, 4 Radio Spectral Imaging of an M8.4 Eruptive Solar Flare: Possible Evidence of a Termination Shock
• Zhang et al. (2021), ApJ, 910, 40 Multiwavelength Observations of the Formation and Eruption of a Complex Filament
• Sharma et al. (2020), ApJ, 904, 94 Radio and X-Ray Observations of Short-lived Episodes of Electron Acceleration in a Solar Microflare
• Chen et al. (2019), ApJ, 884, 63 Radio Spectroscopic Imaging of a Solar Flare Termination Shock: Split-band Feature as Evidence for Shock Compression
• Yu & Chen (2019), ApJ, 872, 71 Possible Detection of Subsecond-period Propagating Magnetohydrodynamics Waves in Post-reconnection Magnetic Loops during a Two-ribbon Solar Flare
• Chen et al. (2018), ApJ, 866, 62 Magnetic Reconnection Null Points as the Origin of Semirelativistic Electron Beams in a Solar Jet

• Wang et al. (2016), ApJ, 848, 77 Dynamic Spectral Imaging of Decimetric Fiber Bursts in an Eruptive Solar Flare
• Chen et al. (2015), Science, 350, 1238 Particle acceleration by a solar flare termination shock
• Chen et al. (2014), ApJ, 794, 149 Direct Evidence of an Eruptive, Filament-hosting Magnetic Flux Rope Leading to a Fast Solar Coronal Mass Ejection
• Chen et al. (2013), ApJL, 763, 21 Tracing Electron Beams in the Sun's Corona with Radio Dynamic Imaging Spectroscopy

Radio Data from Around The Heliosphere

• Radio Data

OVRO-LWA Solar-Dedicated Spectroscopic Imager

The OVRO-LWA (Owens Valley Radio Observatory Long Wavelength Array) has recently been upgraded to include a solar-dedicated beam and two solar imaging modes (slow visibilities of 352 antennas with a 10-s cadence, and fast visibilities of 48 antennas with a 0.1-s cadence). The large collecting area and excellent calibration provide unprecedented high-sensitivity imaging of the quiet Sun and bursts. The array is currently in commissioning and observations are not yet continuous, but they are becoming more so. See the daily realtime data at http://ovsa.njit.edu/status.php for real-time display of the spectrogram and a selection of images, both updated on a 1-min cadence.

Solar-Dedicated Modes

Beamformer

The beamformer uses the 256 core antennas to form a synthesized beam of more than 1 degree in size that tracks the Sun from sunrise to sunset. This permits a continuous record of the full-Stokes total flux (without spatial resolution) of the Sun (a dynamic spectrum) with 24 kHz frequency resolution (3072 frequencies from 15-90 MHz) and as low as 1 ms time resolution.

Slow Visibility Imaging

In this mode, the entire 352-element array is interferometrically correlated to provide visibilities for imaging at all 3072 frequencies at 10-s time resolution. This is ideal for imaging quiet Sun and slowly-varying emission such as coronal mass ejections and active region variability.

Fast Visibility Imaging

In this mode, a subset of 48 antennas (chosen to include mainly outer antennas to maintain good spatial resolution) is interferometrically correlated to provide visibilities for imaging at 768 frequencies (96 kHz frequency resolution) at 0.1-s time resolution. This is ideal for imaging rapidly varying emission such as type II and type III bursts as well as many other solar spectral fine structures.

Inital Data Access

In its current commissioning state, we try to run the beamformer and imaging pipeline every day in real-time since November 2023 (no latency for beamforming spectrograms and 5-10 min latency for images). Quicklook real-time spectrograms/images can be accessed from http://ovsa.njit.edu/status.php. To access data from previous days, use the following links (replace yyyymmdd with the date you desire):

• Quicklook beamformer total-power spectrograms: http://ovsa.njit.edu/lwa-data/1min_spectra/yyyymmdd/. Check this link for additional daily plots Daily OVRO-LWA Beamformer Data.
• Quicklook multi-frequency movies at 1-min cadence: http://ovsa.njit.edu/lwa-data/1min_images/yyyymmdd/movie_yyyy-mm-dd.html

Note our pipeline processing development is still in the early phase. For example, absolute flux calibrations have not been done for the beamformer spectrograms. Also, artificial effects (including ionospheric refraction effects) are present in the images that cause distortions/shifts. We caution interested users only to consider them for quick-look purposes at this point. Please contact the EOVSA PIs (Dale Gary, Bin Chen) if you intend to use them for science.

OVRO-LWA Operation Notes

Starting solar beamforming observations

• Log into lwacalim10 using your own account (this is the only node that allows submissions)
• Activate the deployment conda environment

 conda activate deployment 

• Check what schedules are there

lwaobserving show-schedule

• Submit the schedule for the next 7 days (note that sdf files are written to /tmp/solar_<date>_.sdf and will be owned by you).

 
ipython
cd /home/dgary
import make_solar_sdf
make_solar_sdf.multiday_obs(ndays=7)

• Calibrate the beam (if needed, using the same Python session)

from mnc import control
con=control.Controller('/opt/devel/dgary/lwa_config_calim_std.yaml')
con.configure_xengine(['dr2'], calibratebeams=True)

If the beam is already calibrated, the con.configure_xengine command will say that and return immediately. If for any reason you want to override the current calibration, instead type

con.configure_xengine(['dr2'], calibratebeams=True, force=True)

Starting slow and fast visibility recorders

• Log into lwacalim10 using your own account
• Check the recorder status by going to http://localhost:5006/LWA_dashboard
• Activate the environment and configure

conda activate deployment
ipython
cd /home/pipeline/proj/lwa-shell/mnc_python/
from mnc import control
con=control.Controller('/opt/devel/dgary/lwa_config_calim_std.yaml')

• Start the recorders

con.start_dr(['drvs', 'drvf'])

• Check the recorder status in command line

con.status_dr()

Restart slow and fast visibility recorder services (experts only!)

Occasionally, one would see slow and/or fast images on certain bands showing "No Data" all the time. This is the time to suspect that the recorder services need to be restarted. To check this, do the following:

• Log into lwacalim10 and check the recorder status by going to http://localhost:5006/LWA_dashboard. If the recorder services are okay but not started, they show as "normal, idle." In this case, one can just start the recorders following the previous section. If recorders show up as "shutdown," then we need to restart the recorder services.
• Check if the data are being written to disk. One can run the following script for a given day (format yyyy-mm-dd)

source /opt/devel/dgary/check_recording.sh 2024-09-27

If all data are being recorded, it would list all the hours of the day that have data. Otherwise, something like the following would be shown

ls: cannot access '/lustre/pipeline/slow/32MHz/2024-09-27/': No such file or directory
ls: cannot access '/lustre/pipeline/slow/69MHz/2024-09-27/': No such file or directory
ls: cannot access '/lustre/pipeline/fast/32MHz/2024-09-27/': No such file or directory
ls: cannot access '/lustre/pipeline/fast/69MHz/2024-09-27/': No such file or directory

To determine which server node that hosts the recorders, use the following mapping:

13 MHz, 50 MHz → lwacalim01
18 MHz, 55 MHz → lwacalim02 
23 MHz, 59 MHz → lwacalim03 
27 MHz, 64 MHz → lwacalim04
32 MHz, 69 MHz → lwacalim05 
36 MHz, 73 MHz → lwacalim06
41 MHz, 78 MHz → lwacalim07 
46 MHz, 82 MHz → lwacalim08

In the example above, the problem lies in the slow and fast recorders on node lwacalim05. To fix them, do the following

• Log in to the respective node (lwacalim05 in this example) as the "pipeline" user (only a few of us have the privilege)
• Restart the slow and fast services. Each node hosts two slow recorders and two fast recorders. The slow recorders are named dr-vslow-[m1] and dr-vslow-[m2], where m1=2n-1 and m2=2n, with n the node number (5 in this example). Similarly, the fast recorders are named dr-vfast-[m].

systemctl --user restart dr-vslow-9
systemctl --user restart dr-vslow-10
systemctl --user restart dr-vfast-9
systemctl --user restart dr-vfast-10

Once this is done, check http://localhost:5006/LWA_dashboard again. The recorders in question should show as "normal, idle." The last step is to start the recorders following the steps in the previous section, e.g.,

con.start_dr(['drvs', 'drvf'])

Don't worry if you see messages such as "'Failed to schedule recording start: Operation starts during a previously scheduled operation'" for recorders that are already working. Pay attention to those weren't working, and they should display something like "'drvs8002': {'sequence_id': '7428a3d67cee11ef80113cecef5ef4c6', 'timestamp': 1727454906.4683754, 'status': 'success', 'response': {'filename': '/lustre/pipeline/slow/'}}". Lastly, check if the recorders are back and the data are flowing.

con.status_dr(['drvs', 'drvf'])