View source for OVRO-LWA Operation Notes

==Starting solar beamforming observations==
* Log into lwacalim10 using your own account (this is the only node that allows submissions)
* Activate the deployment conda environment
<pre> conda activate deployment </pre>
* Check what schedules are there
<pre>
lwaobserving show-schedule
</pre>
* Submit the schedule for the next 7 days (note that sdf files are written to /tmp/solar_<date>_<time>.sdf and will be owned by you).
<pre> 
ipython
cd /home/dgary
import make_solar_sdf
make_solar_sdf.multiday_obs(ndays=7)
</pre>
* Calibrate the beam (if needed, using the same Python session)
<pre>
from mnc import control
con=control.Controller('/opt/devel/dgary/lwa_config_calim_std.yaml')
con.configure_xengine(['dr2'], calibratebeams=True)
</pre>
If the beam is already calibrated, the con.configure_xengine command will say that and return immediately.  If for any reason you want to override the current calibration, instead type
<pre>
con.configure_xengine(['dr2'], calibratebeams=True, force=True)
</pre>

==Starting slow and fast visibility recorders ==
* Log into lwacalim10 using your own account
* Check the recorder status by going to http://localhost:5006/LWA_dashboard
* Activate the environment and configure
<pre>
conda activate deployment
ipython
cd /home/pipeline/proj/lwa-shell/mnc_python/
from mnc import control
con=control.Controller('/opt/devel/dgary/lwa_config_calim_std.yaml')
</pre>
* Start the recorders
<pre>
con.start_dr(['drvs', 'drvf'])
</pre>
* Check the recorder status in command line
<pre>
con.status_dr()
</pre>

==Restart slow and fast visibility recorder services (experts only!)==
Occasionally, one would see slow and/or fast images on certain bands showing "No Data" all the time. This is the time to suspect that the recorder services need to be restarted. To check this, do the following:
* Log into lwacalim10 and check the recorder status by going to http://localhost:5006/LWA_dashboard. If the recorder services are okay but not started, they show as "normal, idle." In this case, one can just start the recorders following the previous section. If recorders show up as "shutdown," then we need to restart the recorder services.
* Check if the data are being written to disk. One can run the following script for a given day (format yyyy-mm-dd)
<pre>
source /opt/devel/dgary/check_recording.sh 2024-09-27
</pre>
If all data are being recorded, it would list all the hours of the day that have data. Otherwise, something like the following would be shown
<pre>
ls: cannot access '/lustre/pipeline/slow/32MHz/2024-09-27/': No such file or directory
ls: cannot access '/lustre/pipeline/slow/69MHz/2024-09-27/': No such file or directory
ls: cannot access '/lustre/pipeline/fast/32MHz/2024-09-27/': No such file or directory
ls: cannot access '/lustre/pipeline/fast/69MHz/2024-09-27/': No such file or directory
</pre>
To determine which server node that hosts the recorders, use the following mapping:
<pre>
13 MHz, 50 MHz → lwacalim01
18 MHz, 55 MHz → lwacalim02 
23 MHz, 59 MHz → lwacalim03 
27 MHz, 64 MHz → lwacalim04
32 MHz, 69 MHz → lwacalim05 
36 MHz, 73 MHz → lwacalim06
41 MHz, 78 MHz → lwacalim07 
46 MHz, 82 MHz → lwacalim08
</pre>
In the example above, the problem lies in the slow and fast recorders on node lwacalim05. To fix them, do the following
* Log in to the respective node (lwacalim05 in this example) as the "pipeline" user (only a few of us have the privilege) 
* Restart the slow and fast services. Each node hosts two slow recorders and two fast recorders. The slow recorders are named dr-vslow-[m1] and dr-vslow-[m2], where m1=2n-1 and m2=2n, with n the node number (5 in this example). Similarly, the fast recorders are named dr-vfast-[m]. 
<pre>
systemctl --user restart dr-vslow-9
systemctl --user restart dr-vslow-10
systemctl --user restart dr-vfast-9
systemctl --user restart dr-vfast-10
</pre>
Once this is done, check http://localhost:5006/LWA_dashboard again. The recorders in question should show as "normal, idle." The last step is to start the recorders following the steps in the previous section, e.g.,
<pre>
con.start_dr(['drvs', 'drvf'])
</pre>
Don't worry if you see messages such as "'Failed to schedule recording start: Operation starts during a previously scheduled operation'" for recorders that are already working. Pay attention to those weren't working, and they should display something like "'drvs8002': {'sequence_id': '7428a3d67cee11ef80113cecef5ef4c6', 'timestamp': 1727454906.4683754, 'status': 'success', 'response': {'filename': '/lustre/pipeline/slow/'}}".
Lastly, check if the recorders are back and the data are flowing.
<pre>
con.status_dr(['drvs', 'drvf'])
</pre>

==Restart Xengine==
If for some reason the entire Xengine is not working (e.g., the OVRO-LWA system health board shows that they are all red), one can do the following to restart it. 
<pre>
lwamnc start-xengine --full
</pre>

==Check/Set ARX attenuator settings==

Under the "deployment" environment
<pre>
> cd /home/pipeline/proj/lwa-shell/mnc_python/
> from mnc import settings
> s = settings.Settings()
> last = s.get_last_settings()
# last is a dictionary that contains the last setting information. The time is in mjd
> print(last) 
{'time_loaded': 60607.64228738569, 'user': 'bin.chen', 'filename': '20240922-settingsAll-day.mat'}
> from astropy.time import Time
> Time(last['time_loaded'], format='mjd').isot
Out[25]: '2024-10-24T15:24:53.630'
</pre>

List all the settings:
<pre>
s.list_settings()
</pre>

If last['filename'] != the latest daytime file ("20240922-settingsAll-day.mat" in this case), set it to the latest.
<pre>
settings.update('/home/pipeline/opsdata/20240922-settingsAll-day.mat')
</pre>

==Check bad antennas==
Andrea Isella produces reports of the antenna health at [http://obelix.rice.edu/~ai14/OVRO_LWA/ this link]. The lists can be accessed using the method below on the lwacalim nodes.
<pre>
conda activate deployment
ipython
cd /home/pipeline/proj/lwa-shell/mnc_python/
from mnc import anthealth
badants = anthealth.get_badants('selfcorr')
</pre>

Check the time and antenna list of the latest report
<pre>
from astropy.time import Time
Time(badants[0], format='mjd').isot
print(badants[1])
</pre>

Note that the output antenna list refer to the antenna names, but not the "correlator numbers" or CASA antenna indices.