2017 March

Mar 02

03:05 UT The 27-m is back in service, so I am grabbing a quick 3C84 observation to check, and possibly adjust, the delays. I will plan to do a two-source observation (3C273 and 3C286) overnight, using 3bands.fsq.

03:50 UT I got good delays and set them, so now I will take a few minutes of 3C84 data as a further test with them applied. The HA on the big dish is now 49.5 degrees, and the HA limit is set to 50 degrees so that the cable alarm does not get set again by going too far. So I should see the dish hit the soft limit in a few minutes. Yep, right on schedule.

04:06 UT I have verified the delays are good (except ant 4 is still showing a lot of noise, so no delay solution). I have now set up my two-source observation to start at 07:00 UT.

10:22 UT So far, the observations have gone flawlessly, so this should be good data to get a baseline update for Bx, By and Bz. I will also be able to check pointing and feed unrotating.

Mar 05

13:30 UT Huge news! The 300 MHz correlator is working. The only known glitch is an off-by-one-slot error for the correlated data, which Jack is working on. Meanwhile, I took data all night on 3C273 and 3C286 using the new/complete science-channel definitions. The data recorded okay, and the science channels are good. However, we had no phase coherence. I discovered that the clock rate was set to 0.8 (GHz) in DPPparameters.f90. I have changed to 1.2, and am taking new data on 3C286 (the stronger 3C273 has set...).

Mar 06

15:00 UT I am back to the 200 MHz correlator, and will take some calibration (two-src) observations on 2136+006 and 2253+161, using 3bands.fsq. The scan started at 15:00 UT, but in fact with the new HA limits the source is not reachable yet. Should be reachable by the next file, at 15:10 UT.

15:30 UT Some sort of strange glitch occurred on Ant 14, and it stopped tracking without a visible error indication. I had to reboot it to recover. It is tracking again now.

18:25 UT I got a nice delay solution on source 2136+006, using delay_widget.py, and updated it. I am now taking some additional data on that source to check the delays. I am reasonably confident of the main (X) delays, but the Y-X delays may have the wrong sign.

19:07 UT I verified that delay_widget used the wrong sign of Y-X delays, so I fixed that and did one more test. The results are great! Just because I am paranoid, I am doing one last test after a very minor update in the delays. Then I will move on to test Jack's latest compile of the 300 MHz correlator.

Mar 07

01:32 UT The 300 MHz correlator appears to be fully working! I have been working with dppxmp all afternoon to solve the phase wind issues, and I believe I have finally solved it. With an 800 MHz clock, our 600-1200 MHz IF was in the third nyquist zone, so the order of channels was forward (low subchannels -> low frequency, etc.). Now with our 1200 MHz clock, the IF is in the second nyquist zone, so the order is reversed. I understood that, and adjusted chan_util_bc.py to write the channel assignments in the inverted order. However, that still did not solve the phase wind. Thinking further, I finally realized that the sign of the phase correction should also reverse. I applied that sign reversal, and now some 3C84 data I just took may be right. Oddly, though, it is only good on baseline 8-14, which is a problem I noted last month, which was only corrected by rebooting the ROACHes. I will do that now...

02:20 UT The phases are great now! We definitely have a working correlator. I adjusted the delays using delay_widget.py, but a couple of baselines required huge delays (of order 10 ns), so I am not sure of the sign. Corrections of +10 and -10 were about equally viable. I'll see what the next scan shows...

05:39 UT In looking at the new 3C84 data after application of the delays, I realized that the delays measured with the new correlator need to be applied with the opposite sign also. I updated delay_widget.py accordingly, and updated the delays with the correct sign, but I have not yet verified the new delays. I have set up to observe 3C273 and 3C286 all night, starting at 06:10 UT.

Mar 11

14:18 UT We are continuing with the 300 MHz correlator, but after running for some hours it tends to start dropping packets and requires a reboot. That creates new delay settings that will affect comparison with calibration data taken at other boot cycles. There are also potential problems with the phase correction (fringe stopping) that have not been fully understood, so data with the new correlator may not be scientifically useful. Still, I believe we need to continue with it to learn more about the failure modes and get them solved. I took 1 h of data on 3C84 last night, but had to reboot both just before and just after, so I am not sure how valuable it will be.

Mar 12

01:23 UT Very strange. The correlator ran perfectly all day. At about 01:05 UT I checked, and it was still good, on the last scan of the day (hour long 3C84 run). After those observations ended, I rechecked and found that the correlator is now dropping packets! How does it know anything about ending observations? It is very puzzling what could affect it in any way. I have rebooted it once again, so it will be interesting to see if it runs all night.

Mar 13

Shaheda reported that IDB files stopped at 14:57:32. Checked packages from dpp and the numbers are 154000-155000. Restarted roaches at 16:30. Still no IDB files. dppxmp is running. Captured some packages and the data are all zero.