2016 December

Back to Observing Log

Dec 01

12:52 UT For the past several days we have done normal solar observing, with some interruptions due to testing new fast-correlator designs. The latter is showing improvement, but still some issues to resolve. During the solar observations, there have been a few small events. One M1-class flare occurred, but had almost no radio emission apparently. Activity seems to be subsiding now.

Dec 02

15:00 UT We made an attempt last night to do pointing measurements on Ant 14, but there were several problems. The tunable LO was not switching, but mainly Ant 14 was not in the subarray! After this was fixed, the wind was high, so observations were terminated early. None of the observations will be useful. We will proceed with normal solar observing.

Dec 03

12:04 UT Another attempt is underway to do pointing measurements. Early in the observations, it was discovered that the GPS clock had lost connection, which caused many subsystems to get confused. By about 01:30 UT the system had been brought back into service, and data-taking had started. Ant 12 is missing, because of a possibly spurious Lo Hard Limit in Azimuth (the current Azimuth is reading high, not low...). Also, overnight there have been many episodes of high wind, so many of the scans will be useless. Right now I count 9 potentially good scans. I also found the tunable LO saying Queue Overrun, so tuning on band14 may not have been correct for awhile--not sure when this occurred.

Dec 04

12:22 UT The wind was bad all day yesterday without let-up, hence there were virtually no good scans the entire day--maybe about 9 total. I restarted the pointing schedule last night, for a third night of attempts to get good results, and the wind was good, but this morning I found three anomalies: Ant 14 Dec drive was stuck with a permit, Ant 14 VATTN was set at 5 dB (maybe not a killer), and Ant 14 DCMATTN was 0 (definitely a killer). Also DCMAUTO was not off, which is very strange. I have no idea how these settings get corrupted. It may be that none of these data up to now are any good. The scan starting 12:39 UT may be the first with any chance.

14:30 UT The Ant 14 Dec drive spiked again, so it only successfully did one source. I have reset it, but there appears to be some generic problem with the dish that results in Dec motor-current spikes.

17:30 UT I checked again, and again the Ant 14 drive is stopped. I reset it, but I do not know if we are getting any good data. Perhaps I have to babysit it on each source change.

21:51 UT For what it is worth, the observations from 17:30 UT to now should be good, with 4 more pointings to go.

Dec 05

23:30 UT No observations at all today, because of testing of the fast correlator. The packet headers look better now, so we are getting close, but there are some issues with both number of X packets and the content of the packets, suggesting an issue with the FFT block.

Dec 06

17:20 UT Normal solar observing has started. Kjell tells me that the air conditioning stopped again, but he is cooling the room manually.

18:34 UT It seems that none of the pointing observations we attempted over the weekend are any good, even though at least some were certainly supposed to be tracking, with correct settings. As a test, I am grabbing a quick observation of 3C273--on source by 18:36 UT. Okay, I just checked, and the data on 3C273 is just fine. I cannot imagine what went wrong over the weekend. I guess we have to make another attempt to do pointing, overnight.

18:50 UT Back to solar observing.

23:41 UT Start of 24-hour pointing measurements. Ant12 "to stow". Wind 10 mph.

Dec 07

00:00 UT Continue 4-hour pointing measurements.

00:04 UT Strong wind 21 mph.

00:44 UT Front end temperatures for Ant3 and Ant14 are red.

15:12 UT The pointing measurements are continuing, and despite some losses due to wind, and again a permit error on Ant 14 Dec drive, at least now we are getting good results.

15:25 UT I updated the delay center values according to Bin's measurements on a calibrator, and the scan being taken starting now (at 15:25 UT) should have the new delays, so I can compare before and after to see if I got the sign right!

16:04 UT I checked the delay correction, and found that things got worse, but not just in the sense of getting the wrong sign. In fact, Bin has analyzed some older data taken before I made some delay changes, so his corrections are invalid. I might be able to follow the changes since then and figure out the differences, but there were at least two updates to delays in the meantime, so it sounds prone to error. I just took out the bad corrections, so we are back to the previous table (although this will not take effect until the scan starting 16:23 UT).

Dec 09

11:26 UT Natsuha is running a series of long calibrator scans (70-min on each source, using pcal.fsq, which is 10 frequencies on the Hi receiver), starting at 00:00 UT today. This morning I checked and found Ant 14 with a permit error on the Dec drive, but it just happened on the current source (1130-148). I went ahead and reset it, so data over 11:25-11:40 UT will be okay (10:30-11:25 UT is no good). There are three more scans to go.

Dec 10

00:55 UT Last night's observations went well, and Bin used the first scan to determine delay centers. I took a 10-min scan of 2136+006 just now, with the same delays as last night, and then updated the delays and will compare the next 10-min scan to make sure the sign is correct. Maria also analyzed the latest pointing measurements and I successfully used them to calculate pointing coefficients for Ant 14. I have not entered them yet, but will do so shortly.

01:10 UT Started a pointing measurement without updating Ant 14 pointing coefficients. When done, I will repeat after updating.

01:38 UT First pointing measurement is complete, and Ant 14 pointing coefficients are updated. A new pointing measurement is now underway.

02:30 UT I am repeating my experiment with toggling the PA tracking on and off on 3C84. The first scan is largely missed due to the large slew time for Ant14, but after this they should be okay, starting 02:30 UT. This is band14.

14:31 UT The observations last night went well, but on analyzing it I find that the pointing is not good. However, one way it is not good is in very different pointing when the PA of the feed is rotated 180 degrees, which is something James was asking about. I now realize that this should be balanced first, and then the pointing should be determined. I wish I had thought of that earlier!

16:50 UT Started an observation on 3C273 (band 14), with a new command to sweep the PA between -90 and 90 degrees, at a rate of 1 degree every 12 s. Unfortunately, the sweep stopped at PA = 38, at 16:56:40 UT -- some sort of glitch with reading the stateframe. I then attempted another sweep at a rate of 1 degree every 3 s, but it started doing it again at a rate of 12 s, so the command has a bug.

17:30 UT Meanwhile, I discovered that there was a C-class flare, so I went (much too late) back to the Sun.

Dec 11

04:33 UT I analyzed the earlier 3C273 data, and clearly there is a large variation in amplitude as the PA changes. I plan to get back onto 3C273 when it rises at 11:20 UT, which will give me 4 hours to mess with it before the solar observations need to begin. I want to try a faster rotation like 1 degree every 3 s. If that works, I should be able to get a measurement in 10 min, and iterate quite a few times with different X positions.

11:30 UT I had a little trouble with Ant 14 (Maths Error), but I set up for rotating the PA from -90 to 90 at the start of the file at 11:34:58 UT. Hmm, I forgot that it would take awhile to get to -90, so this is not a complete test. I have to move the PA to the start before starting the count-down, which I could do by issuing an FRM-SET-PA command first. The X-axis position is 500.

12:15 UT That worked well, actually, and I can see that we are way off in the rotation axis. I am redoing the measurement with an X-axis position of 510. I worked out that a 10 mm shift in focus position is equivalent to a pointing shift of 3'. The modulation should get worse if I went the wrong direction, or better (less modulation) is I went the right direction. This will be the file starting at 12:14:58 UT.

12:32 UT Well, that is a bit embarrassing. The modulation at X-axis 510 is almost gone! That means James had the right position after all, and I moved it in order to peak up at some point, thereby putting it out of the center! I am doing another test at 520 mm, just to verify. This will be the file starting at 12:34:58 UT.

12:52 UT That run had exactly the expected behavior, so now the best pointing is on the other side (negative PA), but not nearly as much modulation as for 500. So I have set it to 512 mm, and will do one more run, where I expect it to be exactly right! This will be the file starting at 12:54:58 UT.

13:15 UT I analyzed the result, and there seems to be a clear (although slight) modulation at 512 mm, so I will do another run at 511 mm, starting at 13:14:58 UT. I am aware that some off-pointing of the entire dish over time could contribute to this, although the variation would have to be significant over 10 min, which seems unlikely.

13:45 UT There was some strange glitch in the middle of the above scan, so I am repeating it, starting at 13:44:58 UT. There was a sudden loss of coherence about mid-way through the scan, which seems to have recovered at the very end. It was on all baselines with Ant 14, so was either a problem with Ant 14 or something strange with reading the packets by the dpp.

14:10 UT That time the 511 mm measurement worked, but still shows modulation in the sense that it wants to be a lower X-axis value. It may even be worse that 512 mm. I wonder if there is hysteresis. Anyway, I just started another run, this time on 510 mm, after running it to 490 and then back to 510, to see if a larger move and approach from below somehow changes things... The file starts at 14:10:45 UT.

14:35 UT This is rather frustrating. The 510 mm measurement also shows a need for a smaller setting. I decided to re-zero the FRM by doing an FRM-HOME command, then select the Hi-freq receiver by RX-SELECT HI, and finally refocused by FRM-ABS-Z 0. I am now doing another measurement at the nominal position (510 mm). The file starts at 14:34:44 UT.

15:00 UT This is my last attempt. I set it to 504 mm, and the scan started at 14:59:39 UT. The previous one at 14:34 UT did not change from the 510 measurement at 14:10 UT, despite the homing of the FRM.

Dec 12

Fig. 1: Result of the Dec. 12 observations, suggesting best rotation center is near X=487 mm

14:54 UT I have created a scheme to automate the checking of the rotation axis, which will be much more efficient than my manual attempts yesterday. I have a list of atomic commands in PACAL2.ctl that will move the X-axis to a given location, then start a sweep from -80 to 80 degrees. The reduced range is so that the rotation mechanism can acquire the initial PA and do the sweep at a rate of 3-s/degree, all in less than 10 minutes. I changed PA-SWEEP to wait for the FRM to reach the initial PA before starting the sweep. I then created a schedule called 3c273-pa.scd that will do a series of 5 sweeps, at X = 490, 500, 510, 520 and 530 mm. The first scan started at 14:52:16 UT. It does seem to be working.

15:03 UT Watching the change to a new scan, I saw that the X-axis command got stuck somehow, so although the BRICK got the command, it was sitting at PosErr -10 and not moving. It seems like the mechanism can only drive in one axis at a time, but that does not jive with the HOME command, which clearly drives in both axes at once. I had to manually send the command again to move the X-axis. Not good...

15:18 UT It is clear that the desired tasks cannot be completed in 10 minutes, although it is really close. The lag grows with each scan, and causes problems. I can easily shave off a minute by making the default PA range -70 to 70, which should certainly be sufficient range to do what I want. Another way to solve it is just to space the scans 12-min apart, which has the small downside of writing two files for each scan, the second being only 2 min long. But really, this should be fine.

Dec 13

13:10 UT I am trying out my automated rotation scheme again, this time measuring a range of X-axis coordinates separated by 5 mm, just to see how repeatable it is from yesterday, and how consistent with X-axis position. This will end at 15:07 UT, after which regular solar observing can start.

Dec 15

19:43 UT Ant 14 Maths error, controller reset.

20:19 UT Ant 14 Maths error, controller reset.

Dec 16

03:08 UT Ant 14 Maths error, controller reset.

14:39 UT Ant 14 Maths error, controller reset.

Dec 17

06:39 UT Ant 14 Maths error, controller reset.

15:38 UT Ant 14 Maths error, controller reset.

Dec 18

15:34 UT Ant 14 Maths error, controller reset. Implemented a scheme to help the controller deal with day changes, to see if this has anything to do with the problem. We'll see if the problem occurs again today.

Dec 20

18:13 UT I had to reset the Ant 14 controller again this morning, despite updating the tracktable creation code to help the controllers deal with day and RA wrap issues. I looked further into it and found out that the maths error occurred around the time that the HA became 90 degrees, even when the antenna is stowed and not tracking. There is a bug in the code that does the HA tracking. To get around this problem, I asked Gelu to set the controller to DATAMODE 0 (HORIZON coordinates) and TRACKDATASOURCE 0 (SETPOINT) on a FLUSH command (also done during STOW). This should stop the controller calculation that is causing the problem.

Dec 21

02:32 UT We had a long struggle with the crios after Gelu updated the crio software. Only now are all of the antennas tracking 3C84. Other antennas have been tracking for some time, so some baselines will be good. The problem antennas were 5, 6 and 8. The schedule is set to observe 3C84 until 08:50 UT, then 3C273 from 10:40 - 15:30 UT, in band 14, not tracking PA. The observations will be used to get baseline Bx and By corrections on 6, 8 and 12, and also to test my unrotating of the feeds. One problem is that Ant 14 pointing may not be too good, because the X-axis setting is 510 mm.

02:46 UT Doh! Ant 12 was fooling me by indicating tracking while the drives were actually not tracking. I just now reset the controller and got it tracking again.

03:29 UT I just discovered that Ant 8 was not tracking--hung up again. I think it did so quite soon after 02:46 UT, so there is basically no data yet for Ant 8. I am not at all sure that it will continue, but it is tracking now.

11:44 UT I found ant14 not tracking on 3C273, and just now sent it there, so the first hour on the source will be no good. We will have almost 4 hours on it, though.

Dec 22

00:00 UT Natsuha is running another series of long calibrator scans (70-min on each source, using pcal.fsq, which is 10 frequencies on the Hi receiver). In the startup ctlgo was needed. During the first 20 minutes, she accidentally set the v-pol attenuation of ALL antennas to 0, which then required manual adjustments. As a result, the first 20 minutes of the data (source 2136+006) may have messed up amplitudes.

Figure 2: Ant13 vpol plot (cyan). The jump is the adjustment at 03:43 UT.

03:38 UT Ant12 v-pol dBm value fell below 1, so attenuation was adjusted from (0,9) to (0,7).

03:43 UT Ant13 v-pol seems to be showing oscillation between ~2<dBm<~6, so adjusted. Does not seem to change much, so I will not try to adjust anymore unless it gets worse. (Figure 2)

04:48 UT Ant13 v-pol fluctuation was gone upon entering new source, so it was probably not the oscillation. Because of the attenuation change I made in the last source, now the dBm value fell below 1. Attenuation lessened from (0,6) to (0,4).

06:48 UT Ant13 dBm values for both h- and v-pol are unstable, ~2<dBm<~8 and ~4<dBn<~8, respectively. But the change every second seems to be random, not 2dB, which would be the case for attenuation imbalance between polarizations. Leaving it as is.

11:40 UT I noticed that the LO1A tuning showed an error (overflow), so I rebooted it on this source change. Since the tuning was a single frequency, I am not sure if it was really not tuned correctly, but if so, earlier data will be no good, from whatever time the problem occurred since ~08:30UT (good before this time). Note that Ant13 is stable now. Oops. I just realized that I had thought we were using band14.fsq, so that is what I set it to for this source (11:40-12:50 UT). It should have been pcal.fsq.

15:51 UT Solar observation resumed.

21:47 UT Kjell was working on ant 4, 6, and 8. Not sure if related but 4, 5, 6, 8's cRIOs stopped communicating shortly after that. Dr. Gary brought back 4, Gelu brought back 5, 6, and 8, but 5 kept falling back onto having negative values in TDIF. Gelu says that the problem seems to be coming from the time lag between cRIO and Stateframe. Was instructed by him to keep using sync command. After two sync commands ant5 cRIO came back by itself..but tracking is in orange color...

23:30 UT Ant8 v-pol attenuation value is (31,31) but dBm value was fluctuating between 20 and 50. Also, ant6's v-pol attenuation was (8 29) and dBm value was "nan". Both came back to normal upon entering phasecal.