System Gain Calibration

Gain Control "Knobs"

EOVSA Gain Control "Knobs"

Non-solar radio interferometers can make the assumption that the system noise is dominated by the relatively uniform sky, but this is not at all valid for the Sun--the Sun dominates the system noise, and can be highly variable, especially during flares and other radio outbursts. This is a main reason why it is necessary to design solar-dedicated instruments for observing the Sun. In order to cope with the high and variable noise from the Sun, EOVSA is equipped with a series of attenuators, two RF attenuators in the frontend, and an IF attenuator in the analog downconverter. In addition, it is possible to change the gain via parameters (ADC Attenuation, FFT Shift, and Equalizer Coefficients) in the digital correlator. The table above lists the various gain control points, their purpose, and other relevant information.

Calibrating Front End Power Detectors

There are power detectors in each of the two channels in each front end, just before the optical link, which measure a voltage proportional to the RF power level (integrated over the full 2.5-18 GHz range). To convert these voltages to power measurements, in dBm (decibel-milliwatts), the input of each front end is terminated with a room temperature 50-ohm load, and the output just before the optical link is connected to the E4418B power meter. There is a LabVIEW vi called "E4418B Measurement.vi," to be run on the Win1 computer, that steps the attenuation and performs the measurements both with and without the ND turned on, to range over a wide range of voltages and powers. The vi then writes the result into two text files named, for example,

Antenna 8 HPOL 412016 193307UT.txt
Antenna 8 VPOL 412016 193307UT.txt

There are many more measurements in the text files than needed, and there is no close synchronization between the switched state of the attenuators and the measurement, so one must then edit the file to remove all measurements except one for a given state. Here is an example of an edited file:

Figure 1: Example plot from fem_cal.py showing the voltage to power curves (symbols) for the H and V channels of Antenna 8 front end, overplotted with 4th-degree polynomial fits.

HPOWER ND HATTN1 HATTN2 HVOLT VATTN1 VATTN2 VVOLT
10.629 1.000 0.000 0.000 2.280 0.000 0.000 3.406
9.774 1.000 1.000 0.000 1.917 1.000 0.000 2.981
8.982 1.000 2.000 0.000 1.624 2.000 0.000 2.620
7.063 1.000 4.000 0.000 1.086 4.000 0.000 1.880
3.231 1.000 8.000 0.000 0.576 8.000 0.000 0.879
-4.953 1.000 16.000 0.000 0.173 16.000 0.000 0.271
9.791 1.000 0.000 1.000 1.924 0.000 1.000 2.998
8.960 1.000 0.000 2.000 1.616 0.000 2.000 2.617
7.044 1.000 0.000 4.000 1.082 0.000 4.000 1.868
3.160 1.000 0.000 8.000 0.571 0.000 8.000 0.876
-5.139 1.000 0.000 16.000 0.168 0.000 16.000 0.266
6.886 0.000 0.000 0.000 1.045 0.000 0.000 1.841
5.813 0.000 1.000 0.000 0.859 1.000 0.000 1.487
4.881 0.000 2.000 0.000 0.737 2.000 0.000 1.223
2.777 0.000 4.000 0.000 0.540 4.000 0.000 0.825
-1.246 0.000 8.000 0.000 0.303 8.000 0.000 0.457
-9.454 0.000 16.000 0.000 0.081 16.000 0.000 0.137
5.837 0.000 0.000 1.000 0.862 0.000 1.000 1.501
4.861 0.000 0.000 2.000 0.735 0.000 2.000 1.221
2.752 0.000 0.000 4.000 0.537 0.000 4.000 0.820
-1.343 0.000 0.000 8.000 0.300 0.000 8.000 0.452
-9.886 0.000 0.000 16.000 0.076 0.000 16.000 0.127

When the edited files are ready, the Python script fem_cal.py is run to create the plot shown in Figure 1 for antenna 8, which includes the measured points and a 4th-degree polynomial fits. The parameters of the fit are printed to the terminal, which for the example in Figure 1 are:

HPOL.c0 =  6.6138626
HPOL.c1 =  5.6355898
HPOL.c2 = -1.0031312
HPOL.c3 = -0.1882171
HPOL.c4 =  0.0348016
VPOL.c0 =  5.5092565
VPOL.c1 =  5.4037776
VPOL.c2 = -0.9535324
VPOL.c3 =  0.3611041
VPOL.c4 =  0.2671788

These lines are then entered into the corresponding crio.ini file, which is located in the crio's /ni-rt/startup folder. The new values will take effect on the next reboot of the crio, or in response to a sync command.

Calibrating the FEM Attenuation Settings

First point the antennas off the Sun (e.g. at STOW), and in the schedule, issue the commands

 FEMAUTO-OFF
 FEMATTN 0
 $FEM-INIT 

After approximately 30 s, the new settings will appear.

The purpose of the frontend attenuators is to set the optimum base power level for the optical transmitters on the blank sky. FEM stands for Front End Module. The base power level is then maintained automatically by the system when pointing at the Sun or other strong power source (e.g. a satellite) by means of the automatic gain control system (invoked by issuing the AGC command to set the AGC operating parameters, and issuing FEMAUTO-ON to turn it on). To set the attenuations to an appropriate level requires the following calibration procedure.

The calibration of the FEM attenuation settings is very straight-forward. Because the voltage detectors are calibrated to provide power levels, all that is necessary is to read the current power level and attenuation settings from the 1-s stateframe monitor system and determine what attenuation is needed to achieve the desired, fixed power level. The default desired power level is 3 dBm, so if the current power level for an antenna/polarization channel is, say, 5 dBm, one would insert an additional 2 dB of attenuation on that channel to bring the level to 3 dBm. The commands to set the frontend attenuation are HATTN a1 a2 <ant> and VATTN a1 a2 <ant>, where a1 is the setting for the first attenuator, in dB, and a2 is the setting for the second attenuator. The <ant> string, of the standard form (e.g. ant1, or ant1-5) specifies which antenna(s) to apply the settings to. To calibrate the settings, the current power levels and attenuation levels are simply read from the 1-s stateframe monitor system, and any power-level differences from the target 3 dBm are minimized by issuing HATTN and VATTN commands to increase or decrease the current attenuation settings. This can be done from the schedule by means of the $FEM-INIT command, which spawns a process to run the set_fem_attn() routine in the adc_cal2.py module, or the set_fem_attn() routine can be run manually from ipython.

There are some rules and exceptions:

• The power-level measurement is to be made on blank sky (i.e. NOT pointing at the Sun or other strong source such as a satellite).
• The FEMAUTO-OFF setting should be issued (i.e. no automatic gain control), and FEMATTN 0 set (no additional attenuation in the front end), prior to $FEM-INIT. Issuing the $FEM-INIT does not insure the above conditions--these have to be done manually before issuing that command.
• The $FEM-INIT only adjusts ant1-13, because the ant14 power level is allowed to be higher (front end attenuations should always be zero for calibrators, although some attenuation is needed on satellites).
• The calibration settings are remembered by the crio in each antenna by the simple means of writing any HATTN or VATTN settings into the crio.ini initialization file on the crio. Therefore, such settings DO survive a reboot of the crio.
• Note that any inadvertent issuing of an HATTN or VATTN command by hand will overwrite the calibration, and necessitate rerunning the $FEM-INIT command. To avoid this, use the FEMAUTO-OFF and FEMATTN commands instead, to temporarily set an attenuation.
• A few antennas (e.g. the "prototype" frontends on Ant 1, 8 and 11) may not be able to reach 3 dBm power level, even with 0 attenuation. That is not a problem, since the remainder of the signal chain is adjusted to compensate.

Calibrating the DCM Attenuation Settings

First point the antennas off the Sun (e.g. at STOW), and in the schedule, issue the commands

SUBARRAY1 ant1-14
DCMAUTO-OFF
FSEQ-FILE solar.fsq
FSEQ-ON
$SCAN-STOP
$CAPTURE-1S dcm

to capture the data for the calibration, then analyze it in ipython (on pipeline, say)

import roachcal
tbl = roachcal.DCM_calnew('/dppdata1/PRT/PRT<yyyymmddhhmmss>dcm.dat',\
dcmattn=10, missing='ant15')

tbl

Examine the table. Recently, band 1 has had bad attenuations due to RFI, so it underflows on the Sun. If so

 tbl = roachcal.override(tbl, bandlist=[1], t=Time('2020-05-08'))

where the t argument gives the date whose SQL table is the source of the override information. If not provided, or None, the SQL table for the current date is used. It is useful to compare the new table with an existing table to check for errors/glitches. To do this,

 roachcal.compare_tbl(tbl, t=Time('2020-05-08'))

will print a table of attenuation differences. Normally such differences should be small, so large, unexplained differences should be investigated. When satisfied with the table, write a new record to the SQL database.

import cal_header
cal_header.dcm_master_table2sql(tbl)

The purpose of the DCM attenuator calibration is to ensure optimal ADC (analog to digital converter) levels in the ROACH boards. DCM stands for Down Converter Module. The ADCs take the noise-like variable voltage from the DCMs and convert it to 8-bit digital numbers ranging from -128 to 127. For optimum performance, the ADC level should be as high as possible without clipping. The correlator sends out P (power) packets whose header gives the integrated power for each accumulation. It is useful to record here the relationship between those power levels in the header and the ADC digital numbers. Basically, the time-domain ADC digital values $x_{i}$ are normalized to +/- 1, squared, and summed over $8192n_{acc}$ values, then divided by $2^{9}$. By trial and error, I find that the correlator reports overflows if the power values exceed 2500, while a more-or-less optimum power level is 1600. Working backward, a power-level $P$ can be converted to the typical ADC value by $128{\sqrt {2^{9}P/8192n_{acc}}}$, so for the 200 MHz correlator, where the accumulation length is $n_{acc}=1792$, this is $x_{rms}\approx 128{\sqrt {2^{9}P/8192/1792}}=0.76{\sqrt {P}}$. The above P values give $x_{rms}\approx 38$ and $x_{rms}\approx 30$, for 2500 and 1600, respectively. This agrees quite well with my experimental results, since $3\sigma$ for $x_{rms}=38$ is 114, close to the maximum value of 128.

Unlike the FEM attenuation, which affects the entire RF band, the DCM attenuation is frequency dependent, because it is inserted into the IF signal path after tuning to a specific band. That brings up two complications: (1) a separate DCM attenuation setting is needed for each of the 34 possible IF bands, and (2) the DCM attenuation has to be changed on a 20 ms timescale (50 times/s), in unison with the rapidly switching frequency sequence. Each 20-ms period is called a "slot."

In order to accomplish this feat, whenever the frequency sequence (fsq file) is changed, a new set of attenuations matching that sequence has to be uploaded to the DCMs. This starts with a "master" table of 30 IF channels (2 each for 15 antennas, although Ant 15 is no longer part of the system) for 52 bands. The purpose of the DCM attenuation calibration is to determine these optimal attenuation numbers. Once the master table is defined, the system creates the DCM attenuation table on the fly at the start of each scan, according to the chosen frequency sequence for that scan. The DCM master table size, then, is 30 x 52, from which the DCM table of size 30 x 50 is created at the start of a scan, for the 30 IF channels and 50 "slots." This 30 x 50 DCM table (called dcm.txt) is sent to the ACC (FTP'd to the /parm folder and then commanded to be read by the DCMTABLE command) at the start of each scan. The ACC subsequently uploads the information once to all of the DCMs on the relatively slow RS-485 serial connection. Further modifications to the DCM attenuation are possible on a 1-s cadence, issued by the DPP.

The procedure for creating the master calibration table (the 30 x 52 table) is to first ensure that the dishes are off the Sun, the FEM attenuations are properly set via the $FEM-INIT schedule command, and a suitable frequency sequence is running that samples all 52 bands. Currently, the solar.fsq sequence is defined for this purpose. Stop the DCM attenuation switch by issuing the DCMAUTO-OFF command. This will set the DCM attenuations on Ants 1-13 to a fixed 10 dB. With the dppxmp program idle (i.e. with $SCAN-STOP in effect), perform a 1-s packet capture on the DPP (can be done with the $CAPTURE-1S schedule command, or just type capture in a dpp terminal window). A new capture file named PRTyyyymmddhhmmss.dat will appear in the /data1/PRT folder on the DPP (or /dppdata1/PRT folder on pipeline or tawa). To analyze the packet capture file and update the SQL database, in ipython on the dpp, pipeline, or tawa, issue the following commands:

import roachcal
tbl = roachcal.DCM_calnew(<filename>, dcmattn=10, missing='ant15')
tbl

Note that this is an update from the old 34-band analysis, which was done using the roachcal.DCM_cal() routine. This old routine is kept unchanged in case we ever need to go back to the previous 34-band setup (not anticipated!). The above commands will print the 30 x 52 master table to the terminal window. Examine the table for any errors. In particular, here, it is important to list any non-working antennas (ant15 is always non-working) in the "missing" parameter, so that the routine knows not to update the attenuations for those. A common error is to not specify a non-working antenna, which may show up as no signal, and hence the analysis will result in setting those DCM attenuations to zero (in an attempt to increase the signal level). When the antenna comes back online, the DCM table will indicate zero attenuation, and now the ADC level will be very large and potentially damage the ADC! Once you are happy with the contents of the table, issue these commands to write the result to the SQL database:

import cal_header
cal_header.dcm_master_table2sql(tbl)

Note: cal_header.dcm_master_table2sql() also has a tbl=tbl parameter, but this expects an array of numbers, not an ASCII table of the type returned by roachcal.DCM_cal(), so calling it as tbl=tbl will fail.

Calibrating the Correlator EQ coefficients

The procedure is under development, and will be summarized here when final.

The purpose of the equalizer coefficients is to set the power level in the correlator for optimum 4-bit sampling. The way to verify a correct setting is to compare the high-precision accumulated power with the accumulated 4-bit autocorrelation power. When the settings are correct, these two should be identical. Note that the procedure described here is under development and will likely change in some details. --Dgary (talk) 13:06, 22 April 2017 (UTC)

When the FEM and DCM attenuations are properly set, the power level measured in the correlator should be roughly constant, independent of band, as the system rapidly tunes during the 1-s frequency sequence. However, this is only the integrated power level over the IF band. The power level over the 4096 subchannels within a band can vary, mainly due to the bandpass shape. To keep everything optimum for good linearity of the correlated data, the autocorrelation power on a 4096 subchannel basis should match the power. Therefore, the EQ coefficients have sufficient resolution (128 separate coefficients over the 4096 subchannels, each adjusting 32 contiguous subchannels) to flatten the bandpass, with a separate array for each of the 50 slots of a 1 s sequence. So far, we have only set the EQ coefficients to a constant, with the same value applied to all antennas, all bands, and all 128 values across each band.

The basic procedure envisioned for this calibration is to set an all-band observing sequence (e.g. gaincal.fsq) and then packet-capture data over a range of constant EQ coefficient settings, then analyze the packet-capture files to determine what setting results in the best agreement between power and autocorrelation for each antenna, polarization, band, and 32-subchannel subband. Unfortunately, this has to be done with packet-capture data because we need the 4096-subband resolution, and although it takes only 1 s to capture a file for one EQ setting, it takes about 1 min to read such a file (1 GB) and analyze it. One can imagine that, once things are set close to optimum, a set of captures over 4-5 EQ settings would be sufficient, and so the entire procedure could be done in 5 minutes. It remains to be seen how variable this is over time, and whether the settings are substantially different off and on the Sun.

Curious Correlator Non-Linearity

In January 2021 it came to our attention that although the total power calibration provided good numbers for the attenuator settings in the system that corrected the total power well, those same attenuator settings when applied to auto-correlation often gave poor results. Those same poor results also affect the cross-correlation amplitudes! To understand the problem, the GAINCALTEST observations were analyzed not only for total power, but also auto-correlation, which clearly revealed the source of the problem. As shown in Figure 2.

Figure 2: Comparison of total power attenuation steps from a GAINCALTEST (blue line) with the same steps measured in auto-correlation (orange line).

Parameters of the function $f(\log P)=a{\rm {erf}}[(\log P)/b-c]+d$

[1.22384997, 1.37407714, 2.14633069, 2.1519944 ]

[1.74569198],
[1.82825339],
[1.99329483],
[2.01711336],
[2.04462634],
[2.07359821],
[2.10740063],