Solar Pointing (SOLPNT) Calibration — Operator Runbook

Rollout status

This page documents the full designed workflow for the browser-based Solar Pointing view in caleovsa, plus a manual fallback for recovery. As of this writing, only Scan & Analyze and Review are live (read-only, Phase 0). Preview & Apply is designed but [not yet enabled — Phase 2], and History & Recovery is [Phase 3]. Until those ship, use the manual procedure in Section 4 for recovery, and apply routine corrections through whatever interim process your domain owner currently authorizes — this page does not add a new sending mechanism.

1. Overview

SOLPNT (solar-pointing) calibration measures each antenna's pointing error by sweeping it across the solar disk and fitting the resulting total-power response, per frequency channel, with a Gaussian. The fit centers give the antenna's offset from where it was commanded to point. That offset is corrected by adjusting two pointing-model coefficients:

• P1 — Azimuth offset (AzEl-mounted antennas) or Hour-Angle offset (equatorially mounted antennas)
• P7 — Elevation offset (AzEl) or Declination offset (equatorial)

See Pointing Calibration for the full pointing model and mount types. SOLPNT corrects only these two constant-offset terms — it cannot fix a mispointed model term that varies with sky position (see Section 5).

The four workflow stages

The Solar Pointing view is a fourth top-level view in caleovsa, organized as four tabs that mirror the actual data flow:

1. Scan & Analyze — pick a SOLPNTCAL scan and run the fitting/gating job.
2. Review — read the per-antenna results: status, reason codes, and the linked map/offsets/beam/gain-impact plots.
3. Preview & Apply — build a dry-run of the coefficient write, confirm, send, and verify by readback. [not yet enabled — Phase 2]
4. History & Recovery — browse historical P1/P7 pairs for one antenna and restore a prior pair. [Phase 3]

[Pick scan] → [Analyze (job)] → [Review gating] → [Select Pass/ack Review ants]
     → [Preview (live coeffs, conflict check)] → [Confirm] → [Send + readback verify]
     → [Receipt + audit + post-apply commands] → [Verification scan] → done
                                   ↘ PARTIAL → guided retry / revert
Fail (off-field) ants  ─────────────────────────→ History & Recovery (separate path)

Routine calibration and off-field recovery are different apply paths with different preconditions; they share the same underlying write/verify/ audit machinery.

The read-only-first principle

The rollout is deliberately staged so that nothing writes antenna state until the gating thresholds have been validated against real history:

• Phase 0 ships the job runner, the result-bundle format, and the Scan & Analyze / Review tabs with zero write endpoints in the codebase — there is no code path in this phase that can change a coefficient.
• The analysis job itself is side-effect-averse by default even though it is not the write path: it runs as an isolated subprocess (never imported into the web server), skips saving plot PNGs, and does not persist the legacy SKYCAL/ATTNCAL SQL records for web-triggered runs unless your domain owner has changed that default.
• Phase 1 backtests the gating thresholds over months of historical scans before anything is frozen.
• Phase 2 adds the apply module (Preview & Apply), and Phase 3 adds History & Recovery — both described below for orientation, and both clearly marked as not yet enabled.

2. Routine calibration procedure

Scan & Analyze

1. Open the Solar Pointing view and pick a SOLPNTCAL scan from the scan list (populated automatically by date/date-range — you never type a timestamp).
2. Click Analyze. The job runs on the pipeline host as its own process (not inside the web server), so it survives a browser refresh — it is keyed by a run_id, not your browser session. Expect roughly these phases: metadata (~10 s), trajectory setup, read & background-correct (~85 s), Gaussian fitting (~4 min, with live per-antenna progress), and gating.
3. Only one SOLPNT job runs at a time per host (a single-flight lock). If Analyze is refused, another job is already running — check the Recent Bundles list or the job status before assuming the app is broken.
4. Cancel sends the job a termination signal; it traps this, marks its bundle cancelled, and cleans up.
5. Re-analyzing a scan never overwrites a prior result — each run produces a new, immutable, versioned bundle. The Recent Bundles list shows scan time, run time, engine version, thresholds version, and a digest prefix for each.

Review

1. Open a bundle from Recent Bundles (or from the scan picker) to load it into Review.
2. The summary map shows per-antenna Az/El (or RA/Dec) offsets, with an explicit "Off-field / not fittable" side list naming any antennas excluded from fitting, with their nominal values and exclusion reason.
3. The per-antenna decision table is the primary read: one row per antenna with a status chip (Pass / Review / Fail / No data), reason codes with hover text, key metrics (median Az/El offset, MAD, valid- channel counts, band slope, feed disagreement, width ratio), and — for Pass/Review rows — the proposed P1/P7 increments.
4. Selecting an antenna (map dot, table row, or selector) drives every other panel: offsets-vs-frequency, beam-width-vs-frequency, and the gain-impact strip all update together with a synchronized hover crosshair. See Section 3 for how to read them.

Preview & Apply — [not yet enabled — Phase 2]

Not yet enabled

Everything in this subsection describes the designed Phase 2 behavior for orientation. There is no write endpoint in the deployed app today. Use Section 4 for the interim manual procedure.

When enabled, the tab will work as follows:

1. Selection starts empty. Only Pass antennas are freely selectable. Review antennas require a per-antenna acknowledgment checkbox restating the specific warning. Fail and No data antennas are locked and cannot be selected by any UI action.
2. Preview builds one atomic review table: scan-time P1/P7, current live P1/P7, a drift flag if they differ, the increments, the proposed P1/P7 (live + increment), the correction magnitude, and any warnings.
3. Confirm shows a dry-run table of exactly what will be sent, requires a typed operator name (the app has no user authentication — this is accountability by convention, not access control [TBD — pending Q1: whether SSO hardening becomes a prerequisite before Phase 2 is enabled]), then sends each coefficient and verifies it by reading it back from the stateframe. The receipt reports, per antenna and per coefficient, confirmed / timeout / send-error.
4. A post-apply command block shows the exact commands for the applied set, generated from the actual applied antenna list (not retyped by the operator), with a copy button, e.g.:

reboot 1 ant6-8 ant10
tracktable sun_tab.radec ant6-8 ant10
track ant6-8 ant10

1 is the reboot register value. The antenna list uses the standard space-separated range syntax already documented in Control commands — dashes indicate a range, e.g. ant1 ant3 ant5-9 — and is generated from the actual applied set. This fixes a real class of bug in the old wiki example, where the reboot line silently dropped ant10/ant13 because the list had been typed by hand instead of generated from what was actually sent. These commands are displayed for the operator to run through the normal channel, never executed by the app itself. 5. Applied antennas enter a "verification pending" state until a later SOLPNT analysis passes them [TBD — pending Q5: whether a passing verification scan is mandatory before the next apply, or advisory; design default is mandatory].

3. Reading the plots

The Review visuals share two physical yardsticks so that a mispointing's size is visible at a glance, not just its sign.

The map: solar-radius yardstick

The summary map is a square, equal-scale canvas (unequal axis scales would misrepresent pointing geometry). Its yardstick is the solar radius: R☉ ≈ 16′ = 2,667 units (units are \(1\times10^{-4}\) degree, the same convention used for offsets throughout the bundle and in the trajectory table format). The map draws:

• a pale disk at 1 R☉,
• dashed rings at 2–3 R☉ (this replaces the old, unlabeled 2,500-unit circle from the legacy plots — same idea, now an explicit, legended yardstick),
• gating radii (the Review/Fail off-field boundaries) drawn distinctly and legended so they are never confused with the solar-radius rings.

Selecting an antenna draws that antenna's beam-FWHM circles at the band edges, centered on its measured offset, alongside a focus card with its status, reasons, and proposal.

The offsets panel: beam-envelope logic

The offsets-vs-frequency panel's yardstick is the beam width at each frequency: a shaded ±FWHM/2 envelope plus ±1 R☉ guides. A channel whose offset escapes its own beam envelope is visibly dangerous — that is the plot's whole point. Valid channels are drawn solid; rejected channels are drawn faint, with the robust (median) aggregate line overlaid. The companion beam-width-vs-frequency panel keeps the nominal beam curve plus warn/fail width-ratio guides (see the reason-code tables below for the exact ratio bounds).

The gain-impact strip

A third, compact strip renders the single derived number that answers "what does this mispointing cost?" — the per-channel, per-feed gain impact:

\[ \text{gain impact} = \exp\!\left(-\left(\frac{\text{offset}}{\text{width}}\right)^2\right) \]

This is estimated sensitivity loss vs. frequency, and is the line an operator should read as the bottom-line consequence of a given offset, rather than trying to mentally combine offset and beam-width numbers.

All three panels share one frequency axis and a synchronized hover crosshair with readout, driven by whichever antenna is currently selected (map dot, table row, or selector).

Reason-code tables

Every antenna gets exactly one status — Pass, Review, Fail, or No data — driven by machine-readable reason codes. These thresholds are provisional (seeded from a single run) and versioned (thresholds_version, recorded in every bundle); they will be revised after a backtest against historical scans [TBD — pending Q2: threshold sign-off after the Phase 1 backtest]. Every code carries a human-readable sentence in the bundle's quality.notes.

Failure codes (any of these ⇒ status Fail, locked out of routine apply):

Code	Trigger (initial provisional threshold)
F_NO_DATA	n_valid ≈ 0 on both feeds → status No data (ants 2, 11–14 in the evidence run)
F_SPARSE	n_valid < 200 on either feed
F_OFF_FIELD	any aggregate offset magnitude > 5,000 (beyond the densely sampled region; fits there are extrapolations) — ants 7, 9, 15
F_HIGH_SCATTER	MAD > 1,500 on either axis
F_WIDTH_ANOM	width ratio outside [0.5, 2.0] on either feed, or non-physical structure (band-median undefined)
F_EXCESSIVE	correction magnitude > 5,000 — routine path refuses; use Recovery
F_NO_CLOCK	antenna clock zero throughout the scan (Ante_Cont_SystemClockms gate)

Warning codes (any of these, with no failure code ⇒ status Review, apply allowed only after per-antenna operator acknowledgment):

Code	Trigger (initial provisional threshold)
W_FREQ_STRUCTURE	band drift > 800 units (ant 5's Az "V" shape)
W_HIGH_SCATTER	MAD in 500–1,500
W_FEED_DISAGREE	cross-sweep feed discrepancy > 500
W_WIDTH_RATIO	width ratio outside [0.7, 1.3] but inside hard bounds
W_LARGE_CORRECTION	correction magnitude in 2,500–5,000 (ant 3's −4,200 Az case: well-fit but too large to apply without a human deciding)
W_COEFF_DRIFT	live P1/P7 ≠ scan-time P1/P7 at preview time (someone changed pointing after the scan; the measured offsets may no longer apply)

No codes ⇒ Pass.

4. Recovery of a badly mispointed antenna

History & Recovery panel — [Phase 3]

Not yet enabled

The panel described here is designed but not yet built. Use the manual procedure below until it ships.

For one selected antenna, the panel will show:

• Pair history: P1 and P7 rendered as inseparable, timestamped pairs — a step-plot timeline plus a table. The UI will have no affordance to take P1 from one date and P7 from another.
• Badges: a pair is marked "verified good" only when this system's own audit trail shows it was applied and subsequently passed a verification scan. Pre-system history gets no badge — a record's mere existence is not evidence that it was ever good.
• Restore: select a pair, and a preview shows the current live pair, the selected pair, the difference, and a mandatory reason field; confirming runs the same signature/conflict/readback/audit path as a routine apply. A restore always sets verification-pending and locks the antenna out of routine apply until a verification scan passes.

Manual procedure (for use today)

Never use the old wiki's modulo-timestamp query

The retired wiki snippet filtered on (Timestamp % 86400) = 100 with no bounding time range. On the current wide stateframe table this predicate cannot use the Timestamp index at all, so the database has to scan the entire table and compute the modulo on every row — in practice this hangs rather than returning. The wiki snippet also referenced a cursor variable that was never defined, so it could not have run as written even before that. Always bound your query with an indexed Timestamp BETWEEN range.

Until the History & Recovery panel exists, query the stateframe database directly with eovsapy.dbutil, using the current schema: dimension-15 fields (which include Ante_Cont_PointingCoefficient1/7) now live in the fV70_vD16 table, indexed by column I16 (post-v66, the old dimension-15 table was folded into the dimension-16 table — see Stateframe database). A 1-based UI antenna number N maps to I16 % 16 == N - 1.

The pattern below takes narrow, indexed point-samples at chosen dates instead of scanning the table — each query is bounded by Timestamp BETWEEN, so it hits the index and returns in well under a second:

#!/usr/bin/env python3
"""Query historical PointingCoefficient1/7 pairs for one antenna.

Uses the CURRENT stateframe schema: table fV70_vD16, index column I16
(post-v66, dimension-15 fields moved into the dimension-16 table). Always
bounds the query with an indexed Timestamp BETWEEN window -- never a bare
modulo-on-Timestamp predicate (see the warning in the runbook).
"""
from eovsapy import dbutil as db
from eovsapy.util import Time

ANT = 11  # 1-based UI antenna number
FIELDS = ["Ante_Cont_PointingCoefficient1", "Ante_Cont_PointingCoefficient7"]


def read_pair_at(iso_time, window_s=60):
    """Return (timestamp, P1, P7) for the first record at/after iso_time.

    Uses an indexed Timestamp BETWEEN window, never a modulo scan.
    """
    cnxn, cursor = db.get_cursor()
    t0 = Time(iso_time).lv
    t1 = t0 + window_s
    fields = ",".join(FIELDS)
    query = (
        f"select Timestamp,{fields} from fV70_vD16 "
        f"where (I16 % 16) = {ANT - 1} "
        f"and Timestamp between {int(t0)} and {int(t1)} "
        f"order by Timestamp"
    )
    data, msg = db.do_query(cursor, query)
    if msg != "Success" or len(data.get("Timestamp", [])) == 0:
        return None
    return (data["Timestamp"][0], data[FIELDS[0]][0], data[FIELDS[1]][0])


# Point-sample across a date range (e.g. weekly) instead of scanning the
# whole table with a modulo predicate.
for date in ["2026-06-01", "2026-06-08", "2026-06-15", "2026-06-22"]:
    pair = read_pair_at(date + " 00:00:00")
    print(date, pair)

If two adjacent samples disagree, you can localize the exact change to within a chosen tolerance by bisecting between them (each step is still an indexed, bounded query):

def find_change_epoch(iso_lo, iso_hi, tol_s=3600):
    """Bisect between two point-samples that disagree on P1/P7, to localize
    the coefficient-change timestamp to within tol_s seconds. Both
    endpoints must have data and must disagree, or this raises.
    """
    lo, hi = Time(iso_lo).lv, Time(iso_hi).lv
    pair_lo = read_pair_at(Time(lo, format="lv").iso)
    pair_hi = read_pair_at(Time(hi, format="lv").iso)
    if pair_lo is None or pair_hi is None or pair_lo[1:] == pair_hi[1:]:
        raise ValueError("endpoints must both have data and must disagree")
    while hi - lo > tol_s:
        mid = (lo + hi) / 2
        pair_mid = read_pair_at(Time(mid, format="lv").iso)
        if pair_mid is None:
            break  # no data at mid; narrow the window manually and retry
        if pair_mid[1:] == pair_lo[1:]:
            lo = mid
        else:
            hi = mid
    return lo, hi

db.get_cursor() reads database credentials from your .netrc file — do not hardcode credentials in any script based on this snippet.

Once you have identified the historical pair to restore, treat P1 and P7 as one inseparable unit — never take P1 from one date and P7 from another. Send both coefficients for the same timestamp, then run the same post-apply sequence used for routine calibration (Section 2):

pointingcoefficient1 <P1_value> ant<n>
pointingcoefficient7 <P7_value> ant<n>
reboot 1 ant<n>
tracktable sun_tab.radec ant<n>
track ant<n>

These are the same class of atomic schedule commands documented in Control commands and Schedule commands — send them the same way you send reboot/tracktable/track today. This runbook does not add a new sending mechanism.

Commands are fire-and-forget — verify before you trust them

These commands are sent as plain-text TCP with no acknowledgment from the ACC. A command that was sent is not the same as a command that took effect. Before trusting a restore (or any manual coefficient change), independently read back Ante_Cont_PointingCoefficient1 and Ante_Cont_PointingCoefficient7 from the live stateframe (the same query pattern above, pointed at the current time) and confirm the values match what you sent. "Sent" is never the same as "applied."

5. When to escalate to star pointing

Solar pointing measures and corrects only the constant offsets P1 and P7. Escalate to star pointing (see Star Pointing Notes and the star-pointing section of Pointing Calibration) when:

• (a) a restored historical pair still leaves the antenna off-field on the verification scan,
• (b) no credible historical pair exists — for example, after mechanical or encoder work on the antenna, or
• (c) the offsets are position-dependent across the day — a constant- offset correction cannot fix a mispointed all-sky pointing-model term.

6. Troubleshooting

Job stuck / need to cancel

The analysis job writes its own progress (phase, fraction, message, per-antenna fit progress, process id, and a heartbeat time) as it runs, so the app can tell the difference between "still working" and "actually dead." If Analyze appears to hang:

• Check the job status — it reports both the progress file and liveness (process check plus heartbeat age).
• Use Cancel to send a termination signal; the job traps it, marks its bundle cancelled, and cleans up rather than leaving a half-written bundle behind.
• Remember only one SOLPNT job runs at a time per host (a single-flight lock). If a new Analyze is refused, that is very likely why — check for a job already in progress before assuming something is broken.

/tmp trajectory-table cross-user hazard

The analysis engine's trajectory-table step communicates through hardcoded /tmp/caltraj* files. If two runs (or two different accounts) collide on those paths, one run's table write can be silently blocked by a file owned by another account, and the analysis then reads back a stale, wrong table instead of failing loudly. In a real validation run, this produced zeroed fits specifically for the ant12-type antennas (their fits use the parallactic-angle-rotated trajectory table, so a wrong or stale table corrupts them first and most visibly).

• Symptom: ant12-type antennas all come back No data, while other antennas on the same scan look normal — do not assume this means those antennas are actually dead; check for a stale/blocked trajectory table first.
• Fix: per-user trajectory-table filenames plus a hard failure on any non-success return from the table-writing step (instead of silently reading whatever happens to already be on disk). This is a design requirement for the analysis job, not an optional nicety.

PARTIAL apply state — [Phase 2]

Not yet enabled

Described here for orientation; there is no apply path deployed yet.

Because coefficient sends have no acknowledgment, an apply can succeed on one coefficient and fail on the other. Any antenna where P1 is confirmed by readback but P7 is not (or vice versa) is reported as PARTIAL, shown in red on the receipt, with two one-click recovery actions: retry the missing coefficient, or revert the one that did apply back to its pre-apply value. Both actions go through the same send-then-readback machinery as a normal apply — there is no separate, less-verified code path for recovering from a PARTIAL.

Ant12-type over-correction caution (cross-el vs. azimuth units)

Safety-critical — treat as confirmed, pending final sign-off

Evidence from a 2026-08-25→26 correction pair indicates that the newer-generation ("ant12-type") dishes respond to a P1 change 1:1 in cross-elevation units, while the legacy offset/increment math computes the increment as azoff = cross_eloff / cos(el0) — an over-correction by a factor of 1/cos(el) (about 1.8× at summer elevations) for these mounts. The match is quantitative, not just suggestive: one antenna's applied ΔP1 of +26,156, multiplied by cos(55°), gives 14,300 — matching its measured pre-apply cross-elevation offset of −14,189 to within 0.8%, i.e. a clean overshoot past zero exactly as this hypothesis predicts. Independent supporting evidence: the trajectory code's own commit history notes that "azel 2 m antennas use cross-el rather than az units," from when the divide-by-cos(alt) step was removed for those mounts.

Until this is formally confirmed by the domain owner, treat any gating-engine proposal for ant12-type mounts as suspect, and do not use the legacy offsets2ants routine for them at all — it applies exactly the suspect 1/cos(el) conversion. A verification scan after any correction to one of these mounts is not optional: if the antenna overshoots to a map-Az offset in the same direction the hypothesis predicts, that is further confirmation, not a new problem to debug from scratch.

Which specific antennas count as "ant12-type" is not fully enumerated in the source design review beyond the antennas it names as evidence (ant 9, ant 11, and antenna 12 itself, the mount's namesake) [TBD — pending Q8: confirmation of the current ant12-type roster with the domain owner]. If you are not sure whether a given antenna is affected, ask before applying a routine correction to it.