Six-Hour Fleet Soak

Date: 2026-08-03.

Verdict: PASS. Over 6 h 11 m, all reachable units ran the same clean committed build with zero reboots, zero crashes, zero safety faults, zero heap leaks, and zero unreachability. The single logged anomaly was one transient control-plane reconnect during startup settling, self-healed in seconds.

Setup

• Firmware: all units OTA-updated to 95a948b (a clean, committed, non-dirty build — replacing a prior mix of f4daf48-dirty / 1cfbd9a / 3dcd08d). 95a948b also adds the ml_reconnects telemetry field (see below).

• Units under test (the reachable fleet; 6 other units were offline):

  Unit	ID	Role in soak	Path
  machn-01d77a1c	0x01d77a1c	machine (drives 2 series relays)	Ethernet
  PSTOP54	0x01d7eed0	remote — live safety link to machn	USB-NCM
  PSTOP42	0x01d7ed38	remote — control-plane/network soak	USB-NCM
  DUT	0x01d7f344	remote — DERP-relayed control-plane soak	DERP relay

• Window: 2026-08-03 00:43:37 → 06:54:10 (6 h 11 m monitored; units up ~6 h 15 m continuous). Poll interval 30 s → 716 samples/unit, 2864 total. (Monitoring ran in two windows — the host killed the first background task at 4 h 44 m — but the units ran continuously the whole time and self-track cumulative counters, so coverage is complete. A ~5 min fine-grained polling gap at 05:27 has no reboot/fault in the surrounding cumulative counters.)

• Machine state: fail-safe STOP (unarmed) throughout — the safety link is soaked (heartbeats + status streaming + replies), not the energized-RUN path (which would need a continuously-driven arming gesture).

Results (all 716 samples/unit)

Signal	machn	PSTOP54	PSTOP42	DUT
Reboots (uptime regression)	0	0	0	0
Fault resets (PANIC/WDT/brownout) during soak	0	0	0	0
Lockstep mismatch	0	0	0	0
Dual-core clock xcheck fault	0	0	0	0
GPIO pad-config fault (SR-R-09)	0	0	0	0
Relay faults / spurious STOP	0	0	0	0
Heap leak	none	none	none	none
Unreachable samples	0/716	0/716	0/716	0/716
Control-plane reconnects (ml_reconnects)	2	0	0	0
WiFi/internet disconnects	0	0	0	0

• No restarts: every unit's uptime climbed monotonically to ~6 h 15 m; reset_reason was 3 (SW — the initial OTA boot) for all, with no fault code.
• Heap stable: machn 8019k→8020k; PSTOP54 7917k→7918k (min dip 7852k, recovered); PSTOP42 7918k→7921k; DUT 7917k→7920k. No downward trend on any unit.
• DERP path held: the DERP-relayed DUT stayed ml_state=CONNECTED for all 716 samples — historically the flakiest path, solid here.

The live safety link (PSTOP54 ↔ machn)

• bonded=1 on every one of 716 machn samples — the bond never dropped.
• 110,817 status messages sent, 110,687 replies — a 99.88 % reply rate over 6 h.
• Rebonds: 13 → 32 (+19 total). ~10 of those were a single settling burst in the first 14 min (right after four near-simultaneous OTA reboots); the remaining ~9 were sporadic over 6 h (≈ 1 per 40 min), each a brief self-healing re-bond that never caused a STOP, a mismatch, or a lost bond. pstop_send_fail 11 → 20 (+9).
• wg_dedup_drops climbed on the USB-NCM units (harmless — dropping duplicate WireGuard packets that arrive via both direct and relay paths).

The one logged anomaly

00:49:17 machn: ml_state=3 (RECONNECTING) — a single transient during startup settling, recovered within seconds; it is one of machn's 2 ml_reconnects. Both machn reconnects landed by 00:57; zero reconnects in the following ~6 h.

Notes

• ml_reconnects (new this build) earned its place: it caught machn's 2 control-plane reconnects, which self-recovered between 30 s polls and would otherwise have been invisible. 0 on all three remotes = the control plane never flapped for them.
• rst_hist historical entries are pre-soak: PSTOP42's ring buffer contains older 6 (TASK_WDT) entries and PSTOP54 older 11 (USB) entries — these predate this firmware and this soak (leftover from bench flashing / USB re-enumeration). No unit incurred a watchdog or fault reset during the 6 h.
• Monitor correction: an early false-positive ("machn link stale") came from applying remote-role semantics to the machine (pstop_last_reply_ms is an absolute boot-timestamp, and a machine sends rather than receives replies). The check was switched to a reply-counter-progress test; the device was never stale.

Assessment

The fleet is stable for extended continuous operation on 95a948b. No restarts, no disconnects at the bond level, no safety faults, no resource leaks. The only imperfection is a small rate of brief, self-healing re-bonds on the one active wireless-tunnelled safety link (~1 per 40 min, 99.88 % reply rate) — worth keeping an eye on but well within normal for a WireGuard/DERP-tunnelled link and never consequential to the safety function.