Skip to main content

Crash Hypothesis Catalog

Date: 2026-06-22.

Generated by a 4-angle read-only audit (concurrency, memory/OOM/stack, lwIP/socket/abort, lifecycle/safety-ladder). Ranked by severity × likelihood. Status legend: triggerable = reproducible from outside via network/admin; power-cycle = needs physical power events; instrument = needs a test build to provoke; code-proven = the defect is evident from source and a fix is justified without a live repro.

Test plan: after the soak, a single serialized hardware-test agent confirms the triggerable items on the chip (capturing rst_hist + /api/last_log). code-proven items get fixed by inspection. Then rebuild + OTA + re-validate.

CRITICAL — net_liveness false-abort → rollback ladder (3 of 4 agents converged)​

The deliberate abort() watchdog (dcs_net_liveness.c) is the single largest crash/regression surface. Its purpose is to catch a wedged lwIP stack, but its false-positive surface includes ordinary upstream outages, and a false fire feeds the boot-count → OTA-rollback chain.

  • H1 — pstop cross-check is structurally absent when pstop never bonded. abort() is vetoed only by a fresh pstop reply, but g_dcs_pstop_last_reply_ms stays 0 until bond_with_retry() (infinite retry, main.c) first succeeds. If the pstop machine is down (maintenance / wrong peer) the veto (ps_last > 0) is permanently false, so a gateway ICMP-silence >90s aborts a fully healthy chip. (dcs_net_liveness.c:216-230) — triggerable
  • H2 — abort@90s vs boot-count-clear@120s is unsurvivable. The watchdog can fire ~90s after boot, but bc_clear_task needs 120s of continuous healthy uptime to zero the crash counter. A persistent/slow upstream outage therefore marches boot_count 1→2→3→4 and trips esp_ota_mark_app_invalid_rollback_and_reboot(). Directly contradicts the stated robot reality ("Ethernet takes minutes to come up"). (dcs_safety.c vs dcs_net_liveness.c:42) — triggerable / by-design
  • H3 — "armed" state is sticky across a failover. retarget_ping() re-seeds the silence clock but never re-zeros s_ping_reply_count, so after Eth→WiFi/USB the watchdog stays armed; failing over to a gateway that filters ICMP arms a 90s death timer against a healthy link. (dcs_net_liveness.c:139-179, 200) — triggerable
  • H4 — retarget_ping has NO generation guard (asymmetric with net_inet's fix). A late on_ping_success from the old gateway session can land after retarget and stamp s_last_ping_ok_ms fresh — disarming the watchdog when it should fire (credits the new, possibly-dead gateway with the old one's reply). (dcs_net_liveness.c:139-179) — code-proven

HIGH — rollback-ladder robustness​

  • H5 — ESP_RST_BROWNOUT counts toward the ladder. A PoE/USB power-quality event that browns out 4× before the 120s clear → OTA rollback for a non-firmware reason. (dcs_safety.c:34) — power-cycle / code-proven
  • H6 — rollback boots a possibly-stale slot. 2-slot table, no factory, anti-rollback disabled → mark_invalid_rollback boots the other slot with zero version protection; the other_invalid guard only catches INVALID/ABORTED, not stale-but-valid. (partitions.csv, sdkconfig) — code-proven / config
  • H7 — 120s mark-valid→clear window. Running image is marked valid almost immediately (ml_app), but the crash counter persists 120s regardless of health, so a crash in that window on an otherwise-healthy boot still counts. (dcs_safety.c:104) — code-proven

HIGH — concurrency / lifecycle​

  • H8 — unsynchronized vTaskSuspend(g_dcs_wg_handle). The shutdown hook + api_usb_enable
    • boot-finalize all suspend ml_wg_mgr with no coordination with LOCK_TCPIP_CORE(), which wg_mgr holds across decrypt/ip_input/periodic. Suspend mid-critical-section freezes the lwIP core lock for the whole system → restart hang/panic (the documented "panic in the 500ms window"). Plus g_dcs_wg_handle is cached once and never revalidated → suspend-on-freed-TCB if wg_mgr was recreated. (dcs_support.c:46-48,180,211; dcs_admin_pages.c:385) — triggerable
  • H9 — dcs_usb_set_enabled has no busy guard (asymmetric with dcs_wifi_set_enabled's CAS). The supervisor (net_sup) and an admin POST can drive try_start/stop concurrently → esp_netif_new overlapping esp_netif_destroy on s_netif → UAF/double-free. (dcs_boot.c:109-120) — triggerable

HIGH — OOM / stack (the 21 KB internal-heap floor)​

  • H10 — g_done[] semaphores and g_core_h[] tasks created without NULL checks. Under the 21 KB floor (WiFi + USB resident + DERP TLS re-handshake, all internal heap because CONFIG_MBEDTLS_INTERNAL_MEM_ALLOC=y), creation can fail → xSemaphoreGive(NULL) / xTaskNotifyGive(NULL) in the SAFETY lockstep path → assert/panic. (main.c:435,451) — instrument / code-proven
  • H11 — net_liveness_task stack is 2560 B but calls esp_ping_new_session inline (sets up socket + ping task + ipaddr_ntoa + ESP_LOG). The code's own comment bumped the ping task to 4096 for the same reason; the supervisor task driving it did not get the headroom. A gateway-change-heavy path can overflow → canary panic. (dcs_net_liveness.c:193,237) — triggerable

MEDIUM​

  • H12 — pstop open_socket() == -1 → comparator_task self-deletes with no retry → the safety heartbeat is dead until reboot (fail-safe but unrecoverable). Reachable via fd exhaustion (16 sockets; ~12 steady + ping-session recreate churn). (main.c:291-296) — instrument
  • H13 — no microlink_rebind() on failover (CONFIG_ML_ENABLE_NET_SWITCH unset) → WG/DERP sockets stay on the stale interface after Eth→USB/WiFi → traffic black-holes until DERP self-heals; can manufacture the pstop-silence precondition for H1. (ml_wg_mgr.c, dcs_net_supervisor.c) — triggerable
  • H14 — comparator stale-g_done window. Cores run at the SAME priority (8) as the comparator, so a late g_done give can land after the drain → spurious mismatch (fail-safe send-nothing) or a torn-buffer memcmp. (main.c:333-344) — instrument
  • H15 — s_rx_lock (the new mutex) vs core-lock order. on_usb_rx holds s_rx_lock then takes the core lock; if wg_mgr is suspended while holding the core lock (H8), on_usb_rx holds s_rx_lock forever → ml_dev_tether_stop() hangs. Compound with H8. (ml_dev_tether.c) — triggerable (compound)
  • H16 — on_usb_rx per-packet malloc under the 21 KB floor → RX drops → pstop loss → feeds H1. Heap-pressure amplifier now that USB-NCM is permanently resident. (ml_dev_tether.c:81) — triggerable (amplifier)

Remediation decisions (settled with user, 2026-06-22)​

net_liveness / rollback cluster (H1–H7) — agreed design:

  • net_liveness KEEPS abort() (preserves the panic backtrace for diagnosis), but made conservative:
    • Arm only after BOTH the gateway has replied AND pstop has bonded at least once. A never-bonded pstop => stay DISARMED (degraded/provisioning, not a wedge). Closes H1.
    • Raise NET_LIVENESS_TIMEOUT_MS above the 120 s boot-count-clear window. Addresses H2 timing.
    • Require the active netif UP with a valid (non-link-local) lease for the whole window; disarm on link/lease loss.
    • Reset the gateway-arm on failover/retarget (H3); add the generation guard mirroring net_inet (H4).
  • Rollback ladder counts ONLY genuine code faults (H5/H6/H7): exception PANIC, INT/TASK WDT, lockstep failure. EXCLUDE ESP_RST_BROWNOUT and deliberate net_liveness wedge-aborts. net_liveness sets an RTC_NOINIT marker before abort() so dcs_safety_account_boot distinguishes its deliberate reboot from a real crash and does NOT count it toward the 4-strikes rollback. Net effect: a network/power/upstream condition can never downgrade a deployed safety device; only a bad image can. (The wedge-abort still reboots to recover and still logs a backtrace.)

Other items: H8 (wg-suspend), H9 (usb busy-guard), H10 (safety-task NULL checks), H11 (net_live stack) to be fixed straightforwardly. H12–H16 assessed during the fix pass.

HIL test + fix outcomes (2026-06-22/23)​

  • H1 — CONFIRMED on hardware, FIXED. Chip abort()ed on gateway+pstop silence while HTTP served 200 (healthy lwIP). Fix shipped (66dc2a8): arm-on-both, 180s timeout, gen-guard, reset-arm-on-failover; abort excluded from rollback.
  • H9 — CONFIRMED on hardware, FIXED. Supervisor + admin USB bring-up at the same timestamp -> panic. Busy-guard added (66dc2a8); usb-hammer-alone (50×) and eth-flap-alone both clean afterward.
  • Compound-stress crash (eth-flap + 2/sec usb-hammer simultaneously) — crashed twice on the session-worn 66dc2a8 image, but does NOT reproduce on a clean, cold-booted image (2 aggressive attempts ~130s). Most likely OOM under accumulated heap pressure (~18KB floor) + compound churn — beyond-spec, beyond the passing 2h soak. Flash coredump now armed to backtrace it if it ever recurs.
  • Coredump-to-flash enabled (f5b189c/201562e) — idf.py -p /dev/ttyACM1 coredump-info reads any future panic's ELF backtrace remotely.
  • Cold-boot validated (#6 partial): power-on reset_reason=1, OTA valid, NVS preserved across the repartition full-flash.
  • New HW finding: aggressive /api/iface/eth flapping can wedge the W5500; only a power-cycle recovers it (ESP soft-reset doesn't reset the SPI eth chip).
  • Still unverified empirically: the net_liveness genuine-wedge abort + boot_count=0 exclusion (code-clear, boots clean; no live abort test run yet).

Items checked and judged NOT bugs​

  • /state.json builder: overflow-proof (PSRAM malloc + internal fallback + CLAMP_N).
  • NVS readers: default gracefully; reset-history ring + pstop_unit_num correctly bounded.
  • AUTOIP 0xFEA9 mask: correct for esp_netif's network-order storage.
  • WireGuard/DERP pbuf paths: no reachable UAF/double-free (single-task, freed once).
  • Both-partitions-invalid brick: defended by the other_invalid guard (residual is H6 stale-but-valid).