Skip to main content

Multiple Remotes and Machines

This is the authoritative guide to how pstop remotes bond to pstop machines over Tailscale, how to configure a many-remotes ↔ many-machines fleet, and — most importantly — the peer-cap rule you must follow so a remote can always reach a machine on a large tailnet. Read the "Peer cap" section before adding units.

1. The model in one paragraph​

A remote (the E-stop button unit) bonds to one or more machines (the units that drive the robot's stop relays) and exchanges heartbeats with each over a WireGuard/Tailscale black channel. Safety is fail-safe OR: a machine runs only while every bonded remote says OK; any remote that sends STOP, goes silent past its heartbeat timeout, unbonds, or sends a bad message → the machine STOPs. A remote is either an operator (may STOP and re-arm) or stop-only (may STOP and is heartbeat-monitored, but may never re-arm) — the remote declares which it is (/api/role), live. Machines may optionally restrict who may bond with an admission allow/denylist (/api/admission). See docs/design/failover_and_arming_design.md and the authorization model in docs/guides/http_api.md.

2. Topology​

  • Many remotes → one machine: every bonded remote can STOP the machine (OR). Only an operator remote can re-arm. First operator to STOP owns the re-arm cycle.
  • One remote → many machines (≤4 slots): the remote holds a peer slot per machine (ports 8891–8894), bonds each independently, and a STOP/silence on ANY slot stops that machine. See docs/guides/http_api.md "Multi-machine".
  • Machines and remotes are ordinary Tailscale nodes; they reach each other by their tailnet (VPN) IP over WireGuard — direct when a path can be punched, else DERP-relayed. Cross-site (different LANs) is the normal production case.

3. How to bond a remote to a machine (step by step)​

  1. Point the remote at the machine. On the remote, set a machine peer slot to the machine's tailnet IP + pstop port: POST /api/pstop_peers?slot=N&ip=<machine_vpn_ip>&port=8890 (slot 0..3; admin auth). POST .../api/pstop_peers?slot=N&clear=1 clears a slot. The remote pins each configured machine so the machine can never be trimmed from the remote's netmap (see §5).
  2. Decide operator vs stop-only — on the remote. A new remote announces stop-only (it can stop but not re-arm). To let it re-arm, promote it: POST /api/role?role=operator on the remote (admin auth). Applies live, no reboot; role=stop_only demotes it again (an armed machine keeps running but refuses the next re-arm). The machine has no say in this.
  3. Optionally restrict admission — on the machine. Both lists are empty by default and every remote may bond. POST /api/admission?allow=<remote_id> (id is the 32-bit 0x01<mac24>, e.g. 0x01d7f344; hex or decimal) switches the machine to only listed remotes may bond; ?deny=<id> bans one regardless; ?unallow=/?undeny= remove; GET /api/admission lists. Allowlisting also PINS the remote on the machine (see §5) — it is what makes a cross-site remote reliably bond. A refused remote is answered UNBOND, shows REJECTED on its machine-peer row, and does not retry until its Rebond button (or POST /api/pstop_peers?slot=N&rebond=1) is used.
  4. Verify (see §4).

Provisioning at scale is the fleet's job: it stores each remote's machine slots and role and each machine's admission lists and re-pushes them after any reflash (NVS-preserving, but re-push to be safe). See the fleet-side handoff.

4. Verifying a bond​

  • On the machine — GET /state.json → bonded_remotes[]: each entry has id, state (2 = bonded), age_ms (freshness — small = live), rtt_ms, wg_direct (1 = direct path, 0 = DERP), and stop_only (0 = operator, 1 = stop-only — the remote's current announced role). allowlist[]/denylist[] show the admission lists.
  • On the remote — GET /state.json → pstop_machines[] per slot: state, sent/replies (both advancing = healthy), send_fail, rebonds, hb_ms, last_reply_ms, wg_rtt_ms.
  • A healthy bond: machine state=2, age_ms small; remote sent≈replies and both climbing.

5. The peer cap — the rule that prevents "a remote can't reach the machine"​

Read this. Each chip holds at most CONFIG_ML_MAX_PEERS (= 128, a hard ceiling) WireGuard peers. On a tailnet with more than 128 nodes, every chip trims its peer table, keeping pinned peers and recently-active peers and dropping the rest. WireGuard is symmetric: a session forms only if both ends hold each other's key. Therefore:

  • A remote pins each configured machine → the remote never trims its machine. ✅
  • A machine pins each admission-allowlisted remote → the machine never trims them, so a brand-new remote that has never sent a packet is still kept and can bond. ✅ (Implemented via a microlink peer-wanted hook; the machine matches the incoming netmap FQDN pstop-01<mac24>.<tailnet> against its allowlist.)

Consequence you must design around:

  • An allowlisted remote will always bond to its machine, anywhere on the tailnet, because both sides pin each other. Allowlist any remote that must reliably reach a machine across sites (§3 step 3).
  • An unlisted remote (open admission) is best-effort on a >128-node tailnet: if it has no recent activity it may be trimmed from the machine's netmap and fail to bond (sent=0, send_fail climbing, errno 128, and it won't appear in the machine's /admin/api/peers). This is not a safety hole (the machine isn't relying on a remote it never bonded), but it is why a bench/test remote "won't connect."
  • Keep the tailnet lean. Retire stale/test nodes. Below 128 total nodes, nothing is trimmed and every remote bonds regardless of admission listing. s_extra_pins holds up to ML_EXTRA_PINS (= 16) app-pins per chip (machine slots on a remote, operators on a machine).

5.1 Scaling to hundreds of remotes and machines​

The 128 WG-session ceiling (ML_MAX_PEERS) is per chip, not per site. Hundreds of units are fine as long as no single node needs sessions to more than ~120 others:

  • Remotes never scale-limit — a remote sessions only with its ≤4 configured machines + the fleet (~5 peers), at any site size.
  • A machine sessions with each bonded remote, so a single machine can serve at most ~120 remotes. Beyond that, partition remotes across more machines (a topology choice) or raise ML_MAX_PEERS/WIREGUARD_MAX_PEERS in firmware (bounded by RAM and needs validation).

max_peers (runtime POST /api/settings) is the per-site active-set tuning:

  • Dedicated pstop tailnet (recommended for production): with only real pstop nodes in the netmap there's no noise to trim. A value like max_peers = 32 is a good production default — it comfortably covers a machine's fan-out + fleet + operators while keeping the DISCO/WG probing set small, and it's well under the 128 ceiling. Raise it toward 128 only for a machine that must serve >~28 remotes.
  • Shared/noisy tailnet: a low value (e.g. 16) suppresses churn from unrelated nodes — a test accommodation, not a production setting.
  • ML_EXTRA_PINS (= 16) caps pinned peers per chip. For a machine that must reliably hold >16 operators, raise it (small static-RAM cost) so every operator survives the trim; stop-only remotes remain best-effort.

Do not ship max_peers = 16 — it would cap a machine at ~16 remotes. Set it to the production value (≈32, or higher per fan-out) once the deployment tailnet is sized.

6. DERP / direct-path notes​

  • A chip homes one DERP connection on its priority peer's (the fleet's) region for inbound reachability, and (since 2026-08-04) opens additional aux DERP connections on demand — one per distinct region among its pinned safety peers, up to ML_DERP_MAX_CONNS (6). A DERP server only relays between peers connected to it, so before multi-region a machine on a different region than the chip's home was unreachable outbound (errno 128 — see docs/design/outbound_cold_bond_over_derp.md). Now the chip routes each peer's relayed frames to that peer's region automatically, so cross-region remote↔machine bonds without a manual tailscale ping. If a link still only comes up after a ping, check /admin/api/monitor derp_pool[] — an aux conn for the peer's region should be present. See [[reference_pstop_derp_region]].
  • Same-LAN remote+machine upgrade to a direct path automatically; judge direct-vs-DERP by wg_direct/tailscale status, not by rtt_ms (the pstop loop RTT is hold-time inflated).

7. Troubleshooting: "remote won't bond to the machine"​

Symptom on the remote: pstop_machines[slot] state=1, sent=0, send_fail climbing, errno 128; the remote's /admin/api/peers has the machine but the machine's /admin/api/peers does not have the remote.

errno 128 (ENOTCONN / "no session keys") has TWO causes — disambiguate first. Check the tailnet size before doing anything:

  1. Is the tailnet < 128 nodes? Then it is NOT a peer-cap trim (step 1 does not apply — do not prune the tailnet or add operators for connectivity). On a small tailnet, errno 128 means the chip's single/pooled DERP connection can't relay its WireGuard handshake to a machine on a different DERP region — an outbound cold-bond / region-home failure. Check /admin/api/monitor derp_effective_home_region and derp_pool[] vs the machine's region. With the multi-region relay (2026-08-04) the chip opens an aux DERP conn on the machine's region automatically; if it hasn't, the machine's region is unknown/unreachable. Full details and the fix history: docs/design/outbound_cold_bond_over_derp.md. This is the most-misdiagnosed case — it is not a peer count problem.
  2. Is the tailnet > 128 nodes? (/admin/api/peers length = 128 on both = at the cap.) If so → peer-cap trim. Fix: allowlist the remote on the machine (POST /api/admission?allow=<id>) so the machine pins it, or prune the tailnet below 128, or co-locate them on one LAN (direct path, no relay).
  3. Different failure? sf_route≠0 / errno 118 = no route → machine peer not in the remote's netmap or WG down → check the remote's /admin/api/monitor (derp_home_region, wg_paused, ts_boot_en). wg_paused=1/ts_boot_en=0 = boot-safety pause from crash-boots → POST /api/ts_boot then a clean POST /admin/api/restart (not a power-cycle). See [[reference_pstop_derp_region]].
  4. After adding an operator, the machine keeps the remote on its next netmap sync (seconds). A machine reboot re-reads operators from NVS and re-pins them.

8. Reference​

  • Authorization / arming: docs/design/failover_and_arming_design.md, docs/guides/http_api.md.
  • Machine node: docs/design/machine_on_esp32_s3.md. Multi-machine API: docs/guides/http_api.md.
  • Peer-cap fix: components/microlink/src/ml_wg_mgr.c (s_extra_pins, peer-wanted hook), machn/main/main.c (operator match). Safety case: HARA H-13 / SR-SYS-09.