docs: spec — reliability core + integrated supervisor

docs/superpowers/specs/2026-08-22-reliability-core-design.md

@@ -0,0 +1,202 @@
+# Mark VIII Reliability Core + Integrated Supervisor — Design
+
+Date: 2026-08-22
+Branch: `feature/reliability`
+Status: Approved by user (design sections 1–7)
+
+## Problem
+
+Mark VIII works, but the device does not run reliably or consistently as
+expected. Evidence from 79 prior opencode sessions, the repository history,
+the factory firmware image, and the live device:
+
+1. **Factory defaults are themselves unstable.** The stock firmware image
+   (`pineapplepager-firmware-1.1.0-signed.bin`, OpenWrt 24.10.1,
+   ramips/mt76x8, kernel 6.6.86) ships `/etc/config/pineapd` with every
+   verified crash source enabled: `wlan1mon` bands `'2,5'` fast-hop,
+   `wlan2mon` enabled+hop on a nonexistent interface, SSID pool without an
+   explicit disable, pool target `broadcast`. Any reset, upgrade, or stock-UI
+   reconvergence reintroduces pineapd SIGSEGV crash loops.
+2. **Fixes revert.** Crash-guard UCI values applied at runtime were observed
+   reverting to unsafe defaults after service restarts and deploys.
+3. **Knock-offs.** Enabling client-mode uplink / `wifi reload` mid-operation
+   repeatedly killed management reachability (SSH/HTTP), forcing power
+   cycles and losing engagement state.
+4. **2.4 GHz blindness.** The client uplink STA on phy0 pins the radio's
+   channel; wlan0mon cannot hop, so 2.4 GHz recon goes quiet while appearing
+   "green" in older UI logic.
+5. **Reboot fragility.** After reboots, stale configs and refilled pools
+   produced broken states until v1.3.x added startup checks; ordering is
+   still wrong: Mark VIII starts at S99, *after* pineapd (S50).
+6. **Deploy fragility.** Non-atomic deploys, version confusion across three
+   files, portal refresh failures, and one secret-leak incident.
+
+User decisions: payload-only hardening (no firmware flashing); uplink moves
+to phy1; Reliability Core plus an integrated lightweight supervisor;
+experimental work on a branch.
+
+## Goals
+
+A device that: survives reboot/firmware-upgrade with safe PineAP state;
+never loses management reachability from a UI-initiated operation; keeps
+2.4 GHz operations fully available during engagements; reports radio truth;
+and self-heals known failure modes without human intervention.
+
+## Non-goals
+
+Firmware repacking/flashing (parked as future experiment), new standalone
+processes/daemons, external databases, metrics graphing beyond counters,
+`:1471` takeover.
+
+## Architecture
+
+All changes live inside the existing payload. Three layers, one process:
+
+- **Guard** — `mk8-guard` init script installed by `payload.sh` at START=49
+  (before the S50 pineapple stack that launches `pineapd`): enforces the
+  known-good UCI profile before crash-prone daemons start. Idempotent;
+  commits only differences; logs to syslog and the event journal once Mark
+  VIII is up.
+- **Core** — backend modules in `server.py` (pure stdlib, python3-light
+  compatible): config reconciler, profile store, RF role manager,
+  preflight/rollback gates.
+- **Supervisor** — passive sampler thread inside `server.py`, capped JSONL
+  event journal, UI health panel.
+
+Persistent state lives in `/mmc/mk8/` (ext4, 3.3 GB free) which survives
+reboots *and* firmware upgrades (overlay wipe):
+
+```
+/mmc/mk8/
+  profiles/<name>/{pineapd,wireless,network}.uci   # named snapshots
+  releases/{current,previous}/                      # atomic deploy dirs
+  events.log                                        # rotated JSONL journal
+  boot.marker                                       # boot counter / clean-shutdown flag
+```
+
+## Components
+
+### 1. Boot-time reconciler
+
+Runs on every service start (and `mk8-guard` runs it early at boot).
+Compares live UCI against the built-in known-good profile; commits only
+differences; logs each action to the journal.
+
+Enforced invariants (the five verified crash sources plus v1.3.x rules):
+- `pineapd.@ssidpool[0].disable='1'` and empty `ssid` list
+- `pineapd.wlan2mon.disable='1'`, `hop='0'`
+- `pineapd.wlan1mon.bands='5'`
+- `pineapd.@pineapd[0].autossidpool='0'`
+- `wireless.dummy_radio0` parked per v1.3.1 semantics (disabled unless a
+  scan explicitly borrows it)
+- monitor interfaces present and administratively up
+
+The reconciler never touches AP sections owned by the user (evil twins),
+client sections, or network/firewall config.
+
+### 2. Profiles + knock-off protection
+
+- **Profile store**: `uci export` snapshots under `/mmc/mk8/profiles/`.
+  Save/restore from Settings UI; restore = write files + `wifi reload` +
+  guard re-run. One profile is auto-captured as `lastknown-good` whenever
+  all health checks pass for ≥5 minutes.
+- **Preflight gate** wraps every risky operation: client-mode connect or
+  disconnect, `wifi reload`, any AP enable/disable, enterprise engine
+  start/stop, any UCI commit touching `wireless`/`network`. Sequence:
+  auto-snapshot `pre-<op>-<ts>` → apply → spawn detached watchdog.
+- **Rollback watchdog**: a small POSIX sh script started via `setsid` so it
+  survives SSH/UI death. It probes **local** liveness only — HTTP GET to
+  `127.0.0.1:8080/api/health` and presence/state of the management
+  interface — deliberately ignoring workstation-side reachability, which
+  historically caused false assumptions. If local probes fail on N
+  consecutive checks (default 6 × 5 s), it restores the pre-op snapshot,
+  runs `wifi reload`, writes a `ROLLBACK` journal entry, and exits. Success
+  path: after M consecutive healthy checks it promotes the snapshot to
+  `lastknown-good` and exits.
+
+### 3. RF role manager (uplink on phy1)
+
+Declarative, mutually exclusive radio plan enforced server-side:
+- **phy0 = OPS, always**: monitor hop + PineAP + 2.4 GHz evil twins. Never
+  carries the uplink again.
+- **phy1 ∈ {attack, uplink, idle}**: role switch API + UI control.
+
+`set-role(uplink)`: snapshot config → create/enable a `wifi-iface` STA
+section on `radio1` → pause `wlan1mon` hop (reusing the existing pause/
+resume mechanism used by radio1 APs) → verify association truthfully
+(iw + daemon state). Failure at any step → rollback snapshot + event.
+`set-role(attack)`: STA disabled → monitor/AP stack restored.
+
+Honest tradeoff surfaced in UI text: while the phy1 uplink associates, phy1
+is pinned to the uplink channel — 5 GHz recon is limited to that channel;
+2.4 GHz remains fully hoppable. The dashboard RF chip shows
+`PHY0: OPS · PHY1: UPLINK ch36` style state.
+
+### 4. Supervisor
+
+A sampler thread inside the existing backend process:
+- Every 30 s, passive reads only (`pidof`, `iw dev`, `/proc/meminfo`,
+  interface flags) — no pineapd socket pings (crash source 5).
+- SIGSEGV trend via throttled `logread | grep -c` scan every 5 min.
+- Hysteresis actions: pineapd absent for 2 consecutive samples →
+  `/etc/init.d/pineapd restart` + guard verify; monitor dropped → re-raise
+  (`ip link set <iface> up`); memory >85% sustained 5 samples → alert only
+  (no aggressive action).
+- Event journal: JSONL entries `{ts, kind, sev, msg, meta}` rotated at
+  5 MB × 4 files.
+- Unexpected-reboot detection via `/mmc/mk8/boot.marker` (clean shutdown
+  clears it; boot increments counter when present).
+- UI: Dashboard health panel gains recent-events feed + reliability
+  counters (boots, unexpected boots, rollbacks, restarts, guard fixes);
+  `/api/health` extended accordingly.
+
+### 5. Deploy hardening
+
+- Single-source version: top-level `VERSION` file consumed by build step to
+  stamp `_hak5_manifest.json`, `payload.sh`, and `server.py` banner; no more
+  hand-synced numbers.
+- Atomic deploys in `scripts/deploy.sh`: stage upload to `/tmp/mk8-stage`
+  → sha256 manifest verification → stop service → swap into
+  `/mmc/mk8/releases/current` (previous kept) → start → post-deploy
+  self-check (version match + local health probe). Failed self-check →
+  previous release restored automatically.
+- Payload install continues to work from overlay paths for compatibility;
+  release dir on `/mmc` is symlinked as the service target.
+
+### 6. Testing & verification
+
+Unit tests (existing pattern: stdlib unittest, mocks, one module per
+process): reconciler diff-only idempotence; profile save/restore roundtrip;
+role exclusivity + hop pause/resume; watchdog decision table (probe
+outcomes × thresholds); deploy staging flow with mocked SSH; supervisor
+sampling parsers and hysteresis.
+
+New `scripts/smoke.sh` (on-device, read-only unless explicitly flagged):
+boot persistence of guards, guard enforcement after writing factory-default
+bad values (then restoring), role-switch cycle uplink↔attack, rollback
+watchdog trigger against a deliberately stopped port (safe variant), deploy
+version match, journal integrity.
+
+## Failure modes & handling
+
+| Failure | Handling |
+|---|---|
+| Factory-default bad UCI at boot | Guard fixes before pineapd starts |
+| pineapd crash-loop despite guards | procd respawn + supervisor restart w/ backoff + alert |
+| Risky op kills management plane | Local-liveness rollback watchdog restores snapshot |
+| Deploy uploads corrupt payload | sha256 gate before swap |
+| New payload fails health check | Auto-rollback to previous release |
+| Overlay wiped by firmware upgrade | Reinstall payload; profiles/journal/history survive on /mmc |
+| Memory exhaustion | Sustained-watermark alerts; no destructive automation |
+
+## Security
+
+No new network exposure; all new endpoints behind existing auth; watchdog
+and guard scripts are root-owned, written atomically; no credentials stored
+in repo or journal metadata (SSIDs/BSSIDs only).
+
+## Out-of-scope notes
+
+Custom firmware remains a documented future experiment (extraction recipe
+captured in session history: uImage kernel @0, squashfs-xz rootfs
+@0x2615dc; bootloader signature behavior unverified).