From 4b03ecb1fd955efa182cc2ad38a63bae15b56f5c Mon Sep 17 00:00:00 2001 From: c4ch3c4d3 Date: Sat, 22 Aug 2026 09:52:25 -0600 Subject: [PATCH] =?UTF-8?q?docs:=20spec=20=E2=80=94=20reliability=20core?= =?UTF-8?q?=20+=20integrated=20supervisor?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- .../2026-08-22-reliability-core-design.md | 202 ++++++++++++++++++ 1 file changed, 202 insertions(+) create mode 100644 docs/superpowers/specs/2026-08-22-reliability-core-design.md diff --git a/docs/superpowers/specs/2026-08-22-reliability-core-design.md b/docs/superpowers/specs/2026-08-22-reliability-core-design.md new file mode 100644 index 0000000..a8f2a9d --- /dev/null +++ b/docs/superpowers/specs/2026-08-22-reliability-core-design.md @@ -0,0 +1,202 @@ +# Mark VIII Reliability Core + Integrated Supervisor — Design + +Date: 2026-08-22 +Branch: `feature/reliability` +Status: Approved by user (design sections 1–7) + +## Problem + +Mark VIII works, but the device does not run reliably or consistently as +expected. Evidence from 79 prior opencode sessions, the repository history, +the factory firmware image, and the live device: + +1. **Factory defaults are themselves unstable.** The stock firmware image + (`pineapplepager-firmware-1.1.0-signed.bin`, OpenWrt 24.10.1, + ramips/mt76x8, kernel 6.6.86) ships `/etc/config/pineapd` with every + verified crash source enabled: `wlan1mon` bands `'2,5'` fast-hop, + `wlan2mon` enabled+hop on a nonexistent interface, SSID pool without an + explicit disable, pool target `broadcast`. Any reset, upgrade, or stock-UI + reconvergence reintroduces pineapd SIGSEGV crash loops. +2. **Fixes revert.** Crash-guard UCI values applied at runtime were observed + reverting to unsafe defaults after service restarts and deploys. +3. **Knock-offs.** Enabling client-mode uplink / `wifi reload` mid-operation + repeatedly killed management reachability (SSH/HTTP), forcing power + cycles and losing engagement state. +4. **2.4 GHz blindness.** The client uplink STA on phy0 pins the radio's + channel; wlan0mon cannot hop, so 2.4 GHz recon goes quiet while appearing + "green" in older UI logic. +5. **Reboot fragility.** After reboots, stale configs and refilled pools + produced broken states until v1.3.x added startup checks; ordering is + still wrong: Mark VIII starts at S99, *after* pineapd (S50). +6. **Deploy fragility.** Non-atomic deploys, version confusion across three + files, portal refresh failures, and one secret-leak incident. + +User decisions: payload-only hardening (no firmware flashing); uplink moves +to phy1; Reliability Core plus an integrated lightweight supervisor; +experimental work on a branch. + +## Goals + +A device that: survives reboot/firmware-upgrade with safe PineAP state; +never loses management reachability from a UI-initiated operation; keeps +2.4 GHz operations fully available during engagements; reports radio truth; +and self-heals known failure modes without human intervention. + +## Non-goals + +Firmware repacking/flashing (parked as future experiment), new standalone +processes/daemons, external databases, metrics graphing beyond counters, +`:1471` takeover. + +## Architecture + +All changes live inside the existing payload. Three layers, one process: + +- **Guard** — `mk8-guard` init script installed by `payload.sh` at START=49 + (before the S50 pineapple stack that launches `pineapd`): enforces the + known-good UCI profile before crash-prone daemons start. Idempotent; + commits only differences; logs to syslog and the event journal once Mark + VIII is up. +- **Core** — backend modules in `server.py` (pure stdlib, python3-light + compatible): config reconciler, profile store, RF role manager, + preflight/rollback gates. +- **Supervisor** — passive sampler thread inside `server.py`, capped JSONL + event journal, UI health panel. + +Persistent state lives in `/mmc/mk8/` (ext4, 3.3 GB free) which survives +reboots *and* firmware upgrades (overlay wipe): + +``` +/mmc/mk8/ + profiles//{pineapd,wireless,network}.uci # named snapshots + releases/{current,previous}/ # atomic deploy dirs + events.log # rotated JSONL journal + boot.marker # boot counter / clean-shutdown flag +``` + +## Components + +### 1. Boot-time reconciler + +Runs on every service start (and `mk8-guard` runs it early at boot). +Compares live UCI against the built-in known-good profile; commits only +differences; logs each action to the journal. + +Enforced invariants (the five verified crash sources plus v1.3.x rules): +- `pineapd.@ssidpool[0].disable='1'` and empty `ssid` list +- `pineapd.wlan2mon.disable='1'`, `hop='0'` +- `pineapd.wlan1mon.bands='5'` +- `pineapd.@pineapd[0].autossidpool='0'` +- `wireless.dummy_radio0` parked per v1.3.1 semantics (disabled unless a + scan explicitly borrows it) +- monitor interfaces present and administratively up + +The reconciler never touches AP sections owned by the user (evil twins), +client sections, or network/firewall config. + +### 2. Profiles + knock-off protection + +- **Profile store**: `uci export` snapshots under `/mmc/mk8/profiles/`. + Save/restore from Settings UI; restore = write files + `wifi reload` + + guard re-run. One profile is auto-captured as `lastknown-good` whenever + all health checks pass for ≥5 minutes. +- **Preflight gate** wraps every risky operation: client-mode connect or + disconnect, `wifi reload`, any AP enable/disable, enterprise engine + start/stop, any UCI commit touching `wireless`/`network`. Sequence: + auto-snapshot `pre--` → apply → spawn detached watchdog. +- **Rollback watchdog**: a small POSIX sh script started via `setsid` so it + survives SSH/UI death. It probes **local** liveness only — HTTP GET to + `127.0.0.1:8080/api/health` and presence/state of the management + interface — deliberately ignoring workstation-side reachability, which + historically caused false assumptions. If local probes fail on N + consecutive checks (default 6 × 5 s), it restores the pre-op snapshot, + runs `wifi reload`, writes a `ROLLBACK` journal entry, and exits. Success + path: after M consecutive healthy checks it promotes the snapshot to + `lastknown-good` and exits. + +### 3. RF role manager (uplink on phy1) + +Declarative, mutually exclusive radio plan enforced server-side: +- **phy0 = OPS, always**: monitor hop + PineAP + 2.4 GHz evil twins. Never + carries the uplink again. +- **phy1 ∈ {attack, uplink, idle}**: role switch API + UI control. + +`set-role(uplink)`: snapshot config → create/enable a `wifi-iface` STA +section on `radio1` → pause `wlan1mon` hop (reusing the existing pause/ +resume mechanism used by radio1 APs) → verify association truthfully +(iw + daemon state). Failure at any step → rollback snapshot + event. +`set-role(attack)`: STA disabled → monitor/AP stack restored. + +Honest tradeoff surfaced in UI text: while the phy1 uplink associates, phy1 +is pinned to the uplink channel — 5 GHz recon is limited to that channel; +2.4 GHz remains fully hoppable. The dashboard RF chip shows +`PHY0: OPS · PHY1: UPLINK ch36` style state. + +### 4. Supervisor + +A sampler thread inside the existing backend process: +- Every 30 s, passive reads only (`pidof`, `iw dev`, `/proc/meminfo`, + interface flags) — no pineapd socket pings (crash source 5). +- SIGSEGV trend via throttled `logread | grep -c` scan every 5 min. +- Hysteresis actions: pineapd absent for 2 consecutive samples → + `/etc/init.d/pineapd restart` + guard verify; monitor dropped → re-raise + (`ip link set up`); memory >85% sustained 5 samples → alert only + (no aggressive action). +- Event journal: JSONL entries `{ts, kind, sev, msg, meta}` rotated at + 5 MB × 4 files. +- Unexpected-reboot detection via `/mmc/mk8/boot.marker` (clean shutdown + clears it; boot increments counter when present). +- UI: Dashboard health panel gains recent-events feed + reliability + counters (boots, unexpected boots, rollbacks, restarts, guard fixes); + `/api/health` extended accordingly. + +### 5. Deploy hardening + +- Single-source version: top-level `VERSION` file consumed by build step to + stamp `_hak5_manifest.json`, `payload.sh`, and `server.py` banner; no more + hand-synced numbers. +- Atomic deploys in `scripts/deploy.sh`: stage upload to `/tmp/mk8-stage` + → sha256 manifest verification → stop service → swap into + `/mmc/mk8/releases/current` (previous kept) → start → post-deploy + self-check (version match + local health probe). Failed self-check → + previous release restored automatically. +- Payload install continues to work from overlay paths for compatibility; + release dir on `/mmc` is symlinked as the service target. + +### 6. Testing & verification + +Unit tests (existing pattern: stdlib unittest, mocks, one module per +process): reconciler diff-only idempotence; profile save/restore roundtrip; +role exclusivity + hop pause/resume; watchdog decision table (probe +outcomes × thresholds); deploy staging flow with mocked SSH; supervisor +sampling parsers and hysteresis. + +New `scripts/smoke.sh` (on-device, read-only unless explicitly flagged): +boot persistence of guards, guard enforcement after writing factory-default +bad values (then restoring), role-switch cycle uplink↔attack, rollback +watchdog trigger against a deliberately stopped port (safe variant), deploy +version match, journal integrity. + +## Failure modes & handling + +| Failure | Handling | +|---|---| +| Factory-default bad UCI at boot | Guard fixes before pineapd starts | +| pineapd crash-loop despite guards | procd respawn + supervisor restart w/ backoff + alert | +| Risky op kills management plane | Local-liveness rollback watchdog restores snapshot | +| Deploy uploads corrupt payload | sha256 gate before swap | +| New payload fails health check | Auto-rollback to previous release | +| Overlay wiped by firmware upgrade | Reinstall payload; profiles/journal/history survive on /mmc | +| Memory exhaustion | Sustained-watermark alerts; no destructive automation | + +## Security + +No new network exposure; all new endpoints behind existing auth; watchdog +and guard scripts are root-owned, written atomically; no credentials stored +in repo or journal metadata (SSIDs/BSSIDs only). + +## Out-of-scope notes + +Custom firmware remains a documented future experiment (extraction recipe +captured in session history: uImage kernel @0, squashfs-xz rootfs +@0x2615dc; bootloader signature behavior unverified).