Mark-VIII/docs/superpowers/specs/2026-08-22-reliability-core-design.md

Mark VIII Reliability Core + Integrated Supervisor — Design

Date: 2026-08-22 Branch: feature/reliability Status: Approved by user (design sections 1–7)

Problem

Mark VIII works, but the device does not run reliably or consistently as expected. Evidence from 79 prior opencode sessions, the repository history, the factory firmware image, and the live device:

1. Factory defaults are themselves unstable. The stock firmware image (pineapplepager-firmware-1.1.0-signed.bin, OpenWrt 24.10.1, ramips/mt76x8, kernel 6.6.86) ships /etc/config/pineapd with every verified crash source enabled: wlan1mon bands '2,5' fast-hop, wlan2mon enabled+hop on a nonexistent interface, SSID pool without an explicit disable, pool target broadcast. Any reset, upgrade, or stock-UI reconvergence reintroduces pineapd SIGSEGV crash loops.
2. Fixes revert. Crash-guard UCI values applied at runtime were observed reverting to unsafe defaults after service restarts and deploys.
3. Knock-offs. Enabling client-mode uplink / wifi reload mid-operation repeatedly killed management reachability (SSH/HTTP), forcing power cycles and losing engagement state.
4. 2.4 GHz blindness. The client uplink STA on phy0 pins the radio's channel; wlan0mon cannot hop, so 2.4 GHz recon goes quiet while appearing "green" in older UI logic.
5. Reboot fragility. After reboots, stale configs and refilled pools produced broken states until v1.3.x added startup checks; ordering is still wrong: Mark VIII starts at S99, after pineapd (S50).
6. Deploy fragility. Non-atomic deploys, version confusion across three files, portal refresh failures, and one secret-leak incident.

User decisions: payload-only hardening (no firmware flashing); uplink moves to phy1; Reliability Core plus an integrated lightweight supervisor; experimental work on a branch.

Goals

A device that: survives reboot/firmware-upgrade with safe PineAP state; never loses management reachability from a UI-initiated operation; keeps 2.4 GHz operations fully available during engagements; reports radio truth; and self-heals known failure modes without human intervention.

Non-goals

Firmware repacking/flashing (parked as future experiment), new standalone processes/daemons, external databases, metrics graphing beyond counters, :1471 takeover.

Architecture

All changes live inside the existing payload. Three layers, one process:

• Guard — mk8-guard init script installed by payload.sh at START=49 (before the S50 pineapple stack that launches pineapd): enforces the known-good UCI profile before crash-prone daemons start. Idempotent; commits only differences; logs to syslog and the event journal once Mark VIII is up.
• Core — backend modules in server.py (pure stdlib, python3-light compatible): config reconciler, profile store, RF role manager, preflight/rollback gates.
• Supervisor — passive sampler thread inside server.py, capped JSONL event journal, UI health panel.

Persistent state lives in /mmc/mk8/ (ext4, 3.3 GB free) which survives reboots and firmware upgrades (overlay wipe):

/mmc/mk8/
  profiles/<name>/{pineapd,wireless,network}.uci   # named snapshots
  releases/{current,previous}/                      # atomic deploy dirs
  events.log                                        # rotated JSONL journal
  boot.marker                                       # boot counter / clean-shutdown flag

Components

1. Boot-time reconciler

Runs on every service start (and mk8-guard runs it early at boot). Compares live UCI against the built-in known-good profile; commits only differences; logs each action to the journal.

Enforced invariants (the five verified crash sources plus v1.3.x rules):

• pineapd.@ssidpool[0].disable='1' and empty ssid list
• pineapd.wlan2mon.disable='1', hop='0'
• pineapd.wlan1mon.bands='5'
• pineapd.@pineapd[0].autossidpool='0'
• wireless.dummy_radio0 parked per v1.3.1 semantics (disabled unless a scan explicitly borrows it)
• monitor interfaces present and administratively up

The reconciler never touches AP sections owned by the user (evil twins), client sections, or network/firewall config.

2. Profiles + knock-off protection

• Profile store: uci export snapshots under /mmc/mk8/profiles/. Save/restore from Settings UI; restore = write files + wifi reload + guard re-run. One profile is auto-captured as lastknown-good whenever all health checks pass for ≥5 minutes.
• Preflight gate wraps every risky operation: client-mode connect or disconnect, wifi reload, any AP enable/disable, enterprise engine start/stop, any UCI commit touching wireless/network. Sequence: auto-snapshot pre-<op>-<ts> → apply → spawn detached watchdog.
• Rollback watchdog: a small POSIX sh script started via setsid so it survives SSH/UI death. It probes local liveness only — HTTP GET to 127.0.0.1:8080/api/health and presence/state of the management interface — deliberately ignoring workstation-side reachability, which historically caused false assumptions. If local probes fail on N consecutive checks (default 6 × 5 s), it restores the pre-op snapshot, runs wifi reload, writes a ROLLBACK journal entry, and exits. Success path: after M consecutive healthy checks it promotes the snapshot to lastknown-good and exits.

3. RF role manager (uplink on phy1)

Declarative, mutually exclusive radio plan enforced server-side:

• phy0 = OPS, always: monitor hop + PineAP + 2.4 GHz evil twins. Never carries the uplink again.
• phy1 ∈ {attack, uplink, idle}: role switch API + UI control.

set-role(uplink): snapshot config → create/enable a wifi-iface STA section on radio1 → pause wlan1mon hop (reusing the existing pause/ resume mechanism used by radio1 APs) → verify association truthfully (iw + daemon state). Failure at any step → rollback snapshot + event. set-role(attack): STA disabled → monitor/AP stack restored.

Honest tradeoff surfaced in UI text: while the phy1 uplink associates, phy1 is pinned to the uplink channel — 5 GHz recon is limited to that channel; 2.4 GHz remains fully hoppable. The dashboard RF chip shows PHY0: OPS · PHY1: UPLINK ch36 style state.

4. Supervisor

A sampler thread inside the existing backend process:

• Every 30 s, passive reads only (pidof, iw dev, /proc/meminfo, interface flags) — no pineapd socket pings (crash source 5).
• SIGSEGV trend via throttled logread | grep -c scan every 5 min.
• Hysteresis actions: pineapd absent for 2 consecutive samples → /etc/init.d/pineapd restart + guard verify; monitor dropped → re-raise (ip link set <iface> up); memory >85% sustained 5 samples → alert only (no aggressive action).
• Event journal: JSONL entries {ts, kind, sev, msg, meta} rotated at 5 MB × 4 files.
• Unexpected-reboot detection via /mmc/mk8/boot.marker (clean shutdown clears it; boot increments counter when present).
• UI: Dashboard health panel gains recent-events feed + reliability counters (boots, unexpected boots, rollbacks, restarts, guard fixes); /api/health extended accordingly.

5. Deploy hardening

• Single-source version: top-level VERSION file consumed by build step to stamp _hak5_manifest.json, payload.sh, and server.py banner; no more hand-synced numbers.
• Atomic deploys in scripts/deploy.sh: stage upload to /tmp/mk8-stage → sha256 manifest verification → stop service → swap into /mmc/mk8/releases/current (previous kept) → start → post-deploy self-check (version match + local health probe). Failed self-check → previous release restored automatically.
• Payload install continues to work from overlay paths for compatibility; release dir on /mmc is symlinked as the service target.

6. Testing & verification

Unit tests (existing pattern: stdlib unittest, mocks, one module per process): reconciler diff-only idempotence; profile save/restore roundtrip; role exclusivity + hop pause/resume; watchdog decision table (probe outcomes × thresholds); deploy staging flow with mocked SSH; supervisor sampling parsers and hysteresis.

New scripts/smoke.sh (on-device, read-only unless explicitly flagged): boot persistence of guards, guard enforcement after writing factory-default bad values (then restoring), role-switch cycle uplink↔attack, rollback watchdog trigger against a deliberately stopped port (safe variant), deploy version match, journal integrity.

Failure modes & handling

Failure	Handling
Factory-default bad UCI at boot	Guard fixes before pineapd starts
pineapd crash-loop despite guards	procd respawn + supervisor restart w/ backoff + alert
Risky op kills management plane	Local-liveness rollback watchdog restores snapshot
Deploy uploads corrupt payload	sha256 gate before swap
New payload fails health check	Auto-rollback to previous release
Overlay wiped by firmware upgrade	Reinstall payload; profiles/journal/history survive on /mmc
Memory exhaustion	Sustained-watermark alerts; no destructive automation

Security

No new network exposure; all new endpoints behind existing auth; watchdog and guard scripts are root-owned, written atomically; no credentials stored in repo or journal metadata (SSIDs/BSSIDs only).

Out-of-scope notes

Custom firmware remains a documented future experiment (extraction recipe captured in session history: uImage kernel @0, squashfs-xz rootfs @0x2615dc; bootloader signature behavior unverified).