# Mark VIII Reliability Core + Integrated Supervisor — Design

Date: 2026-08-22
Branch: `feature/reliability`
Status: Approved by user (design sections 1–7)

## Problem

Mark VIII works, but the device does not run reliably or consistently as
expected. Evidence from 79 prior opencode sessions, the repository history,
the factory firmware image, and the live device:

1. **Factory defaults are themselves unstable.** The stock firmware image
   (`pineapplepager-firmware-1.1.0-signed.bin`, OpenWrt 24.10.1,
   ramips/mt76x8, kernel 6.6.86) ships `/etc/config/pineapd` with every
   verified crash source enabled: `wlan1mon` bands `'2,5'` fast-hop,
   `wlan2mon` enabled+hop on a nonexistent interface, SSID pool without an
   explicit disable, pool target `broadcast`. Any reset, upgrade, or stock-UI
   reconvergence reintroduces pineapd SIGSEGV crash loops.
2. **Fixes revert.** Crash-guard UCI values applied at runtime were observed
   reverting to unsafe defaults after service restarts and deploys.
3. **Knock-offs.** Enabling client-mode uplink / `wifi reload` mid-operation
   repeatedly killed management reachability (SSH/HTTP), forcing power
   cycles and losing engagement state.
4. **2.4 GHz blindness.** The client uplink STA on phy0 pins the radio's
   channel; wlan0mon cannot hop, so 2.4 GHz recon goes quiet while appearing
   "green" in older UI logic.
5. **Reboot fragility.** After reboots, stale configs and refilled pools
   produced broken states until v1.3.x added startup checks; ordering is
   still wrong: Mark VIII starts at S99, *after* pineapd (S50).
6. **Deploy fragility.** Non-atomic deploys, version confusion across three
   files, portal refresh failures, and one secret-leak incident.

User decisions: payload-only hardening (no firmware flashing); uplink moves
to phy1; Reliability Core plus an integrated lightweight supervisor;
experimental work on a branch.

## Goals

A device that: survives reboot/firmware-upgrade with safe PineAP state;
never loses management reachability from a UI-initiated operation; keeps
2.4 GHz operations fully available during engagements; reports radio truth;
and self-heals known failure modes without human intervention.

## Non-goals

Firmware repacking/flashing (parked as future experiment), new standalone
processes/daemons, external databases, metrics graphing beyond counters,
`:1471` takeover.

## Architecture

All changes live inside the existing payload. Three layers, one process:

- **Guard** — `mk8-guard` init script installed by `payload.sh` at START=49
  (before the S50 pineapple stack that launches `pineapd`): enforces the
  known-good UCI profile before crash-prone daemons start. Idempotent;
  commits only differences; logs to syslog and the event journal once Mark
  VIII is up.
- **Core** — backend modules in `server.py` (pure stdlib, python3-light
  compatible): config reconciler, profile store, RF role manager,
  preflight/rollback gates.
- **Supervisor** — passive sampler thread inside `server.py`, capped JSONL
  event journal, UI health panel.

Persistent state lives in `/mmc/mk8/` (ext4, 3.3 GB free) which survives
reboots *and* firmware upgrades (overlay wipe):

```
/mmc/mk8/
  profiles/<name>/{pineapd,wireless,network}.uci   # named snapshots
  releases/{current,previous}/                      # atomic deploy dirs
  events.log                                        # rotated JSONL journal
  boot.marker                                       # boot counter / clean-shutdown flag
```

## Components

### 1. Boot-time reconciler

Runs on every service start (and `mk8-guard` runs it early at boot).
Compares live UCI against the built-in known-good profile; commits only
differences; logs each action to the journal.

Enforced invariants (the five verified crash sources plus v1.3.x rules):
- `pineapd.@ssidpool[0].disable='1'` and empty `ssid` list
- `pineapd.wlan2mon.disable='1'`, `hop='0'`
- `pineapd.wlan1mon.bands='5'`
- `pineapd.@pineapd[0].autossidpool='0'`
- `wireless.dummy_radio0` parked per v1.3.1 semantics (disabled unless a
  scan explicitly borrows it)
- monitor interfaces present and administratively up

The reconciler never touches AP sections owned by the user (evil twins),
client sections, or network/firewall config.

### 2. Profiles + knock-off protection

- **Profile store**: `uci export` snapshots under `/mmc/mk8/profiles/`.
  Save/restore from Settings UI; restore = write files + `wifi reload` +
  guard re-run. One profile is auto-captured as `lastknown-good` whenever
  all health checks pass for ≥5 minutes.
- **Preflight gate** wraps every risky operation: client-mode connect or
  disconnect, `wifi reload`, any AP enable/disable, enterprise engine
  start/stop, any UCI commit touching `wireless`/`network`. Sequence:
  auto-snapshot `pre-<op>-<ts>` → apply → spawn detached watchdog.
- **Rollback watchdog**: a small POSIX sh script started via `setsid` so it
  survives SSH/UI death. It probes **local** liveness only — HTTP GET to
  `127.0.0.1:8080/api/health` and presence/state of the management
  interface — deliberately ignoring workstation-side reachability, which
  historically caused false assumptions. If local probes fail on N
  consecutive checks (default 6 × 5 s), it restores the pre-op snapshot,
  runs `wifi reload`, writes a `ROLLBACK` journal entry, and exits. Success
  path: after M consecutive healthy checks it promotes the snapshot to
  `lastknown-good` and exits.

### 3. RF role manager (uplink on phy1)

Declarative, mutually exclusive radio plan enforced server-side:
- **phy0 = OPS, always**: monitor hop + PineAP + 2.4 GHz evil twins. Never
  carries the uplink again.
- **phy1 ∈ {attack, uplink, idle}**: role switch API + UI control.

`set-role(uplink)`: snapshot config → create/enable a `wifi-iface` STA
section on `radio1` → pause `wlan1mon` hop (reusing the existing pause/
resume mechanism used by radio1 APs) → verify association truthfully
(iw + daemon state). Failure at any step → rollback snapshot + event.
`set-role(attack)`: STA disabled → monitor/AP stack restored.

Honest tradeoff surfaced in UI text: while the phy1 uplink associates, phy1
is pinned to the uplink channel — 5 GHz recon is limited to that channel;
2.4 GHz remains fully hoppable. The dashboard RF chip shows
`PHY0: OPS · PHY1: UPLINK ch36` style state.

### 4. Supervisor

A sampler thread inside the existing backend process:
- Every 30 s, passive reads only (`pidof`, `iw dev`, `/proc/meminfo`,
  interface flags) — no pineapd socket pings (crash source 5).
- SIGSEGV trend via throttled `logread | grep -c` scan every 5 min.
- Hysteresis actions: pineapd absent for 2 consecutive samples →
  `/etc/init.d/pineapd restart` + guard verify; monitor dropped → re-raise
  (`ip link set <iface> up`); memory >85% sustained 5 samples → alert only
  (no aggressive action).
- Event journal: JSONL entries `{ts, kind, sev, msg, meta}` rotated at
  5 MB × 4 files.
- Unexpected-reboot detection via `/mmc/mk8/boot.marker` (clean shutdown
  clears it; boot increments counter when present).
- UI: Dashboard health panel gains recent-events feed + reliability
  counters (boots, unexpected boots, rollbacks, restarts, guard fixes);
  `/api/health` extended accordingly.

### 5. Deploy hardening

- Single-source version: top-level `VERSION` file consumed by build step to
  stamp `_hak5_manifest.json`, `payload.sh`, and `server.py` banner; no more
  hand-synced numbers.
- Atomic deploys in `scripts/deploy.sh`: stage upload to `/tmp/mk8-stage`
  → sha256 manifest verification → stop service → swap into
  `/mmc/mk8/releases/current` (previous kept) → start → post-deploy
  self-check (version match + local health probe). Failed self-check →
  previous release restored automatically.
- Payload install continues to work from overlay paths for compatibility;
  release dir on `/mmc` is symlinked as the service target.

### 6. Testing & verification

Unit tests (existing pattern: stdlib unittest, mocks, one module per
process): reconciler diff-only idempotence; profile save/restore roundtrip;
role exclusivity + hop pause/resume; watchdog decision table (probe
outcomes × thresholds); deploy staging flow with mocked SSH; supervisor
sampling parsers and hysteresis.

New `scripts/smoke.sh` (on-device, read-only unless explicitly flagged):
boot persistence of guards, guard enforcement after writing factory-default
bad values (then restoring), role-switch cycle uplink↔attack, rollback
watchdog trigger against a deliberately stopped port (safe variant), deploy
version match, journal integrity.

## Failure modes & handling

| Failure | Handling |
|---|---|
| Factory-default bad UCI at boot | Guard fixes before pineapd starts |
| pineapd crash-loop despite guards | procd respawn + supervisor restart w/ backoff + alert |
| Risky op kills management plane | Local-liveness rollback watchdog restores snapshot |
| Deploy uploads corrupt payload | sha256 gate before swap |
| New payload fails health check | Auto-rollback to previous release |
| Overlay wiped by firmware upgrade | Reinstall payload; profiles/journal/history survive on /mmc |
| Memory exhaustion | Sustained-watermark alerts; no destructive automation |

## Security

No new network exposure; all new endpoints behind existing auth; watchdog
and guard scripts are root-owned, written atomically; no credentials stored
in repo or journal metadata (SSIDs/BSSIDs only).

## Out-of-scope notes

Custom firmware remains a documented future experiment (extraction recipe
captured in session history: uImage kernel @0, squashfs-xz rootfs
@0x2615dc; bootloader signature behavior unverified).