203 lines
9.5 KiB
Markdown
203 lines
9.5 KiB
Markdown
# Mark VIII Reliability Core + Integrated Supervisor — Design
|
||
|
||
Date: 2026-08-22
|
||
Branch: `feature/reliability`
|
||
Status: Approved by user (design sections 1–7)
|
||
|
||
## Problem
|
||
|
||
Mark VIII works, but the device does not run reliably or consistently as
|
||
expected. Evidence from 79 prior opencode sessions, the repository history,
|
||
the factory firmware image, and the live device:
|
||
|
||
1. **Factory defaults are themselves unstable.** The stock firmware image
|
||
(`pineapplepager-firmware-1.1.0-signed.bin`, OpenWrt 24.10.1,
|
||
ramips/mt76x8, kernel 6.6.86) ships `/etc/config/pineapd` with every
|
||
verified crash source enabled: `wlan1mon` bands `'2,5'` fast-hop,
|
||
`wlan2mon` enabled+hop on a nonexistent interface, SSID pool without an
|
||
explicit disable, pool target `broadcast`. Any reset, upgrade, or stock-UI
|
||
reconvergence reintroduces pineapd SIGSEGV crash loops.
|
||
2. **Fixes revert.** Crash-guard UCI values applied at runtime were observed
|
||
reverting to unsafe defaults after service restarts and deploys.
|
||
3. **Knock-offs.** Enabling client-mode uplink / `wifi reload` mid-operation
|
||
repeatedly killed management reachability (SSH/HTTP), forcing power
|
||
cycles and losing engagement state.
|
||
4. **2.4 GHz blindness.** The client uplink STA on phy0 pins the radio's
|
||
channel; wlan0mon cannot hop, so 2.4 GHz recon goes quiet while appearing
|
||
"green" in older UI logic.
|
||
5. **Reboot fragility.** After reboots, stale configs and refilled pools
|
||
produced broken states until v1.3.x added startup checks; ordering is
|
||
still wrong: Mark VIII starts at S99, *after* pineapd (S50).
|
||
6. **Deploy fragility.** Non-atomic deploys, version confusion across three
|
||
files, portal refresh failures, and one secret-leak incident.
|
||
|
||
User decisions: payload-only hardening (no firmware flashing); uplink moves
|
||
to phy1; Reliability Core plus an integrated lightweight supervisor;
|
||
experimental work on a branch.
|
||
|
||
## Goals
|
||
|
||
A device that: survives reboot/firmware-upgrade with safe PineAP state;
|
||
never loses management reachability from a UI-initiated operation; keeps
|
||
2.4 GHz operations fully available during engagements; reports radio truth;
|
||
and self-heals known failure modes without human intervention.
|
||
|
||
## Non-goals
|
||
|
||
Firmware repacking/flashing (parked as future experiment), new standalone
|
||
processes/daemons, external databases, metrics graphing beyond counters,
|
||
`:1471` takeover.
|
||
|
||
## Architecture
|
||
|
||
All changes live inside the existing payload. Three layers, one process:
|
||
|
||
- **Guard** — `mk8-guard` init script installed by `payload.sh` at START=49
|
||
(before the S50 pineapple stack that launches `pineapd`): enforces the
|
||
known-good UCI profile before crash-prone daemons start. Idempotent;
|
||
commits only differences; logs to syslog and the event journal once Mark
|
||
VIII is up.
|
||
- **Core** — backend modules in `server.py` (pure stdlib, python3-light
|
||
compatible): config reconciler, profile store, RF role manager,
|
||
preflight/rollback gates.
|
||
- **Supervisor** — passive sampler thread inside `server.py`, capped JSONL
|
||
event journal, UI health panel.
|
||
|
||
Persistent state lives in `/mmc/mk8/` (ext4, 3.3 GB free) which survives
|
||
reboots *and* firmware upgrades (overlay wipe):
|
||
|
||
```
|
||
/mmc/mk8/
|
||
profiles/<name>/{pineapd,wireless,network}.uci # named snapshots
|
||
releases/{current,previous}/ # atomic deploy dirs
|
||
events.log # rotated JSONL journal
|
||
boot.marker # boot counter / clean-shutdown flag
|
||
```
|
||
|
||
## Components
|
||
|
||
### 1. Boot-time reconciler
|
||
|
||
Runs on every service start (and `mk8-guard` runs it early at boot).
|
||
Compares live UCI against the built-in known-good profile; commits only
|
||
differences; logs each action to the journal.
|
||
|
||
Enforced invariants (the five verified crash sources plus v1.3.x rules):
|
||
- `pineapd.@ssidpool[0].disable='1'` and empty `ssid` list
|
||
- `pineapd.wlan2mon.disable='1'`, `hop='0'`
|
||
- `pineapd.wlan1mon.bands='5'`
|
||
- `pineapd.@pineapd[0].autossidpool='0'`
|
||
- `wireless.dummy_radio0` parked per v1.3.1 semantics (disabled unless a
|
||
scan explicitly borrows it)
|
||
- monitor interfaces present and administratively up
|
||
|
||
The reconciler never touches AP sections owned by the user (evil twins),
|
||
client sections, or network/firewall config.
|
||
|
||
### 2. Profiles + knock-off protection
|
||
|
||
- **Profile store**: `uci export` snapshots under `/mmc/mk8/profiles/`.
|
||
Save/restore from Settings UI; restore = write files + `wifi reload` +
|
||
guard re-run. One profile is auto-captured as `lastknown-good` whenever
|
||
all health checks pass for ≥5 minutes.
|
||
- **Preflight gate** wraps every risky operation: client-mode connect or
|
||
disconnect, `wifi reload`, any AP enable/disable, enterprise engine
|
||
start/stop, any UCI commit touching `wireless`/`network`. Sequence:
|
||
auto-snapshot `pre-<op>-<ts>` → apply → spawn detached watchdog.
|
||
- **Rollback watchdog**: a small POSIX sh script started via `setsid` so it
|
||
survives SSH/UI death. It probes **local** liveness only — HTTP GET to
|
||
`127.0.0.1:8080/api/health` and presence/state of the management
|
||
interface — deliberately ignoring workstation-side reachability, which
|
||
historically caused false assumptions. If local probes fail on N
|
||
consecutive checks (default 6 × 5 s), it restores the pre-op snapshot,
|
||
runs `wifi reload`, writes a `ROLLBACK` journal entry, and exits. Success
|
||
path: after M consecutive healthy checks it promotes the snapshot to
|
||
`lastknown-good` and exits.
|
||
|
||
### 3. RF role manager (uplink on phy1)
|
||
|
||
Declarative, mutually exclusive radio plan enforced server-side:
|
||
- **phy0 = OPS, always**: monitor hop + PineAP + 2.4 GHz evil twins. Never
|
||
carries the uplink again.
|
||
- **phy1 ∈ {attack, uplink, idle}**: role switch API + UI control.
|
||
|
||
`set-role(uplink)`: snapshot config → create/enable a `wifi-iface` STA
|
||
section on `radio1` → pause `wlan1mon` hop (reusing the existing pause/
|
||
resume mechanism used by radio1 APs) → verify association truthfully
|
||
(iw + daemon state). Failure at any step → rollback snapshot + event.
|
||
`set-role(attack)`: STA disabled → monitor/AP stack restored.
|
||
|
||
Honest tradeoff surfaced in UI text: while the phy1 uplink associates, phy1
|
||
is pinned to the uplink channel — 5 GHz recon is limited to that channel;
|
||
2.4 GHz remains fully hoppable. The dashboard RF chip shows
|
||
`PHY0: OPS · PHY1: UPLINK ch36` style state.
|
||
|
||
### 4. Supervisor
|
||
|
||
A sampler thread inside the existing backend process:
|
||
- Every 30 s, passive reads only (`pidof`, `iw dev`, `/proc/meminfo`,
|
||
interface flags) — no pineapd socket pings (crash source 5).
|
||
- SIGSEGV trend via throttled `logread | grep -c` scan every 5 min.
|
||
- Hysteresis actions: pineapd absent for 2 consecutive samples →
|
||
`/etc/init.d/pineapd restart` + guard verify; monitor dropped → re-raise
|
||
(`ip link set <iface> up`); memory >85% sustained 5 samples → alert only
|
||
(no aggressive action).
|
||
- Event journal: JSONL entries `{ts, kind, sev, msg, meta}` rotated at
|
||
5 MB × 4 files.
|
||
- Unexpected-reboot detection via `/mmc/mk8/boot.marker` (clean shutdown
|
||
clears it; boot increments counter when present).
|
||
- UI: Dashboard health panel gains recent-events feed + reliability
|
||
counters (boots, unexpected boots, rollbacks, restarts, guard fixes);
|
||
`/api/health` extended accordingly.
|
||
|
||
### 5. Deploy hardening
|
||
|
||
- Single-source version: top-level `VERSION` file consumed by build step to
|
||
stamp `_hak5_manifest.json`, `payload.sh`, and `server.py` banner; no more
|
||
hand-synced numbers.
|
||
- Atomic deploys in `scripts/deploy.sh`: stage upload to `/tmp/mk8-stage`
|
||
→ sha256 manifest verification → stop service → swap into
|
||
`/mmc/mk8/releases/current` (previous kept) → start → post-deploy
|
||
self-check (version match + local health probe). Failed self-check →
|
||
previous release restored automatically.
|
||
- Payload install continues to work from overlay paths for compatibility;
|
||
release dir on `/mmc` is symlinked as the service target.
|
||
|
||
### 6. Testing & verification
|
||
|
||
Unit tests (existing pattern: stdlib unittest, mocks, one module per
|
||
process): reconciler diff-only idempotence; profile save/restore roundtrip;
|
||
role exclusivity + hop pause/resume; watchdog decision table (probe
|
||
outcomes × thresholds); deploy staging flow with mocked SSH; supervisor
|
||
sampling parsers and hysteresis.
|
||
|
||
New `scripts/smoke.sh` (on-device, read-only unless explicitly flagged):
|
||
boot persistence of guards, guard enforcement after writing factory-default
|
||
bad values (then restoring), role-switch cycle uplink↔attack, rollback
|
||
watchdog trigger against a deliberately stopped port (safe variant), deploy
|
||
version match, journal integrity.
|
||
|
||
## Failure modes & handling
|
||
|
||
| Failure | Handling |
|
||
|---|---|
|
||
| Factory-default bad UCI at boot | Guard fixes before pineapd starts |
|
||
| pineapd crash-loop despite guards | procd respawn + supervisor restart w/ backoff + alert |
|
||
| Risky op kills management plane | Local-liveness rollback watchdog restores snapshot |
|
||
| Deploy uploads corrupt payload | sha256 gate before swap |
|
||
| New payload fails health check | Auto-rollback to previous release |
|
||
| Overlay wiped by firmware upgrade | Reinstall payload; profiles/journal/history survive on /mmc |
|
||
| Memory exhaustion | Sustained-watermark alerts; no destructive automation |
|
||
|
||
## Security
|
||
|
||
No new network exposure; all new endpoints behind existing auth; watchdog
|
||
and guard scripts are root-owned, written atomically; no credentials stored
|
||
in repo or journal metadata (SSIDs/BSSIDs only).
|
||
|
||
## Out-of-scope notes
|
||
|
||
Custom firmware remains a documented future experiment (extraction recipe
|
||
captured in session history: uImage kernel @0, squashfs-xz rootfs
|
||
@0x2615dc; bootloader signature behavior unverified).
|