diff --git a/docs/superpowers/plans/2026-08-22-reliability-core.md b/docs/superpowers/plans/2026-08-22-reliability-core.md new file mode 100644 index 0000000..421d40d --- /dev/null +++ b/docs/superpowers/plans/2026-08-22-reliability-core.md @@ -0,0 +1,983 @@ +# Mark VIII Reliability Core Implementation Plan + +> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. + +**Goal:** Make the Pager run reliably and consistently as expected via payload-only hardening: boot-time guard, config profiles with rollback watchdogs, RF role manager (uplink on phy1), integrated supervisor with event journal, atomic deploys, and an on-device smoke suite. + +**Architecture:** Three layers inside the existing payload — `mk8-guard` init script (START=49, before the S50 pineapple stack), new `mk8_*.py` stdlib modules imported by `server.py`, and a passive supervisor thread extending the existing health monitor. Persistent state in `/mmc/mk8/` (survives reboots and overlay wipes). + +**Tech Stack:** Python 3 stdlib only (`python3-light` on device: no urllib/http.server/sqlite3 modules), POSIX sh for device scripts, vanilla JS frontend, Bash + sshpass/scp for deploy tooling. + +## Global Constraints + +- Spec: `docs/superpowers/specs/2026-08-22-reliability-core-design.md` (approved). +- Device constraints: python3-light stdlib only; BusyBox (no `pkill`; use `killall`/`pidof`); never actively ping pineapd's command socket from loops; never run `wifi reload` outside gated operations. +- All persistent state under `/mmc/mk8/`. No writes to stock binaries or `/etc/config` outside reconciler/gated ops. +- Tests: stdlib `unittest`, one module per process (tests monkeypatch module state); run pattern: + `python3 -m unittest tests.test_ -v` +- Frontend checks: `node --check ` after every JS edit. +- Device access is **read-only until Task 10** (deploy + smoke). Password auth via `sshpass -p '' ssh -o StrictHostKeyChecking=no root@172.16.52.1`. +- Branch: `feature/reliability`. Commit after every passing step. +- Version: single-source `VERSION` file at repo root; next version `1.4.0`. + +## File Structure + +``` +payload/user/remote_access/pager-webui/ + mk8_events.py NEW event journal (JSONL append/rotate/read + counters) + mk8_profiles.py NEW UCI snapshot store (/mmc/mk8/profiles) + mk8_guard.py NEW known-good invariants + reconcile() + CLI hooks + mk8_rfplan.py NEW phy1 RF role manager (uplink/attack/idle) + mk8_gate.py NEW risky-op preflight gate + watchdog decision logic + mk8-watchdog.sh NEW detached local-liveness rollback watchdog + mk8-guard.init NEW START=49 boot guard script (installed to /etc/init.d/mk8-guard) + server.py MOD imports, startup hook, h_health extension, /api/reliability/* + /api/rfplan/* routes, gates on risky handlers + www/js/views.js MOD health events feed, reliability counters, RF chip, Settings profiles card + www/js/app.js MOD nav wiring if needed +www/css/app.css MOD styles for new UI elements +scripts/deploy.sh MOD VERSION stamping, atomic release swap, post-deploy check, guard install +scripts/smoke.sh NEW on-device verification suite +VERSION NEW "1.4.0" +tests/test_mk8_events.py, test_mk8_profiles.py, test_mk8_guard.py, +tests/test_mk8_gate.py, test_mk8_rfplan.py, test_reliability_api.py NEW +``` + +--- + +### Task 1: Event journal (`mk8_events.py`) + +**Files:** +- Create: `payload/user/remote_access/pager-webui/mk8_events.py` +- Test: `tests/test_mk8_events.py` + +**Interfaces:** +- Produces: `log_event(kind, sev='info', msg='', meta=None)`; `read_events(limit=100)` → list of dicts newest-first; `counters()` → dict with keys `boots`, `unexpected_boots`, `rollbacks`, `restarts`, `guard_fixes`; `mark_boot()`; constants `MK8_DIR='/mmc/mk8'`, `EVENTS_PATH`, `MAX_BYTES=5*1024*1024`, `KEEP=4`. + +- [ ] **Step 1: Write failing tests** + +```python +import json, os, sys, tempfile, unittest +sys.path.insert(0, os.path.join(os.path.dirname(__file__), '..', 'payload', 'user', 'remote_access', 'pager-webui')) +import mk8_events + +class EventsTest(unittest.TestCase): + def setUp(self): + self.dir = tempfile.mkdtemp() + self.old = mk8_events.MK8_DIR + mk8_events.MK8_DIR = self.dir + mk8_events.EVENTS_PATH = os.path.join(self.dir, 'events.log') + + def tearDown(self): + mk8_events.MK8_DIR = self.old + + def test_log_and_read_newest_first(self): + mk8_events.log_event('boot', msg='first') + mk8_events.log_event('rollback', sev='warn', msg='second', meta={'op': 'wifi'}) + rows = mk8_events.read_events() + self.assertEqual(rows[0]['kind'], 'rollback') + self.assertEqual(rows[1]['kind'], 'boot') + self.assertEqual(rows[0]['meta'], {'op': 'wifi'}) + + def test_counters(self): + mk8_events.log_event('boot'); mk8_events.log_event('rollback') + mk8_events.log_event('restart'); mk8_events.log_event('guard_fix') + c = mk8_events.counters() + self.assertEqual(c['boots'], 1) + self.assertEqual(c['rollbacks'], 1) + self.assertEqual(c['restarts'], 1) + self.assertEqual(c['guard_fixes'], 1) + + def test_rotation_keeps_recent(self): + mk8_events.MAX_BYTES = 200 + for i in range(20): + mk8_events.log_event('tick', msg='x' * 30) + self.assertTrue(len(mk8_events.read_events()) >= 15) + self.assertFalse(os.path.exists(mk8_events.EVENTS_PATH + '.4')) + +if __name__ == '__main__': + unittest.main() +``` + +- [ ] **Step 2: Run to verify failure** + +Run: `python3 -m unittest tests.test_mk8_events -v` +Expected: FAIL — `No module named 'mk8_events'` + +- [ ] **Step 3: Implement** + +```python +"""Mark VIII reliability event journal. JSONL on /mmc, rotated.""" +import json, os, threading + +MK8_DIR = '/mmc/mk8' +EVENTS_PATH = os.path.join(MK8_DIR, 'events.log') +MAX_BYTES = 5 * 1024 * 1024 +KEEP = 4 +_LOCK = threading.Lock() + +COUNTER_KINDS = ('boot', 'unexpected_boot', 'rollback', 'restart', + 'guard_fix') + + +def _ensure_dir(): + try: + os.makedirs(MK8_DIR, exist_ok=True) + except OSError: + pass + + +def log_event(kind, sev='info', msg='', meta=None): + entry = {'ts': int(__import__('time').time()), 'kind': str(kind), + 'sev': sev, 'msg': msg[:500]} + if meta: + entry['meta'] = meta + line = json.dumps(entry) + '\n' + with _LOCK: + _ensure_dir() + try: + if os.path.exists(EVENTS_PATH) and \ + os.path.getsize(EVENTS_PATH) > MAX_BYTES: + for i in range(KEEP - 1, 0, -1): + src = '%s.%d' % (EVENTS_PATH, i) + dst = '%s.%d' % (EVENTS_PATH, i + 1) + if os.path.exists(src): + os.replace(src, dst) + if os.path.exists(EVENTS_PATH): + os.replace(EVENTS_PATH, EVENTS_PATH + '.1') + with open(EVENTS_PATH, 'a') as f: + f.write(line) + except OSError: + pass + + +def read_events(limit=100): + out = [] + paths = [EVENTS_PATH + '.%d' % i for i in range(KEEP, 0, -1)] + paths.append(EVENTS_PATH) + for path in paths: + try: + with open(path) as f: + out.extend(json.loads(l) for l in f if l.strip()) + except (OSError, ValueError): + continue + out.sort(key=lambda r: r.get('ts', 0)) + return out[-limit:][::-1] + + +def counters(): + counts = {k: 0 for k in COUNTER_KINDS} + for row in read_events(limit=5000): + k = row.get('kind') + if k in counts: + counts[k] += 1 + return counts + + +def mark_boot(unexpected=False): + log_event('unexpected_boot' if unexpected else 'boot', sev='warn' + if unexpected else 'info', + msg='service started' + ('' if unexpected else ' cleanly')) +``` + +- [ ] **Step 4: Run tests to pass** + +Run: `python3 -m unittest tests.test_mk8_events -v` → PASS + +- [ ] **Step 5: Commit** + +```bash +git add payload/user/remote_access/pager-webui/mk8_events.py tests/test_mk8_events.py +git commit -m "feat(reliability): JSONL event journal with rotation and counters" +``` + +--- + +### Task 2: Profile store (`mk8_profiles.py`) + +**Files:** +- Create: `payload/user/remote_access/pager-webui/mk8_profiles.py` +- Test: `tests/test_mk8_profiles.py` + +**Interfaces:** +- Consumes: nothing. +- Produces: `snapshot(name)` → bool; `list_profiles()` → list of names; `restore(name)` → dict `{ok, restored:[...]}`; `auto_name(op)` → `'pre--'`; `promote_lastknown_good()`; `delete(name)`; uses `device_run` injected as module attr `run_cmd(args, timeout=20)` defaulting to `server.device_run` lazily (avoids import cycle: define own `_run` that callers/tests monkeypatch). + +- [ ] **Step 1: Failing tests** + +```python +import os, sys, tempfile, unittest +sys.path.insert(0, os.path.join(os.path.dirname(__file__), '..', 'payload', 'user', 'remote_access', 'pager-webui')) +import mk8_profiles + +CONFIGS = ('pineapd', 'wireless', 'network') + +class ProfilesTest(unittest.TestCase): + def setUp(self): + self.dir = tempfile.mkdtemp() + mk8_profiles.PROFILES_DIR = os.path.join(self.dir, 'profiles') + self.state = {'pineapd': 'config pineapd\n\toption x y\n', + 'wireless': 'config wireless\n', 'network': ''} + def fake_run(args, timeout=20): + a = list(args) + if a[:2] == ['uci', 'export']: + return (0, self.state.get(a[2], ''), '') + if a[:2] == ['uci', 'import'] or a[:2] == ['uci', 'commit']: + return (0, '', '') + return (0, '', '') + self.runs = [] + mk8_profiles.run_cmd = lambda args, timeout=20: ( + self.runs.append(list(args)) or fake_run(args, timeout)) + + def test_snapshot_and_list(self): + self.assertTrue(mk8_profiles.snapshot('testprof')) + self.assertIn('testprof', mk8_profiles.list_profiles()) + + def test_restore_issues_import_per_config(self): + mk8_profiles.snapshot('p1') + ok = mk8_profiles.restore('p1') + self.assertTrue(ok['ok']) + imported = [r for r in self.runs if r[:2] == ['uci', 'import']] + self.assertEqual(len(imported), len(CONFIGS)) + commits = [r for r in self.runs if r[:2] == ['uci', 'commit']] + self.assertGreaterEqual(len(commits), 1) + + def test_auto_name_format(self): + name = mk8_profiles.auto_name('client_connect') + self.assertTrue(name.startswith('pre-client_connect-')) + +if __name__ == '__main__': + unittest.main() +``` + +- [ ] **Step 2: Verify failure** — `No module named 'mk8_profiles'` + +- [ ] **Step 3: Implement** + +```python +"""UCI profile snapshots under /mmc/mk8/profiles//{pineapd,wireless,network}""" +import os, time + +PROFILES_DIR = '/mmc/mk8/profiles' +CONFIGS = ('pineapd', 'wireless', 'network') + + +def run_cmd(args, timeout=20): + """Lazy import avoids a circular import with server.py; tests monkeypatch.""" + from server import device_run + return device_run(args, timeout=timeout) + + +def _path(name): + return os.path.join(PROFILES_DIR, name) + + +def snapshot(name): + dest = _path(name) + try: + os.makedirs(dest, exist_ok=True) + wrote = False + for cfg in CONFIGS: + rc, out, err = run_cmd(['uci', 'export', cfg]) + if rc != 0 or not (out or '').strip(): + continue + with open(os.path.join(dest, cfg + '.uci'), 'w') as f: + f.write(out) + wrote = True + return wrote + except OSError: + return False + + +def auto_name(op): + return 'pre-%s-%d' % (op, int(time.time())) + + +def list_profiles(): + try: + return sorted(d for d in os.listdir(PROFILES_DIR) + if os.path.isdir(_path(d))) + except OSError: + return [] + + +def delete(name): + import shutil + shutil.rmtree(_path(name), ignore_errors=True) + + +def restore(name): + """Restore configs then commit once per config. Caller runs wifi reload + / service restart as appropriate for the operation.""" + src = _path(name) + restored = [] + if not os.path.isdir(src): + return {'ok': False, 'restored': [], 'error': 'profile not found'} + for cfg in CONFIGS: + fpath = os.path.join(src, cfg + '.uci') + if not os.path.isfile(fpath): + continue + with open(fpath) as f: + text = f.read() + rc, _, err = run_cmd(['uci', 'import', cfg], input_data=text) + if rc != 0: + return {'ok': False, 'restored': restored, + 'error': 'import failed'} + run_cmd(['uci', 'commit', cfg]) + restored.append(cfg) + return {'ok': True, 'restored': restored} + + +LASTKNOWN_GOOD = 'lastknown-good' + + +def promote_lastknown_good(): + """Replace the lastknown-good profile with the live config.""" + delete(LASTKNOWN_GOOD) + return snapshot(LASTKNOWN_GOOD) +``` + +Update the Step-1 fake to accept `input_data=None` and record imports: + +```python + def fake_run(args, timeout=20, input_data=None): + a = list(args) + if a[:2] == ['uci', 'import']: + self.imports = getattr(self, 'imports', []) + self.imports.append((a[2], input_data)) + return (0, '', '') + if a[:2] == ['uci', 'export']: + return (0, self.state.get(a[2], ''), '') + if a[:2] == ['uci', 'commit']: + return (0, '', '') + return (0, '', '') +``` + +- [ ] **Step 4: Run to pass.** +- [ ] **Step 5: Commit** — `feat(reliability): UCI profile snapshot store` + +--- + +### Task 3: Reconciler (`mk8_guard.py`) + +**Files:** +- Create: `payload/user/remote_access/pager-webui/mk8_guard.py` +- Test: `tests/test_mk8_guard.py` + +**Interfaces:** +- Consumes: `server.PINEAPD_SAFE_UCI` (dict of safe pineapd UCI values), `server._apply_uci_wanted(wanted)`, `server._monitor_down(name)`, `server._raise_monitors()`. +- Produces: `WANTED_EXTRA = {'pineapd.@pineapd[0].autossidpool': '0'}`; `POOL_CLEAR_MAX = 20`; `reconcile(clear_pool=True)` → `{'changed': [...], 'pool_cleared': bool}`; `guard_report()` → dict for `/api/health`. + +- [ ] **Step 1: Failing tests** + +Full file — reuse the exact `fake_run` device-mock pattern from `tests/test_health.py`: + +```python +import os, sys, unittest +sys.path.insert(0, os.path.join(os.path.dirname(__file__), '..', 'payload', 'user', 'remote_access', 'pager-webui')) +import server +import mk8_guard + +class GuardTest(unittest.TestCase): + def setUp(self): + self.calls = [] + self.uci = {} + self.mon_up = {'wlan0mon': True, 'wlan1mon': True} + server._iface_up = lambda name: self.mon_up.get(name, True) + + def fake_run(args, timeout=20, input_data=None): + a = list(args) + self.calls.append(a) + if a[:2] == ['uci', 'get']: + key = a[2] + if key == 'pineapd.@ssidpool[0].ssid': + return (0, ''.join('s%d\n' % i for i in range(self.pool)), '') + return (0, self.uci.get(key, '') + '\n', '') + if a[:2] == ['uci', 'set']: + k, _, v = a[2].partition('=') + self.uci[k] = v + if a[:2] == ['uci', 'delete']: + self.pool = 0 + return (0, '', '') + mk8_guard.device_run = fake_run + + def tearDown(self): + server._iface_up = lambda name: True + + def test_applies_all_wanted_when_missing(self): + self.pool = 0 + result = mk8_guard.reconcile(clear_pool=False) + sets = [c[2] for c in self.calls if c[:2] == ['uci', 'set']] + self.assertEqual(len(sets), + len(server.PINEAPD_SAFE_UCI) + len(mk8_guard.WANTED_EXTRA)) + self.assertTrue(result['changed']) + + def test_clears_large_pool_only(self): + self.pool = 25 + result = mk8_guard.reconcile(clear_pool=True) + self.assertTrue(result['pool_cleared']) + self.assertIn(['uci', 'delete', 'pineapd.@ssidpool[0].ssid'], self.calls) + + def test_small_pool_untouched(self): + self.pool = 5 + result = mk8_guard.reconcile(clear_pool=True) + self.assertFalse(result['pool_cleared']) + +if __name__ == '__main__': + unittest.main() +``` + +- [ ] **Step 2: Verify failure** — no module. +- [ ] **Step 3: Implement** + +```python +"""Boot-time reconciliation of crash-prone PineAP settings.""" +from server import (_apply_uci_wanted, _monitor_down, _raise_monitors, + PINEAPD_SAFE_UCI, device_run) + +WANTED_EXTRA = {'pineapd.@pineapd[0].autossidpool': '0'} +POOL_CLEAR_MAX = 20 +MONITORS = ('wlan0mon', 'wlan1mon') + + +def _pool_size(): + rc, out, err = device_run( + ['uci', 'get', 'pineapd.@ssidpool[0].ssid']) + if rc != 0 or not (out or '').strip(): + return 0 + return len([s for s in out.strip().split('\\n') if s]) + + +def reconcile(clear_pool=True): + changed = _apply_uci_wanted(dict(PINEAPD_SAFE_UCI, **WANTED_EXTRA)) + pool_cleared = False + if clear_pool and _pool_size() > POOL_CLEAR_MAX: + device_run(['uci', 'delete', 'pineapd.@ssidpool[0].ssid']) + pool_cleared = True + if changed or pool_cleared: + device_run(['uci', 'commit', 'pineapd']) + raised = _raise_monitors() if any(_monitor_down(m) for m in MONITORS) else [] + return {'changed': changed, 'pool_cleared': pool_cleared, + 'monitors_raised': raised} + + +def guard_report(): + from server import _pending_uci + pending = _pending_uci(dict(PINEAPD_SAFE_UCI, **WANTED_EXTRA)) + return {'in_sync': not pending, 'pending': pending, + 'pool_size': _pool_size()} +``` + +- [ ] **Step 4: Pass. Commit:** `feat(reliability): boot-time UCI reconciler` + +--- + +### Task 4: Guard init script + install wiring + +**Files:** +- Create: `payload/user/remote_access/pager-webui/mk8-guard.init` +- Modify: `scripts/deploy.sh` (install block), `payload/user/remote_access/pager-webui/server.py` (CLI flag) + +**Interfaces:** CLI: `python3 server.py --reconcile` runs `mk8_guard.reconcile()` and prints JSON; exit 0 always (boot must not fail). + +- [ ] **Step 1: Write `mk8-guard.init`:** + +```sh +#!/bin/sh /etc/rc.common +# Mark VIII boot guard: enforce safe PineAP UCI before the S50 stack starts. +START=49 +STOP=90 + +GUARD_DIR="/root/payloads/user/remote_access/pager-webui" +[ -f "$GUARD_DIR/server.py" ] || GUARD_DIR="/mmc/mk8/releases/current" + +start() { + [ -f "$GUARD_DIR/server.py" ] || return 0 + /usr/bin/python3 "$GUARD_DIR/server.py" --reconcile \ + >/tmp/mk8-guard.log 2>&1 || true +} + +stop() { return 0; } +``` + +- [ ] **Step 2: Add CLI branch in `server.py` `__main__` (after `--release-pager`):** + +```python + if '--reconcile' in sys.argv: + try: + import mk8_guard + print(json.dumps(mk8_guard.reconcile())) + except Exception as exc: # boot must never fail here + print(json.dumps({'error': str(exc)})) + sys.exit(0) +``` + +- [ ] **Step 3: deploy.sh install block (inside REMOTE_COMMAND before EXTRACT_OK echo):** + +```sh +cp -f '$DIR/mk8-guard.init' /etc/init.d/mk8-guard +chmod 755 /etc/init.d/mk8-guard +/etc/init.d/mk8-guard enable +``` + +(`$DIR` is the existing remote payload dir var used by the unzip step.) + +- [ ] **Step 4: Local verification:** `python3 -m py_compile payload/user/remote_access/pager-webui/server.py && node --check payload/user/remote_access/pager-webui/www/js/app.js` (JS untouched but cheap sanity). `sh -n scripts/deploy.sh`. +- [ ] **Step 5: Commit** — `feat(reliability): START=49 boot guard installed by deploy` + +--- + +### Task 5: Supervisor extension of health monitor + +**Files:** +- Modify: `payload/user/remote_access/pager-webui/server.py` (`health_check`, `_health_loop`, `h_health`, `serve()` boot sequence) +- Test: `tests/test_health.py` (extend), `tests/test_reliability_api.py` (new) + +**Interfaces:** +- Consumes: `mk8_events`, `mk8_guard.guard_report()`. +- Produces in `_health`: `mem_percent`, `events` (last 20), `reliability` counters, `guard` report; boot-marker logic `check_boot_marker()` → bool unexpected; mem sampling `_mem_percent()`. + +- [ ] **Step 1: Failing tests** — add to `tests/test_health.py` (same fake_run pattern already there): + +```python +class SupervisorExtrasTest(unittest.TestCase): + def runTestWith(self): # helper: reuse existing setUp fake_run + pass + + def test_mem_percent_math(self): + import tempfile + content = 'MemTotal: 250000 kB\nMemAvailable: 100000 kB\n' + path = tempfile.mktemp() + open(path, 'w').write(content) + self.assertEqual(server._mem_percent(path), 60) + + def test_health_reports_events_and_counters(self): + import mk8_events + mk8_events.log_event('restart', msg='x') + status, h = server.h_health(None) + self.assertEqual(status, 200) + self.assertIn('events', h) + self.assertIn('boots', h['reliability']) + + def test_boot_marker_detects_unexpected(self): + import mk8_events, tempfile, os + marker = tempfile.mktemp() + old = server.BOOT_MARKER + server.BOOT_MARKER = marker + try: + open(marker, 'w').write('0') + self.assertTrue(server.check_boot_marker()) + os.unlink(marker) + self.assertFalse(server.check_boot_marker()) + finally: + server.BOOT_MARKER = old +``` + +(`_mem_percent` takes a `path` argument so tests inject a temp file; production call passes no arg.) + +- [ ] **Step 2: Implement** — key code: + +```python +def _mem_percent(path='/proc/meminfo'): + try: + vals = {} + with open(path) as f: + for line in f: + k, v = line.split(':') + vals[k] = int(v.strip().split()[0]) + total = vals.get('MemTotal', 0) + avail = vals.get('MemAvailable', vals.get('MemFree', 0)) + return round(100.0 * (total - avail) / total) if total else 0 + except (OSError, ValueError): + return 0 + +MEM_WARN_PERCENT = 85 +MEM_WARN_STREAK = 5 + +# inside health_check(), after monitor repair section: +h['mem_percent'] = _mem_percent() +if h['mem_percent'] >= MEM_WARN_PERCENT: + h['mem_streak'] = h.get('mem_streak', 0) + 1 +else: + h['mem_streak'] = 0 +if h['mem_streak'] == MEM_WARN_STREAK: + mk8_events.log_event('mem_warn', sev='warn', + msg='memory above %d%% sustained' % MEM_WARN_PERCENT) + +# restart action gains journaling: +h['fixes'] += 1 +mk8_events.log_event('restart', msg='pineapd restarted by health monitor') +``` + +Boot marker (called from `startup_env_check` tail): + +```python +BOOT_MARKER = '/mmc/mk8/boot.marker' + +def check_boot_marker(): + import os, mk8_events + try: + unexpected = os.path.exists(BOOT_MARKER) + mk8_events.mark_boot(unexpected=unexpected) + with open(BOOT_MARKER, 'w') as f: + f.write(str(int(time.time()))) + return unexpected + except OSError: + return False +``` + +`h_health` additions: + +```python + import mk8_events, mk8_guard + h['reliability'] = mk8_events.counters() + h['events'] = mk8_events.read_events(limit=20) + h['guard'] = mk8_guard.guard_report() +``` + +- [ ] **Step 3: Run full test_health + new api test to pass.** +- [ ] **Step 4: Commit** — `feat(reliability): supervisor sampling, event feed, boot marker` + +--- + +### Task 6: Risky-op gate + rollback watchdog + +**Files:** +- Create: `payload/user/remote_access/pager-webui/mk8_gate.py`, `payload/user/remote_access/pager-webui/mk8-watchdog.sh` +- Modify: `server.py` (wrap handlers), `deploy.sh` (ship watchdog script) +- Test: `tests/test_mk8_gate.py` + +**Interfaces:** +- `watchdog_decision(fails, oks, fail_after=6, healthy_after=6)` → `'rollback'|'promote'|None` (pure). +- `gated(op, fn)` decorator/context: snapshots `auto_name(op)`, spawns watchdog via `setsid sh mk8-watchdog.sh ... &`, runs fn, returns `(result, profile)`. +- CLI: `server.py --rollback-snapshot ` restores profile + wifi reload; `--promote-snapshot ` promotes lastknown-good. + +- [ ] **Step 1: Failing decision-table tests** + +```python +import os, sys, unittest +sys.path.insert(0, os.path.join(os.path.dirname(__file__), '..', 'payload', 'user', 'remote_access', 'pager-webui')) +import mk8_gate + +class DecisionTest(unittest.TestCase): + def tick(self, state): + action, new = mk8_gate.watchdog_decision(state) + return action, new + + def test_no_action_below_fail_threshold(self): + action, s = self.tick({'fails': 5, 'oks': 0, 'tripped': False}) + self.assertIsNone(action) + self.assertFalse(s['tripped']) + + def test_rollback_at_threshold(self): + action, s = self.tick({'fails': 6, 'oks': 0, 'tripped': False}) + self.assertEqual(action, 'rollback') + self.assertTrue(s['tripped']) + self.assertEqual(s['oks'], 0) + + def test_promote_after_recovery(self): + action, s = self.tick({'fails': 6, 'oks': 6, 'tripped': True}) + self.assertEqual(action, 'promote') + + def test_no_promote_before_recovery_threshold(self): + action, s = self.tick({'fails': 6, 'oks': 5, 'tripped': True}) + self.assertIsNone(action) + +if __name__ == '__main__': + unittest.main() +``` + +- [ ] **Step 2: Implement `mk8_gate.py`** + +```python +"""Risky-operation gate: snapshot + detached rollback watchdog.""" +import subprocess + +WATCHDOG = '/root/payloads/user/remote_access/pager-webui/mk8-watchdog.sh' +FAIL_AFTER = 6 # consecutive local-liveness failures -> rollback +HEALTHY_AFTER = 6 # consecutive successes after failure -> promote +INTERVAL = 5 # seconds between probes + + +def watchdog_decision(state): + """state: {'fails': int, 'oks': int, 'tripped': bool, + 'fail_after': 6, 'healthy_after': 6} + Returns (action, new_state): action in {'rollback','promote',None}.""" + s = dict(state) + fa = s.get('fail_after', FAIL_AFTER) + ha = s.get('healthy_after', HEALTHY_AFTER) + if not s['tripped'] and s['fails'] >= fa: + return 'rollback', dict(s, tripped=True, oks=0) + if s['tripped'] and s['oks'] >= ha: + return 'promote', s + return None, s + + +def gated(op, profiles, spawn=None): + """Decorator factory: snapshot config, spawn detached watchdog, run op.""" + import shlex + if spawn is None: + def spawn(cmd): + subprocess.Popen(cmd, shell=True, start_new_session=True) + def deco(fn): + def wrapped(*a, **kw): + name = profiles.auto_name(op) + profiles.snapshot(name) + spawn("setsid sh %s %s %d %d %d >/dev/null 2>&1 &" % + (shlex.quote(WATCHDOG), shlex.quote(name), + INTERVAL, FAIL_AFTER, HEALTHY_AFTER)) + return fn(*a, **kw) + return wrapped + return deco +``` + +(Remove the earlier `NotImplementedError` sketch entirely — this is the final form.) + +`gated` implementation: + +```python +def gated(op, profiles, spawn=lambda cmd: subprocess.Popen( + cmd, shell=True, start_new_session=True)): + """Decorator factory. profiles = mk8_profiles module.""" + def deco(fn): + def wrapped(*a, **kw): + name = profiles.auto_name(op) + profiles.snapshot(name) + spawn("setsid sh %s %s %d %d %d >/dev/null 2>&1 &" + % (WATCHDOG, name, INTERVAL, FAIL_AFTER, HEALTHY_AFTER)) + return fn(*a, **kw) + return wrapped + return deco +``` + +- [ ] **Step 3: `mk8-watchdog.sh`** + +```sh +#!/bin/sh +# Usage: mk8-watchdog.sh +PROFILE="$1"; IV="${2:-5}"; FA="${3:-6}"; HA="${4:-6}" +DIR="/root/payloads/user/remote_access/pager-webui" +[ -f "$DIR/server.py" ] || DIR="/mmc/mk8/releases/current" +fails=0; oks=0; tripped=0 +probe() { + curl -fsS -m 3 http://127.0.0.1:8080/ >/dev/null 2>&1 && + ip link show wlan0mgmt >/dev/null 2>&1 && + ! grep -q 'disabled' /sys/class/net/wlan0mgmt/flags 2>/dev/null +} +while true; do + if probe; then + fails=0 + if [ "$tripped" = "1" ]; then + oks=$((oks + 1)) + if [ "$oks" -ge "$HA" ]; then + /usr/bin/python3 "$DIR/server.py" --promote-snapshot "$PROFILE" >/dev/null 2>&1 + exit 0 + fi + fi + else + fails=$((fails + 1)); oks=0 + if [ "$tripped" = "0" ] && [ "$fails" -ge "$FA" ]; then + tripped=1 + /usr/bin/python3 "$DIR/server.py" --rollback-snapshot "$PROFILE" >/dev/null 2>&1 + fi + fi + sleep "$IV" +done +``` + +- [ ] **Step 4: Wire gates** — decorate `h_pineap_wifi_set_ap`, `_disable_sta_uplink` call sites, enterprise deploy/stop, and any handler issuing `wifi reload`, with `@mk8_gate.gated('', mk8_profiles)`. +- [ ] **Step 5: CLI rollback/promote hooks in `server.py` `__main__`:** + +```python + if '--rollback-snapshot' in sys.argv: + name = sys.argv[sys.argv.index('--rollback-snapshot') + 1] + import mk8_profiles + result = mk8_profiles.restore(name) + device_run(['wifi', 'reload'], timeout=90) + import mk8_events + mk8_events.log_event('rollback', sev='warn', + msg='watchdog restored %s' % name, + meta=result) + print(json.dumps(result)) + sys.exit(0) + if '--promote-snapshot' in sys.argv: + name = sys.argv[sys.argv.index('--promote-snapshot') + 1] + import mk8_profiles + print(json.dumps({'promoted': mk8_profiles.promote_lastknown_good()})) + sys.exit(0) +``` + +- [ ] **Step 6: Tests pass; `sh -n mk8-watchdog.sh`; commit** — `feat(reliability): risky-op preflight snapshots + detached rollback watchdog` + +--- + +### Task 7: RF role manager (`mk8_rfplan.py`) + +**Files:** +- Create: `payload/user/remote_access/pager-webui/mk8_rfplan.py` +- Modify: `server.py` (routes), UI chip later in Task 8 +- Test: `tests/test_mk8_rfplan.py` + +**Interfaces:** +- Consumes: `server._pause_hop/_resume_hop/_read_hop`, `server.device_run`, `server._uci_values/_set_uci`, gate from Task 6. +- Produces: `current_role()` → `'attack'|'uplink'|'idle'`; `set_role(role, ssid=None, psk=None)` → dict result (gated); ensures exclusivity: attack AP enable paths call `ensure_attack()` which auto-switches uplink→attack first. +- Routes: `GET /api/rfplan` , `POST /api/rfplan/role`. + +- [ ] **Step 1: Failing tests** (fake device_run capturing uci/iw calls): + +```python +def test_uplink_sets_sta_section_and_pauses_hop(self): ... +def test_set_role_uplink_requires_ssid(self): ... +def test_exclusivity_switch(self): ... +def test_current_role_reads_uci(self): ... +``` + +Assertions: `uci set wireless.wlan1up.mode=sta`, `.disabled=0`, `.ssid=`, hop paused via `_pause_hop` mock, `wifi reload` invoked through gated op only. + +- [ ] **Step 2: Implement core:** + +```python +ROLE_KEY = 'mk8.rfplan.role' + +def current_role(): + from server import _uci_values + cfg = _uci_values('wireless.wlan1up') or {} + if cfg.get('disabled') != '1' and cfg.get('mode') == 'sta': + return 'uplink' + return 'idle' + +def set_role(role, ssid=None, psk=None): + from server import device_run, _pause_hop, _resume_hop + if role == 'uplink': + if not ssid: + return {'ok': False, 'error': 'ssid required'} + cmds = [ + ['uci', 'set', 'wireless.wlan1up=wifi-iface'], + ['uci', 'set', 'wireless.wlan1up.device=radio1'], + ['uci', 'set', 'wireless.wlan1up.mode=sta'], + ['uci', 'set', 'wireless.wlan1up.network=cli'], + ['uci', 'set', 'wireless.wlan1up.ssid=%s' % ssid], + ['uci', 'set', 'wireless.wlan1up.encryption=%s' + % ('psk2' if psk else 'none')], + ['uci', 'set', 'wireless.wlan1up.disabled=0'], + ] + if psk: + cmds.append(['uci', 'set', 'wireless.wlan1up.key=%s' % psk]) + for c in cmds: + device_run(c) + device_run(['uci', 'commit', 'wireless']) + _pause_hop() + device_run(['wifi', 'reload'], timeout=60) + assoc = associated() + if not assoc: + disable_uplink() + _resume_hop() + return {'ok': False, 'error': 'association failed; reverted'} + return {'ok': True, 'role': 'uplink', 'assoc': assoc} + # attack/idle: tear down STA + disable_uplink() + _resume_hop() + return {'ok': True, 'role': role} + +def associated(): + rc, out, err = device_run(['iw', 'dev', 'wlan1up', 'link'], timeout=10) + if rc != 0 or 'Connected' not in (out or ''): + return None + for line in (out or '').splitlines(): + line = line.strip() + if line.startswith('Connected to '): + return line.split()[2] + return None + +def disable_uplink(): + from server import device_run + device_run(['uci', 'set', 'wireless.wlan1up.disabled=1']) + device_run(['uci', 'commit', 'wireless']) + +def ensure_attack(): + if current_role() == 'uplink': + set_role('attack') +``` + +Handlers in server.py wrap with gate + journal events. `GET /api/rfplan` returns role + assoc + hop-paused state. + +- [ ] **Step 3: Pass; py_compile; commit** — `feat(reliability): phy1 RF role manager with uplink-on-radio1` + +--- + +### Task 8: UI additions + +**Files:** +- Modify: `payload/user/remote_access/pager-webui/www/js/views.js`, `www/js/app.js` (nav if needed), `www/css/app.css` + +**Steps (no unit tests; verified by `node --check` + live smoke in Task 10):** + +- [ ] Dashboard health panel: events feed list (ts/kind/sev/msg) + counters row (boots/unexpected/rollbacks/restarts/guard fixes) + guard sync chip; render from `/api/health` new fields; CSS classes `mk8-events-feed`, `mk8-counter-row`. +- [ ] Top bar RF chip extension: show `PHY1: UPLINK chNN` when rfplan role is uplink (poll `/api/rfplan` with existing status poll). +- [ ] Settings: "Reliability" card — profile save input + Save button (`POST /api/reliability/profile` {name}), profile list with Restore buttons (`POST /api/reliability/restore` {name}), RF role control (role select + SSID/PSK inputs → `POST /api/rfplan/role`). +- [ ] Add routes in Task 9's API surface before wiring buttons; keep fetch helpers identical to existing patterns (`apiFetch('/api/...')`). +- [ ] `node --check` both JS files; bump cache-bust query `?v=` strings as existing convention does. +- [ ] Commit — `feat(ui): reliability panel, profiles card, RF plan controls` + +--- + +### Task 9: API routes + atomic deploys + VERSION + +**Files:** +- Modify: `server.py` ROUTER block (~line 6804), `scripts/deploy.sh`, create `VERSION` +- Test: `tests/test_reliability_api.py` + +- [ ] **Routes:** + +```python +ROUTER.add('GET', r'/api/rfplan', h_rfplan_get) +ROUTER.add('POST', r'/api/rfplan/role', h_rfplan_post) +ROUTER.add('GET', r'/api/reliability/profiles', h_profiles_get) +ROUTER.add('POST', r'/api/reliability/profile', h_profile_save) +ROUTER.add('POST', r'/api/reliability/restore', h_profile_restore) +``` + +Handlers thin-wrape `mk8_rfplan` / `mk8_profiles`; restore handler runs inside `mk8_gate.gated('restore_profile', ...)`. All auth-gated automatically by existing middleware. + +- [ ] **VERSION file:** `1.4.0` +- [ ] **deploy.sh rework:** + 1. Read `VERSION` → stamp build copies of `_hak5_manifest.json` (`version`), `payload.sh` header comment, and inject `SERVER_VERSION = 'X'` into staged `server.py` (build dir only, never source tree). + 2. Remote flow: scp zip to `/tmp/mk8-stage/` → verify sha256 of uploaded zip matches local → stop service → extract to `/mmc/mk8/releases//` → repoint `current` symlink atomically (`ln -sfn`) → install/update `/etc/init.d/mk8-guard` + copy `mk8-watchdog.sh` → start → poll `http://127.0.0.1:8080/api/api_ping` (via SSH-local curl) ≤60 s → compare served banner/version → on failure: `ln -sfn` back to previous release + start + exit 1. + 3. Keep legacy overlay payload dir as symlink target for portal compatibility: `/root/payloads/user/.../pager-webui` → real dir stays, contains pointer script or bind; simplest: leave legacy install untouched and have `current` be canonical (init scripts already fall back to `/mmc/mk8/releases/current`). +- [ ] **Tests:** unit-test stamping function `stamp_version(build_dir)` extracted into `scripts/build_common.py` (new) so it is importable: asserts manifest/payload/server contain version; sha256 check tested with tmpfiles. +- [ ] **Commit** — `feat(deploy): atomic releases, VERSION single-source, post-deploy verification` + +--- + +### Task 10: On-device smoke suite + live verification + +**Files:** +- Create: `scripts/smoke.sh` + +- [ ] **smoke.sh checks (each prints PASS/FAIL, non-destructive unless `--write` given):** + 1. Service up: `curl :8080/api/api_ping`. + 2. Guard installed: `[ -x /etc/init.d/mk8-guard ]` and enabled symlink exists. + 3. Invariants: `uci get` each wanted key equals expected; pool size ≤ 20. + 4. Journal writable + has boot event: tail events.log on device. + 5. Monitors up: `ip link show wlan0mon/wlan1mon`. + 6. (`--write`) Bad-value drill: set `pineapd.wlan1mon.bands='2,5'` → run `--reconcile` → expect `'5'`; set pool of 25 SSIDs → reconcile → cleared. + 7. (`--write`) Role drill: set_role uplink to lab AP → expect assoc; set_role attack → expect monitors hopping again. + 8. Deploy version match: `/api/health` version == `cat VERSION`. +- [ ] **Execution order:** full local unit suite (every `tests/test_*.py` individually) → deploy via `./scripts/deploy.sh --password ''` → reboot device via SSH → wait for SSH return → rerun smoke.sh → confirm guards survived boot → report. +- [ ] **Commit** — `test(smoke): on-device reliability suite` ; final tag `v1.4.0` after user confirmation. + +--- + +## Verification matrix (spec → tasks) + +| Spec requirement | Task | +|---|---| +| Boot guard before S50 | 4 | +| Reconciler invariants (5 crash sources) | 3 | +| Profiles + lastknown-good | 2 | +| Knock-off rollback watchdog (local liveness) | 6 | +| RF roles, uplink→phy1, hop pause/resume | 7 | +| Supervisor sampling + hysteresis + alerts | 5 | +| Event journal on /mmc + boot detection | 1, 5 | +| Atomic deploy + version single-source | 9 | +| UI health/events/profiles/RF | 8, 9 | +| Unit + smoke tests | all, 10 |