Mark-VIII/docs/superpowers/plans/2026-08-22-reliability-core.md

Mark VIII Reliability Core Implementation Plan

For agentic workers: REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (- [ ]) syntax for tracking.

Goal: Make the Pager run reliably and consistently as expected via payload-only hardening: boot-time guard, config profiles with rollback watchdogs, RF role manager (uplink on phy1), integrated supervisor with event journal, atomic deploys, and an on-device smoke suite.

Architecture: Three layers inside the existing payload — mk8-guard init script (START=49, before the S50 pineapple stack), new mk8_*.py stdlib modules imported by server.py, and a passive supervisor thread extending the existing health monitor. Persistent state in /mmc/mk8/ (survives reboots and overlay wipes).

Tech Stack: Python 3 stdlib only (python3-light on device: no urllib/http.server/sqlite3 modules), POSIX sh for device scripts, vanilla JS frontend, Bash + sshpass/scp for deploy tooling.

Global Constraints

• Spec: docs/superpowers/specs/2026-08-22-reliability-core-design.md (approved).
• Device constraints: python3-light stdlib only; BusyBox (no pkill; use killall/pidof); never actively ping pineapd's command socket from loops; never run wifi reload outside gated operations.
• All persistent state under /mmc/mk8/. No writes to stock binaries or /etc/config outside reconciler/gated ops.
• Tests: stdlib unittest, one module per process (tests monkeypatch module state); run pattern: python3 -m unittest tests.test_<name> -v
• Frontend checks: node --check <file> after every JS edit.
• Device access is read-only until Task 10 (deploy + smoke). Password auth via sshpass -p '<pw>' ssh -o StrictHostKeyChecking=no root@172.16.52.1.
• Branch: feature/reliability. Commit after every passing step.
• Version: single-source VERSION file at repo root; next version 1.4.0.

File Structure

payload/user/remote_access/pager-webui/
  mk8_events.py      NEW  event journal (JSONL append/rotate/read + counters)
  mk8_profiles.py    NEW  UCI snapshot store (/mmc/mk8/profiles)
  mk8_guard.py       NEW  known-good invariants + reconcile() + CLI hooks
  mk8_rfplan.py      NEW  phy1 RF role manager (uplink/attack/idle)
  mk8_gate.py        NEW  risky-op preflight gate + watchdog decision logic
  mk8-watchdog.sh    NEW  detached local-liveness rollback watchdog
  mk8-guard.init     NEW  START=49 boot guard script (installed to /etc/init.d/mk8-guard)
  server.py          MOD  imports, startup hook, h_health extension, /api/reliability/* + /api/rfplan/* routes, gates on risky handlers
  www/js/views.js    MOD  health events feed, reliability counters, RF chip, Settings profiles card
  www/js/app.js      MOD  nav wiring if needed
www/css/app.css      MOD  styles for new UI elements
scripts/deploy.sh    MOD  VERSION stamping, atomic release swap, post-deploy check, guard install
scripts/smoke.sh     NEW  on-device verification suite
VERSION              NEW  "1.4.0"
tests/test_mk8_events.py, test_mk8_profiles.py, test_mk8_guard.py,
tests/test_mk8_gate.py, test_mk8_rfplan.py, test_reliability_api.py   NEW

---

Task 1: Event journal (mk8_events.py)

Files:

• Create: payload/user/remote_access/pager-webui/mk8_events.py
• Test: tests/test_mk8_events.py

Interfaces:

• Produces: log_event(kind, sev='info', msg='', meta=None); read_events(limit=100) → list of dicts newest-first; counters() → dict with keys boots, unexpected_boots, rollbacks, restarts, guard_fixes; mark_boot(); constants MK8_DIR='/mmc/mk8', EVENTS_PATH, MAX_BYTES=5*1024*1024, KEEP=4.

• Step 1: Write failing tests

import json, os, sys, tempfile, unittest
sys.path.insert(0, os.path.join(os.path.dirname(__file__), '..', 'payload', 'user', 'remote_access', 'pager-webui'))
import mk8_events

class EventsTest(unittest.TestCase):
    def setUp(self):
        self.dir = tempfile.mkdtemp()
        self.old = mk8_events.MK8_DIR
        mk8_events.MK8_DIR = self.dir
        mk8_events.EVENTS_PATH = os.path.join(self.dir, 'events.log')

    def tearDown(self):
        mk8_events.MK8_DIR = self.old

    def test_log_and_read_newest_first(self):
        mk8_events.log_event('boot', msg='first')
        mk8_events.log_event('rollback', sev='warn', msg='second', meta={'op': 'wifi'})
        rows = mk8_events.read_events()
        self.assertEqual(rows[0]['kind'], 'rollback')
        self.assertEqual(rows[1]['kind'], 'boot')
        self.assertEqual(rows[0]['meta'], {'op': 'wifi'})

    def test_counters(self):
        mk8_events.log_event('boot'); mk8_events.log_event('rollback')
        mk8_events.log_event('restart'); mk8_events.log_event('guard_fix')
        c = mk8_events.counters()
        self.assertEqual(c['boots'], 1)
        self.assertEqual(c['rollbacks'], 1)
        self.assertEqual(c['restarts'], 1)
        self.assertEqual(c['guard_fixes'], 1)

    def test_rotation_keeps_recent(self):
        mk8_events.MAX_BYTES = 200
        for i in range(20):
            mk8_events.log_event('tick', msg='x' * 30)
        self.assertTrue(len(mk8_events.read_events()) >= 15)
        self.assertFalse(os.path.exists(mk8_events.EVENTS_PATH + '.4'))

if __name__ == '__main__':
    unittest.main()

• Step 2: Run to verify failure

Run: python3 -m unittest tests.test_mk8_events -v Expected: FAIL — No module named 'mk8_events'

• Step 3: Implement

"""Mark VIII reliability event journal. JSONL on /mmc, rotated."""
import json, os, threading

MK8_DIR = '/mmc/mk8'
EVENTS_PATH = os.path.join(MK8_DIR, 'events.log')
MAX_BYTES = 5 * 1024 * 1024
KEEP = 4
_LOCK = threading.Lock()

COUNTER_KINDS = ('boot', 'unexpected_boot', 'rollback', 'restart',
                 'guard_fix')


def _ensure_dir():
    try:
        os.makedirs(MK8_DIR, exist_ok=True)
    except OSError:
        pass


def log_event(kind, sev='info', msg='', meta=None):
    entry = {'ts': int(__import__('time').time()), 'kind': str(kind),
             'sev': sev, 'msg': msg[:500]}
    if meta:
        entry['meta'] = meta
    line = json.dumps(entry) + '\n'
    with _LOCK:
        _ensure_dir()
        try:
            if os.path.exists(EVENTS_PATH) and \
                    os.path.getsize(EVENTS_PATH) > MAX_BYTES:
                for i in range(KEEP - 1, 0, -1):
                    src = '%s.%d' % (EVENTS_PATH, i)
                    dst = '%s.%d' % (EVENTS_PATH, i + 1)
                    if os.path.exists(src):
                        os.replace(src, dst)
                if os.path.exists(EVENTS_PATH):
                    os.replace(EVENTS_PATH, EVENTS_PATH + '.1')
            with open(EVENTS_PATH, 'a') as f:
                f.write(line)
        except OSError:
            pass


def read_events(limit=100):
    out = []
    paths = [EVENTS_PATH + '.%d' % i for i in range(KEEP, 0, -1)]
    paths.append(EVENTS_PATH)
    for path in paths:
        try:
            with open(path) as f:
                out.extend(json.loads(l) for l in f if l.strip())
        except (OSError, ValueError):
            continue
    out.sort(key=lambda r: r.get('ts', 0))
    return out[-limit:][::-1]


def counters():
    counts = {k: 0 for k in COUNTER_KINDS}
    for row in read_events(limit=5000):
        k = row.get('kind')
        if k in counts:
            counts[k] += 1
    return counts


def mark_boot(unexpected=False):
    log_event('unexpected_boot' if unexpected else 'boot', sev='warn'
              if unexpected else 'info',
              msg='service started' + ('' if unexpected else ' cleanly'))

• Step 4: Run tests to pass

Run: python3 -m unittest tests.test_mk8_events -v → PASS

• Step 5: Commit

git add payload/user/remote_access/pager-webui/mk8_events.py tests/test_mk8_events.py
git commit -m "feat(reliability): JSONL event journal with rotation and counters"

---

Task 2: Profile store (mk8_profiles.py)

Files:

• Create: payload/user/remote_access/pager-webui/mk8_profiles.py
• Test: tests/test_mk8_profiles.py

Interfaces:

• Consumes: nothing.

• Produces: snapshot(name) → bool; list_profiles() → list of names; restore(name) → dict {ok, restored:[...]}; auto_name(op) → 'pre-<op>-<ts>'; promote_lastknown_good(); delete(name); uses device_run injected as module attr run_cmd(args, timeout=20) defaulting to server.device_run lazily (avoids import cycle: define own _run that callers/tests monkeypatch).

• Step 1: Failing tests

import os, sys, tempfile, unittest
sys.path.insert(0, os.path.join(os.path.dirname(__file__), '..', 'payload', 'user', 'remote_access', 'pager-webui'))
import mk8_profiles

CONFIGS = ('pineapd', 'wireless', 'network')

class ProfilesTest(unittest.TestCase):
    def setUp(self):
        self.dir = tempfile.mkdtemp()
        mk8_profiles.PROFILES_DIR = os.path.join(self.dir, 'profiles')
        self.state = {'pineapd': 'config pineapd\n\toption x y\n',
                      'wireless': 'config wireless\n', 'network': ''}
        def fake_run(args, timeout=20):
            a = list(args)
            if a[:2] == ['uci', 'export']:
                return (0, self.state.get(a[2], ''), '')
            if a[:2] == ['uci', 'import'] or a[:2] == ['uci', 'commit']:
                return (0, '', '')
            return (0, '', '')
        self.runs = []
        mk8_profiles.run_cmd = lambda args, timeout=20: (
            self.runs.append(list(args)) or fake_run(args, timeout))

    def test_snapshot_and_list(self):
        self.assertTrue(mk8_profiles.snapshot('testprof'))
        self.assertIn('testprof', mk8_profiles.list_profiles())

    def test_restore_issues_import_per_config(self):
        mk8_profiles.snapshot('p1')
        ok = mk8_profiles.restore('p1')
        self.assertTrue(ok['ok'])
        imported = [r for r in self.runs if r[:2] == ['uci', 'import']]
        self.assertEqual(len(imported), len(CONFIGS))
        commits = [r for r in self.runs if r[:2] == ['uci', 'commit']]
        self.assertGreaterEqual(len(commits), 1)

    def test_auto_name_format(self):
        name = mk8_profiles.auto_name('client_connect')
        self.assertTrue(name.startswith('pre-client_connect-'))

if __name__ == '__main__':
    unittest.main()

• Step 2: Verify failure — No module named 'mk8_profiles'

• Step 3: Implement

"""UCI profile snapshots under /mmc/mk8/profiles/<name>/{pineapd,wireless,network}"""
import os, time

PROFILES_DIR = '/mmc/mk8/profiles'
CONFIGS = ('pineapd', 'wireless', 'network')


def run_cmd(args, timeout=20):
    """Lazy import avoids a circular import with server.py; tests monkeypatch."""
    from server import device_run
    return device_run(args, timeout=timeout)


def _path(name):
    return os.path.join(PROFILES_DIR, name)


def snapshot(name):
    dest = _path(name)
    try:
        os.makedirs(dest, exist_ok=True)
        wrote = False
        for cfg in CONFIGS:
            rc, out, err = run_cmd(['uci', 'export', cfg])
            if rc != 0 or not (out or '').strip():
                continue
            with open(os.path.join(dest, cfg + '.uci'), 'w') as f:
                f.write(out)
            wrote = True
        return wrote
    except OSError:
        return False


def auto_name(op):
    return 'pre-%s-%d' % (op, int(time.time()))


def list_profiles():
    try:
        return sorted(d for d in os.listdir(PROFILES_DIR)
                      if os.path.isdir(_path(d)))
    except OSError:
        return []


def delete(name):
    import shutil
    shutil.rmtree(_path(name), ignore_errors=True)


def restore(name):
    """Restore configs then commit once per config. Caller runs wifi reload
    / service restart as appropriate for the operation."""
    src = _path(name)
    restored = []
    if not os.path.isdir(src):
        return {'ok': False, 'restored': [], 'error': 'profile not found'}
    for cfg in CONFIGS:
        fpath = os.path.join(src, cfg + '.uci')
        if not os.path.isfile(fpath):
            continue
        with open(fpath) as f:
            text = f.read()
        rc, _, err = run_cmd(['uci', 'import', cfg], input_data=text)
        if rc != 0:
            return {'ok': False, 'restored': restored,
                    'error': 'import failed'}
        run_cmd(['uci', 'commit', cfg])
        restored.append(cfg)
    return {'ok': True, 'restored': restored}


LASTKNOWN_GOOD = 'lastknown-good'


def promote_lastknown_good():
    """Replace the lastknown-good profile with the live config."""
    delete(LASTKNOWN_GOOD)
    return snapshot(LASTKNOWN_GOOD)

Update the Step-1 fake to accept input_data=None and record imports:

        def fake_run(args, timeout=20, input_data=None):
            a = list(args)
            if a[:2] == ['uci', 'import']:
                self.imports = getattr(self, 'imports', [])
                self.imports.append((a[2], input_data))
                return (0, '', '')
            if a[:2] == ['uci', 'export']:
                return (0, self.state.get(a[2], ''), '')
            if a[:2] == ['uci', 'commit']:
                return (0, '', '')
            return (0, '', '')

• Step 4: Run to pass.
• Step 5: Commit — feat(reliability): UCI profile snapshot store

---

Task 3: Reconciler (mk8_guard.py)

Files:

• Create: payload/user/remote_access/pager-webui/mk8_guard.py
• Test: tests/test_mk8_guard.py

Interfaces:

• Consumes: server.PINEAPD_SAFE_UCI (dict of safe pineapd UCI values), server._apply_uci_wanted(wanted), server._monitor_down(name), server._raise_monitors().

• Produces: WANTED_EXTRA = {'pineapd.@pineapd[0].autossidpool': '0'}; POOL_CLEAR_MAX = 20; reconcile(clear_pool=True) → {'changed': [...], 'pool_cleared': bool}; guard_report() → dict for /api/health.

• Step 1: Failing tests

Full file — reuse the exact fake_run device-mock pattern from tests/test_health.py:

import os, sys, unittest
sys.path.insert(0, os.path.join(os.path.dirname(__file__), '..', 'payload', 'user', 'remote_access', 'pager-webui'))
import server
import mk8_guard

class GuardTest(unittest.TestCase):
    def setUp(self):
        self.calls = []
        self.uci = {}
        self.mon_up = {'wlan0mon': True, 'wlan1mon': True}
        server._iface_up = lambda name: self.mon_up.get(name, True)

        def fake_run(args, timeout=20, input_data=None):
            a = list(args)
            self.calls.append(a)
            if a[:2] == ['uci', 'get']:
                key = a[2]
                if key == 'pineapd.@ssidpool[0].ssid':
                    return (0, ''.join('s%d\n' % i for i in range(self.pool)), '')
                return (0, self.uci.get(key, '') + '\n', '')
            if a[:2] == ['uci', 'set']:
                k, _, v = a[2].partition('=')
                self.uci[k] = v
            if a[:2] == ['uci', 'delete']:
                self.pool = 0
            return (0, '', '')
        mk8_guard.device_run = fake_run

    def tearDown(self):
        server._iface_up = lambda name: True

    def test_applies_all_wanted_when_missing(self):
        self.pool = 0
        result = mk8_guard.reconcile(clear_pool=False)
        sets = [c[2] for c in self.calls if c[:2] == ['uci', 'set']]
        self.assertEqual(len(sets),
                         len(server.PINEAPD_SAFE_UCI) + len(mk8_guard.WANTED_EXTRA))
        self.assertTrue(result['changed'])

    def test_clears_large_pool_only(self):
        self.pool = 25
        result = mk8_guard.reconcile(clear_pool=True)
        self.assertTrue(result['pool_cleared'])
        self.assertIn(['uci', 'delete', 'pineapd.@ssidpool[0].ssid'], self.calls)

    def test_small_pool_untouched(self):
        self.pool = 5
        result = mk8_guard.reconcile(clear_pool=True)
        self.assertFalse(result['pool_cleared'])

if __name__ == '__main__':
    unittest.main()

• Step 2: Verify failure — no module.
• Step 3: Implement

"""Boot-time reconciliation of crash-prone PineAP settings."""
from server import (_apply_uci_wanted, _monitor_down, _raise_monitors,
                    PINEAPD_SAFE_UCI, device_run)

WANTED_EXTRA = {'pineapd.@pineapd[0].autossidpool': '0'}
POOL_CLEAR_MAX = 20
MONITORS = ('wlan0mon', 'wlan1mon')


def _pool_size():
    rc, out, err = device_run(
        ['uci', 'get', 'pineapd.@ssidpool[0].ssid'])
    if rc != 0 or not (out or '').strip():
        return 0
    return len([s for s in out.strip().split('\\n') if s])


def reconcile(clear_pool=True):
    changed = _apply_uci_wanted(dict(PINEAPD_SAFE_UCI, **WANTED_EXTRA))
    pool_cleared = False
    if clear_pool and _pool_size() > POOL_CLEAR_MAX:
        device_run(['uci', 'delete', 'pineapd.@ssidpool[0].ssid'])
        pool_cleared = True
    if changed or pool_cleared:
        device_run(['uci', 'commit', 'pineapd'])
    raised = _raise_monitors() if any(_monitor_down(m) for m in MONITORS) else []
    return {'changed': changed, 'pool_cleared': pool_cleared,
            'monitors_raised': raised}


def guard_report():
    from server import _pending_uci
    pending = _pending_uci(dict(PINEAPD_SAFE_UCI, **WANTED_EXTRA))
    return {'in_sync': not pending, 'pending': pending,
            'pool_size': _pool_size()}

• Step 4: Pass. Commit: feat(reliability): boot-time UCI reconciler

---

Task 4: Guard init script + install wiring

Files:

• Create: payload/user/remote_access/pager-webui/mk8-guard.init
• Modify: scripts/deploy.sh (install block), payload/user/remote_access/pager-webui/server.py (CLI flag)

Interfaces: CLI: python3 server.py --reconcile runs mk8_guard.reconcile() and prints JSON; exit 0 always (boot must not fail).

• Step 1: Write mk8-guard.init:

#!/bin/sh /etc/rc.common
# Mark VIII boot guard: enforce safe PineAP UCI before the S50 stack starts.
START=49
STOP=90

GUARD_DIR="/root/payloads/user/remote_access/pager-webui"
[ -f "$GUARD_DIR/server.py" ] || GUARD_DIR="/mmc/mk8/releases/current"

start() {
    [ -f "$GUARD_DIR/server.py" ] || return 0
    /usr/bin/python3 "$GUARD_DIR/server.py" --reconcile \
        >/tmp/mk8-guard.log 2>&1 || true
}

stop() { return 0; }

• Step 2: Add CLI branch in server.py __main__ (after --release-pager):

    if '--reconcile' in sys.argv:
        try:
            import mk8_guard
            print(json.dumps(mk8_guard.reconcile()))
        except Exception as exc:  # boot must never fail here
            print(json.dumps({'error': str(exc)}))
        sys.exit(0)

• Step 3: deploy.sh install block (inside REMOTE_COMMAND before EXTRACT_OK echo):

cp -f '$DIR/mk8-guard.init' /etc/init.d/mk8-guard
chmod 755 /etc/init.d/mk8-guard
/etc/init.d/mk8-guard enable

($DIR is the existing remote payload dir var used by the unzip step.)

• Step 4: Local verification: python3 -m py_compile payload/user/remote_access/pager-webui/server.py && node --check payload/user/remote_access/pager-webui/www/js/app.js (JS untouched but cheap sanity). sh -n scripts/deploy.sh.
• Step 5: Commit — feat(reliability): START=49 boot guard installed by deploy

---

Task 5: Supervisor extension of health monitor

Files:

• Modify: payload/user/remote_access/pager-webui/server.py (health_check, _health_loop, h_health, serve() boot sequence)
• Test: tests/test_health.py (extend), tests/test_reliability_api.py (new)

Interfaces:

• Consumes: mk8_events, mk8_guard.guard_report().

• Produces in _health: mem_percent, events (last 20), reliability counters, guard report; boot-marker logic check_boot_marker() → bool unexpected; mem sampling _mem_percent().

• Step 1: Failing tests — add to tests/test_health.py (same fake_run pattern already there):

class SupervisorExtrasTest(unittest.TestCase):
    def runTestWith(self):  # helper: reuse existing setUp fake_run
        pass

    def test_mem_percent_math(self):
        import tempfile
        content = 'MemTotal:  250000 kB\nMemAvailable: 100000 kB\n'
        path = tempfile.mktemp()
        open(path, 'w').write(content)
        self.assertEqual(server._mem_percent(path), 60)

    def test_health_reports_events_and_counters(self):
        import mk8_events
        mk8_events.log_event('restart', msg='x')
        status, h = server.h_health(None)
        self.assertEqual(status, 200)
        self.assertIn('events', h)
        self.assertIn('boots', h['reliability'])

    def test_boot_marker_detects_unexpected(self):
        import mk8_events, tempfile, os
        marker = tempfile.mktemp()
        old = server.BOOT_MARKER
        server.BOOT_MARKER = marker
        try:
            open(marker, 'w').write('0')
            self.assertTrue(server.check_boot_marker())
            os.unlink(marker)
            self.assertFalse(server.check_boot_marker())
        finally:
            server.BOOT_MARKER = old

(_mem_percent takes a path argument so tests inject a temp file; production call passes no arg.)

• Step 2: Implement — key code:

def _mem_percent(path='/proc/meminfo'):
    try:
        vals = {}
        with open(path) as f:
            for line in f:
                k, v = line.split(':')
                vals[k] = int(v.strip().split()[0])
        total = vals.get('MemTotal', 0)
        avail = vals.get('MemAvailable', vals.get('MemFree', 0))
        return round(100.0 * (total - avail) / total) if total else 0
    except (OSError, ValueError):
        return 0

MEM_WARN_PERCENT = 85
MEM_WARN_STREAK = 5

# inside health_check(), after monitor repair section:
h['mem_percent'] = _mem_percent()
if h['mem_percent'] >= MEM_WARN_PERCENT:
    h['mem_streak'] = h.get('mem_streak', 0) + 1
else:
    h['mem_streak'] = 0
if h['mem_streak'] == MEM_WARN_STREAK:
    mk8_events.log_event('mem_warn', sev='warn',
                         msg='memory above %d%% sustained' % MEM_WARN_PERCENT)

# restart action gains journaling:
h['fixes'] += 1
mk8_events.log_event('restart', msg='pineapd restarted by health monitor')

Boot marker (called from startup_env_check tail):

BOOT_MARKER = '/mmc/mk8/boot.marker'

def check_boot_marker():
    import os, mk8_events
    try:
        unexpected = os.path.exists(BOOT_MARKER)
        mk8_events.mark_boot(unexpected=unexpected)
        with open(BOOT_MARKER, 'w') as f:
            f.write(str(int(time.time())))
        return unexpected
    except OSError:
        return False

h_health additions:

    import mk8_events, mk8_guard
    h['reliability'] = mk8_events.counters()
    h['events'] = mk8_events.read_events(limit=20)
    h['guard'] = mk8_guard.guard_report()

• Step 3: Run full test_health + new api test to pass.
• Step 4: Commit — feat(reliability): supervisor sampling, event feed, boot marker

---

Task 6: Risky-op gate + rollback watchdog

Files:

• Create: payload/user/remote_access/pager-webui/mk8_gate.py, payload/user/remote_access/pager-webui/mk8-watchdog.sh
• Modify: server.py (wrap handlers), deploy.sh (ship watchdog script)
• Test: tests/test_mk8_gate.py

Interfaces:

• watchdog_decision(fails, oks, fail_after=6, healthy_after=6) → 'rollback'|'promote'|None (pure).

• gated(op, fn) decorator/context: snapshots auto_name(op), spawns watchdog via setsid sh mk8-watchdog.sh <profile> ... &, runs fn, returns (result, profile).

• CLI: server.py --rollback-snapshot <name> restores profile + wifi reload; --promote-snapshot <name> promotes lastknown-good.

• Step 1: Failing decision-table tests

import os, sys, unittest
sys.path.insert(0, os.path.join(os.path.dirname(__file__), '..', 'payload', 'user', 'remote_access', 'pager-webui'))
import mk8_gate

class DecisionTest(unittest.TestCase):
    def tick(self, state):
        action, new = mk8_gate.watchdog_decision(state)
        return action, new

    def test_no_action_below_fail_threshold(self):
        action, s = self.tick({'fails': 5, 'oks': 0, 'tripped': False})
        self.assertIsNone(action)
        self.assertFalse(s['tripped'])

    def test_rollback_at_threshold(self):
        action, s = self.tick({'fails': 6, 'oks': 0, 'tripped': False})
        self.assertEqual(action, 'rollback')
        self.assertTrue(s['tripped'])
        self.assertEqual(s['oks'], 0)

    def test_promote_after_recovery(self):
        action, s = self.tick({'fails': 6, 'oks': 6, 'tripped': True})
        self.assertEqual(action, 'promote')

    def test_no_promote_before_recovery_threshold(self):
        action, s = self.tick({'fails': 6, 'oks': 5, 'tripped': True})
        self.assertIsNone(action)

if __name__ == '__main__':
    unittest.main()

• Step 2: Implement mk8_gate.py

"""Risky-operation gate: snapshot + detached rollback watchdog."""
import subprocess

WATCHDOG = '/root/payloads/user/remote_access/pager-webui/mk8-watchdog.sh'
FAIL_AFTER = 6          # consecutive local-liveness failures -> rollback
HEALTHY_AFTER = 6       # consecutive successes after failure -> promote
INTERVAL = 5            # seconds between probes


def watchdog_decision(state):
    """state: {'fails': int, 'oks': int, 'tripped': bool,
               'fail_after': 6, 'healthy_after': 6}
    Returns (action, new_state): action in {'rollback','promote',None}."""
    s = dict(state)
    fa = s.get('fail_after', FAIL_AFTER)
    ha = s.get('healthy_after', HEALTHY_AFTER)
    if not s['tripped'] and s['fails'] >= fa:
        return 'rollback', dict(s, tripped=True, oks=0)
    if s['tripped'] and s['oks'] >= ha:
        return 'promote', s
    return None, s


def gated(op, profiles, spawn=None):
    """Decorator factory: snapshot config, spawn detached watchdog, run op."""
    import shlex
    if spawn is None:
        def spawn(cmd):
            subprocess.Popen(cmd, shell=True, start_new_session=True)
    def deco(fn):
        def wrapped(*a, **kw):
            name = profiles.auto_name(op)
            profiles.snapshot(name)
            spawn("setsid sh %s %s %d %d %d >/dev/null 2>&1 &" %
                  (shlex.quote(WATCHDOG), shlex.quote(name),
                   INTERVAL, FAIL_AFTER, HEALTHY_AFTER))
            return fn(*a, **kw)
        return wrapped
    return deco

(Remove the earlier NotImplementedError sketch entirely — this is the final form.)

gated implementation:

def gated(op, profiles, spawn=lambda cmd: subprocess.Popen(
        cmd, shell=True, start_new_session=True)):
    """Decorator factory. profiles = mk8_profiles module."""
    def deco(fn):
        def wrapped(*a, **kw):
            name = profiles.auto_name(op)
            profiles.snapshot(name)
            spawn("setsid sh %s %s %d %d %d >/dev/null 2>&1 &"
                  % (WATCHDOG, name, INTERVAL, FAIL_AFTER, HEALTHY_AFTER))
            return fn(*a, **kw)
        return wrapped
    return deco

• Step 3: mk8-watchdog.sh

#!/bin/sh
# Usage: mk8-watchdog.sh <profile> <interval> <fail_after> <healthy_after>
PROFILE="$1"; IV="${2:-5}"; FA="${3:-6}"; HA="${4:-6}"
DIR="/root/payloads/user/remote_access/pager-webui"
[ -f "$DIR/server.py" ] || DIR="/mmc/mk8/releases/current"
fails=0; oks=0; tripped=0
probe() {
    curl -fsS -m 3 http://127.0.0.1:8080/ >/dev/null 2>&1 &&
    ip link show wlan0mgmt >/dev/null 2>&1 &&
    ! grep -q 'disabled' /sys/class/net/wlan0mgmt/flags 2>/dev/null
}
while true; do
    if probe; then
        fails=0
        if [ "$tripped" = "1" ]; then
            oks=$((oks + 1))
            if [ "$oks" -ge "$HA" ]; then
                /usr/bin/python3 "$DIR/server.py" --promote-snapshot "$PROFILE" >/dev/null 2>&1
                exit 0
            fi
        fi
    else
        fails=$((fails + 1)); oks=0
        if [ "$tripped" = "0" ] && [ "$fails" -ge "$FA" ]; then
            tripped=1
            /usr/bin/python3 "$DIR/server.py" --rollback-snapshot "$PROFILE" >/dev/null 2>&1
        fi
    fi
    sleep "$IV"
done

• Step 4: Wire gates — decorate h_pineap_wifi_set_ap, _disable_sta_uplink call sites, enterprise deploy/stop, and any handler issuing wifi reload, with @mk8_gate.gated('<op>', mk8_profiles).
• Step 5: CLI rollback/promote hooks in server.py __main__:

    if '--rollback-snapshot' in sys.argv:
        name = sys.argv[sys.argv.index('--rollback-snapshot') + 1]
        import mk8_profiles
        result = mk8_profiles.restore(name)
        device_run(['wifi', 'reload'], timeout=90)
        import mk8_events
        mk8_events.log_event('rollback', sev='warn',
                             msg='watchdog restored %s' % name,
                             meta=result)
        print(json.dumps(result))
        sys.exit(0)
    if '--promote-snapshot' in sys.argv:
        name = sys.argv[sys.argv.index('--promote-snapshot') + 1]
        import mk8_profiles
        print(json.dumps({'promoted': mk8_profiles.promote_lastknown_good()}))
        sys.exit(0)

• Step 6: Tests pass; sh -n mk8-watchdog.sh; commit — feat(reliability): risky-op preflight snapshots + detached rollback watchdog

---

Task 7: RF role manager (mk8_rfplan.py)

Files:

• Create: payload/user/remote_access/pager-webui/mk8_rfplan.py
• Modify: server.py (routes), UI chip later in Task 8
• Test: tests/test_mk8_rfplan.py

Interfaces:

• Consumes: server._pause_hop/_resume_hop/_read_hop, server.device_run, server._uci_values/_set_uci, gate from Task 6.

• Produces: current_role() → 'attack'|'uplink'|'idle'; set_role(role, ssid=None, psk=None) → dict result (gated); ensures exclusivity: attack AP enable paths call ensure_attack() which auto-switches uplink→attack first.

• Routes: GET /api/rfplan , POST /api/rfplan/role.

• Step 1: Failing tests (fake device_run capturing uci/iw calls):

def test_uplink_sets_sta_section_and_pauses_hop(self): ...
def test_set_role_uplink_requires_ssid(self): ...
def test_exclusivity_switch(self): ...
def test_current_role_reads_uci(self): ...

Assertions: uci set wireless.wlan1up.mode=sta, .disabled=0, .ssid=<ssid>, hop paused via _pause_hop mock, wifi reload invoked through gated op only.

• Step 2: Implement core:

ROLE_KEY = 'mk8.rfplan.role'

def current_role():
    from server import _uci_values
    cfg = _uci_values('wireless.wlan1up') or {}
    if cfg.get('disabled') != '1' and cfg.get('mode') == 'sta':
        return 'uplink'
    return 'idle'

def set_role(role, ssid=None, psk=None):
    from server import device_run, _pause_hop, _resume_hop
    if role == 'uplink':
        if not ssid:
            return {'ok': False, 'error': 'ssid required'}
        cmds = [
            ['uci', 'set', 'wireless.wlan1up=wifi-iface'],
            ['uci', 'set', 'wireless.wlan1up.device=radio1'],
            ['uci', 'set', 'wireless.wlan1up.mode=sta'],
            ['uci', 'set', 'wireless.wlan1up.network=cli'],
            ['uci', 'set', 'wireless.wlan1up.ssid=%s' % ssid],
            ['uci', 'set', 'wireless.wlan1up.encryption=%s'
             % ('psk2' if psk else 'none')],
            ['uci', 'set', 'wireless.wlan1up.disabled=0'],
        ]
        if psk:
            cmds.append(['uci', 'set', 'wireless.wlan1up.key=%s' % psk])
        for c in cmds:
            device_run(c)
        device_run(['uci', 'commit', 'wireless'])
        _pause_hop()
        device_run(['wifi', 'reload'], timeout=60)
        assoc = associated()
        if not assoc:
            disable_uplink()
            _resume_hop()
            return {'ok': False, 'error': 'association failed; reverted'}
        return {'ok': True, 'role': 'uplink', 'assoc': assoc}
    # attack/idle: tear down STA
    disable_uplink()
    _resume_hop()
    return {'ok': True, 'role': role}

def associated():
    rc, out, err = device_run(['iw', 'dev', 'wlan1up', 'link'], timeout=10)
    if rc != 0 or 'Connected' not in (out or ''):
        return None
    for line in (out or '').splitlines():
        line = line.strip()
        if line.startswith('Connected to '):
            return line.split()[2]
    return None

def disable_uplink():
    from server import device_run
    device_run(['uci', 'set', 'wireless.wlan1up.disabled=1'])
    device_run(['uci', 'commit', 'wireless'])

def ensure_attack():
    if current_role() == 'uplink':
        set_role('attack')

Handlers in server.py wrap with gate + journal events. GET /api/rfplan returns role + assoc + hop-paused state.

• Step 3: Pass; py_compile; commit — feat(reliability): phy1 RF role manager with uplink-on-radio1

---

Task 8: UI additions

Files:

• Modify: payload/user/remote_access/pager-webui/www/js/views.js, www/js/app.js (nav if needed), www/css/app.css

Steps (no unit tests; verified by node --check + live smoke in Task 10):

• Dashboard health panel: events feed list (ts/kind/sev/msg) + counters row (boots/unexpected/rollbacks/restarts/guard fixes) + guard sync chip; render from /api/health new fields; CSS classes mk8-events-feed, mk8-counter-row.
• Top bar RF chip extension: show PHY1: UPLINK chNN when rfplan role is uplink (poll /api/rfplan with existing status poll).
• Settings: "Reliability" card — profile save input + Save button (POST /api/reliability/profile {name}), profile list with Restore buttons (POST /api/reliability/restore {name}), RF role control (role select + SSID/PSK inputs → POST /api/rfplan/role).
• Add routes in Task 9's API surface before wiring buttons; keep fetch helpers identical to existing patterns (apiFetch('/api/...')).
• node --check both JS files; bump cache-bust query ?v= strings as existing convention does.
• Commit — feat(ui): reliability panel, profiles card, RF plan controls

---

Task 9: API routes + atomic deploys + VERSION

Files:

• Modify: server.py ROUTER block (~line 6804), scripts/deploy.sh, create VERSION

• Test: tests/test_reliability_api.py

• Routes:

ROUTER.add('GET', r'/api/rfplan', h_rfplan_get)
ROUTER.add('POST', r'/api/rfplan/role', h_rfplan_post)
ROUTER.add('GET', r'/api/reliability/profiles', h_profiles_get)
ROUTER.add('POST', r'/api/reliability/profile', h_profile_save)
ROUTER.add('POST', r'/api/reliability/restore', h_profile_restore)

Handlers thin-wrape mk8_rfplan / mk8_profiles; restore handler runs inside mk8_gate.gated('restore_profile', ...). All auth-gated automatically by existing middleware.

• VERSION file: 1.4.0
• deploy.sh rework:
  1. Read VERSION → stamp build copies of _hak5_manifest.json (version), payload.sh header comment, and inject SERVER_VERSION = 'X' into staged server.py (build dir only, never source tree).
  2. Remote flow: scp zip to /tmp/mk8-stage/ → verify sha256 of uploaded zip matches local → stop service → extract to /mmc/mk8/releases/<ts>/ → repoint current symlink atomically (ln -sfn) → install/update /etc/init.d/mk8-guard + copy mk8-watchdog.sh → start → poll http://127.0.0.1:8080/api/api_ping (via SSH-local curl) ≤60 s → compare served banner/version → on failure: ln -sfn back to previous release + start + exit 1.
  3. Keep legacy overlay payload dir as symlink target for portal compatibility: /root/payloads/user/.../pager-webui → real dir stays, contains pointer script or bind; simplest: leave legacy install untouched and have current be canonical (init scripts already fall back to /mmc/mk8/releases/current).
• Tests: unit-test stamping function stamp_version(build_dir) extracted into scripts/build_common.py (new) so it is importable: asserts manifest/payload/server contain version; sha256 check tested with tmpfiles.
• Commit — feat(deploy): atomic releases, VERSION single-source, post-deploy verification

---

Task 10: On-device smoke suite + live verification

Files:

• Create: scripts/smoke.sh

• smoke.sh checks (each prints PASS/FAIL, non-destructive unless --write given):

  1. Service up: curl :8080/api/api_ping.
  2. Guard installed: [ -x /etc/init.d/mk8-guard ] and enabled symlink exists.
  3. Invariants: uci get each wanted key equals expected; pool size ≤ 20.
  4. Journal writable + has boot event: tail events.log on device.
  5. Monitors up: ip link show wlan0mon/wlan1mon.
  6. (--write) Bad-value drill: set pineapd.wlan1mon.bands='2,5' → run --reconcile → expect '5'; set pool of 25 SSIDs → reconcile → cleared.
  7. (--write) Role drill: set_role uplink to lab AP → expect assoc; set_role attack → expect monitors hopping again.
  8. Deploy version match: /api/health version == cat VERSION.

• Execution order: full local unit suite (every tests/test_*.py individually) → deploy via ./scripts/deploy.sh --password '<pw>' → reboot device via SSH → wait for SSH return → rerun smoke.sh → confirm guards survived boot → report.

• Commit — test(smoke): on-device reliability suite ; final tag v1.4.0 after user confirmation.

---

Verification matrix (spec → tasks)

Spec requirement	Task
Boot guard before S50	4
Reconciler invariants (5 crash sources)	3
Profiles + lastknown-good	2
Knock-off rollback watchdog (local liveness)	6
RF roles, uplink→phy1, hop pause/resume	7
Supervisor sampling + hysteresis + alerts	5
Event journal on /mmc + boot detection	1, 5
Atomic deploy + version single-source	9
UI health/events/profiles/RF	8, 9
Unit + smoke tests	all, 10