# Mark VIII Reliability Core Implementation Plan

> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.

**Goal:** Make the Pager run reliably and consistently as expected via payload-only hardening: boot-time guard, config profiles with rollback watchdogs, RF role manager (uplink on phy1), integrated supervisor with event journal, atomic deploys, and an on-device smoke suite.

**Architecture:** Three layers inside the existing payload — `mk8-guard` init script (START=49, before the S50 pineapple stack), new `mk8_*.py` stdlib modules imported by `server.py`, and a passive supervisor thread extending the existing health monitor. Persistent state in `/mmc/mk8/` (survives reboots and overlay wipes).

**Tech Stack:** Python 3 stdlib only (`python3-light` on device: no urllib/http.server/sqlite3 modules), POSIX sh for device scripts, vanilla JS frontend, Bash + sshpass/scp for deploy tooling.

## Global Constraints

- Spec: `docs/superpowers/specs/2026-08-22-reliability-core-design.md` (approved).
- Device constraints: python3-light stdlib only; BusyBox (no `pkill`; use `killall`/`pidof`); never actively ping pineapd's command socket from loops; never run `wifi reload` outside gated operations.
- All persistent state under `/mmc/mk8/`. No writes to stock binaries or `/etc/config` outside reconciler/gated ops.
- Tests: stdlib `unittest`, one module per process (tests monkeypatch module state); run pattern:
  `python3 -m unittest tests.test_<name> -v`
- Frontend checks: `node --check <file>` after every JS edit.
- Device access is **read-only until Task 10** (deploy + smoke). Password auth via `sshpass -p '<pw>' ssh -o StrictHostKeyChecking=no root@172.16.52.1`.
- Branch: `feature/reliability`. Commit after every passing step.
- Version: single-source `VERSION` file at repo root; next version `1.4.0`.

## File Structure

```
payload/user/remote_access/pager-webui/
  mk8_events.py      NEW  event journal (JSONL append/rotate/read + counters)
  mk8_profiles.py    NEW  UCI snapshot store (/mmc/mk8/profiles)
  mk8_guard.py       NEW  known-good invariants + reconcile() + CLI hooks
  mk8_rfplan.py      NEW  phy1 RF role manager (uplink/attack/idle)
  mk8_gate.py        NEW  risky-op preflight gate + watchdog decision logic
  mk8-watchdog.sh    NEW  detached local-liveness rollback watchdog
  mk8-guard.init     NEW  START=49 boot guard script (installed to /etc/init.d/mk8-guard)
  server.py          MOD  imports, startup hook, h_health extension, /api/reliability/* + /api/rfplan/* routes, gates on risky handlers
  www/js/views.js    MOD  health events feed, reliability counters, RF chip, Settings profiles card
  www/js/app.js      MOD  nav wiring if needed
www/css/app.css      MOD  styles for new UI elements
scripts/deploy.sh    MOD  VERSION stamping, atomic release swap, post-deploy check, guard install
scripts/smoke.sh     NEW  on-device verification suite
VERSION              NEW  "1.4.0"
tests/test_mk8_events.py, test_mk8_profiles.py, test_mk8_guard.py,
tests/test_mk8_gate.py, test_mk8_rfplan.py, test_reliability_api.py   NEW
```

---

### Task 1: Event journal (`mk8_events.py`)

**Files:**
- Create: `payload/user/remote_access/pager-webui/mk8_events.py`
- Test: `tests/test_mk8_events.py`

**Interfaces:**
- Produces: `log_event(kind, sev='info', msg='', meta=None)`; `read_events(limit=100)` → list of dicts newest-first; `counters()` → dict with keys `boots`, `unexpected_boots`, `rollbacks`, `restarts`, `guard_fixes`; `mark_boot()`; constants `MK8_DIR='/mmc/mk8'`, `EVENTS_PATH`, `MAX_BYTES=5*1024*1024`, `KEEP=4`.

- [ ] **Step 1: Write failing tests**

```python
import json, os, sys, tempfile, unittest
sys.path.insert(0, os.path.join(os.path.dirname(__file__), '..', 'payload', 'user', 'remote_access', 'pager-webui'))
import mk8_events

class EventsTest(unittest.TestCase):
    def setUp(self):
        self.dir = tempfile.mkdtemp()
        self.old = mk8_events.MK8_DIR
        mk8_events.MK8_DIR = self.dir
        mk8_events.EVENTS_PATH = os.path.join(self.dir, 'events.log')

    def tearDown(self):
        mk8_events.MK8_DIR = self.old

    def test_log_and_read_newest_first(self):
        mk8_events.log_event('boot', msg='first')
        mk8_events.log_event('rollback', sev='warn', msg='second', meta={'op': 'wifi'})
        rows = mk8_events.read_events()
        self.assertEqual(rows[0]['kind'], 'rollback')
        self.assertEqual(rows[1]['kind'], 'boot')
        self.assertEqual(rows[0]['meta'], {'op': 'wifi'})

    def test_counters(self):
        mk8_events.log_event('boot'); mk8_events.log_event('rollback')
        mk8_events.log_event('restart'); mk8_events.log_event('guard_fix')
        c = mk8_events.counters()
        self.assertEqual(c['boots'], 1)
        self.assertEqual(c['rollbacks'], 1)
        self.assertEqual(c['restarts'], 1)
        self.assertEqual(c['guard_fixes'], 1)

    def test_rotation_keeps_recent(self):
        mk8_events.MAX_BYTES = 200
        for i in range(20):
            mk8_events.log_event('tick', msg='x' * 30)
        self.assertTrue(len(mk8_events.read_events()) >= 15)
        self.assertFalse(os.path.exists(mk8_events.EVENTS_PATH + '.4'))

if __name__ == '__main__':
    unittest.main()
```

- [ ] **Step 2: Run to verify failure**

Run: `python3 -m unittest tests.test_mk8_events -v`
Expected: FAIL — `No module named 'mk8_events'`

- [ ] **Step 3: Implement**

```python
"""Mark VIII reliability event journal. JSONL on /mmc, rotated."""
import json, os, threading

MK8_DIR = '/mmc/mk8'
EVENTS_PATH = os.path.join(MK8_DIR, 'events.log')
MAX_BYTES = 5 * 1024 * 1024
KEEP = 4
_LOCK = threading.Lock()

COUNTER_KINDS = ('boot', 'unexpected_boot', 'rollback', 'restart',
                 'guard_fix')


def _ensure_dir():
    try:
        os.makedirs(MK8_DIR, exist_ok=True)
    except OSError:
        pass


def log_event(kind, sev='info', msg='', meta=None):
    entry = {'ts': int(__import__('time').time()), 'kind': str(kind),
             'sev': sev, 'msg': msg[:500]}
    if meta:
        entry['meta'] = meta
    line = json.dumps(entry) + '\n'
    with _LOCK:
        _ensure_dir()
        try:
            if os.path.exists(EVENTS_PATH) and \
                    os.path.getsize(EVENTS_PATH) > MAX_BYTES:
                for i in range(KEEP - 1, 0, -1):
                    src = '%s.%d' % (EVENTS_PATH, i)
                    dst = '%s.%d' % (EVENTS_PATH, i + 1)
                    if os.path.exists(src):
                        os.replace(src, dst)
                if os.path.exists(EVENTS_PATH):
                    os.replace(EVENTS_PATH, EVENTS_PATH + '.1')
            with open(EVENTS_PATH, 'a') as f:
                f.write(line)
        except OSError:
            pass


def read_events(limit=100):
    out = []
    paths = [EVENTS_PATH + '.%d' % i for i in range(KEEP, 0, -1)]
    paths.append(EVENTS_PATH)
    for path in paths:
        try:
            with open(path) as f:
                out.extend(json.loads(l) for l in f if l.strip())
        except (OSError, ValueError):
            continue
    out.sort(key=lambda r: r.get('ts', 0))
    return out[-limit:][::-1]


def counters():
    counts = {k: 0 for k in COUNTER_KINDS}
    for row in read_events(limit=5000):
        k = row.get('kind')
        if k in counts:
            counts[k] += 1
    return counts


def mark_boot(unexpected=False):
    log_event('unexpected_boot' if unexpected else 'boot', sev='warn'
              if unexpected else 'info',
              msg='service started' + ('' if unexpected else ' cleanly'))
```

- [ ] **Step 4: Run tests to pass**

Run: `python3 -m unittest tests.test_mk8_events -v` → PASS

- [ ] **Step 5: Commit**

```bash
git add payload/user/remote_access/pager-webui/mk8_events.py tests/test_mk8_events.py
git commit -m "feat(reliability): JSONL event journal with rotation and counters"
```

---

### Task 2: Profile store (`mk8_profiles.py`)

**Files:**
- Create: `payload/user/remote_access/pager-webui/mk8_profiles.py`
- Test: `tests/test_mk8_profiles.py`

**Interfaces:**
- Consumes: nothing.
- Produces: `snapshot(name)` → bool; `list_profiles()` → list of names; `restore(name)` → dict `{ok, restored:[...]}`; `auto_name(op)` → `'pre-<op>-<ts>'`; `promote_lastknown_good()`; `delete(name)`; uses `device_run` injected as module attr `run_cmd(args, timeout=20)` defaulting to `server.device_run` lazily (avoids import cycle: define own `_run` that callers/tests monkeypatch).

- [ ] **Step 1: Failing tests**

```python
import os, sys, tempfile, unittest
sys.path.insert(0, os.path.join(os.path.dirname(__file__), '..', 'payload', 'user', 'remote_access', 'pager-webui'))
import mk8_profiles

CONFIGS = ('pineapd', 'wireless', 'network')

class ProfilesTest(unittest.TestCase):
    def setUp(self):
        self.dir = tempfile.mkdtemp()
        mk8_profiles.PROFILES_DIR = os.path.join(self.dir, 'profiles')
        self.state = {'pineapd': 'config pineapd\n\toption x y\n',
                      'wireless': 'config wireless\n', 'network': ''}
        def fake_run(args, timeout=20):
            a = list(args)
            if a[:2] == ['uci', 'export']:
                return (0, self.state.get(a[2], ''), '')
            if a[:2] == ['uci', 'import'] or a[:2] == ['uci', 'commit']:
                return (0, '', '')
            return (0, '', '')
        self.runs = []
        mk8_profiles.run_cmd = lambda args, timeout=20: (
            self.runs.append(list(args)) or fake_run(args, timeout))

    def test_snapshot_and_list(self):
        self.assertTrue(mk8_profiles.snapshot('testprof'))
        self.assertIn('testprof', mk8_profiles.list_profiles())

    def test_restore_issues_import_per_config(self):
        mk8_profiles.snapshot('p1')
        ok = mk8_profiles.restore('p1')
        self.assertTrue(ok['ok'])
        imported = [r for r in self.runs if r[:2] == ['uci', 'import']]
        self.assertEqual(len(imported), len(CONFIGS))
        commits = [r for r in self.runs if r[:2] == ['uci', 'commit']]
        self.assertGreaterEqual(len(commits), 1)

    def test_auto_name_format(self):
        name = mk8_profiles.auto_name('client_connect')
        self.assertTrue(name.startswith('pre-client_connect-'))

if __name__ == '__main__':
    unittest.main()
```

- [ ] **Step 2: Verify failure** — `No module named 'mk8_profiles'`

- [ ] **Step 3: Implement**

```python
"""UCI profile snapshots under /mmc/mk8/profiles/<name>/{pineapd,wireless,network}"""
import os, time

PROFILES_DIR = '/mmc/mk8/profiles'
CONFIGS = ('pineapd', 'wireless', 'network')


def run_cmd(args, timeout=20):
    """Lazy import avoids a circular import with server.py; tests monkeypatch."""
    from server import device_run
    return device_run(args, timeout=timeout)


def _path(name):
    return os.path.join(PROFILES_DIR, name)


def snapshot(name):
    dest = _path(name)
    try:
        os.makedirs(dest, exist_ok=True)
        wrote = False
        for cfg in CONFIGS:
            rc, out, err = run_cmd(['uci', 'export', cfg])
            if rc != 0 or not (out or '').strip():
                continue
            with open(os.path.join(dest, cfg + '.uci'), 'w') as f:
                f.write(out)
            wrote = True
        return wrote
    except OSError:
        return False


def auto_name(op):
    return 'pre-%s-%d' % (op, int(time.time()))


def list_profiles():
    try:
        return sorted(d for d in os.listdir(PROFILES_DIR)
                      if os.path.isdir(_path(d)))
    except OSError:
        return []


def delete(name):
    import shutil
    shutil.rmtree(_path(name), ignore_errors=True)


def restore(name):
    """Restore configs then commit once per config. Caller runs wifi reload
    / service restart as appropriate for the operation."""
    src = _path(name)
    restored = []
    if not os.path.isdir(src):
        return {'ok': False, 'restored': [], 'error': 'profile not found'}
    for cfg in CONFIGS:
        fpath = os.path.join(src, cfg + '.uci')
        if not os.path.isfile(fpath):
            continue
        with open(fpath) as f:
            text = f.read()
        rc, _, err = run_cmd(['uci', 'import', cfg], input_data=text)
        if rc != 0:
            return {'ok': False, 'restored': restored,
                    'error': 'import failed'}
        run_cmd(['uci', 'commit', cfg])
        restored.append(cfg)
    return {'ok': True, 'restored': restored}


LASTKNOWN_GOOD = 'lastknown-good'


def promote_lastknown_good():
    """Replace the lastknown-good profile with the live config."""
    delete(LASTKNOWN_GOOD)
    return snapshot(LASTKNOWN_GOOD)
```

Update the Step-1 fake to accept `input_data=None` and record imports:

```python
        def fake_run(args, timeout=20, input_data=None):
            a = list(args)
            if a[:2] == ['uci', 'import']:
                self.imports = getattr(self, 'imports', [])
                self.imports.append((a[2], input_data))
                return (0, '', '')
            if a[:2] == ['uci', 'export']:
                return (0, self.state.get(a[2], ''), '')
            if a[:2] == ['uci', 'commit']:
                return (0, '', '')
            return (0, '', '')
```

- [ ] **Step 4: Run to pass.**
- [ ] **Step 5: Commit** — `feat(reliability): UCI profile snapshot store`

---

### Task 3: Reconciler (`mk8_guard.py`)

**Files:**
- Create: `payload/user/remote_access/pager-webui/mk8_guard.py`
- Test: `tests/test_mk8_guard.py`

**Interfaces:**
- Consumes: `server.PINEAPD_SAFE_UCI` (dict of safe pineapd UCI values), `server._apply_uci_wanted(wanted)`, `server._monitor_down(name)`, `server._raise_monitors()`.
- Produces: `WANTED_EXTRA = {'pineapd.@pineapd[0].autossidpool': '0'}`; `POOL_CLEAR_MAX = 20`; `reconcile(clear_pool=True)` → `{'changed': [...], 'pool_cleared': bool}`; `guard_report()` → dict for `/api/health`.

- [ ] **Step 1: Failing tests**

Full file — reuse the exact `fake_run` device-mock pattern from `tests/test_health.py`:

```python
import os, sys, unittest
sys.path.insert(0, os.path.join(os.path.dirname(__file__), '..', 'payload', 'user', 'remote_access', 'pager-webui'))
import server
import mk8_guard

class GuardTest(unittest.TestCase):
    def setUp(self):
        self.calls = []
        self.uci = {}
        self.mon_up = {'wlan0mon': True, 'wlan1mon': True}
        server._iface_up = lambda name: self.mon_up.get(name, True)

        def fake_run(args, timeout=20, input_data=None):
            a = list(args)
            self.calls.append(a)
            if a[:2] == ['uci', 'get']:
                key = a[2]
                if key == 'pineapd.@ssidpool[0].ssid':
                    return (0, ''.join('s%d\n' % i for i in range(self.pool)), '')
                return (0, self.uci.get(key, '') + '\n', '')
            if a[:2] == ['uci', 'set']:
                k, _, v = a[2].partition('=')
                self.uci[k] = v
            if a[:2] == ['uci', 'delete']:
                self.pool = 0
            return (0, '', '')
        mk8_guard.device_run = fake_run

    def tearDown(self):
        server._iface_up = lambda name: True

    def test_applies_all_wanted_when_missing(self):
        self.pool = 0
        result = mk8_guard.reconcile(clear_pool=False)
        sets = [c[2] for c in self.calls if c[:2] == ['uci', 'set']]
        self.assertEqual(len(sets),
                         len(server.PINEAPD_SAFE_UCI) + len(mk8_guard.WANTED_EXTRA))
        self.assertTrue(result['changed'])

    def test_clears_large_pool_only(self):
        self.pool = 25
        result = mk8_guard.reconcile(clear_pool=True)
        self.assertTrue(result['pool_cleared'])
        self.assertIn(['uci', 'delete', 'pineapd.@ssidpool[0].ssid'], self.calls)

    def test_small_pool_untouched(self):
        self.pool = 5
        result = mk8_guard.reconcile(clear_pool=True)
        self.assertFalse(result['pool_cleared'])

if __name__ == '__main__':
    unittest.main()
```

- [ ] **Step 2: Verify failure** — no module.
- [ ] **Step 3: Implement**

```python
"""Boot-time reconciliation of crash-prone PineAP settings."""
from server import (_apply_uci_wanted, _monitor_down, _raise_monitors,
                    PINEAPD_SAFE_UCI, device_run)

WANTED_EXTRA = {'pineapd.@pineapd[0].autossidpool': '0'}
POOL_CLEAR_MAX = 20
MONITORS = ('wlan0mon', 'wlan1mon')


def _pool_size():
    rc, out, err = device_run(
        ['uci', 'get', 'pineapd.@ssidpool[0].ssid'])
    if rc != 0 or not (out or '').strip():
        return 0
    return len([s for s in out.strip().split('\\n') if s])


def reconcile(clear_pool=True):
    changed = _apply_uci_wanted(dict(PINEAPD_SAFE_UCI, **WANTED_EXTRA))
    pool_cleared = False
    if clear_pool and _pool_size() > POOL_CLEAR_MAX:
        device_run(['uci', 'delete', 'pineapd.@ssidpool[0].ssid'])
        pool_cleared = True
    if changed or pool_cleared:
        device_run(['uci', 'commit', 'pineapd'])
    raised = _raise_monitors() if any(_monitor_down(m) for m in MONITORS) else []
    return {'changed': changed, 'pool_cleared': pool_cleared,
            'monitors_raised': raised}


def guard_report():
    from server import _pending_uci
    pending = _pending_uci(dict(PINEAPD_SAFE_UCI, **WANTED_EXTRA))
    return {'in_sync': not pending, 'pending': pending,
            'pool_size': _pool_size()}
```

- [ ] **Step 4: Pass. Commit:** `feat(reliability): boot-time UCI reconciler`

---

### Task 4: Guard init script + install wiring

**Files:**
- Create: `payload/user/remote_access/pager-webui/mk8-guard.init`
- Modify: `scripts/deploy.sh` (install block), `payload/user/remote_access/pager-webui/server.py` (CLI flag)

**Interfaces:** CLI: `python3 server.py --reconcile` runs `mk8_guard.reconcile()` and prints JSON; exit 0 always (boot must not fail).

- [ ] **Step 1: Write `mk8-guard.init`:**

```sh
#!/bin/sh /etc/rc.common
# Mark VIII boot guard: enforce safe PineAP UCI before the S50 stack starts.
START=49
STOP=90

GUARD_DIR="/root/payloads/user/remote_access/pager-webui"
[ -f "$GUARD_DIR/server.py" ] || GUARD_DIR="/mmc/mk8/releases/current"

start() {
    [ -f "$GUARD_DIR/server.py" ] || return 0
    /usr/bin/python3 "$GUARD_DIR/server.py" --reconcile \
        >/tmp/mk8-guard.log 2>&1 || true
}

stop() { return 0; }
```

- [ ] **Step 2: Add CLI branch in `server.py` `__main__` (after `--release-pager`):**

```python
    if '--reconcile' in sys.argv:
        try:
            import mk8_guard
            print(json.dumps(mk8_guard.reconcile()))
        except Exception as exc:  # boot must never fail here
            print(json.dumps({'error': str(exc)}))
        sys.exit(0)
```

- [ ] **Step 3: deploy.sh install block (inside REMOTE_COMMAND before EXTRACT_OK echo):**

```sh
cp -f '$DIR/mk8-guard.init' /etc/init.d/mk8-guard
chmod 755 /etc/init.d/mk8-guard
/etc/init.d/mk8-guard enable
```

(`$DIR` is the existing remote payload dir var used by the unzip step.)

- [ ] **Step 4: Local verification:** `python3 -m py_compile payload/user/remote_access/pager-webui/server.py && node --check payload/user/remote_access/pager-webui/www/js/app.js` (JS untouched but cheap sanity). `sh -n scripts/deploy.sh`.
- [ ] **Step 5: Commit** — `feat(reliability): START=49 boot guard installed by deploy`

---

### Task 5: Supervisor extension of health monitor

**Files:**
- Modify: `payload/user/remote_access/pager-webui/server.py` (`health_check`, `_health_loop`, `h_health`, `serve()` boot sequence)
- Test: `tests/test_health.py` (extend), `tests/test_reliability_api.py` (new)

**Interfaces:**
- Consumes: `mk8_events`, `mk8_guard.guard_report()`.
- Produces in `_health`: `mem_percent`, `events` (last 20), `reliability` counters, `guard` report; boot-marker logic `check_boot_marker()` → bool unexpected; mem sampling `_mem_percent()`.

- [ ] **Step 1: Failing tests** — add to `tests/test_health.py` (same fake_run pattern already there):

```python
class SupervisorExtrasTest(unittest.TestCase):
    def runTestWith(self):  # helper: reuse existing setUp fake_run
        pass

    def test_mem_percent_math(self):
        import tempfile
        content = 'MemTotal:  250000 kB\nMemAvailable: 100000 kB\n'
        path = tempfile.mktemp()
        open(path, 'w').write(content)
        self.assertEqual(server._mem_percent(path), 60)

    def test_health_reports_events_and_counters(self):
        import mk8_events
        mk8_events.log_event('restart', msg='x')
        status, h = server.h_health(None)
        self.assertEqual(status, 200)
        self.assertIn('events', h)
        self.assertIn('boots', h['reliability'])

    def test_boot_marker_detects_unexpected(self):
        import mk8_events, tempfile, os
        marker = tempfile.mktemp()
        old = server.BOOT_MARKER
        server.BOOT_MARKER = marker
        try:
            open(marker, 'w').write('0')
            self.assertTrue(server.check_boot_marker())
            os.unlink(marker)
            self.assertFalse(server.check_boot_marker())
        finally:
            server.BOOT_MARKER = old
```

(`_mem_percent` takes a `path` argument so tests inject a temp file; production call passes no arg.)

- [ ] **Step 2: Implement** — key code:

```python
def _mem_percent(path='/proc/meminfo'):
    try:
        vals = {}
        with open(path) as f:
            for line in f:
                k, v = line.split(':')
                vals[k] = int(v.strip().split()[0])
        total = vals.get('MemTotal', 0)
        avail = vals.get('MemAvailable', vals.get('MemFree', 0))
        return round(100.0 * (total - avail) / total) if total else 0
    except (OSError, ValueError):
        return 0

MEM_WARN_PERCENT = 85
MEM_WARN_STREAK = 5

# inside health_check(), after monitor repair section:
h['mem_percent'] = _mem_percent()
if h['mem_percent'] >= MEM_WARN_PERCENT:
    h['mem_streak'] = h.get('mem_streak', 0) + 1
else:
    h['mem_streak'] = 0
if h['mem_streak'] == MEM_WARN_STREAK:
    mk8_events.log_event('mem_warn', sev='warn',
                         msg='memory above %d%% sustained' % MEM_WARN_PERCENT)

# restart action gains journaling:
h['fixes'] += 1
mk8_events.log_event('restart', msg='pineapd restarted by health monitor')
```

Boot marker (called from `startup_env_check` tail):

```python
BOOT_MARKER = '/mmc/mk8/boot.marker'

def check_boot_marker():
    import os, mk8_events
    try:
        unexpected = os.path.exists(BOOT_MARKER)
        mk8_events.mark_boot(unexpected=unexpected)
        with open(BOOT_MARKER, 'w') as f:
            f.write(str(int(time.time())))
        return unexpected
    except OSError:
        return False
```

`h_health` additions:

```python
    import mk8_events, mk8_guard
    h['reliability'] = mk8_events.counters()
    h['events'] = mk8_events.read_events(limit=20)
    h['guard'] = mk8_guard.guard_report()
```

- [ ] **Step 3: Run full test_health + new api test to pass.**
- [ ] **Step 4: Commit** — `feat(reliability): supervisor sampling, event feed, boot marker`

---

### Task 6: Risky-op gate + rollback watchdog

**Files:**
- Create: `payload/user/remote_access/pager-webui/mk8_gate.py`, `payload/user/remote_access/pager-webui/mk8-watchdog.sh`
- Modify: `server.py` (wrap handlers), `deploy.sh` (ship watchdog script)
- Test: `tests/test_mk8_gate.py`

**Interfaces:**
- `watchdog_decision(fails, oks, fail_after=6, healthy_after=6)` → `'rollback'|'promote'|None` (pure).
- `gated(op, fn)` decorator/context: snapshots `auto_name(op)`, spawns watchdog via `setsid sh mk8-watchdog.sh <profile> ... &`, runs fn, returns `(result, profile)`.
- CLI: `server.py --rollback-snapshot <name>` restores profile + wifi reload; `--promote-snapshot <name>` promotes lastknown-good.

- [ ] **Step 1: Failing decision-table tests**

```python
import os, sys, unittest
sys.path.insert(0, os.path.join(os.path.dirname(__file__), '..', 'payload', 'user', 'remote_access', 'pager-webui'))
import mk8_gate

class DecisionTest(unittest.TestCase):
    def tick(self, state):
        action, new = mk8_gate.watchdog_decision(state)
        return action, new

    def test_no_action_below_fail_threshold(self):
        action, s = self.tick({'fails': 5, 'oks': 0, 'tripped': False})
        self.assertIsNone(action)
        self.assertFalse(s['tripped'])

    def test_rollback_at_threshold(self):
        action, s = self.tick({'fails': 6, 'oks': 0, 'tripped': False})
        self.assertEqual(action, 'rollback')
        self.assertTrue(s['tripped'])
        self.assertEqual(s['oks'], 0)

    def test_promote_after_recovery(self):
        action, s = self.tick({'fails': 6, 'oks': 6, 'tripped': True})
        self.assertEqual(action, 'promote')

    def test_no_promote_before_recovery_threshold(self):
        action, s = self.tick({'fails': 6, 'oks': 5, 'tripped': True})
        self.assertIsNone(action)

if __name__ == '__main__':
    unittest.main()
```

- [ ] **Step 2: Implement `mk8_gate.py`**

```python
"""Risky-operation gate: snapshot + detached rollback watchdog."""
import subprocess

WATCHDOG = '/root/payloads/user/remote_access/pager-webui/mk8-watchdog.sh'
FAIL_AFTER = 6          # consecutive local-liveness failures -> rollback
HEALTHY_AFTER = 6       # consecutive successes after failure -> promote
INTERVAL = 5            # seconds between probes


def watchdog_decision(state):
    """state: {'fails': int, 'oks': int, 'tripped': bool,
               'fail_after': 6, 'healthy_after': 6}
    Returns (action, new_state): action in {'rollback','promote',None}."""
    s = dict(state)
    fa = s.get('fail_after', FAIL_AFTER)
    ha = s.get('healthy_after', HEALTHY_AFTER)
    if not s['tripped'] and s['fails'] >= fa:
        return 'rollback', dict(s, tripped=True, oks=0)
    if s['tripped'] and s['oks'] >= ha:
        return 'promote', s
    return None, s


def gated(op, profiles, spawn=None):
    """Decorator factory: snapshot config, spawn detached watchdog, run op."""
    import shlex
    if spawn is None:
        def spawn(cmd):
            subprocess.Popen(cmd, shell=True, start_new_session=True)
    def deco(fn):
        def wrapped(*a, **kw):
            name = profiles.auto_name(op)
            profiles.snapshot(name)
            spawn("setsid sh %s %s %d %d %d >/dev/null 2>&1 &" %
                  (shlex.quote(WATCHDOG), shlex.quote(name),
                   INTERVAL, FAIL_AFTER, HEALTHY_AFTER))
            return fn(*a, **kw)
        return wrapped
    return deco
```

(Remove the earlier `NotImplementedError` sketch entirely — this is the final form.)

`gated` implementation:

```python
def gated(op, profiles, spawn=lambda cmd: subprocess.Popen(
        cmd, shell=True, start_new_session=True)):
    """Decorator factory. profiles = mk8_profiles module."""
    def deco(fn):
        def wrapped(*a, **kw):
            name = profiles.auto_name(op)
            profiles.snapshot(name)
            spawn("setsid sh %s %s %d %d %d >/dev/null 2>&1 &"
                  % (WATCHDOG, name, INTERVAL, FAIL_AFTER, HEALTHY_AFTER))
            return fn(*a, **kw)
        return wrapped
    return deco
```

- [ ] **Step 3: `mk8-watchdog.sh`**

```sh
#!/bin/sh
# Usage: mk8-watchdog.sh <profile> <interval> <fail_after> <healthy_after>
PROFILE="$1"; IV="${2:-5}"; FA="${3:-6}"; HA="${4:-6}"
DIR="/root/payloads/user/remote_access/pager-webui"
[ -f "$DIR/server.py" ] || DIR="/mmc/mk8/releases/current"
fails=0; oks=0; tripped=0
probe() {
    curl -fsS -m 3 http://127.0.0.1:8080/ >/dev/null 2>&1 &&
    { ip link show wlan0mon >/dev/null 2>&1 ||
      ip link show wlan1mon >/dev/null 2>&1; }
}
while true; do
    if probe; then
        fails=0
        if [ "$tripped" = "1" ]; then
            oks=$((oks + 1))
            if [ "$oks" -ge "$HA" ]; then
                /usr/bin/python3 "$DIR/server.py" --promote-snapshot "$PROFILE" >/dev/null 2>&1
                exit 0
            fi
        fi
    else
        fails=$((fails + 1)); oks=0
        if [ "$tripped" = "0" ] && [ "$fails" -ge "$FA" ]; then
            tripped=1
            /usr/bin/python3 "$DIR/server.py" --rollback-snapshot "$PROFILE" >/dev/null 2>&1
        fi
    fi
    sleep "$IV"
done
```

- [ ] **Step 4: Wire gates** — decorate `h_pineap_wifi_set_ap`, `_disable_sta_uplink` call sites, enterprise deploy/stop, and any handler issuing `wifi reload`, with `@mk8_gate.gated('<op>', mk8_profiles)`.
- [ ] **Step 5: CLI rollback/promote hooks in `server.py` `__main__`:**

```python
    if '--rollback-snapshot' in sys.argv:
        name = sys.argv[sys.argv.index('--rollback-snapshot') + 1]
        import mk8_profiles
        result = mk8_profiles.restore(name)
        device_run(['wifi', 'reload'], timeout=90)
        import mk8_events
        mk8_events.log_event('rollback', sev='warn',
                             msg='watchdog restored %s' % name,
                             meta=result)
        print(json.dumps(result))
        sys.exit(0)
    if '--promote-snapshot' in sys.argv:
        name = sys.argv[sys.argv.index('--promote-snapshot') + 1]
        import mk8_profiles
        print(json.dumps({'promoted': mk8_profiles.promote_lastknown_good()}))
        sys.exit(0)
```

- [ ] **Step 6: Tests pass; `sh -n mk8-watchdog.sh`; commit** — `feat(reliability): risky-op preflight snapshots + detached rollback watchdog`

---

### Task 7: RF role manager (`mk8_rfplan.py`)

**Files:**
- Create: `payload/user/remote_access/pager-webui/mk8_rfplan.py`
- Modify: `server.py` (routes), UI chip later in Task 8
- Test: `tests/test_mk8_rfplan.py`

**Interfaces:**
- Consumes: `server._pause_hop/_resume_hop/_read_hop`, `server.device_run`, `server._uci_values/_set_uci`, gate from Task 6.
- Produces: `current_role()` → `'attack'|'uplink'|'idle'`; `set_role(role, ssid=None, psk=None)` → dict result (gated); ensures exclusivity: attack AP enable paths call `ensure_attack()` which auto-switches uplink→attack first.
- Routes: `GET /api/rfplan` , `POST /api/rfplan/role`.

- [ ] **Step 1: Failing tests** (fake device_run capturing uci/iw calls):

```python
def test_uplink_sets_sta_section_and_pauses_hop(self): ...
def test_set_role_uplink_requires_ssid(self): ...
def test_exclusivity_switch(self): ...
def test_current_role_reads_uci(self): ...
```

Assertions: `uci set wireless.wlan1up.mode=sta`, `.disabled=0`, `.ssid=<ssid>`, hop paused via `_pause_hop` mock, `wifi reload` invoked through gated op only.

- [ ] **Step 2: Implement core:**

```python
ROLE_KEY = 'mk8.rfplan.role'

def current_role():
    from server import _uci_values
    cfg = _uci_values('wireless.wlan1up') or {}
    if cfg.get('disabled') != '1' and cfg.get('mode') == 'sta':
        return 'uplink'
    return 'idle'

def set_role(role, ssid=None, psk=None):
    from server import device_run, _pause_hop, _resume_hop
    if role == 'uplink':
        if not ssid:
            return {'ok': False, 'error': 'ssid required'}
        cmds = [
            ['uci', 'set', 'wireless.wlan1up=wifi-iface'],
            ['uci', 'set', 'wireless.wlan1up.device=radio1'],
            ['uci', 'set', 'wireless.wlan1up.mode=sta'],
            ['uci', 'set', 'wireless.wlan1up.network=cli'],
            ['uci', 'set', 'wireless.wlan1up.ssid=%s' % ssid],
            ['uci', 'set', 'wireless.wlan1up.encryption=%s'
             % ('psk2' if psk else 'none')],
            ['uci', 'set', 'wireless.wlan1up.disabled=0'],
        ]
        if psk:
            cmds.append(['uci', 'set', 'wireless.wlan1up.key=%s' % psk])
        for c in cmds:
            device_run(c)
        device_run(['uci', 'commit', 'wireless'])
        _pause_hop()
        device_run(['wifi', 'reload'], timeout=60)
        assoc = associated()
        if not assoc:
            disable_uplink()
            _resume_hop()
            return {'ok': False, 'error': 'association failed; reverted'}
        return {'ok': True, 'role': 'uplink', 'assoc': assoc}
    # attack/idle: tear down STA
    disable_uplink()
    _resume_hop()
    return {'ok': True, 'role': role}

def associated():
    rc, out, err = device_run(['iw', 'dev', 'wlan1up', 'link'], timeout=10)
    if rc != 0 or 'Connected' not in (out or ''):
        return None
    for line in (out or '').splitlines():
        line = line.strip()
        if line.startswith('Connected to '):
            return line.split()[2]
    return None

def disable_uplink():
    from server import device_run
    device_run(['uci', 'set', 'wireless.wlan1up.disabled=1'])
    device_run(['uci', 'commit', 'wireless'])

def ensure_attack():
    if current_role() == 'uplink':
        set_role('attack')
```

Handlers in server.py wrap with gate + journal events. `GET /api/rfplan` returns role + assoc + hop-paused state.

- [ ] **Step 3: Pass; py_compile; commit** — `feat(reliability): phy1 RF role manager with uplink-on-radio1`

---

### Task 8: UI additions

**Files:**
- Modify: `payload/user/remote_access/pager-webui/www/js/views.js`, `www/js/app.js` (nav if needed), `www/css/app.css`

**Steps (no unit tests; verified by `node --check` + live smoke in Task 10):**

- [ ] Dashboard health panel: events feed list (ts/kind/sev/msg) + counters row (boots/unexpected/rollbacks/restarts/guard fixes) + guard sync chip; render from `/api/health` new fields; CSS classes `mk8-events-feed`, `mk8-counter-row`.
- [ ] Top bar RF chip extension: show `PHY1: UPLINK chNN` when rfplan role is uplink (poll `/api/rfplan` with existing status poll).
- [ ] Settings: "Reliability" card — profile save input + Save button (`POST /api/reliability/profile` {name}), profile list with Restore buttons (`POST /api/reliability/restore` {name}), RF role control (role select + SSID/PSK inputs → `POST /api/rfplan/role`).
- [ ] Add routes in Task 9's API surface before wiring buttons; keep fetch helpers identical to existing patterns (`apiFetch('/api/...')`).
- [ ] `node --check` both JS files; bump cache-bust query `?v=` strings as existing convention does.
- [ ] Commit — `feat(ui): reliability panel, profiles card, RF plan controls`

---

### Task 9: API routes + atomic deploys + VERSION

**Files:**
- Modify: `server.py` ROUTER block (~line 6804), `scripts/deploy.sh`, create `VERSION`
- Test: `tests/test_reliability_api.py`

- [ ] **Routes:**

```python
ROUTER.add('GET', r'/api/rfplan', h_rfplan_get)
ROUTER.add('POST', r'/api/rfplan/role', h_rfplan_post)
ROUTER.add('GET', r'/api/reliability/profiles', h_profiles_get)
ROUTER.add('POST', r'/api/reliability/profile', h_profile_save)
ROUTER.add('POST', r'/api/reliability/restore', h_profile_restore)
```

Handlers thin-wrape `mk8_rfplan` / `mk8_profiles`; restore handler runs inside `mk8_gate.gated('restore_profile', ...)`. All auth-gated automatically by existing middleware.

- [ ] **VERSION file:** `1.4.0`
- [ ] **deploy.sh rework:**
  1. Read `VERSION` → stamp build copies of `_hak5_manifest.json` (`version`), `payload.sh` header comment, and inject `SERVER_VERSION = 'X'` into staged `server.py` (build dir only, never source tree).
  2. Remote flow: scp zip to `/tmp/mk8-stage/` → verify sha256 of uploaded zip matches local → stop service → extract to `/mmc/mk8/releases/<ts>/` → repoint `current` symlink atomically (`ln -sfn`) → install/update `/etc/init.d/mk8-guard` + copy `mk8-watchdog.sh` → start → poll `http://127.0.0.1:8080/api/api_ping` (via SSH-local curl) ≤60 s → compare served banner/version → on failure: `ln -sfn` back to previous release + start + exit 1.
  3. Keep legacy overlay payload dir as symlink target for portal compatibility: `/root/payloads/user/.../pager-webui` → real dir stays, contains pointer script or bind; simplest: leave legacy install untouched and have `current` be canonical (init scripts already fall back to `/mmc/mk8/releases/current`).
- [ ] **Tests:** unit-test stamping function `stamp_version(build_dir)` extracted into `scripts/build_common.py` (new) so it is importable: asserts manifest/payload/server contain version; sha256 check tested with tmpfiles.
- [ ] **Commit** — `feat(deploy): atomic releases, VERSION single-source, post-deploy verification`

---

### Task 10: On-device smoke suite + live verification

**Files:**
- Create: `scripts/smoke.sh`

- [ ] **smoke.sh checks (each prints PASS/FAIL, non-destructive unless `--write` given):**
  1. Service up: `curl :8080/api/api_ping`.
  2. Guard installed: `[ -x /etc/init.d/mk8-guard ]` and enabled symlink exists.
  3. Invariants: `uci get` each wanted key equals expected; pool size ≤ 20.
  4. Journal writable + has boot event: tail events.log on device.
  5. Monitors up: `ip link show wlan0mon/wlan1mon`.
  6. (`--write`) Bad-value drill: set `pineapd.wlan1mon.bands='2,5'` → run `--reconcile` → expect `'5'`; set pool of 25 SSIDs → reconcile → cleared.
  7. (`--write`) Role drill: set_role uplink to lab AP → expect assoc; set_role attack → expect monitors hopping again.
  8. Deploy version match: `/api/health` version == `cat VERSION`.
- [ ] **Execution order:** full local unit suite (every `tests/test_*.py` individually) → deploy via `./scripts/deploy.sh --password '<pw>'` → reboot device via SSH → wait for SSH return → rerun smoke.sh → confirm guards survived boot → report.
- [ ] **Commit** — `test(smoke): on-device reliability suite` ; final tag `v1.4.0` after user confirmation.

---

## Verification matrix (spec → tasks)

| Spec requirement | Task |
|---|---|
| Boot guard before S50 | 4 |
| Reconciler invariants (5 crash sources) | 3 |
| Profiles + lastknown-good | 2 |
| Knock-off rollback watchdog (local liveness) | 6 |
| RF roles, uplink→phy1, hop pause/resume | 7 |
| Supervisor sampling + hysteresis + alerts | 5 |
| Event journal on /mmc + boot detection | 1, 5 |
| Atomic deploy + version single-source | 9 |
| UI health/events/profiles/RF | 8, 9 |
| Unit + smoke tests | all, 10 |