llama-watch/docs/superpowers/specs/2026-08-14-usage-stats-gpu-combos-design.md

Usage Stats + GPU Combinations

Date: 2026-08-14 Status: Approved (design)

Summary

Two extensions to the existing llama-watch actions:

1. The In-Flight Monitor action gains a Usage stats display mode showing token/request totals and the generation-speed P95 from llama-swap's /api/metrics/stats endpoint.
2. The GPU Graph action gains GPU combinations: a key can aggregate an arbitrary subset of GPUs (e.g. only the two Blackwells).

Data Source (verified against llama-swap source)

• GET /api/metrics/stats (no query param) → global aggregates.
• GET /api/metrics/stats?model=<id> → per-model aggregates.
• Response fields used: total_requests, total_input_tokens, total_output_tokens, gen_histogram.p95.
• p95 is a tokens/sec generation-speed percentile, not latency.
• Totals cover llama-swap's in-memory activity retention (default ~1000 requests), not lifetime counters.
• The /api/events SSE activity event (payload {"id": N}) fires per completed request and is used as a "re-poll stats" trigger. It carries no data itself.
• The Prometheus /metrics endpoint exposes no token/request/latency data and is not used for this feature.

1. Usage stats data layer

• src/lib/stats.ts (new):
  • interface UsageStats { totalRequests: number; totalInputTokens: number; totalOutputTokens: number; genP95: number }.
  • parseStats(json: unknown): UsageStats | null — defensive; returns null on malformed input; missing fields default to 0.
  • fetchStats(cfg: LlamaSwapConfig, modelId: string): Promise<UsageStats | null> — fetches /api/metrics/stats (with ?model= when modelId is not "all"), returns parsed stats or null on HTTP error/parse failure.
• src/lib/stats-cache.ts (new):
  • StatsCache holds Set<string> of registered model keys ("all" or a model id) and a Map<key, UsageStats | null> of last-known values.
  • One shared 5s interval polls every registered key via fetchStats.
  • register(key) / unregister(key) start/stop the interval as needed (no ref counting — each key registers/unregisters once).
  • get(key): UsageStats | undefined.
  • refresh() re-polls all registered keys.
  • Change listener invoked after each poll completes (success or failure).
  • Fetch failure keeps the last-known value.
• src/lib/event-feed.ts: decodeEvent learns the activity event — { type: "activity", id: number }. FeedEvent (in inflight-tracker.ts) gains the activity variant.
• src/lib/runtime.ts: runtime owns a StatsCache (created alongside feed/poller, reset on config change). Feed activity events trigger statsCache.refresh() throttled to at most once per 2s. Runtime exposes:
  • getStats(modelId: string): UsageStats | undefined
  • watchStats(modelId: string): () => void (register; returns unsubscribe).
  • Stats changes flow through the existing subscribe/emit mechanism so actions re-render on the normal tick.

2. In-Flight action modes

• New settings (per key, in inflight-monitor.ts):
  • display: "count" | "usage" (default count).
  • primaryStat: "requests" | "input_tokens" | "output_tokens" | "gen_p95" (default requests).
• Model setting semantics:
  • Model select (PI datasource) gains an "All models" option, value all.
  • Count mode: all → total in-flight across all models; label ALL MODELS.
  • Usage mode: all → global stats; otherwise per-model stats.
• Count mode: unchanged rendering (count number, spark, IDLE/ LOADING/OFF/OFFLINE). The 1s spark sampler runs only in count mode.
• Usage mode (renderUsage in render.ts, dark #10131a bg):
  • top: model short name or ALL MODELS (7px).
  • center (24px bold): the primaryStat value, compact-formatted — < 1000 integer, else x.xk, else x.xM (genP95 shows integer t/s).
  • three small rows (7px): the remaining three stats, labeled REQ, IN, OUT, P95.
  • offline → OFFLINE overlay; stats unknown → --.
• Press opens the UI (unchanged).

3. GPU combinations

• New setting gpuCombo?: string — comma-separated GPU ids, e.g. "0,2". When non-empty it overrides gpuId.
• Pure helper in metrics-poller.ts (exported, tested): combineSeries(histories: number[][], kind): { value?: number; history: number[] } — element-wise across the per-GPU rings, aligned by index, applying the existing aggregate rules: util_percent/memory_util_percent/fan → average, temperature → max, power → sum. A trailing undefined value from an empty ring leaves that element undefined.
• gpu-graph.ts render: for a combo, build per-GPU histories via poller.getHistory(id, metric), combine, use the combined value/history. Unknown ids are filtered out; if none remain → --.
• Label: GPU 0+2 (ids joined with +).

4. Property inspectors

• ui/inflight.html: add "Display" select (count/usage) and "Primary stat" select. Both always visible; primaryStat has no effect in count mode.
• src/lib/datasources.ts: the models datasource prepends { label: "All models", value: "all" } to its items (so the model select offers both the aggregate and individual models).
• ui/gpu.html: add "GPU combination (comma-separated ids)" textfield, optional, placeholder e.g. 0,2.

Error Handling / Edge Cases

• Stats fetch error → keep last-known values; no OFFLINE flip (offline is driven by feed status as today).
• Server offline → existing OFFLINE overlay in both modes.
• Stats not yet loaded → -- placeholders.
• Activity-triggered refresh throttled to 1 per 2s.
• Combo with no valid ids → --.
• No manifest change (both features extend existing actions).

Testing

• parseStats: happy path; malformed input; missing fields → 0.
• StatsCache: register/unregister lifecycle (interval start/stop), single poll per model, values cached, refresh on demand, failure keeps last value. Uses an injected fetch function.
• decodeEvent: activity event decodes to { type: "activity", id }.
• renderUsage: big primary stat, small labeled rows, compact formatting (52.1k), -- when no data, ALL MODELS label, offline overlay.
• combineSeries: avg/max/sum per metric; empty/mismatched histories.
• In-Flight action: all count sums models.