1122 lines
35 KiB
Markdown
1122 lines
35 KiB
Markdown
# Usage Stats + GPU Combinations Implementation Plan
|
|
|
|
> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.
|
|
|
|
**Goal:** Add a Usage-stats display mode to the In-Flight Monitor action (request/token totals + gen-speed P95 from `/api/metrics/stats`) and GPU combination support to the GPU Graph action.
|
|
|
|
**Architecture:** A shared runtime-owned `StatsCache` polls llama-swap's `/api/metrics/stats` for registered model keys (single interval, `activity` SSE events trigger throttled refreshes) and feeds a new `renderUsage` view in the In-Flight action. GPU combinations are aggregated in the GPU action via a pure `combineSeries` helper over the poller's existing per-GPU rings.
|
|
|
|
**Tech Stack:** TypeScript ESM, `node:test`+`tsx`, `@elgato/streamdeck@2.1.1`. Tests run with `npm test` (i.e. `tsx --test`). Build: `npm run build` (rollup). Typecheck: `npx tsc --noEmit`.
|
|
|
|
## Global Constraints
|
|
|
|
- No code comments (project convention).
|
|
- Work committed on `main` (user-approved workflow; no remote).
|
|
- `feed.ts`'s `FeedEvent` type lives in `src/lib/inflight-tracker.ts`.
|
|
- Default base URL `http://localhost:9292`; llama-swap `/api/metrics/stats` returns `{ total_requests, total_input_tokens, total_output_tokens, total_cache_tokens, prompt_histogram, gen_histogram }`; `gen_histogram.p95` is a **tokens/sec** percentile (not latency).
|
|
- Key images are SVG strings via `svgDataUrl`; dark background `#10131a`.
|
|
- Multi-key safety: per-key state lives in a `Map` keyed on `ev.action.id`.
|
|
- `npm test` runs ALL test files via `tsx --test`; `AggregateError` style failures indicate a failing test file.
|
|
|
|
---
|
|
|
|
### Task 1: Stats types, parsing, and fetching
|
|
|
|
**Files:**
|
|
- Create: `src/lib/stats.ts`
|
|
- Test: `tests/stats.test.ts`
|
|
|
|
**Interfaces:**
|
|
- Produces:
|
|
- `interface UsageStats { totalRequests: number; totalInputTokens: number; totalOutputTokens: number; genP95: number }`
|
|
- `parseStats(json: unknown): UsageStats | null`
|
|
- `fetchStats(cfg: LlamaSwapConfig, modelId: string): Promise<UsageStats | null>`
|
|
|
|
- [ ] **Step 1: Write the failing test**
|
|
|
|
Create `tests/stats.test.ts`:
|
|
|
|
```ts
|
|
import assert from "node:assert/strict";
|
|
import { test } from "node:test";
|
|
import { fetchStats, parseStats, type UsageStats } from "../src/lib/stats";
|
|
|
|
test("parseStats extracts totals and gen p95", () => {
|
|
const stats = parseStats({
|
|
total_requests: 1985,
|
|
total_input_tokens: 312327705,
|
|
total_output_tokens: 932710,
|
|
total_cache_tokens: 308144640,
|
|
gen_histogram: { p50: 341, p95: 378.86, p99: 392 },
|
|
});
|
|
assert.deepEqual(stats, { totalRequests: 1985, totalInputTokens: 312327705, totalOutputTokens: 932710, genP95: 378.86 });
|
|
});
|
|
|
|
test("parseStats defaults missing fields and rejects malformed input", () => {
|
|
assert.deepEqual(parseStats({}), { totalRequests: 0, totalInputTokens: 0, totalOutputTokens: 0, genP95: 0 });
|
|
assert.equal(parseStats({ total_requests: "nope" })!.totalRequests, 0);
|
|
assert.equal(parseStats(null), null);
|
|
assert.equal(parseStats("x"), null);
|
|
});
|
|
|
|
test("fetchStats calls /api/metrics/stats with a model query", async () => {
|
|
const calls: string[] = [];
|
|
const orig = globalThis.fetch;
|
|
globalThis.fetch = (async (url: RequestInfo | URL) => {
|
|
calls.push(String(url));
|
|
return { ok: true, json: async () => ({ total_requests: 7 }) } as unknown as Response;
|
|
}) as typeof fetch;
|
|
try {
|
|
await fetchStats({ baseUrl: "http://x" }, "DeepSeek-V4-Flash-0731");
|
|
assert.equal(calls[0], "http://x/api/metrics/stats?model=DeepSeek-V4-Flash-0731");
|
|
} finally {
|
|
globalThis.fetch = orig;
|
|
}
|
|
});
|
|
|
|
test("fetchStats omits the model query for 'all' and returns null on HTTP error", async () => {
|
|
const calls: string[] = [];
|
|
const orig = globalThis.fetch;
|
|
globalThis.fetch = (async (url: RequestInfo | URL) => {
|
|
calls.push(String(url));
|
|
return { ok: false, status: 500 } as unknown as Response;
|
|
}) as typeof fetch;
|
|
try {
|
|
const stats = await fetchStats({ baseUrl: "http://x" }, "all");
|
|
assert.equal(stats, null);
|
|
assert.equal(calls[0], "http://x/api/metrics/stats");
|
|
} finally {
|
|
globalThis.fetch = orig;
|
|
}
|
|
});
|
|
```
|
|
|
|
- [ ] **Step 2: Run test to verify it fails**
|
|
|
|
Run: `npx tsx --test tests/stats.test.ts`
|
|
Expected: FAIL — `Cannot find module '../src/lib/stats'`.
|
|
|
|
- [ ] **Step 3: Write minimal implementation**
|
|
|
|
Create `src/lib/stats.ts`:
|
|
|
|
```ts
|
|
import { type LlamaSwapConfig } from "./util";
|
|
|
|
export interface UsageStats {
|
|
totalRequests: number;
|
|
totalInputTokens: number;
|
|
totalOutputTokens: number;
|
|
genP95: number;
|
|
}
|
|
|
|
export function parseStats(json: unknown): UsageStats | null {
|
|
if (!json || typeof json !== "object") return null;
|
|
const obj = json as Record<string, unknown>;
|
|
const genHist = (obj.gen_histogram ?? {}) as Record<string, unknown>;
|
|
const num = (v: unknown): number => (typeof v === "number" && Number.isFinite(v) ? v : 0);
|
|
return {
|
|
totalRequests: num(obj.total_requests),
|
|
totalInputTokens: num(obj.total_input_tokens),
|
|
totalOutputTokens: num(obj.total_output_tokens),
|
|
genP95: num(genHist.p95),
|
|
};
|
|
}
|
|
|
|
export async function fetchStats(cfg: LlamaSwapConfig, modelId: string): Promise<UsageStats | null> {
|
|
const query = modelId === "all" ? "" : `?model=${encodeURIComponent(modelId)}`;
|
|
const headers: Record<string, string> = {};
|
|
if (cfg.apiKey) headers["Authorization"] = `Bearer ${cfg.apiKey}`;
|
|
const res = await fetch(`${cfg.baseUrl}/api/metrics/stats${query}`, {
|
|
headers,
|
|
signal: AbortSignal.timeout(5000),
|
|
});
|
|
if (!res.ok) throw new Error(`stats HTTP ${res.status}`);
|
|
return parseStats(await res.json());
|
|
}
|
|
```
|
|
|
|
- [ ] **Step 4: Run test to verify it passes**
|
|
|
|
Run: `npx tsx --test tests/stats.test.ts`
|
|
Expected: PASS (4 tests).
|
|
|
|
- [ ] **Step 5: Commit**
|
|
|
|
```bash
|
|
git add src/lib/stats.ts tests/stats.test.ts
|
|
git commit -m "feat: add usage stats parsing and fetching from /api/metrics/stats"
|
|
```
|
|
|
|
---
|
|
|
|
### Task 2: StatsCache
|
|
|
|
**Files:**
|
|
- Create: `src/lib/stats-cache.ts`
|
|
- Test: `tests/stats-cache.test.ts`
|
|
|
|
**Interfaces:**
|
|
- Consumes: `fetchStats`, `UsageStats` from `./stats`; `LlamaSwapConfig` from `./util`.
|
|
- Produces:
|
|
- `class StatsCache`
|
|
- `constructor(cfg: LlamaSwapConfig, fetchFn?: FetchFn, pollMs?: number, throttleMs?: number)` where `FetchFn = (cfg: LlamaSwapConfig, modelId: string) => Promise<UsageStats | null>`
|
|
- `register(key: string): void`
|
|
- `unregister(key: string): void`
|
|
- `get(key: string): UsageStats | undefined`
|
|
- `scheduleRefresh(): void` — throttled (trailing, min `throttleMs` between polls)
|
|
- `refresh(): Promise<void>`
|
|
- `setConfig(cfg: LlamaSwapConfig): void`
|
|
- `onChange(listener: () => void): () => void`
|
|
|
|
- [ ] **Step 1: Write the failing test**
|
|
|
|
Create `tests/stats-cache.test.ts`:
|
|
|
|
```ts
|
|
import assert from "node:assert/strict";
|
|
import { test } from "node:test";
|
|
import { StatsCache, type FetchFn } from "../src/lib/stats-cache";
|
|
import { type LlamaSwapConfig } from "../src/lib/util";
|
|
|
|
const cfg: LlamaSwapConfig = { baseUrl: "http://x" };
|
|
|
|
function stubFetch(result: unknown, counter: { count: number }): FetchFn {
|
|
return async () => {
|
|
counter.count++;
|
|
return result as never;
|
|
};
|
|
}
|
|
|
|
async function flush(): Promise<void> {
|
|
await new Promise((r) => setTimeout(r, 0));
|
|
}
|
|
|
|
test("register polls once immediately and caches the value", async () => {
|
|
const calls = { count: 0 };
|
|
const cache = new StatsCache(cfg, stubFetch({ totalRequests: 3, totalInputTokens: 1, totalOutputTokens: 2, genP95: 4 }, calls), 1000, 30);
|
|
cache.register("all");
|
|
await flush();
|
|
assert.equal(calls.count, 1);
|
|
assert.equal(cache.get("all")!.totalRequests, 3);
|
|
});
|
|
|
|
test("multiple keys are each polled", async () => {
|
|
const keys: string[] = [];
|
|
const cache = new StatsCache(
|
|
cfg,
|
|
async (_c, key) => {
|
|
keys.push(key);
|
|
return { totalRequests: 1, totalInputTokens: 1, totalOutputTokens: 1, genP95: 1 };
|
|
},
|
|
1000,
|
|
30,
|
|
);
|
|
cache.register("a");
|
|
cache.register("b");
|
|
await flush();
|
|
assert.deepEqual(keys.sort(), ["a", "b"]);
|
|
});
|
|
|
|
test("unregister stops the interval and clears the value", async () => {
|
|
const calls = { count: 0 };
|
|
const cache = new StatsCache(cfg, stubFetch({ totalRequests: 1, totalInputTokens: 1, totalOutputTokens: 1, genP95: 1 }, calls), 10, 30);
|
|
cache.register("a");
|
|
await flush();
|
|
assert.equal(calls.count, 1);
|
|
cache.unregister("a");
|
|
await new Promise((r) => setTimeout(r, 40));
|
|
assert.equal(cache.get("a"), undefined);
|
|
assert.equal(calls.count, 1);
|
|
});
|
|
|
|
test("scheduleRefresh throttles to one poll per window", async () => {
|
|
const calls = { count: 0 };
|
|
const cache = new StatsCache(cfg, stubFetch({ totalRequests: 1, totalInputTokens: 1, totalOutputTokens: 1, genP95: 1 }, calls), 10000, 30);
|
|
cache.register("a");
|
|
await flush();
|
|
const before = calls.count;
|
|
cache.scheduleRefresh();
|
|
cache.scheduleRefresh();
|
|
cache.scheduleRefresh();
|
|
await flush();
|
|
assert.equal(calls.count, before);
|
|
await new Promise((r) => setTimeout(r, 60));
|
|
assert.equal(calls.count, before + 1);
|
|
cache.unregister("a");
|
|
});
|
|
|
|
test("fetch failure keeps the last-known value", async () => {
|
|
const fail: FetchFn = async () => {
|
|
throw new Error("boom");
|
|
};
|
|
const cache = new StatsCache(cfg, fail, 10000, 30);
|
|
cache.register("a");
|
|
await flush();
|
|
assert.equal(cache.get("a"), undefined);
|
|
cache.unregister("a");
|
|
});
|
|
```
|
|
|
|
- [ ] **Step 2: Run test to verify it fails**
|
|
|
|
Run: `npx tsx --test tests/stats-cache.test.ts`
|
|
Expected: FAIL — `Cannot find module '../src/lib/stats-cache'`.
|
|
|
|
- [ ] **Step 3: Write minimal implementation**
|
|
|
|
Create `src/lib/stats-cache.ts`:
|
|
|
|
```ts
|
|
import { fetchStats, type UsageStats } from "./stats";
|
|
import { type LlamaSwapConfig } from "./util";
|
|
|
|
const POLL_MS = 5000;
|
|
const ACTIVITY_THROTTLE_MS = 2000;
|
|
|
|
export type FetchFn = (cfg: LlamaSwapConfig, modelId: string) => Promise<UsageStats | null>;
|
|
|
|
export class StatsCache {
|
|
private keys = new Set<string>();
|
|
private values = new Map<string, UsageStats | undefined>();
|
|
private listeners = new Set<() => void>();
|
|
private timer?: ReturnType<typeof setInterval>;
|
|
private throttleTimer?: ReturnType<typeof setTimeout>;
|
|
private lastRefresh = 0;
|
|
|
|
constructor(
|
|
private cfg: LlamaSwapConfig,
|
|
private fetchFn: FetchFn = fetchStats,
|
|
private pollMs = POLL_MS,
|
|
private throttleMs = ACTIVITY_THROTTLE_MS,
|
|
) {}
|
|
|
|
setConfig(cfg: LlamaSwapConfig): void {
|
|
this.cfg = cfg;
|
|
this.values.clear();
|
|
}
|
|
|
|
register(key: string): void {
|
|
if (this.keys.has(key)) return;
|
|
this.keys.add(key);
|
|
if (!this.timer) {
|
|
void this.refresh();
|
|
this.timer = setInterval(() => void this.refresh(), this.pollMs);
|
|
}
|
|
}
|
|
|
|
unregister(key: string): void {
|
|
this.keys.delete(key);
|
|
this.values.delete(key);
|
|
if (this.keys.size === 0 && this.timer) {
|
|
clearInterval(this.timer);
|
|
this.timer = undefined;
|
|
}
|
|
}
|
|
|
|
get(key: string): UsageStats | undefined {
|
|
return this.values.get(key);
|
|
}
|
|
|
|
scheduleRefresh(): void {
|
|
const now = Date.now();
|
|
const wait = Math.max(0, this.throttleMs - (now - this.lastRefresh));
|
|
if (this.throttleTimer) clearTimeout(this.throttleTimer);
|
|
this.throttleTimer = setTimeout(() => {
|
|
this.throttleTimer = undefined;
|
|
void this.refresh();
|
|
}, wait);
|
|
}
|
|
|
|
async refresh(): Promise<void> {
|
|
this.lastRefresh = Date.now();
|
|
for (const key of this.keys) {
|
|
try {
|
|
const stats = await this.fetchFn(this.cfg, key);
|
|
if (stats) this.values.set(key, stats);
|
|
} catch {
|
|
}
|
|
}
|
|
this.emit();
|
|
}
|
|
|
|
onChange(listener: () => void): () => void {
|
|
this.listeners.add(listener);
|
|
return () => {
|
|
this.listeners.delete(listener);
|
|
};
|
|
}
|
|
|
|
private emit(): void {
|
|
for (const listener of this.listeners) listener();
|
|
}
|
|
}
|
|
```
|
|
|
|
- [ ] **Step 4: Run test to verify it passes**
|
|
|
|
Run: `npx tsx --test tests/stats-cache.test.ts`
|
|
Expected: PASS (5 tests).
|
|
|
|
- [ ] **Step 5: Commit**
|
|
|
|
```bash
|
|
git add src/lib/stats-cache.ts tests/stats-cache.test.ts
|
|
git commit -m "feat: add StatsCache for shared per-model usage polling"
|
|
```
|
|
|
|
---
|
|
|
|
### Task 3: Decode `activity` SSE events + tracker total
|
|
|
|
**Files:**
|
|
- Modify: `src/lib/inflight-tracker.ts`, `src/lib/event-feed.ts`
|
|
- Test: `tests/event-feed.test.ts`, `tests/inflight-tracker.test.ts`
|
|
|
|
**Interfaces:**
|
|
- Consumes: existing `FeedEvent` in `inflight-tracker.ts`.
|
|
- Produces:
|
|
- `FeedEvent` gains `| { type: "activity"; id: number }`
|
|
- `InflightTracker.total(): number`
|
|
- `decodeEvent` handles `{"type":"activity","data":"{\"id\":817}"}`.
|
|
|
|
- [ ] **Step 1: Write the failing tests**
|
|
|
|
Append to `tests/event-feed.test.ts`:
|
|
|
|
```ts
|
|
test("decodeEvent parses an activity event with an id", () => {
|
|
const msg: SseMessage = {
|
|
event: "message",
|
|
data: JSON.stringify({ type: "activity", data: JSON.stringify({ id: 817 }) }),
|
|
};
|
|
const ev = decodeEvent(msg);
|
|
assert.deepEqual(ev, { type: "activity", id: 817 });
|
|
});
|
|
```
|
|
|
|
Append to `tests/inflight-tracker.test.ts`:
|
|
|
|
```ts
|
|
test("total sums in-flight requests across all models", () => {
|
|
const tracker = new InflightTracker();
|
|
assert.equal(tracker.total(), 0);
|
|
tracker.apply({ type: "inflight", operation: "add", requests: [{ model: "A", id: "1" }, { model: "B", id: "2" }] });
|
|
tracker.apply({ type: "inflight", operation: "add", requests: [{ model: "A", id: "3" }] });
|
|
assert.equal(tracker.total(), 3);
|
|
tracker.apply({ type: "inflight", operation: "remove", id: "1" });
|
|
assert.equal(tracker.total(), 2);
|
|
});
|
|
```
|
|
|
|
- [ ] **Step 2: Run tests to verify they fail**
|
|
|
|
Run: `npx tsx --test tests/event-feed.test.ts tests/inflight-tracker.test.ts`
|
|
Expected: FAIL — activity event decodes to `null`; `tracker.total is not a function`.
|
|
|
|
- [ ] **Step 3: Write minimal implementation**
|
|
|
|
In `src/lib/inflight-tracker.ts`, change the `FeedEvent` union to:
|
|
|
|
```ts
|
|
export type FeedEvent =
|
|
| { type: "inflight"; operation: "snapshot" | "add"; requests: InflightRequest[] }
|
|
| { type: "inflight"; operation: "remove"; id: string }
|
|
| { type: "activity"; id: number }
|
|
| { type: "modelStatus"; models: ModelState[] };
|
|
```
|
|
|
|
and add a method to `InflightTracker` (after `count`):
|
|
|
|
```ts
|
|
total(): number {
|
|
return this.requests.size;
|
|
}
|
|
```
|
|
|
|
In `src/lib/event-feed.ts`, inside `decodeEvent`, before the `if (outer.type === "inflight")` block, add:
|
|
|
|
```ts
|
|
if (outer.type === "activity") {
|
|
const id = typeof inner.id === "number" ? inner.id : Number.NaN;
|
|
if (Number.isFinite(id)) return { type: "activity", id };
|
|
return null;
|
|
}
|
|
```
|
|
|
|
- [ ] **Step 4: Run tests to verify they pass**
|
|
|
|
Run: `npx tsx --test tests/event-feed.test.ts tests/inflight-tracker.test.ts`
|
|
Expected: PASS.
|
|
|
|
- [ ] **Step 5: Commit**
|
|
|
|
```bash
|
|
git add src/lib/inflight-tracker.ts src/lib/event-feed.ts tests/event-feed.test.ts tests/inflight-tracker.test.ts
|
|
git commit -m "feat: decode activity SSE events and expose tracker total"
|
|
```
|
|
|
|
---
|
|
|
|
### Task 4: Runtime integration
|
|
|
|
**Files:**
|
|
- Modify: `src/lib/runtime.ts`
|
|
|
|
**Interfaces:**
|
|
- Consumes: `StatsCache` from `./stats-cache`; `UsageStats` from `./stats`; `FeedEvent` (already imported via tracker).
|
|
- Produces (public API used by the In-Flight action):
|
|
- `runtime.getStats(modelId: string): UsageStats | undefined`
|
|
- `runtime.watchStats(modelId: string): () => void`
|
|
|
|
- [ ] **Step 1: Modify `src/lib/runtime.ts`**
|
|
|
|
Replace the whole file body after the class opening as shown:
|
|
|
|
```ts
|
|
import { EventFeed } from "./event-feed";
|
|
import { InflightTracker } from "./inflight-tracker";
|
|
import { MetricsPoller } from "./metrics-poller";
|
|
import { StatsCache } from "./stats-cache";
|
|
import { type UsageStats } from "./stats";
|
|
import { type LlamaSwapConfig } from "./util";
|
|
|
|
class Runtime {
|
|
readonly tracker = new InflightTracker();
|
|
offline = true;
|
|
private cfg?: LlamaSwapConfig;
|
|
private feed?: EventFeed;
|
|
private poller?: MetricsPoller;
|
|
private statsCache?: StatsCache;
|
|
private statsUnsub?: () => void;
|
|
private listeners = new Set<() => void>();
|
|
|
|
ensureConnections(cfg: LlamaSwapConfig): void {
|
|
const changed = !this.cfg || this.cfg.baseUrl !== cfg.baseUrl || this.cfg.apiKey !== cfg.apiKey;
|
|
if (changed) {
|
|
this.feed?.stop();
|
|
this.poller?.stop();
|
|
this.feed = undefined;
|
|
this.poller = undefined;
|
|
this.statsUnsub?.();
|
|
this.statsCache = undefined;
|
|
this.statsUnsub = undefined;
|
|
this.cfg = cfg;
|
|
}
|
|
if (!this.feed) {
|
|
this.feed = new EventFeed(cfg, (ev) => {
|
|
this.tracker.apply(ev);
|
|
if (ev.type === "activity") this.statsCache?.scheduleRefresh();
|
|
this.emit();
|
|
});
|
|
this.feed.setStatusHandler((connected) => {
|
|
this.offline = !connected;
|
|
this.emit();
|
|
});
|
|
this.feed.start();
|
|
}
|
|
if (!this.poller) {
|
|
this.poller = new MetricsPoller(cfg);
|
|
this.poller.on(() => this.emit());
|
|
this.poller.start();
|
|
}
|
|
if (!this.statsCache) {
|
|
this.statsCache = new StatsCache(cfg);
|
|
this.statsUnsub = this.statsCache.onChange(() => this.emit());
|
|
}
|
|
}
|
|
|
|
get pollerInstance(): MetricsPoller | undefined {
|
|
return this.poller;
|
|
}
|
|
|
|
getStats(modelId: string): UsageStats | undefined {
|
|
return this.statsCache?.get(modelId);
|
|
}
|
|
|
|
watchStats(modelId: string): () => void {
|
|
this.statsCache?.register(modelId);
|
|
return () => this.statsCache?.unregister(modelId);
|
|
}
|
|
|
|
subscribe(listener: () => void): () => void {
|
|
this.listeners.add(listener);
|
|
return () => {
|
|
this.listeners.delete(listener);
|
|
};
|
|
}
|
|
|
|
private emit(): void {
|
|
for (const listener of this.listeners) listener();
|
|
}
|
|
}
|
|
|
|
export const runtime = new Runtime();
|
|
```
|
|
|
|
- [ ] **Step 2: Typecheck and run the full suite**
|
|
|
|
Run: `npx tsc --noEmit && npm test`
|
|
Expected: `TYPECHECK OK` equivalent (no errors) and all tests pass.
|
|
|
|
- [ ] **Step 3: Commit**
|
|
|
|
```bash
|
|
git add src/lib/runtime.ts
|
|
git commit -m "feat: expose usage stats via shared runtime StatsCache"
|
|
```
|
|
|
|
---
|
|
|
|
### Task 5: `renderUsage` in render.ts
|
|
|
|
**Files:**
|
|
- Modify: `src/lib/render.ts`
|
|
- Test: `tests/render.test.ts`
|
|
|
|
**Interfaces:**
|
|
- Consumes: `UsageStats` from `./stats`.
|
|
- Produces:
|
|
- `interface UsageRenderOptions { modelName: string; stats?: UsageStats; primaryStat: "requests" | "input_tokens" | "output_tokens" | "gen_p95"; offline: boolean }`
|
|
- `renderUsage(opts: UsageRenderOptions): string`
|
|
- `formatCompact(n: number): string` (exported for tests)
|
|
|
|
- [ ] **Step 1: Write the failing tests**
|
|
|
|
Append to `tests/render.test.ts`:
|
|
|
|
```ts
|
|
import { renderGpuGraph, renderInflight, renderUsage, svgDataUrl, formatCompact } from "../src/lib/render";
|
|
```
|
|
|
|
(replace the existing import line) and append:
|
|
|
|
```ts
|
|
test("formatCompact renders integers, k, and M", () => {
|
|
assert.equal(formatCompact(0), "0");
|
|
assert.equal(formatCompact(999), "999");
|
|
assert.equal(formatCompact(52100), "52.1k");
|
|
assert.equal(formatCompact(1200000), "1.2M");
|
|
});
|
|
|
|
test("renderUsage shows the primary stat big and the rest small", () => {
|
|
const svg = renderUsage({
|
|
modelName: "DeepSeek-V4-Flash-0731",
|
|
stats: { totalRequests: 1950, totalInputTokens: 312177540, totalOutputTokens: 889193, genP95: 378.86 },
|
|
primaryStat: "gen_p95",
|
|
offline: false,
|
|
});
|
|
assert.match(svg, />379</);
|
|
assert.match(svg, /REQ/);
|
|
assert.match(svg, /IN/);
|
|
assert.match(svg, /OUT/);
|
|
assert.match(svg, /312.2M/);
|
|
assert.match(svg, /#10131a/);
|
|
});
|
|
|
|
test("renderUsage renders -- when stats are unknown and OFFLINE when offline", () => {
|
|
const none = renderUsage({ modelName: "A", stats: undefined, primaryStat: "requests", offline: false });
|
|
assert.match(none, />--</);
|
|
const off = renderUsage({ modelName: "A", stats: undefined, primaryStat: "requests", offline: true });
|
|
assert.match(off, /OFFLINE/);
|
|
});
|
|
|
|
test("renderUsage labels the aggregate view ALL MODELS", () => {
|
|
const svg = renderUsage({ modelName: "all", stats: { totalRequests: 1, totalInputTokens: 1, totalOutputTokens: 1, genP95: 1 }, primaryStat: "requests", offline: false });
|
|
assert.match(svg, /ALL MODELS/);
|
|
});
|
|
```
|
|
|
|
- [ ] **Step 2: Run test to verify it fails**
|
|
|
|
Run: `npx tsx --test tests/render.test.ts`
|
|
Expected: FAIL — `Cannot find name 'renderUsage'` / `formatCompact`.
|
|
|
|
- [ ] **Step 3: Write minimal implementation**
|
|
|
|
In `src/lib/render.ts`, add an import of `UsageStats` and append the following at the end of the file (before helper functions or after — helpers can be placed after `renderUsage`):
|
|
|
|
```ts
|
|
export function formatCompact(n: number): string {
|
|
if (!Number.isFinite(n)) return "--";
|
|
if (n < 1000) return `${Math.round(n)}`;
|
|
if (n < 1_000_000) return `${(n / 1000).toFixed(1)}k`;
|
|
return `${(n / 1_000_000).toFixed(1)}M`;
|
|
}
|
|
|
|
export interface UsageRenderOptions {
|
|
modelName: string;
|
|
stats?: UsageStats;
|
|
primaryStat: "requests" | "input_tokens" | "output_tokens" | "gen_p95";
|
|
offline: boolean;
|
|
}
|
|
|
|
export function renderUsage(opts: UsageRenderOptions): string {
|
|
const name = escapeXml(shorten(opts.modelName === "all" ? "ALL MODELS" : opts.modelName));
|
|
|
|
if (opts.offline) {
|
|
return frame("#10131a", [
|
|
centerText("!!", 34, 20, "bold", "#e0e0e0"),
|
|
centerText("OFFLINE", 52, 9, "normal", "#bdbdbd"),
|
|
centerText(name, 64, 7, "normal", "#ffffff"),
|
|
]);
|
|
}
|
|
|
|
const s = opts.stats;
|
|
const big = s ? formatCompact(primaryValue(s, opts.primaryStat)) : "--";
|
|
const rows = usageRows(s, opts.primaryStat);
|
|
const parts: string[] = [
|
|
centerText(name, 10, 7, "normal", "#ffffff"),
|
|
centerText(big, 36, 24, "bold", "#ffffff"),
|
|
];
|
|
for (const row of rows) {
|
|
parts.push(
|
|
`<text x="4" y="${row.y}" font-family="Arial,sans-serif" font-size="7" fill="#8b93a5">${row.label}</text>`,
|
|
`<text x="68" y="${row.y}" text-anchor="end" font-family="Arial,sans-serif" font-size="7" fill="#ffffff">${row.value}</text>`,
|
|
);
|
|
}
|
|
return frame("#10131a", parts);
|
|
}
|
|
|
|
function primaryValue(s: UsageStats, primary: "requests" | "input_tokens" | "output_tokens" | "gen_p95"): number {
|
|
switch (primary) {
|
|
case "requests":
|
|
return s.totalRequests;
|
|
case "input_tokens":
|
|
return s.totalInputTokens;
|
|
case "output_tokens":
|
|
return s.totalOutputTokens;
|
|
case "gen_p95":
|
|
return s.genP95;
|
|
}
|
|
}
|
|
|
|
function usageRows(
|
|
s: UsageStats | undefined,
|
|
primary: "requests" | "input_tokens" | "output_tokens" | "gen_p95",
|
|
): { label: string; value: string; y: number }[] {
|
|
const items: { label: string; value: string }[] = [];
|
|
if (primary !== "requests") items.push({ label: "REQ", value: s ? formatCompact(s.totalRequests) : "--" });
|
|
if (primary !== "input_tokens") items.push({ label: "IN", value: s ? formatCompact(s.totalInputTokens) : "--" });
|
|
if (primary !== "output_tokens") items.push({ label: "OUT", value: s ? formatCompact(s.totalOutputTokens) : "--" });
|
|
if (primary !== "gen_p95") items.push({ label: "P95", value: s ? `${Math.round(s.genP95)} t/s` : "--" });
|
|
return items.map((it, i) => ({ ...it, y: 52 + i * 7 }));
|
|
}
|
|
```
|
|
|
|
- [ ] **Step 4: Run test to verify it passes**
|
|
|
|
Run: `npx tsx --test tests/render.test.ts`
|
|
Expected: PASS.
|
|
|
|
- [ ] **Step 5: Commit**
|
|
|
|
```bash
|
|
git add src/lib/render.ts tests/render.test.ts
|
|
git commit -m "feat: add usage stats key renderer"
|
|
```
|
|
|
|
---
|
|
|
|
### Task 6: In-Flight action display modes + property inspector
|
|
|
|
**Files:**
|
|
- Modify: `src/actions/inflight-monitor.ts`, `com.bryce.llamawatch.sdPlugin/ui/inflight.html`, `src/lib/datasources.ts`
|
|
|
|
**Interfaces:**
|
|
- Consumes: `renderUsage`, `renderInflight`, `formatCompact` (render); `runtime.getStats`/`watchStats`; `InflightTracker.total()`.
|
|
- Produces:
|
|
- `InflightSettings` gains `display?: "count" | "usage"` and `primaryStat?: "requests" | "input_tokens" | "output_tokens" | "gen_p95"`.
|
|
- `InflightState` gains `unwatchStats?: () => void`.
|
|
- Models datasource gains an `all` option.
|
|
|
|
- [ ] **Step 1: Update the property inspector**
|
|
|
|
In `com.bryce.llamawatch.sdPlugin/ui/inflight.html`, after the API Key item, add:
|
|
|
|
```html
|
|
<sdpi-item label="Display">
|
|
<sdpi-select setting="display" default="count" placeholder="Select a display mode">
|
|
<option value="count">Request count + activity</option>
|
|
<option value="usage">Usage stats</option>
|
|
</sdpi-select>
|
|
</sdpi-item>
|
|
<sdpi-item label="Primary stat (usage view)">
|
|
<sdpi-select setting="primaryStat" default="requests" placeholder="Select a primary stat">
|
|
<option value="requests">Requests</option>
|
|
<option value="input_tokens">Processed tokens</option>
|
|
<option value="output_tokens">Generated tokens</option>
|
|
<option value="gen_p95">Generation speed P95</option>
|
|
</sdpi-select>
|
|
</sdpi-item>
|
|
```
|
|
|
|
- [ ] **Step 2: Update the models datasource**
|
|
|
|
In `src/lib/datasources.ts`, inside the `event === "models"` branch, change the assignment to prepend the aggregate option:
|
|
|
|
```ts
|
|
items = [{ label: "All models", value: "all" }, ...models.map((m) => ({ label: `${m.id} (${m.status})`, value: m.id }))];
|
|
```
|
|
|
|
- [ ] **Step 3: Update the In-Flight action**
|
|
|
|
In `src/actions/inflight-monitor.ts`:
|
|
|
|
1. Add to the settings type:
|
|
|
|
```ts
|
|
type InflightSettings = CfgSettings & {
|
|
modelId?: string;
|
|
display?: "count" | "usage";
|
|
primaryStat?: "requests" | "input_tokens" | "output_tokens" | "gen_p95";
|
|
};
|
|
```
|
|
|
|
2. Add to `InflightState`:
|
|
|
|
```ts
|
|
unwatchStats?: () => void;
|
|
```
|
|
|
|
3. In `onWillAppear`, after `state.settings = ev.payload.settings;`, replace the sampler setup block:
|
|
|
|
```ts
|
|
state.unsubscribe = runtime.subscribe(() => this.render(state));
|
|
if (this.displayOf(state) === "usage") {
|
|
state.unwatchStats = runtime.watchStats(this.modelKeyOf(state));
|
|
} else {
|
|
state.sampler = setInterval(() => this.sample(state), 1000);
|
|
}
|
|
this.render(state);
|
|
```
|
|
|
|
4. In `onWillDisappear`, add cleanup after `state.unsubscribe?.();`:
|
|
|
|
```ts
|
|
state.unwatchStats?.();
|
|
state.unwatchStats = undefined;
|
|
```
|
|
|
|
5. In `onDidReceiveSettings`, after `state.settings = ev.payload.settings;`, add re-registration:
|
|
|
|
```ts
|
|
state.unwatchStats?.();
|
|
state.unwatchStats = undefined;
|
|
if (this.displayOf(state) === "usage") {
|
|
state.unwatchStats = runtime.watchStats(this.modelKeyOf(state));
|
|
} else if (!state.sampler) {
|
|
state.sampler = setInterval(() => this.sample(state), 1000);
|
|
}
|
|
```
|
|
|
|
6. Replace `render` with a mode-switching version and add helpers:
|
|
|
|
```ts
|
|
private displayOf(state: InflightState): "count" | "usage" {
|
|
return state.settings.display ?? "count";
|
|
}
|
|
|
|
private modelKeyOf(state: InflightState): string {
|
|
const modelId = state.settings.modelId ?? "";
|
|
return modelId === "" ? "all" : modelId;
|
|
}
|
|
|
|
private primaryStatOf(state: InflightState): "requests" | "input_tokens" | "output_tokens" | "gen_p95" {
|
|
return state.settings.primaryStat ?? "requests";
|
|
}
|
|
|
|
private render(state: InflightState): void {
|
|
const action = state.action;
|
|
if (!action) return;
|
|
if (this.displayOf(state) === "usage") {
|
|
const modelKey = this.modelKeyOf(state);
|
|
const offline = runtime.offline;
|
|
void action.setImage(
|
|
svgDataUrl(
|
|
renderUsage({
|
|
modelName: modelKey,
|
|
stats: runtime.getStats(modelKey),
|
|
primaryStat: this.primaryStatOf(state),
|
|
offline,
|
|
}),
|
|
),
|
|
);
|
|
return;
|
|
}
|
|
|
|
const modelId = state.settings.modelId ?? "";
|
|
const trackerState = modelId === "all" ? "ready" : runtime.tracker.state(modelId);
|
|
const count = modelId === "all" ? runtime.tracker.total() : runtime.tracker.count(modelId);
|
|
const offline = runtime.offline && modelId.length > 0;
|
|
const modelName = modelId === "" ? "unset" : modelId;
|
|
void action.setImage(
|
|
svgDataUrl(
|
|
renderInflight({
|
|
modelName,
|
|
state: trackerState,
|
|
count,
|
|
offline,
|
|
history: state.history,
|
|
}),
|
|
),
|
|
);
|
|
}
|
|
|
|
private sample(state: InflightState): void {
|
|
const modelId = state.settings.modelId ?? "";
|
|
state.history.push(modelId === "all" ? runtime.tracker.total() : runtime.tracker.count(modelId));
|
|
if (state.history.length > HISTORY_LIMIT) state.history.shift();
|
|
this.render(state);
|
|
}
|
|
```
|
|
|
|
Note: for `modelId === "all"` in count mode, `trackerState` is forced to `"ready"` so the key renders the total number rather than `NO MODEL`/`OFF`. The label comes from `renderInflight`'s `name` (`ALL MODELS` won't match — see step 4).
|
|
|
|
- [ ] **Step 4: Ensure the count-mode "all" label renders correctly**
|
|
|
|
In `src/lib/render.ts`, `renderInflight` shortens the model name. For `modelName === "all"` pass the display name through `shorten`. Update the line in `renderInflight`:
|
|
|
|
```ts
|
|
const name = escapeXml(shorten(opts.modelName === "all" ? "ALL MODELS" : opts.modelName));
|
|
```
|
|
|
|
(one-line change; mirrors the `renderUsage` approach).
|
|
|
|
- [ ] **Step 5: Typecheck, build, and run the suite**
|
|
|
|
Run: `npx tsc --noEmit && npm run build && npm test`
|
|
Expected: no type errors, build succeeds, all tests pass.
|
|
|
|
- [ ] **Step 6: Commit**
|
|
|
|
```bash
|
|
git add src/actions/inflight-monitor.ts src/lib/render.ts src/lib/datasources.ts com.bryce.llamawatch.sdPlugin/ui/inflight.html
|
|
git commit -m "feat: add usage-stats display mode to In-Flight Monitor action"
|
|
```
|
|
|
|
---
|
|
|
|
### Task 7: GPU `combineSeries` helper
|
|
|
|
**Files:**
|
|
- Modify: `src/lib/metrics-poller.ts`
|
|
- Test: `tests/metrics-poller.test.ts`
|
|
|
|
**Interfaces:**
|
|
- Consumes: `AGGREGATE` (already defined in `metrics-poller.ts`).
|
|
- Produces: `combineSeries(histories: number[][], kind: GpuMetricKind): { value?: number; history: number[] }` (exported).
|
|
|
|
- [ ] **Step 1: Write the failing test**
|
|
|
|
Append to `tests/metrics-poller.test.ts`:
|
|
|
|
```ts
|
|
import { combineSeries, MetricsPoller } from "../src/lib/metrics-poller";
|
|
```
|
|
|
|
(update the existing import) and append:
|
|
|
|
```ts
|
|
test("combineSeries averages util and sums power", () => {
|
|
const util = combineSeries(
|
|
[
|
|
[10, 20, 30],
|
|
[20, 40, 60],
|
|
],
|
|
"util_percent",
|
|
);
|
|
assert.deepEqual(util, { value: 45, history: [15, 30, 45] });
|
|
|
|
const power = combineSeries(
|
|
[
|
|
[100, 200],
|
|
[150, 250],
|
|
],
|
|
"power",
|
|
);
|
|
assert.deepEqual(power, { value: 450, history: [250, 450] });
|
|
});
|
|
|
|
test("combineSeries takes the max temperature and skips gaps", () => {
|
|
const temp = combineSeries([[50, 70], [60]], "temperature");
|
|
assert.deepEqual(temp, { value: 70, history: [60, 70] });
|
|
|
|
const empty = combineSeries([[], []], "temperature");
|
|
assert.deepEqual(empty, { value: undefined, history: [] });
|
|
});
|
|
```
|
|
|
|
- [ ] **Step 2: Run test to verify it fails**
|
|
|
|
Run: `npx tsx --test tests/metrics-poller.test.ts`
|
|
Expected: FAIL — `Cannot find name 'combineSeries'`.
|
|
|
|
- [ ] **Step 3: Write minimal implementation**
|
|
|
|
In `src/lib/metrics-poller.ts`, after the `AGGREGATE` const, add and export:
|
|
|
|
```ts
|
|
export function combineSeries(histories: number[][], kind: GpuMetricKind): { value?: number; history: number[] } {
|
|
const n = Math.max(...histories.map((h) => h.length), 0);
|
|
if (n === 0) return { value: undefined, history: [] };
|
|
const history: number[] = [];
|
|
for (let i = 0; i < n; i++) {
|
|
const at = histories.map((h) => h[h.length - n + i]).filter((v): v is number => v !== undefined);
|
|
if (at.length === 0) continue;
|
|
history.push(AGGREGATE[kind](at));
|
|
}
|
|
return { value: history.length > 0 ? history[history.length - 1] : undefined, history };
|
|
}
|
|
```
|
|
|
|
- [ ] **Step 4: Run test to verify it passes**
|
|
|
|
Run: `npx tsx --test tests/metrics-poller.test.ts`
|
|
Expected: PASS.
|
|
|
|
- [ ] **Step 5: Commit**
|
|
|
|
```bash
|
|
git add src/lib/metrics-poller.ts tests/metrics-poller.test.ts
|
|
git commit -m "feat: add combineSeries for GPU combination aggregation"
|
|
```
|
|
|
|
---
|
|
|
|
### Task 8: GPU action combinations + property inspector
|
|
|
|
**Files:**
|
|
- Modify: `src/actions/gpu-graph.ts`, `com.bryce.llamawatch.sdPlugin/ui/gpu.html`
|
|
|
|
**Interfaces:**
|
|
- Consumes: `combineSeries` from `./metrics-poller`.
|
|
- Produces:
|
|
- `GpuSettings` gains `gpuCombo?: string`.
|
|
- When `gpuCombo` is a non-empty comma-separated list of ids, the key shows the combined series of those GPUs, labeled `GPU 0+2`; otherwise existing `gpuId` behavior.
|
|
|
|
- [ ] **Step 1: Update the property inspector**
|
|
|
|
In `com.bryce.llamawatch.sdPlugin/ui/gpu.html`, after the GPU select item, add:
|
|
|
|
```html
|
|
<sdpi-item label="GPU combination">
|
|
<sdpi-textfield setting="gpuCombo" placeholder="e.g. 0,2 (overrides GPU)" />
|
|
</sdpi-item>
|
|
```
|
|
|
|
- [ ] **Step 2: Update the GPU action**
|
|
|
|
In `src/actions/gpu-graph.ts`:
|
|
|
|
1. Add to the settings type:
|
|
|
|
```ts
|
|
type GpuSettings = CfgSettings & {
|
|
gpuId?: string;
|
|
metric?: GpuMetricKind;
|
|
gpuCombo?: string;
|
|
};
|
|
```
|
|
|
|
2. Replace the `render` method and add a `parseCombo` helper:
|
|
|
|
```ts
|
|
private render(state: GpuState): void {
|
|
const action = state.action;
|
|
if (!action) return;
|
|
const poller = runtime.pollerInstance;
|
|
const metric = state.settings.metric ?? "util_percent";
|
|
const combo = parseCombo(state.settings.gpuCombo);
|
|
const offline = !poller || poller.isOffline();
|
|
|
|
let value: number | undefined;
|
|
let history: number[] = [];
|
|
let gpuName: string;
|
|
if (combo) {
|
|
const known = new Set((poller?.gpus() ?? []).map((g) => g.id));
|
|
const ids = combo.filter((id) => known.has(id));
|
|
const histories = ids.map((id) => poller?.getHistory(id, metric) ?? []);
|
|
const combined = combineSeries(histories, metric);
|
|
value = combined.value;
|
|
history = combined.history;
|
|
gpuName = `GPU ${ids.join("+")}`;
|
|
} else {
|
|
const gpuId = state.settings.gpuId ?? "all";
|
|
value = poller?.getValue(gpuId, metric);
|
|
history = poller?.getHistory(gpuId, metric) ?? [];
|
|
gpuName = gpuId === "all" ? "ALL GPUS" : this.gpuLabel(gpuId);
|
|
}
|
|
|
|
const svg = renderGpuGraph({ gpuName, metric, value, history, offline });
|
|
if (svg === state.lastSvg) return;
|
|
state.lastSvg = svg;
|
|
void action.setImage(svgDataUrl(svg));
|
|
}
|
|
```
|
|
|
|
3. Add the helper at the bottom of the file:
|
|
|
|
```ts
|
|
function parseCombo(raw: string | undefined): string[] | undefined {
|
|
const ids = (raw ?? "")
|
|
.split(",")
|
|
.map((s) => s.trim())
|
|
.filter(Boolean);
|
|
return ids.length > 0 ? ids : undefined;
|
|
}
|
|
```
|
|
|
|
and import `combineSeries`:
|
|
|
|
```ts
|
|
import { combineSeries, type GpuMetricKind } from "../lib/metrics-poller";
|
|
```
|
|
|
|
- [ ] **Step 3: Typecheck, build, and run the suite**
|
|
|
|
Run: `npx tsc --noEmit && npm run build && npm test`
|
|
Expected: no type errors, build succeeds, all tests pass.
|
|
|
|
- [ ] **Step 4: Commit**
|
|
|
|
```bash
|
|
git add src/actions/gpu-graph.ts com.bryce.llamawatch.sdPlugin/ui/gpu.html
|
|
git commit -m "feat: support GPU combinations on the GPU Graph action"
|
|
```
|
|
|
|
---
|
|
|
|
### Task 9: Final verification
|
|
|
|
- [ ] **Step 1: Full verification**
|
|
|
|
Run:
|
|
```bash
|
|
npm test
|
|
npm run build
|
|
npx tsc --noEmit
|
|
streamdeck validate
|
|
```
|
|
Expected: all tests pass, build succeeds, typecheck clean, validation passes.
|
|
|
|
- [ ] **Step 2: Visual smoke test**
|
|
|
|
Run `npx tsx -e` to render a usage SVG and a combo GPU SVG and confirm they are well-formed (no malformed attributes), e.g.:
|
|
|
|
```bash
|
|
npx tsx -e '
|
|
import { renderUsage } from "./src/lib/render";
|
|
console.log(renderUsage({ modelName: "all", stats: { totalRequests: 1950, totalInputTokens: 312177540, totalOutputTokens: 889193, genP95: 378.86 }, primaryStat: "gen_p95", offline: false }).slice(0, 300));
|
|
'
|
|
```
|
|
|
|
- [ ] **Step 3: Restart the plugin**
|
|
|
|
Run: `streamdeck restart com.bryce.llamawatch`
|
|
Expected: `✔ Restarted com.bryce.llamawatch`.
|
|
|
|
- [ ] **Step 4: Update the progress ledger**
|
|
|
|
Append a short note to `.superpowers/sdd/progress.md` summarizing the feature and verification results (one or two lines, matching existing format).
|