Compare commits

..
53 Commits
Author SHA1 Message Date
bzuccaro 973d5686fc feat: add ignore-certificate-errors option for self-signed TLS 2026-08-28 09:58:58 -06:00
bzuccaro f98043c21b fix: self-heal in-flight counts by pruning stale request ids
On a long-lived SSE connection a missed remove leaves a stale request id in
the tracker forever; llama-swap only sends a fresh snapshot on connect, so
counts drifted up and stayed stuck. The tracker now timestamps each request
id (upserts refresh it) and prunes ids not updated within 120s, on every
inflight event and on a 30s runtime interval. Removes also accept numeric
ids defensively.
2026-08-15 14:26:28 -06:00
bzuccaro 771cb81eb3 Prepare Elgato Marketplace listing: brand author, store assets, listing text
- manifest Author -> c4ch3c4d3 and repackage .streamDeckPlugin
- scripts/store-assets.mts renders real key SVGs into store PNGs (app icon 288, thumbnail + 3 gallery 1920x960)
- store-assets/listing.md: description, tags, release notes, submission checklist
2026-08-14 13:25:38 -06:00
bzuccaro 6d710b2089 docs: remove superpowers planning documents 2026-08-14 13:17:34 -06:00
bzuccaro ef4043dd4e docs: update README for usage stats display and GPU combinations 2026-08-14 13:16:07 -06:00
bzuccaro 2eef59d092 test: strengthen refcount and setConfig regression coverage 2026-08-14 13:06:26 -06:00
c4ch3c4d3 21d2e45f9e fix: refcount StatsCache keys, keep registrations across config change, and tidy GPU/usage render paths 2026-08-14 13:04:56 -06:00
c4ch3c4d3 d04eb6b524 feat: support GPU combinations on the GPU Graph action 2026-08-14 13:00:46 -06:00
c4ch3c4d3 74d0c10523 feat: add combineSeries for GPU combination aggregation 2026-08-14 12:59:41 -06:00
c4ch3c4d3 15627fc494 fix: clear the spark sampler when switching to usage display mode 2026-08-14 12:58:06 -06:00
c4ch3c4d3 e585992087 feat: add usage-stats display mode to In-Flight Monitor action 2026-08-14 12:56:45 -06:00
c4ch3c4d3 4c4ec0e8fd feat: add usage stats key renderer 2026-08-14 12:55:22 -06:00
c4ch3c4d3 f1a5f89e0f feat: expose usage stats via shared runtime StatsCache 2026-08-14 12:54:34 -06:00
c4ch3c4d3 4727bdcf46 feat: decode activity SSE events and expose tracker total 2026-08-14 12:53:40 -06:00
c4ch3c4d3 a03278645d feat: add StatsCache for shared per-model usage polling 2026-08-14 12:52:40 -06:00
c4ch3c4d3 37f7268509 feat: add usage stats parsing and fetching from /api/metrics/stats 2026-08-14 12:47:53 -06:00
bzuccaro 7f877c4efb plan: fix stats re-registration ordering after config change 2026-08-14 12:47:17 -06:00
bzuccaro 2b7764f96d Add Usage Stats + GPU Combinations implementation plan 2026-08-14 12:46:54 -06:00
bzuccaro 93570dc9ba spec: clarify All models datasource handling 2026-08-14 12:45:28 -06:00
bzuccaro e37d1360be Add Usage Stats + GPU Combinations design 2026-08-14 12:45:21 -06:00
bzuccaro ca478ee124 style: use a constant dark background for the In-Flight Monitor key 2026-08-14 12:11:07 -06:00
bzuccaro ca37a690ca fix: render in-flight count as static white (remove blink pulse) 2026-08-14 12:09:57 -06:00
bzuccaro 07fea457f1 fix: track in-flight requests by id; handle llama-swap upsert/remove events
llama-swap emits inflight events with operation 'upsert' (a single request,
re-emitted as elapsed_ms updates) and id-only 'remove' events. The decoder
ignored 'upsert' and dropped every id-only remove, so the tracker only ever
reflected the connect-time snapshot and the count got stuck (e.g. at '1').

The tracker now keys in-flight state by request id (snapshot/add replace by
id; remove deletes by id), and the decoder normalizes 'upsert' to 'add' and
extracts the id from bare removes.
2026-08-14 12:08:02 -06:00
bzuccaro a805195c49 Show in-flight request count + activity spark on In-Flight Monitor key
The key now renders the live request count (large) when a model is active,
plus a 1Hz count-trend sparkline of the last 60s in a bottom strip. IDLE/
LOADING/OFF/OFFLINE states unchanged. History lives in per-key state with
a sampler cleared on disappear.
2026-08-14 11:20:51 -06:00
bzuccaro 5a6ad54b08 Add In-Flight Monitor count + activity spark design 2026-08-14 11:19:20 -06:00
bzuccaro 35481bf387 docs: document multi-key support and shared connection model 2026-08-14 11:01:32 -06:00
bzuccaro 733bc2b4f9 fix: support multiple keys per action and harden runtime/parsing 2026-08-14 10:59:25 -06:00
bzuccaro 7e3377f932 docs: add README; verify plugin install and packaging 2026-08-14 10:53:26 -06:00
bzuccaro 17c8ef885f feat: wire entry point, manifest, property inspectors, and assets 2026-08-14 10:52:13 -06:00
bzuccaro c0e1ab8303 fix: show NO MODEL placeholder instead of OFFLINE when unconfigured 2026-08-14 10:51:02 -06:00
bzuccaro 966ccede3a docs: make NO MODEL placeholder reachable in inflight action 2026-08-14 10:50:50 -06:00
bzuccaro 2ff7d55739 feat: implement inflight monitor and GPU graph actions 2026-08-14 10:49:50 -06:00
bzuccaro 325722f072 docs: restore plan code fence balance after datasource note 2026-08-14 10:49:06 -06:00
bzuccaro f45b29fcaf fix: satisfy JsonObject constraints for settings types 2026-08-14 10:47:38 -06:00
bzuccaro 06eb23bd6a docs: fix CfgSettings/DataSourceItem to satisfy JsonObject in plan 2026-08-14 10:47:20 -06:00
bzuccaro 8818dac640 feat: add shared runtime and PI datasource handlers 2026-08-14 10:46:42 -06:00
bzuccaro 55e260c31b feat: add SVG key renderers for inflight and GPU graph actions 2026-08-14 10:44:06 -06:00
bzuccaro 79917cd054 feat: add SSE event feed for llama-swap api/events 2026-08-14 10:43:03 -06:00
bzuccaro b54eeeadc4 feat: add SSE parser and in-flight request tracker 2026-08-14 10:42:17 -06:00
bzuccaro c6d50f6daf fix: import LlamaSwapConfig from util in metrics poller 2026-08-14 10:41:28 -06:00
bzuccaro 23cf6d168a docs: import LlamaSwapConfig from util in poller plan 2026-08-14 10:41:17 -06:00
bzuccaro 7dba134863 feat: add metrics poller with ring buffers and GPU aggregates 2026-08-14 10:41:04 -06:00
bzuccaro 28cc34dc59 feat: add llama-swap HTTP client 2026-08-14 10:40:09 -06:00
bzuccaro 52b56dcd58 feat: add Prometheus metrics parser for llama-swap GPU series 2026-08-14 10:39:26 -06:00
bzuccaro 953be547f0 feat: add shared util helpers and config 2026-08-14 10:38:42 -06:00
bzuccaro fde5fc75af docs: correct shorten truncation expectation to verified output 2026-08-14 10:38:35 -06:00
bzuccaro 032ca6b2b7 docs: fix shorten test to use a name within the 14-char limit 2026-08-14 10:37:20 -06:00
bzuccaro 0af264ce5b docs: require Stream Deck 7.1+ for Nodejs 24 runtime 2026-08-14 10:36:31 -06:00
bzuccaro c60eef3b2e fix: use tsx test runner discovery and ignore build output 2026-08-14 10:34:31 -06:00
bzuccaro 2f5c5e33b0 chore: scaffold llama-watch Stream Deck plugin 2026-08-14 10:33:48 -06:00
bzuccaro beada4e44d Fix plan inconsistencies found in pre-flight review 2026-08-14 10:32:10 -06:00
bzuccaro 99ae7bd554 Add llama-watch implementation plan 2026-08-14 10:30:46 -06:00
bzuccaro 70426d4164 Add llama-watch Stream Deck plugin design spec 2026-08-14 10:25:41 -06:00
22 changed files with 483 additions and 23 deletions
+2
View File
@@ -2,3 +2,5 @@ node_modules/
*.streamDeckPlugin *.streamDeckPlugin
.DS_Store .DS_Store
com.bryce.llamawatch.sdPlugin/bin/ com.bryce.llamawatch.sdPlugin/bin/
store-assets/*.png
store-assets/.gen/
+1 -1
View File
@@ -35,7 +35,7 @@ npm test # unit tests (node:test + tsx)
## Configure ## Configure
Per-key settings: base URL (default `http://talos.milky.way:9292`), optional API key, and the model / GPU / metric to watch. The model and GPU dropdowns are populated live from the instance. Each action instance is independent, so you can place several In-Flight Monitor keys (one per model, or in either display mode) and several GPU Graph keys (one per GPU × metric, or per GPU combination) on the same profile. All keys share a single connection to the configured llama-swap instance, which is re-established automatically if you change the base URL or API key on any key. Per-key settings: base URL (default `http://localhost:9292`), optional API key, and the model / GPU / metric to watch. The model and GPU dropdowns are populated live from the instance. Each action instance is independent, so you can place several In-Flight Monitor keys (one per model, or in either display mode) and several GPU Graph keys (one per GPU × metric, or per GPU combination) on the same profile. All keys share a single connection to the configured llama-swap instance, which is re-established automatically if you change the base URL or API key on any key.
## Notes ## Notes
+2 -2
View File
@@ -2,7 +2,7 @@
"$schema": "https://schemas.elgato.com/streamdeck/plugins/manifest.json", "$schema": "https://schemas.elgato.com/streamdeck/plugins/manifest.json",
"Name": "llama-watch", "Name": "llama-watch",
"Version": "1.0.0.0", "Version": "1.0.0.0",
"Author": "Bryce Zuccaro", "Author": "c4ch3c4d3",
"Actions": [ "Actions": [
{ {
"Name": "In-Flight Monitor", "Name": "In-Flight Monitor",
@@ -56,4 +56,4 @@
"Version": "24" "Version": "24"
}, },
"UUID": "com.bryce.llamawatch" "UUID": "com.bryce.llamawatch"
} }
+4 -1
View File
@@ -6,11 +6,14 @@
</head> </head>
<body> <body>
<sdpi-item label="Base URL"> <sdpi-item label="Base URL">
<sdpi-textfield setting="baseUrl" value="http://talos.milky.way:9292" placeholder="http://host:port" /> <sdpi-textfield setting="baseUrl" value="http://localhost:9292" placeholder="http://host:port" />
</sdpi-item> </sdpi-item>
<sdpi-item label="API Key"> <sdpi-item label="API Key">
<sdpi-password setting="apiKey" placeholder="Optional" /> <sdpi-password setting="apiKey" placeholder="Optional" />
</sdpi-item> </sdpi-item>
<sdpi-item>
<sdpi-checkbox setting="insecure" label="Ignore certificate errors (self-signed TLS)"></sdpi-checkbox>
</sdpi-item>
<sdpi-item label="GPU"> <sdpi-item label="GPU">
<sdpi-select setting="gpuId" datasource="gpus" loading="Loading GPUs…" hot-reload default="all" placeholder="Select a GPU" /> <sdpi-select setting="gpuId" datasource="gpus" loading="Loading GPUs…" hot-reload default="all" placeholder="Select a GPU" />
</sdpi-item> </sdpi-item>
@@ -6,11 +6,14 @@
</head> </head>
<body> <body>
<sdpi-item label="Base URL"> <sdpi-item label="Base URL">
<sdpi-textfield setting="baseUrl" value="http://talos.milky.way:9292" placeholder="http://host:port" /> <sdpi-textfield setting="baseUrl" value="http://localhost:9292" placeholder="http://host:port" />
</sdpi-item> </sdpi-item>
<sdpi-item label="API Key"> <sdpi-item label="API Key">
<sdpi-password setting="apiKey" placeholder="Optional" /> <sdpi-password setting="apiKey" placeholder="Optional" />
</sdpi-item> </sdpi-item>
<sdpi-item>
<sdpi-checkbox setting="insecure" label="Ignore certificate errors (self-signed TLS)"></sdpi-checkbox>
</sdpi-item>
<sdpi-item label="Display"> <sdpi-item label="Display">
<sdpi-select setting="display" default="count" placeholder="Select a display mode"> <sdpi-select setting="display" default="count" placeholder="Select a display mode">
<option value="count">Request count + activity</option> <option value="count">Request count + activity</option>
+11 -1
View File
@@ -8,7 +8,8 @@
"name": "llama-watch", "name": "llama-watch",
"version": "1.0.0", "version": "1.0.0",
"dependencies": { "dependencies": {
"@elgato/streamdeck": "^2.1.1" "@elgato/streamdeck": "^2.1.1",
"undici": "^6.28.0"
}, },
"devDependencies": { "devDependencies": {
"@elgato/cli": "^1.8.1", "@elgato/cli": "^1.8.1",
@@ -2477,6 +2478,15 @@
"node": ">=14.17" "node": ">=14.17"
} }
}, },
"node_modules/undici": {
"version": "6.28.0",
"resolved": "https://registry.npmjs.org/undici/-/undici-6.28.0.tgz",
"integrity": "sha512-LIY910g9TI13YS95lrMFrs8Rm/u/irgHeTWoKCoteeJ04CUJ92eEfj0rVn+7VKMPBpUPiUoBKfhNyLI23EE/KA==",
"license": "MIT",
"engines": {
"node": ">=18.17"
}
},
"node_modules/undici-types": { "node_modules/undici-types": {
"version": "6.20.0", "version": "6.20.0",
"resolved": "https://registry.npmjs.org/undici-types/-/undici-types-6.20.0.tgz", "resolved": "https://registry.npmjs.org/undici-types/-/undici-types-6.20.0.tgz",
+2 -1
View File
@@ -23,6 +23,7 @@
"typescript": "^5.7.3" "typescript": "^5.7.3"
}, },
"dependencies": { "dependencies": {
"@elgato/streamdeck": "^2.1.1" "@elgato/streamdeck": "^2.1.1",
"undici": "^6.28.0"
} }
} }
+188
View File
@@ -0,0 +1,188 @@
import { mkdirSync, rmSync, writeFileSync } from "node:fs";
import { execFileSync } from "node:child_process";
import { fileURLToPath } from "node:url";
import path from "node:path";
import { renderGpuGraph, renderInflight, renderUsage } from "../src/lib/render";
const ROOT = path.resolve(path.dirname(fileURLToPath(import.meta.url)), "..");
const OUT = path.join(ROOT, "store-assets");
const TMP = path.join(OUT, ".gen");
const FONT = "Helvetica, Arial, sans-serif";
const BG = "#0e1117";
const PANEL = "#161b26";
const STROKE = "#242b38";
const MUTED = "#8b93a5";
const WHITE = "#e6e9ef";
const GREEN = "#3fae5a";
const RED = "#e0453a";
const AMBER = "#d9a02a";
rmSync(TMP, { recursive: true, force: true });
mkdirSync(TMP, { recursive: true });
function convert(src: string, dst: string, w: number, h: number): void {
execFileSync("rsvg-convert", ["--unlimited", "-w", String(w), "-h", String(h), "-o", dst, src]);
}
function rasterizeKey(name: string, svg: string, scale = 3): string {
const svgPath = path.join(TMP, `${name}.svg`);
const pngPath = path.join(TMP, `${name}.png`);
writeFileSync(svgPath, svg);
convert(svgPath, pngPath, 72 * scale, 72 * scale);
return `${name}.png`;
}
function svg(w: number, h: number, body: string): string {
return `<svg xmlns="http://www.w3.org/2000/svg" width="${w}" height="${h}" viewBox="0 0 ${w} ${h}">
<rect width="${w}" height="${h}" fill="${BG}"/>
${body}
</svg>`;
}
function text(content: string, x: number, y: number, size: number, fill: string, weight = "normal", anchor = "middle"): string {
return ` <text x="${x}" y="${y}" text-anchor="${anchor}" font-family="${FONT}" font-size="${size}" font-weight="${weight}" fill="${fill}">${content}</text>`;
}
function keyImage(png: string, cx: number, cy: number, size: number): string {
return ` <image href="${png}" x="${cx - size / 2}" y="${cy - size / 2}" width="${size}" height="${size}"/>`;
}
function keyWithLabel(png: string, label: string, cx: number, cy: number, size: number): string {
return `${keyImage(png, cx, cy, size)}
<rect x="${cx - size / 2 - 6}" y="${cy + size / 2 + 14}" width="${size + 12}" height="1" fill="${STROKE}"/>
${text(label, cx, cy + size / 2 + 40, 26, WHITE)}`;
}
function header(title: string, subtitle: string): string {
return `${text(title, 960, 108, 56, WHITE, "bold")}
${text(subtitle, 960, 164, 28, MUTED)}`;
}
function footer(caption: string): string {
return `${text(caption, 960, 872, 24, MUTED)}`;
}
const keys: Record<string, string> = {
inflightIdle: renderInflight({ modelName: "DeepSeek-V4-Flash-0731", state: "ready", count: 0, offline: false }),
inflightActive: renderInflight({ modelName: "DeepSeek-V4-Flash-0731", state: "ready", count: 3, offline: false, history: [0, 1, 2, 3, 2, 3, 4, 3, 3, 2, 3, 2, 1] }),
usageP95: renderUsage({ modelName: "DeepSeek-V4-Flash-0731", stats: { totalRequests: 1950, totalInputTokens: 312177540, totalOutputTokens: 889193, genP95: 378.86 }, primaryStat: "gen_p95", offline: false }),
usageReq: renderUsage({ modelName: "All Models", stats: { totalRequests: 1985, totalInputTokens: 312327705, totalOutputTokens: 932710, genP95: 378.86 }, primaryStat: "requests", offline: false }),
gpuUtil: renderGpuGraph({ gpuName: "GPU 0 · RTX 5090", metric: "util_percent", value: 67, history: [10, 20, 40, 67, 55, 80, 72, 45, 60], offline: false }),
gpuPower: renderGpuGraph({ gpuName: "GPU 1 · RTX 5090", metric: "power", value: 312, history: [100, 150, 200, 312, 280, 350, 220, 190, 260], offline: false }),
gpuAll: renderGpuGraph({ gpuName: "ALL GPUS", metric: "temperature", value: 63, history: [55, 58, 61, 63, 60, 65, 64, 62, 66], offline: false }),
gpuCombo: renderGpuGraph({ gpuName: "GPU 0+2", metric: "util_percent", value: 71, history: [30, 45, 52, 71, 64, 78, 70, 58, 66], offline: false }),
};
const pngs: Record<string, string> = {};
for (const [name, svg] of Object.entries(keys)) {
pngs[name] = rasterizeKey(name, svg);
}
function writePng(name: string, w: number, h: number, body: string): void {
const svgPath = path.join(TMP, `${name}.svg`);
writeFileSync(svgPath, svg(w, h, body));
convert(svgPath, path.join(OUT, name), w, h);
console.log(`wrote store-assets/${name} (${w}x${h})`);
}
// Thumbnail — 1920x960
{
const s = 240;
const cx = [1250, 1570];
const cy = [360, 700];
const cols = [
keyWithLabel(pngs.inflightActive, "In-Flight Monitor", cx[0], cy[0], s),
keyWithLabel(pngs.usageReq, "Usage stats", cx[1], cy[0], s),
keyWithLabel(pngs.gpuUtil, "GPU Graph", cx[0], cy[1], s),
keyWithLabel(pngs.gpuCombo, "GPU combination", cx[1], cy[1], s),
];
const bullets = [
["#3fae5a", "In-flight request counts with a 60-second activity spark"],
["#d9a02a", "Usage stats: requests, tokens, and generation-speed P95"],
["#e0453a", "Live GPU graphs — per GPU, all GPUs, or a combination"],
]
.map(
([color, t], i) =>
` <circle cx="116" cy="${480 + i * 78}" r="10" fill="${color}"/>\n` +
text(t, 146, 488 + i * 78, 28, WHITE, "normal", "start"),
)
.join("\n");
writePng(
"thumbnail.png",
1920,
960,
`${text("llama-watch", 116, 250, 96, WHITE, "bold", "start")}
${text("Stream Deck keys for your llama-swap LLM server", 116, 322, 30, MUTED, "normal", "start")}
<rect x="116" y="350" width="620" height="3" fill="${GREEN}"/>
${bullets}
${cols.join("\n")}`,
);
}
// Gallery 1 — In-Flight Monitor
{
const s = 240;
const centers = [470, 810, 1150, 1490];
const items = [
[pngs.inflightIdle, "IDLE — no requests"],
[pngs.inflightActive, "3 in flight + 60s spark"],
[pngs.usageP95, "Usage · gen speed P95"],
[pngs.usageReq, "Usage · totals"],
];
writePng(
"gallery-1-inflight.png",
1920,
960,
`${header("In-Flight Monitor", "Live request counts, a 60-second activity spark, and usage stats")}
${items.map(([png, label], i) => keyWithLabel(png as string, label as string, centers[i], 460, s)).join("\n")}
${footer("Counts update instantly from the SSE feed; the spark shows the last 60 seconds of activity.")}`,
);
}
// Gallery 2 — GPU Graph
{
const s = 240;
const centers = [470, 810, 1150, 1490];
const items = [
[pngs.gpuUtil, "GPU 0 · utilization"],
[pngs.gpuPower, "GPU 1 · power draw"],
[pngs.gpuAll, "All GPUs · temperature"],
[pngs.gpuCombo, "GPU 0+2 · utilization combo"],
];
writePng(
"gallery-2-gpu.png",
1920,
960,
`${header("GPU Graph", "Live metric charts — one GPU, all GPUs, or a custom combination")}
${items.map(([png, label], i) => keyWithLabel(png as string, label as string, centers[i], 460, s)).join("\n")}
${footer("Sampled every 5 seconds: utilization, VRAM, temperature, power draw, and fan speed.")}`,
);
}
// Gallery 3 — Setup
{
const field = (label: string, value: string, y: number): string =>
`${text(label, 200, y, 22, MUTED, "normal", "start")}
<rect x="200" y="${y + 22}" width="700" height="44" rx="10" fill="${BG}" stroke="${STROKE}" stroke-width="2"/>
${text(value, 220, y + 22 + 29, 22, WHITE, "normal", "start")}`;
const panel = `${text("In-Flight Monitor settings", 200, 240, 30, WHITE, "bold", "start")}
${field("Base URL", "http://localhost:9292", 300)}
${field("API Key", "Optional", 395)}
${field("Model", "DeepSeek-V4-Flash-0731 (ready)", 490)}
${field("Display", "Usage stats", 585)}
${field("Primary stat", "Generation speed P95", 680)}
<rect x="160" y="170" width="760" height="640" rx="24" fill="${PANEL}" stroke="${STROKE}" stroke-width="3"/>`;
writePng(
"gallery-3-setup.png",
1920,
960,
`${header("Point it at your llama-swap instance", "Per-key settings in the property inspector")}
<g transform="translate(0,0)">${panel}</g>
${keyWithLabel(pngs.usageReq, "Usage stats key", 1420, 430, 300)}
${footer("Every key is independent: model, display mode, GPU, metric, and combination.")}`,
);
}
// App icon — 288x288
convert(path.join(ROOT, "com.bryce.llamawatch.sdPlugin/imgs/plugin/icon.svg"), path.join(OUT, "app-icon-288.png"), 288, 288);
console.log("wrote store-assets/app-icon-288.png (288x288)");
+3 -2
View File
@@ -1,3 +1,4 @@
import { fetchWith } from "./http";
import { type LlamaSwapConfig } from "./util"; import { type LlamaSwapConfig } from "./util";
import { parseSse, type SseMessage } from "./sse"; import { parseSse, type SseMessage } from "./sse";
import { type FeedEvent, type InflightRequest, type ModelState } from "./inflight-tracker"; import { type FeedEvent, type InflightRequest, type ModelState } from "./inflight-tracker";
@@ -17,7 +18,7 @@ export function decodeEvent(msg: SseMessage): FeedEvent | null {
if (outer.type === "inflight") { if (outer.type === "inflight") {
const operation = inner.operation as string; const operation = inner.operation as string;
if (operation === "remove") { if (operation === "remove") {
const id = typeof inner.id === "string" ? inner.id : undefined; const id = typeof inner.id === "string" ? inner.id : typeof inner.id === "number" ? String(inner.id) : undefined;
if (id) return { type: "inflight", operation: "remove", id }; if (id) return { type: "inflight", operation: "remove", id };
return null; return null;
} }
@@ -80,7 +81,7 @@ export class EventFeed {
try { try {
const headers: Record<string, string> = { Accept: "text/event-stream" }; const headers: Record<string, string> = { Accept: "text/event-stream" };
if (this.cfg.apiKey) headers["Authorization"] = `Bearer ${this.cfg.apiKey}`; if (this.cfg.apiKey) headers["Authorization"] = `Bearer ${this.cfg.apiKey}`;
const res = await fetch(`${this.cfg.baseUrl}/api/events`, { const res = await fetchWith(this.cfg, `${this.cfg.baseUrl}/api/events`, {
headers, headers,
signal: this.controller.signal, signal: this.controller.signal,
}); });
+9
View File
@@ -0,0 +1,9 @@
import { Agent, fetch as undiciFetch } from "undici";
import { type LlamaSwapConfig } from "./util";
const INSECURE_AGENT = new Agent({ connect: { rejectUnauthorized: false } });
export function fetchWith(cfg: LlamaSwapConfig, input: string, init?: RequestInit): Promise<Response> {
if (cfg.insecure !== true) return fetch(input, init);
return undiciFetch(input, { ...(init ?? {}), dispatcher: INSECURE_AGENT }) as unknown as Promise<Response>;
}
+13 -5
View File
@@ -21,27 +21,35 @@ export type FeedEvent =
export type ModelRuntimeState = "stopped" | "loading" | "ready"; export type ModelRuntimeState = "stopped" | "loading" | "ready";
export class InflightTracker { export class InflightTracker {
private requests = new Map<string, string>(); private requests = new Map<string, { model: string; seen: number }>();
private states = new Map<string, ModelRuntimeState>(); private states = new Map<string, ModelRuntimeState>();
constructor(private now: () => number = () => Date.now()) {}
apply(event: FeedEvent): void { apply(event: FeedEvent): void {
if (event.type === "inflight") { if (event.type === "inflight") {
if (event.operation === "snapshot") { if (event.operation === "snapshot") {
this.requests = new Map<string, string>(); this.requests = new Map<string, { model: string; seen: number }>();
for (const r of event.requests) if (r.id) this.requests.set(r.id, r.model); for (const r of event.requests) if (r.id) this.requests.set(r.id, { model: r.model, seen: this.now() });
} else if (event.operation === "add") { } else if (event.operation === "add") {
for (const r of event.requests) if (r.id) this.requests.set(r.id, r.model); for (const r of event.requests) if (r.id) this.requests.set(r.id, { model: r.model, seen: this.now() });
} else if (event.operation === "remove") { } else if (event.operation === "remove") {
this.requests.delete(event.id); this.requests.delete(event.id);
} }
this.prune();
} else if (event.type === "modelStatus") { } else if (event.type === "modelStatus") {
for (const m of event.models) this.states.set(m.id, normalizeState(m.state)); for (const m of event.models) this.states.set(m.id, normalizeState(m.state));
} }
} }
prune(maxAgeMs = 120_000): void {
const cutoff = this.now() - maxAgeMs;
for (const [id, entry] of this.requests) if (entry.seen < cutoff) this.requests.delete(id);
}
count(modelId: string): number { count(modelId: string): number {
let n = 0; let n = 0;
for (const model of this.requests.values()) if (model === modelId) n++; for (const entry of this.requests.values()) if (entry.model === modelId) n++;
return n; return n;
} }
+3 -2
View File
@@ -1,3 +1,4 @@
import { fetchWith } from "./http";
import { type LlamaSwapConfig } from "./util"; import { type LlamaSwapConfig } from "./util";
const TIMEOUT_MS = 5000; const TIMEOUT_MS = 5000;
@@ -14,7 +15,7 @@ function headersFor(cfg: LlamaSwapConfig): Record<string, string> {
} }
export async function fetchMetrics(cfg: LlamaSwapConfig): Promise<string> { export async function fetchMetrics(cfg: LlamaSwapConfig): Promise<string> {
const res = await fetch(`${cfg.baseUrl}/metrics`, { const res = await fetchWith(cfg, `${cfg.baseUrl}/metrics`, {
headers: headersFor(cfg), headers: headersFor(cfg),
signal: AbortSignal.timeout(TIMEOUT_MS), signal: AbortSignal.timeout(TIMEOUT_MS),
}); });
@@ -23,7 +24,7 @@ export async function fetchMetrics(cfg: LlamaSwapConfig): Promise<string> {
} }
export async function fetchModels(cfg: LlamaSwapConfig): Promise<ModelInfo[]> { export async function fetchModels(cfg: LlamaSwapConfig): Promise<ModelInfo[]> {
const res = await fetch(`${cfg.baseUrl}/v1/models`, { const res = await fetchWith(cfg, `${cfg.baseUrl}/v1/models`, {
headers: headersFor(cfg), headers: headersFor(cfg),
signal: AbortSignal.timeout(TIMEOUT_MS), signal: AbortSignal.timeout(TIMEOUT_MS),
}); });
+11 -1
View File
@@ -12,10 +12,15 @@ class Runtime {
private feed?: EventFeed; private feed?: EventFeed;
private poller?: MetricsPoller; private poller?: MetricsPoller;
private statsCache?: StatsCache; private statsCache?: StatsCache;
private pruneTimer?: ReturnType<typeof setInterval>;
private listeners = new Set<() => void>(); private listeners = new Set<() => void>();
ensureConnections(cfg: LlamaSwapConfig): void { ensureConnections(cfg: LlamaSwapConfig): void {
const changed = !this.cfg || this.cfg.baseUrl !== cfg.baseUrl || this.cfg.apiKey !== cfg.apiKey; const changed =
!this.cfg ||
this.cfg.baseUrl !== cfg.baseUrl ||
this.cfg.apiKey !== cfg.apiKey ||
this.cfg.insecure !== cfg.insecure;
if (changed) { if (changed) {
this.feed?.stop(); this.feed?.stop();
this.poller?.stop(); this.poller?.stop();
@@ -45,6 +50,9 @@ class Runtime {
this.statsCache = new StatsCache(cfg); this.statsCache = new StatsCache(cfg);
this.statsCache.onChange(() => this.emit()); this.statsCache.onChange(() => this.emit());
} }
if (!this.pruneTimer) {
this.pruneTimer = setInterval(() => this.tracker.prune(), PRUNE_INTERVAL_MS);
}
} }
get pollerInstance(): MetricsPoller | undefined { get pollerInstance(): MetricsPoller | undefined {
@@ -73,3 +81,5 @@ class Runtime {
} }
export const runtime = new Runtime(); export const runtime = new Runtime();
const PRUNE_INTERVAL_MS = 30_000;
+2 -1
View File
@@ -1,3 +1,4 @@
import { fetchWith } from "./http";
import { type LlamaSwapConfig } from "./util"; import { type LlamaSwapConfig } from "./util";
export interface UsageStats { export interface UsageStats {
@@ -24,7 +25,7 @@ export async function fetchStats(cfg: LlamaSwapConfig, modelId: string): Promise
const query = modelId === "all" ? "" : `?model=${encodeURIComponent(modelId)}`; const query = modelId === "all" ? "" : `?model=${encodeURIComponent(modelId)}`;
const headers: Record<string, string> = {}; const headers: Record<string, string> = {};
if (cfg.apiKey) headers["Authorization"] = `Bearer ${cfg.apiKey}`; if (cfg.apiKey) headers["Authorization"] = `Bearer ${cfg.apiKey}`;
const res = await fetch(`${cfg.baseUrl}/api/metrics/stats${query}`, { const res = await fetchWith(cfg, `${cfg.baseUrl}/api/metrics/stats${query}`, {
headers, headers,
signal: AbortSignal.timeout(5000), signal: AbortSignal.timeout(5000),
}); });
+4 -1
View File
@@ -1,21 +1,24 @@
import type { JsonObject } from "@elgato/utils"; import type { JsonObject } from "@elgato/utils";
export const DEFAULT_BASE_URL = "http://talos.milky.way:9292"; export const DEFAULT_BASE_URL = "http://localhost:9292";
export type CfgSettings = { export type CfgSettings = {
baseUrl?: string; baseUrl?: string;
apiKey?: string; apiKey?: string;
insecure?: boolean;
} & JsonObject; } & JsonObject;
export interface LlamaSwapConfig { export interface LlamaSwapConfig {
baseUrl: string; baseUrl: string;
apiKey?: string; apiKey?: string;
insecure?: boolean;
} }
export function cfgFromSettings(settings: CfgSettings): LlamaSwapConfig { export function cfgFromSettings(settings: CfgSettings): LlamaSwapConfig {
return { return {
baseUrl: normalizeBaseUrl(settings.baseUrl ?? DEFAULT_BASE_URL), baseUrl: normalizeBaseUrl(settings.baseUrl ?? DEFAULT_BASE_URL),
apiKey: settings.apiKey || undefined, apiKey: settings.apiKey || undefined,
insecure: settings.insecure === true,
}; };
} }
+75
View File
@@ -0,0 +1,75 @@
# Marketplace Listing — llama-watch
Copy the content below into Maker Console when creating the product.
## Product type
Stream Deck plugin
## Name
`llama-watch`
## Author / Organization
Organization: `c4ch3c4d3` (create this in Maker Console; sign the Maker Agreement)
## Description
> Copy exactly (254 chars first 250-char segment is unformatted; total ~900 chars, within the 2501500 limit):
```
llama-watch brings live monitoring of your llama-swap LLM server to your Stream Deck. Watch in-flight request counts per model with a real-time activity spark, switch to Usage stats to see request totals, processed and generated tokens, and the generation-speed P95 — per model or across all models. GPU Graph keys render live line charts of GPU utilization, VRAM, temperature, power draw, and fan speed — for one GPU, all GPUs, or a custom combination.
Features
- In-Flight Monitor: live per-model request count with a 60-second activity spark; Usage stats display (requests, processed/generated tokens, generation-speed P95) per model or all models.
- GPU Graph: live charts for utilization, VRAM, temperature, power draw, and fan speed — single GPU, all GPUs, or a custom combination.
- Per-key configuration: base URL and optional API key; every key is independent.
Requirements
- A running llama-swap instance (open source) and its base URL — default http://localhost:9292.
- Optional API key for protected instances.
- macOS 13 or later; Stream Deck 7.1 or later.
Privacy: the plugin reads data only from your configured llama-swap server and sends nothing elsewhere. No analytics, no telemetry.
```
## Tags
`monitoring`, `gpu`, `llm`, `llama-swap`, `developer tools`
## Price
Free
## Release notes (v1.0.0)
```
Initial release.
- In-Flight Monitor: live per-model request count with a 60-second activity spark; Usage stats display (requests, processed/generated tokens, generation-speed P95) per model or all models.
- GPU Graph: live charts for utilization, VRAM, temperature, power draw, and fan speed — single GPU, all GPUs, or a custom combination.
- Per-key base URL and optional API key settings.
```
## Media checklist (generated by scripts/store-assets.mts)
- `store-assets/app-icon-288.png` — app icon, 288×288 PNG
- `store-assets/thumbnail.png` — thumbnail, 1920×960 PNG
- `store-assets/gallery-1-inflight.png` — gallery, 1920×960 PNG
- `store-assets/gallery-2-gpu.png` — gallery, 1920×960 PNG
- `store-assets/gallery-3-setup.png` — gallery, 1920×960 PNG
## Submission steps (Maker Console)
1. maker.elgato.com → create organization `c4ch3c4d3` → sign the Maker Agreement.
2. Home → Create product → Stream Deck plugin.
3. Upload `com.bryce.llamawatch.streamDeckPlugin` (rebuilt with Author `c4ch3c4d3`).
4. Details: name `llama-watch`, description above, tags, price Free.
5. Media: upload the 5 PNGs above as app icon + thumbnail + 3 gallery items.
6. Release notes: v1.0.0 above. Submit.
7. Review: allow 410 business days; handle feedback by submitting a revision.
Notes
- The product name and monetization cannot be changed after creation without contacting maker@elgato.com — confirm `llama-watch` is still available when you create it.
- No demo video is required (the plugin integrates with the user's own software, not a paid service/hardware).
+9
View File
@@ -67,6 +67,15 @@ test("decodeEvent parses an inflight remove with a bare id", () => {
assert.deepEqual(ev, { type: "inflight", operation: "remove", id: "817" }); assert.deepEqual(ev, { type: "inflight", operation: "remove", id: "817" });
}); });
test("decodeEvent accepts a numeric id on remove", () => {
const msg: SseMessage = {
event: "message",
data: JSON.stringify({ type: "inflight", data: JSON.stringify({ operation: "remove", id: 820 }) }),
};
const ev = decodeEvent(msg);
assert.deepEqual(ev, { type: "inflight", operation: "remove", id: "820" });
});
test("decodeEvent parses modelStatus", () => { test("decodeEvent parses modelStatus", () => {
const msg: SseMessage = { const msg: SseMessage = {
event: "message", event: "message",
+19
View File
@@ -0,0 +1,19 @@
-----BEGIN CERTIFICATE-----
MIIDCTCCAfGgAwIBAgIUXQ30Z78Fhhti2Ct6DXczY/jvDjcwDQYJKoZIhvcNAQEL
BQAwFDESMBAGA1UEAwwJbG9jYWxob3N0MB4XDTI2MDgyNzIxMTAyNloXDTM2MDgy
NDIxMTAyNlowFDESMBAGA1UEAwwJbG9jYWxob3N0MIIBIjANBgkqhkiG9w0BAQEF
AAOCAQ8AMIIBCgKCAQEAtJNuhOYp4Qi7W17LRvph2EZH7WfQsbELROrT3ijJ2ys0
iA+Bee0ATpc/1R+NgSGfQZsPlAYh2TlYK0UWBIA1BGKm5tSHpBvfxs+GXcgmFM0k
nWFE4Kof6d8zvcd3l7aU+8w8Tz/LnhLpaYQiixZ4ZIoAC1SEWKO0wm+AoMSSeLjH
CBaGmLuD1MQ+I6kZPwDK2YluZ28LHMH9ZCbgfVKXR0FVjgF0fEw6645IzgWGUznN
8pcEisVG2Veyr83Q7uqEL3CtvrdYFjgFJP0Qzez2zgCGXKB+tiBsWIzL2rwVnDcV
ss2bwAVz9JMUxaj8aoMq0Bx5hJkzK9JipGRB9C58CQIDAQABo1MwUTAdBgNVHQ4E
FgQUCtzsN0GW6A2iKrJF28+UVJOY7zkwHwYDVR0jBBgwFoAUCtzsN0GW6A2iKrJF
28+UVJOY7zkwDwYDVR0TAQH/BAUwAwEB/zANBgkqhkiG9w0BAQsFAAOCAQEApZfd
QLXrtRfu6NG1BPfzaROjwTLs5O0tU8+/SSyi3WjoIZmYUBkP7nJm/sR2ZxpSUEfq
OFZwux8bDJTccS9Au0/OR9vYbHWmDoY1e14v2GehNRWXz8vvaD3AURluAYXcmgol
5WOnSB4W8rp6A5gEKX7n4hsHrkUx/Mt+uSg7KZv/fFnwQqyu9OthPKVNV1v79dgm
C0ZUNhpaDDX5Ae+khY88kGADwCKY9BTNzftQG0/4ZJgu7O34eHNgC+2MAgXDtLxm
TIk3XSEaFDoT+HOX38Lh3JYnzJCFO59tQfMMGrPuWfgOISxCe5tkZ22zTtPId+wB
+TptauhacNtwwpc1Jw==
-----END CERTIFICATE-----
+28
View File
@@ -0,0 +1,28 @@
-----BEGIN PRIVATE KEY-----
MIIEvQIBADANBgkqhkiG9w0BAQEFAASCBKcwggSjAgEAAoIBAQC0k26E5inhCLtb
XstG+mHYRkftZ9CxsQtE6tPeKMnbKzSID4F57QBOlz/VH42BIZ9Bmw+UBiHZOVgr
RRYEgDUEYqbm1IekG9/Gz4ZdyCYUzSSdYUTgqh/p3zO9x3eXtpT7zDxPP8ueEulp
hCKLFnhkigALVIRYo7TCb4CgxJJ4uMcIFoaYu4PUxD4jqRk/AMrZiW5nbwscwf1k
JuB9UpdHQVWOAXR8TDrrjkjOBYZTOc3ylwSKxUbZV7KvzdDu6oQvcK2+t1gWOAUk
/RDN7PbOAIZcoH62IGxYjMvavBWcNxWyzZvABXP0kxTFqPxqgyrQHHmEmTMr0mKk
ZEH0LnwJAgMBAAECggEADgjFVqvixlwc36GS7+3Gy/3OWkuuwxis9QrBM6t84L1P
ZGG8IONEGleT/PbqUwZvb7Ri9hCx8cWMrjQ83VWviSs3qIoNDrqh3jxDx6cmGojF
Fzw3k7R1LYKM7WuCxnZIxvcdGtWs+BilLm+4FZJGAh5dmYPUk2UJx/DNkPEmJx6n
4aSEwUnV1QGPR7IjchGEHImBj8T8L3HBQnrsRsCRMwgUs3z0XMgfZoFm9/dq08YA
NPzyxAgSsQQrtVOyJMOFj+wi2NrUFtQ0OfS617cUMQXOyceQPpUECn2zPU1UIoPy
VgXfkxSXVdraetgNACELJvBI1bvvZctmkJYkEkY5lwKBgQDXcBDCHn4Any+T4gV8
Jp9XQf9lCWQnyegP7sSB9vLxzFKU49tuEDDG3m1wOPw17OV/Cs1sOzbiaFXSMM78
rD2ljRIODaewr3/Js8B6XbYmmkJqth8ByYuKV6kyq1aFPb4zTmEtcWbnnx45mUHr
yZr+8H33yl1iWhR7rdgAtvNP4wKBgQDWkw1C6qR9T8v8sf/vNbwJUHC5xD6B5d18
XIV8+Uax1tgmIupy/1hVllLBcaizBpE5eFPsWMMNw2pujnSKNER2hfADSFXYUlMO
Ror6e+Dfhr9MctiJjqhNW9kIPBvvnqzpOqwHun6P1GEC/qOi4GIdDDP95OqDaDsV
BvKNKVAwIwKBgQCS+iKEvNbLx85mvrFtRNA6cI0zuhd5Sbcnf4bS/845BmNkrpsa
WLNeSYsyH755b7gWVyFUcIV+Kx45uxDLsxqPolGqAsjfsqukyRxMnzhQ17buJHe8
+WpYpHuLVPc/CaOETznfDdndtWGifBtMKIu02A+oiIfzPG9y/WQ7AJW4bwKBgFmq
FW6TEq1yvPEZiLNzJuJVhOV7xgsN/SHMn9N7bzk9aBF3obTwUv9g07AWSMKWyfTT
/W3UIZ4MvNr6GGTwNnO4wHT+szC0JhTfEZBeV7fQXPwbObUxsc6xxN2WEK5vBh5n
8B9CpUSBIRDZS5PyY81znf5IvF6xHY9J2e13CBU1AoGARfyoIH6gF92bI1DNZ4cT
mtgPYEGbWYr28/ADF25CrlZ3HvWDqwt2Oe1EvWWyALLjTyKdZXuc0iFiKUSsgUSm
qlBjoh1lWXFrtm+uu8rtGaWrp9q0xjNhcmdM0TqjXpvifDXLALGUiRGvUjZHGo8J
mgw1KgJviTZzZu2cy7iFCqg=
-----END PRIVATE KEY-----
+55
View File
@@ -0,0 +1,55 @@
import assert from "node:assert/strict";
import { readFileSync } from "node:fs";
import { createServer, type Server } from "node:https";
import { after, before, test } from "node:test";
import { fetchWith } from "../src/lib/http";
const TLS_OPTIONS = {
cert: readFileSync(new URL("./fixtures/selfsigned-cert.pem", import.meta.url)),
key: readFileSync(new URL("./fixtures/selfsigned-key.pem", import.meta.url)),
};
let server: Server;
let base: string;
before(async () => {
server = createServer(TLS_OPTIONS, (req, res) => {
res.writeHead(200, { "content-type": "text/plain" });
res.end("ok");
});
await new Promise<void>((resolve) => server.listen(0, "127.0.0.1", resolve));
base = `https://127.0.0.1:${(server.address() as { port: number }).port}`;
});
after(async () => new Promise((resolve) => server.close(() => resolve())));
test("fetchWith verifies certificates by default", async () => {
await assert.rejects(fetchWith({ baseUrl: base }, `${base}/`));
});
test("fetchWith accepts self-signed certificates when insecure", async () => {
const res = await fetchWith({ baseUrl: base, insecure: true }, `${base}/`);
assert.equal(res.status, 200);
assert.equal(await res.text(), "ok");
});
test("fetchWith honors init headers and signal when insecure", async () => {
let seen: string | undefined;
const spy = createServer(TLS_OPTIONS, (req, res) => {
seen = req.headers.authorization;
res.writeHead(200);
res.end("ok");
});
await new Promise<void>((resolve) => spy.listen(0, "127.0.0.1", resolve));
const spyBase = `https://127.0.0.1:${(spy.address() as { port: number }).port}`;
try {
const res = await fetchWith({ baseUrl: spyBase, insecure: true }, `${spyBase}/`, {
headers: { Authorization: "Bearer tok" },
signal: AbortSignal.timeout(3000),
});
assert.equal(res.status, 200);
assert.equal(seen, "Bearer tok");
} finally {
await new Promise<void>((resolve) => spy.close(() => resolve()));
}
});
+31
View File
@@ -73,3 +73,34 @@ test("total sums in-flight requests across all models", () => {
tracker.apply({ type: "inflight", operation: "remove", id: "1" }); tracker.apply({ type: "inflight", operation: "remove", id: "1" });
assert.equal(tracker.total(), 2); assert.equal(tracker.total(), 2);
}); });
test("prune drops request ids that stopped receiving updates", () => {
let now = 0;
const tracker = new InflightTracker(() => now);
tracker.apply({ type: "inflight", operation: "add", requests: [{ model: "A", id: "1" }] });
now = 200_000;
tracker.apply({ type: "inflight", operation: "add", requests: [{ model: "A", id: "2" }] });
tracker.prune(60_000);
assert.equal(tracker.total(), 1);
assert.equal(tracker.count("A"), 1);
});
test("active requests refreshed by upserts survive pruning", () => {
let now = 0;
const tracker = new InflightTracker(() => now);
tracker.apply({ type: "inflight", operation: "add", requests: [{ model: "A", id: "1" }] });
now = 200_000;
tracker.apply({ type: "inflight", operation: "add", requests: [{ model: "A", id: "1" }] });
tracker.prune(60_000);
assert.equal(tracker.total(), 1);
});
test("prune is applied on every inflight event", () => {
let now = 0;
const tracker = new InflightTracker(() => now);
tracker.apply({ type: "inflight", operation: "add", requests: [{ model: "A", id: "stale" }] });
now = 200_000;
tracker.apply({ type: "inflight", operation: "add", requests: [{ model: "B", id: "live" }] });
assert.equal(tracker.total(), 1);
assert.equal(tracker.count("B"), 1);
});
+7 -4
View File
@@ -8,18 +8,21 @@ import {
} from "../src/lib/util"; } from "../src/lib/util";
test("normalizeBaseUrl strips trailing slashes", () => { test("normalizeBaseUrl strips trailing slashes", () => {
assert.equal(normalizeBaseUrl("http://talos.milky.way:9292/"), "http://talos.milky.way:9292"); assert.equal(normalizeBaseUrl("http://localhost:9292/"), "http://localhost:9292");
assert.equal(normalizeBaseUrl("http://talos.milky.way:9292"), "http://talos.milky.way:9292"); assert.equal(normalizeBaseUrl("http://localhost:9292"), "http://localhost:9292");
}); });
test("cfgFromSettings defaults baseUrl and omits empty apiKey", () => { test("cfgFromSettings defaults baseUrl and omits empty apiKey", () => {
const cfg = cfgFromSettings({}); const cfg = cfgFromSettings({});
assert.equal(cfg.baseUrl, "http://talos.milky.way:9292"); assert.equal(cfg.baseUrl, "http://localhost:9292");
assert.equal(cfg.apiKey, undefined); assert.equal(cfg.apiKey, undefined);
assert.deepEqual(cfgFromSettings({ baseUrl: "http://x/", apiKey: "abc" }), { assert.equal(cfg.insecure, false);
assert.deepEqual(cfgFromSettings({ baseUrl: "http://x/", apiKey: "abc", insecure: true }), {
baseUrl: "http://x", baseUrl: "http://x",
apiKey: "abc", apiKey: "abc",
insecure: true,
}); });
assert.equal(cfgFromSettings({ insecure: "yes" }).insecure, false);
}); });
test("shorten keeps short names and truncates long ones", () => { test("shorten keeps short names and truncates long ones", () => {