Parallelize the clone queue: in-flight dedup + WORKER_CONCURRENCY

Two changes that multiply effective queue throughput:

- In-flight dedup: POST /v1/clones now attaches identical submits (same
  cacheKey) to the already queued/running job instead of enqueueing a
  duplicate capture. Retry-spam previously multiplied queue load (observed
  4x same-URL submissions in prod); now it's free. Best-effort — a
  same-instant race can still double-submit. Gated on !noCache.

- WORKER_CONCURRENCY (default 1): the worker registers N independent
  pg-boss pollers, running up to N captures concurrently per process with
  no head-of-line blocking. Each slot gets its own verify harness dir
  (harnessDir/slot-i) so concurrent framework builds never collide;
  concurrency 1 keeps the flat harnessDir layout.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
Samraaj Bath
2026-07-06 22:44:42 -07:00
co-authored by Claude Fable 5
parent 99aca7d43d
commit 4c1dea77e1
5 changed files with 68 additions and 13 deletions
+7
View File
@@ -25,6 +25,13 @@ export class DbBackend implements Backend {
return { jobId: hit.jobId, status: "cached", httpStatus: 200, result: { ...r.result, status: "cached" } };
}
}
// In-flight dedup: an identical clone is already queued/running — attach the
// caller to that job instead of enqueueing a duplicate capture. (Best-effort:
// no unique constraint, so a same-instant race can still double-submit.)
const active = await repo.findActiveJobByCacheKey(this.deps.db, key);
if (active) {
return { jobId: active.id, status: "queued", httpStatus: 202 };
}
}
const job = await repo.createJob(this.deps.db, { kind, url, options: options ?? {}, status: "queued", cacheKey: key });