Parallelize the clone queue: in-flight dedup + WORKER_CONCURRENCY
Two changes that multiply effective queue throughput: - In-flight dedup: POST /v1/clones now attaches identical submits (same cacheKey) to the already queued/running job instead of enqueueing a duplicate capture. Retry-spam previously multiplied queue load (observed 4x same-URL submissions in prod); now it's free. Best-effort — a same-instant race can still double-submit. Gated on !noCache. - WORKER_CONCURRENCY (default 1): the worker registers N independent pg-boss pollers, running up to N captures concurrently per process with no head-of-line blocking. Each slot gets its own verify harness dir (harnessDir/slot-i) so concurrent framework builds never collide; concurrency 1 keeps the flat harnessDir layout. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
co-authored by
Claude Fable 5
parent
99aca7d43d
commit
4c1dea77e1
@@ -25,6 +25,13 @@ export class DbBackend implements Backend {
|
||||
return { jobId: hit.jobId, status: "cached", httpStatus: 200, result: { ...r.result, status: "cached" } };
|
||||
}
|
||||
}
|
||||
// In-flight dedup: an identical clone is already queued/running — attach the
|
||||
// caller to that job instead of enqueueing a duplicate capture. (Best-effort:
|
||||
// no unique constraint, so a same-instant race can still double-submit.)
|
||||
const active = await repo.findActiveJobByCacheKey(this.deps.db, key);
|
||||
if (active) {
|
||||
return { jobId: active.id, status: "queued", httpStatus: 202 };
|
||||
}
|
||||
}
|
||||
|
||||
const job = await repo.createJob(this.deps.db, { kind, url, options: options ?? {}, status: "queued", cacheKey: key });
|
||||
|
||||
Reference in New Issue
Block a user