Two changes that multiply effective queue throughput:
- In-flight dedup: POST /v1/clones now attaches identical submits (same
cacheKey) to the already queued/running job instead of enqueueing a
duplicate capture. Retry-spam previously multiplied queue load (observed
4x same-URL submissions in prod); now it's free. Best-effort — a
same-instant race can still double-submit. Gated on !noCache.
- WORKER_CONCURRENCY (default 1): the worker registers N independent
pg-boss pollers, running up to N captures concurrently per process with
no head-of-line blocking. Each slot gets its own verify harness dir
(harnessDir/slot-i) so concurrent framework builds never collide;
concurrency 1 keeps the flat harnessDir layout.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Ported from PR #15 (packages-side surface only; the fork's dev UI page is
intentionally dropped - the API surface is the product):
- job_events table + repository, worker emits stage transitions, additive
GET /v1/clones/:id/events polling route. The fork's hand-written
migration was never registered in Drizzle's journal (it silently never
applied); regenerated properly via drizzle-kit from schema.ts with the
(job_id, seq) index declared, and verified against a live Postgres
- In-memory backend moves to the documented enqueue-202 + poll contract
with single-flight BUSY guard; POST honors backend httpStatus
- app-preview static serving routes; the preview BUILD half is deferred
until the compiler exports buildApp/DEFAULT_HARNESS_DIR - routes 404
cleanly until then
- preview option threaded through core types; cache key intentionally
unchanged while the flag is inert
All workspaces typecheck; api 18/18, core 13/13, worker 1/1 pass;
migration verified applying (table + index + FK) on postgres:16.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The REST API returns a clone as a giant `files` map JSON blob with no
first-party way to use it — users were left staring at kilobytes of
escaped JSON with no path from "you got JSON" to "here's a project".
Add packages/cli: a zero-dependency `ditto` CLI whose `unpack` command
walks the files{} map from POST /v1/clones (or GET .../result) and
writes a real project tree to disk:
- text files written from inline content;
- binary assets materialized from inline base64, else fetched from their
reference URL using $DITTO_API_URL / $DITTO_API_KEY (--no-fetch writes
just the text tree and lists skipped assets);
- path-traversal guards (clone results are untrusted) and sha256
integrity checks;
- reads JSON from stdin ("-") so a curl response pipes straight in;
- clear errors for queued jobs with no files yet, and a tip pointing at
the /bundle endpoint when assets can't be fetched.
Document the pipe-from-curl one-liner prominently next to the REST
examples in README.md and docs/SERVICE.md, add a package README, a
Repository Map row, a root `unpack` script, and a CHANGELOG entry.
Covered by 9 tests (text tree, stdin, inline base64, live URL fetch with
auth + relative-URL resolution, --no-fetch, traversal refusal, queued-job
detection, sha256 mismatch).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
