# ditto.site Service — REST + MCP API A hosted service layer around the deterministic compiler in [`compiler/`](../compiler): **`POST a URL → get back the generated app as a file map`**, over both a REST API and an MCP server, with a Postgres-backed job queue, result caching, and S3/R2 blob storage. The compiler's clone semantics are unchanged — this only wraps it. See the root [`README.md`](../README.md) for the compiler architecture and [`DEPLOY.md`](DEPLOY.md) for production deployment. ## Architecture (monorepo / npm workspaces) ``` compiler/ # the deterministic compiler (unchanged) + a src/index.ts library barrel packages/ core/ # the ONLY package that imports the compiler: # runCloneJob() (temp dir → clone → [verify] → file map), collectFileMap(), cacheKey() db/ # Drizzle schema + migrations + repo + pg-boss queue wrapper storage/ # ArtifactStore: LocalArtifactStore (disk) | ObjectArtifactStore (S3/R2); tgz/zip bundles api/ # Hono app: REST routes + MCP (Streamable-HTTP) + auth/rate-limit/SSRF worker/ # queue consumer: dequeue → runCloneJob → store artifacts → persist; verify harness test-utils/ # shared test helpers (fixture server, Chromium probe, ephemeral Postgres) ``` Two run modes, same HTTP surface: - **In-memory (no `DATABASE_URL`)** — `POST` enqueues (202) and runs the clone in the background within the API process, holding results in memory. Handy for a quick local demo; same poll-to-completion contract as production. - **DB + queue (`DATABASE_URL` set)** — `POST` enqueues a job (202); a separate **worker** process consumes the queue, runs the clone, stores artifacts, and the client polls to completion. This is the production mode. In both modes the job emits structured progress events as it runs — the compiler's granular log stream (`{ event: "captured", ... }`, `{ event: "generated", ... }`, `{ event: "build_start", ... }`, etc.) plus service phases (`{ event: "clone_done" }`, `{ event: "clone_error", error }`). Every event carries a millisecond wall-clock timestamp `t`; the rest of each object is the payload the emitter logged, so field sets vary by event. There is no fixed enum of event types — treat unknown `event` strings as opaque progress and drive UI off the coarse job `status` from `GET /v1/clones/:id`. Poll `GET /v1/clones/:id/events?after=N` → `{ jobId, events: [{ t, ... }] }`. `after` is a **sequence cursor** (a 1-based event count, not a timestamp): pass the number of events you have already consumed to get only newer ones. Caveats: - The returned event bodies do **not** include the `seq` value, so track the cursor yourself as a running count of events received. - The in-memory backend **ignores `after`** and returns the full list on every poll — so in that mode you always get all events and must de-duplicate by count. The DB backend honors `after` (`seq > N`). In production (DB) mode events persist in the `job_events` table (migration `0002_productive_wallflower.sql`) with a per-job `seq`, so they survive worker restarts and remain readable after completion. ## Local development ```bash # 1. start Postgres + MinIO docker compose up -d # 2. install + migrate npm install DATABASE_URL=postgresql://postgres:postgres@localhost:5432/ditto_site npm run db:migrate # 3. run the API + a worker (two terminals), pointing at the local stack cp .env.example .env # then edit DATABASE_URL=... S3_BUCKET=ditto-site-artifacts S3_ENDPOINT=http://localhost:9000 \ S3_ACCESS_KEY_ID=minioadmin S3_SECRET_ACCESS_KEY=minioadmin S3_FORCE_PATH_STYLE=true \ npm run dev:api # ... and the worker with the same env: ... npm run dev:worker # Or, fully containerized: docker compose --profile app up --build ``` Quick demo without a DB (inline single-page clone): ```bash SSRF_ALLOW_LOOPBACK=true npm run dev:api # then: curl -s -X POST localhost:8787/v1/clones -H 'content-type: application/json' \ -d '{"url":"https://example.com/","options":{"mode":"single","styling":"tailwind"}}' | jq '.files | keys' ``` To turn that `files` map into a project on disk, pipe the response into the repo-local `ditto` CLI (`packages/cli`) instead of inspecting the JSON by hand: ```bash curl -s -X POST localhost:8787/v1/clones -H 'content-type: application/json' \ -d '{"url":"https://example.com/","options":{"mode":"single"}}' \ | npm run --silent unpack -- - ./out # binary assets: set DITTO_API_URL (and DITTO_API_KEY when authenticated) so the # CLI can fetch each file's reference URL; --no-fetch writes only the text tree. ``` The CLI package is intentionally private for now; run this command from a checked-out `ditto.site` repo with dependencies installed. Do not use `npx ditto` until this package is published. ## REST surface ``` POST /v1/clones { url, options? } → 202 {jobId,status} | 200 {cached result | inline result} POST /v1/signup { email, label? } → 201 {apiKey,message} (direct public signup when enabled) POST /v1/signup/request { email } → 202 {message} (send verification email) POST /v1/signup/verify { token } → 201 {apiKey,message} (consume email token) GET /v1/clones → list (metadata) GET /v1/clones/:id → status + metadata (fileCount, totalBytes, capture, timings) GET /v1/clones/:id/events?after=N → { jobId, events } — progress events after cursor N (poll while running) GET /v1/clones/:id/result → the eager CloneResult (text files inline; binaries by URL) GET /v1/clones/:id/files/* → stream one generated file (e.g. .../files/preview.html — see Preview) GET /v1/clones/:id/app-preview/* → serve the clone's built static export when present (404 until built) GET /v1/clones/:id/bundle?format=tgz|zip → the whole app as one archive (302 → S3 when configured) DELETE /v1/clones/:id → purge artifacts GET /healthz → { ok: true } (unauthenticated) ``` `/v1/clones*` and `/mcp` are authenticated when `API_KEYS` is set or DB-backed keys exist. Use `Authorization: Bearer ` or `x-api-key: `. > **Treat keys as secrets.** In any snippet you copy or share, template the key as > an environment variable (`Authorization: Bearer $DITTO_API_KEY`) rather than an > inline `dtto_live_...` token — inline keys leak into shell history, logs, and > pasted transcripts. Never commit a key; rotate a leaked one from the dashboard. Signup routes are intentionally public only when `SIGNUP_ENABLED=true` **and** `DATABASE_URL` is set. Direct `POST /v1/signup` mints a `dtto_live_...` key immediately when `SIGNUP_DIRECT_ENABLED=true`. For public production signup, prefer the Resend-backed verified flow: `POST /v1/signup/request` sends a one-time email link, and `POST /v1/signup/verify` consumes the token, stores only the API key's SHA-256 hash in Postgres, stores the verified email in the key label for attribution, and returns the raw key once. Normal product `options` are `{ mode?: "single" | "multi", styling?: "tailwind" | "css", framework?: "next" | "vite", preview?: boolean }`. `mode` defaults to `"single"`, `styling` defaults to `"tailwind"`, and `framework` defaults to `"next"`. `preview` controls the browsable built export served at `/v1/clones/:id/app-preview/` (see **Preview** below); it is on by default for single-page clones, so pass `preview:true` to force it for a multi-page clone. Operational options remain `{ verify?, asyncVerify?, maxRoutes?, maxCollection?, captureConcurrency?, validationConcurrency?, viewportConcurrency?, noCache? }`; `noCache` is service-level and bypasses the cache. Deprecated aliases (`multiPage`, `humanizeMode`) and dev-only escape hatches are still accepted for compatibility, but are not part of the normal product surface. `Cache-Control: no-cache` is honored as an alias. For fast production responses, keep `verify:false` on the delivery job. The service skips validation-only full-page screenshots in that mode. When `verify:true`, multi-page validation can render routes and viewports concurrently with `validationConcurrency` and `viewportConcurrency`; source route capture concurrency is controlled by `captureConcurrency`. When running with the DB worker, `asyncVerify:true` persists the clone result first and then attaches the verify report afterward while the worker still has the run artifacts. This is the current async QA path; a post-hoc verify endpoint would require persisting full capture artifacts, not just the generated app bundle. ## Preview Every clone emits a flat, self-contained **`preview.html`** at generate time — it is a regular entry in the result file map (path `preview.html`, at the app root), and the manifest records it as `preview_html: "preview.html"` (always present). It exists as soon as the compiler's `generate` step finishes emitting the file map — well before any Next build or deploy — so a consumer can show the cloned page within seconds. Its arrival is visible in the event stream as a `{ event: "generated", ... }` event. Fetch it through the ordinary files route: ```bash curl -sS -H "authorization: Bearer $DITTO_API_KEY" \ "$DITTO_API_URL/v1/clones/$JOB_ID/files/preview.html" -o preview.html ``` **What it is — and isn't.** `preview.html` is a single HTML file with inline CSS and **no JavaScript** (no runtime scripts). It renders the captured page statically: animations are frozen at their captured start frame, Lottie/`