ditto.site/docs/SERVICE.md

ditto.site Service — REST + MCP API

A hosted service layer around the deterministic compiler in compiler/: POST a URL → get back the generated app as a file map, over both a REST API and an MCP server, with a Postgres-backed job queue, result caching, and S3/R2 blob storage. The compiler's clone semantics are unchanged — this only wraps it.

See the root README.md for the compiler architecture and DEPLOY.md for production deployment.

Architecture (monorepo / npm workspaces)

compiler/            # the deterministic compiler (unchanged) + a src/index.ts library barrel
packages/
  core/              # the ONLY package that imports the compiler:
                     #   runCloneJob() (temp dir → clone → [verify] → file map), collectFileMap(), cacheKey()
  db/                # Drizzle schema + migrations + repo + pg-boss queue wrapper
  storage/           # ArtifactStore: LocalArtifactStore (disk) | ObjectArtifactStore (S3/R2); tgz/zip bundles
  api/               # Hono app: REST routes + MCP (Streamable-HTTP) + auth/rate-limit/SSRF
  worker/            # queue consumer: dequeue → runCloneJob → store artifacts → persist; verify harness
  test-utils/        # shared test helpers (fixture server, Chromium probe, ephemeral Postgres)

Two run modes, same HTTP surface:

• In-memory (no DATABASE_URL) — POST enqueues (202) and runs the clone in the background within the API process, holding results in memory. Handy for a quick local demo; same poll-to-completion contract as production.
• DB + queue (DATABASE_URL set) — POST enqueues a job (202); a separate worker process consumes the queue, runs the clone, stores artifacts, and the client polls to completion. This is the production mode.

In both modes the job emits structured progress events (stage transitions such as capture, generate, verify) as it runs. Poll GET /v1/clones/:id/events?after=N with the last sequence number you have seen; each event is {seq, t, ...payload}. In production mode events persist in the job_events table (migration 0002_productive_wallflower.sql), so they survive worker restarts and remain readable after completion.

Local development

# 1. start Postgres + MinIO
docker compose up -d

# 2. install + migrate
npm install
DATABASE_URL=postgresql://postgres:postgres@localhost:5432/ditto_site npm run db:migrate

# 3. run the API + a worker (two terminals), pointing at the local stack
cp .env.example .env   # then edit
DATABASE_URL=... S3_BUCKET=ditto-site-artifacts S3_ENDPOINT=http://localhost:9000 \
  S3_ACCESS_KEY_ID=minioadmin S3_SECRET_ACCESS_KEY=minioadmin S3_FORCE_PATH_STYLE=true \
  npm run dev:api
# ... and the worker with the same env:
... npm run dev:worker

# Or, fully containerized:
docker compose --profile app up --build

Quick demo without a DB (inline single-page clone):

SSRF_ALLOW_LOOPBACK=true npm run dev:api    # then:
curl -s -X POST localhost:8787/v1/clones -H 'content-type: application/json' \
  -d '{"url":"https://example.com/","options":{"mode":"single","styling":"tailwind"}}' | jq '.files | keys'

To turn that files map into a project on disk, pipe the response into the repo-local ditto CLI (packages/cli) instead of inspecting the JSON by hand:

curl -s -X POST localhost:8787/v1/clones -H 'content-type: application/json' \
  -d '{"url":"https://example.com/","options":{"mode":"single"}}' \
  | npm run --silent unpack -- - ./out
# binary assets: set DITTO_API_URL (and DITTO_API_KEY when authenticated) so the
# CLI can fetch each file's reference URL; --no-fetch writes only the text tree.

The CLI package is intentionally private for now; run this command from a checked-out ditto.site repo with dependencies installed. Do not use npx ditto until this package is published.

REST surface

POST   /v1/clones                 { url, options? }  → 202 {jobId,status} | 200 {cached result | inline result}
POST   /v1/signup                 { email, label? } → 201 {apiKey,message}  (direct public signup when enabled)
POST   /v1/signup/request         { email } → 202 {message}                 (send verification email)
POST   /v1/signup/verify          { token } → 201 {apiKey,message}          (consume email token)
GET    /v1/clones                  → list (metadata)
GET    /v1/clones/:id              → status + metadata (fileCount, totalBytes, capture, timings)
GET    /v1/clones/:id/events?after=N → structured progress events since seq N (poll while running)
GET    /v1/clones/:id/result       → the eager CloneResult (text files inline; binaries by URL)
GET    /v1/clones/:id/app-preview/*  → serve the clone's built static export when present (404 until built)
GET    /v1/clones/:id/files/*      → stream one file
GET    /v1/clones/:id/bundle?format=tgz|zip  → the whole app as one archive (302 → S3 when configured)
DELETE /v1/clones/:id              → purge artifacts
GET    /healthz                    → { ok: true }  (unauthenticated)

/v1/clones* and /mcp are authenticated when API_KEYS is set or DB-backed keys exist. Use Authorization: Bearer <key> or x-api-key: <key>.

Treat keys as secrets. In any snippet you copy or share, template the key as an environment variable (Authorization: Bearer $DITTO_API_KEY) rather than an inline dtto_live_... token — inline keys leak into shell history, logs, and pasted transcripts. Never commit a key; rotate a leaked one from the dashboard. Signup routes are intentionally public only when SIGNUP_ENABLED=true and DATABASE_URL is set. Direct POST /v1/signup mints a dtto_live_... key immediately when SIGNUP_DIRECT_ENABLED=true. For public production signup, prefer the Resend-backed verified flow: POST /v1/signup/request sends a one-time email link, and POST /v1/signup/verify consumes the token, stores only the API key's SHA-256 hash in Postgres, stores the verified email in the key label for attribution, and returns the raw key once.

Normal product options are { mode?: "single" | "multi", styling?: "tailwind" | "css", framework?: "next" | "vite" }. mode defaults to "single", styling defaults to "tailwind", and framework defaults to "next". Operational options remain { verify?, asyncVerify?, maxRoutes?, maxCollection?, captureConcurrency?, validationConcurrency?, viewportConcurrency?, noCache? }; noCache is service-level and bypasses the cache. Deprecated aliases (multiPage, humanizeMode) and dev-only escape hatches are still accepted for compatibility, but are not part of the normal product surface. Cache-Control: no-cache is honored as an alias.

For fast production responses, keep verify:false on the delivery job. The service skips validation-only full-page screenshots in that mode. When verify:true, multi-page validation can render routes and viewports concurrently with validationConcurrency and viewportConcurrency; source route capture concurrency is controlled by captureConcurrency. When running with the DB worker, asyncVerify:true persists the clone result first and then attaches the verify report afterward while the worker still has the run artifacts. This is the current async QA path; a post-hoc verify endpoint would require persisting full capture artifacts, not just the generated app bundle.

Incremental clone (single → multi, for speed). Clone one URL single-page first (fast app back), then POST the same URL with { mode: "multi" } — the second call reuses the first's entry capture (no re-capture of page 1), crawls + captures only the remaining routes, and regenerates the whole site on top (shared chrome / tokens / components preserved). The result carries captureReused: true when this fired. Backed by a persistent per-URL capture cache (CAPTURE_CACHE_DIR, default on under local-data/; staleness bounded by the cache TTL). The two calls have distinct cache keys (single vs multi), so neither shadows the other.

MCP surface (Streamable-HTTP at /mcp)

List-then-read so a clone never floods the agent's context:

• clone_website({ url, options }) → { jobId, status } (returns immediately).
• get_clone_status({ jobId }) → { status, timings, capture }.
• get_clone_result({ jobId }) → metadata only (routes, verify summary, capture, fileCount, totalBytes, bundleUrl).
• list_clone_files({ jobId, glob?, route?, cursor?, limit? }) → manifest [{path,type,bytes,sha256}], no content.
• read_clone_files({ jobId, paths[], maxBytes? }) → contents for specific files (text inline; binaries as URLs; per-call size budget).
• get_clone_bundle({ jobId, format? }) → a download reference { url, format, bytes, sha256 } (not bytes).
• list_clones() / cancel_clone({ jobId }).

Environment reference

Var	Used by	Default	Notes
PORT	api	8787
DATABASE_URL	api, worker	—	set ⇒ async DB+queue mode
ARTIFACTS_DIR	worker, api	./local-data/artifacts	local blob root (when not using S3)
CACHE_STALE_AFTER	worker	24h	cache TTL (ms/s/m/h/d; 0 disables)
HARNESS_DIR	worker	./local-data/harness	per-worker Next/Vite build harness (verify)
CAPTURE_CACHE_DIR	worker, api	./local-data/capture-cache	per-URL entry-capture cache for the single→multi reuse path ("" disables)
VERIFY_TIER	worker	stage2	perceptual gate tier for verify
PUBLIC_BASE_URL	api	—	absolute base for MCP-returned URLs
API_KEYS	api	—	comma-separated keys; empty = open
RATE_LIMIT_PER_MINUTE	api	0	per key/IP cap (0 = unlimited)
SIGNUP_ENABLED	api	false	DB mode only: expose public API-key signup routes
SIGNUP_RATE_LIMIT_PER_HOUR	api	3	per-IP signup cap; 0 disables signup throttling
DEFAULT_SIGNUP_KEY_RATE_LIMIT	api	30	stored on keys minted by signup; service-wide enforcement still uses RATE_LIMIT_PER_MINUTE
SIGNUP_DIRECT_ENABLED	api	true	keep direct POST /v1/signup enabled; set false when Resend verification is configured
RESEND_API_KEY	api	—	enables verified-email signup request/verify endpoints
SIGNUP_FROM_EMAIL	api	—	verified sender, e.g. Ditto <hello@ditto.site>
SIGNUP_VERIFY_URL	api	—	landing-page URL that receives ?token=..., e.g. https://www.ditto.site/api-key
SIGNUP_TOKEN_TTL_MINUTES	api	30	one-time email verification token lifetime
SIGNUP_CORS_ORIGINS	api	https://ditto.site,https://www.ditto.site	comma-separated browser origins allowed to call public signup routes
SSRF_DISABLE	api	false	turn off the SSRF guard (not recommended)
SSRF_ALLOW_LOOPBACK	api	false	allow cloning localhost (local dev)
S3_BUCKET / S3_ENDPOINT / S3_REGION / S3_ACCESS_KEY_ID / S3_SECRET_ACCESS_KEY / S3_FORCE_PATH_STYLE / S3_PUBLIC_URL	api, worker	—	set S3_BUCKET ⇒ object storage

Testing

npm test runs all workspace suites (node:test via tsx). Tests gate themselves on their dependencies: browser tests skip without Chromium (npx playwright install chromium), Postgres tests use TEST_DATABASE_URL or a throwaway local Postgres (root only). The compiler's byte-determinism (Gate 6) makes the golden-file tests exact.