Initial commit

2026-06-29 15:11:48 -07:00
commit 66dfdcc58d
404 changed files with 45970 additions and 0 deletions
@@ -0,0 +1,137 @@
+# ditto.site Service — REST + MCP API
+
+A hosted service layer around the deterministic compiler in [`compiler/`](../compiler):
+**`POST a URL → get back the generated app as a file map`**, over both a REST
+API and an MCP server, with a Postgres-backed job queue, result caching, and S3/R2
+blob storage. The compiler's clone semantics are unchanged — this only wraps it.
+
+See the root [`README.md`](../README.md) for the compiler architecture and
+[`DEPLOY.md`](DEPLOY.md) for production deployment.
+
+## Architecture (monorepo / npm workspaces)
+
+```
+compiler/            # the deterministic compiler (unchanged) + a src/index.ts library barrel
+packages/
+  core/              # the ONLY package that imports the compiler:
+                     #   runCloneJob() (temp dir → clone → [verify] → file map), collectFileMap(), cacheKey()
+  db/                # Drizzle schema + migrations + repo + pg-boss queue wrapper
+  storage/           # ArtifactStore: LocalArtifactStore (disk) | ObjectArtifactStore (S3/R2); tgz/zip bundles
+  api/               # Hono app: REST routes + MCP (Streamable-HTTP) + auth/rate-limit/SSRF
+  worker/            # queue consumer: dequeue → runCloneJob → store artifacts → persist; verify harness
+  test-utils/        # shared test helpers (fixture server, Chromium probe, ephemeral Postgres)
+```
+
+Two run modes, same HTTP surface:
+
+- **In-memory (no `DATABASE_URL`)** — the API runs single-page clones **inline** and
+  holds results in memory. Handy for a quick local demo.
+- **DB + queue (`DATABASE_URL` set)** — `POST` enqueues a job (202); a separate
+  **worker** process consumes the queue, runs the clone, stores artifacts, and the
+  client polls to completion. This is the production mode.
+
+## Local development
+
+```bash
+# 1. start Postgres + MinIO
+docker compose up -d
+
+# 2. install + migrate
+npm install
+DATABASE_URL=postgresql://postgres:postgres@localhost:5432/ditto_site npm run db:migrate
+
+# 3. run the API + a worker (two terminals), pointing at the local stack
+cp .env.example .env   # then edit
+DATABASE_URL=... S3_BUCKET=ditto-site-artifacts S3_ENDPOINT=http://localhost:9000 \
+  S3_ACCESS_KEY_ID=minioadmin S3_SECRET_ACCESS_KEY=minioadmin S3_FORCE_PATH_STYLE=true \
+  npm run dev:api
+# ... and the worker with the same env:
+... npm run dev:worker
+
+# Or, fully containerized:
+docker compose --profile app up --build
+```
+
+Quick demo without a DB (inline single-page clone):
+
+```bash
+SSRF_ALLOW_LOOPBACK=true npm run dev:api    # then:
+curl -s -X POST localhost:8787/v1/clones -H 'content-type: application/json' \
+  -d '{"url":"https://example.com/","options":{"mode":"single","styling":"tailwind"}}' | jq '.files | keys'
+```
+
+## REST surface
+
+```
+POST   /v1/clones                 { url, options? }  → 202 {jobId,status} | 200 {cached result | inline result}
+GET    /v1/clones                  → list (metadata)
+GET    /v1/clones/:id              → status + metadata (fileCount, totalBytes, capture, timings)
+GET    /v1/clones/:id/result       → the eager CloneResult (text files inline; binaries by URL)
+GET    /v1/clones/:id/files/*      → stream one file
+GET    /v1/clones/:id/bundle?format=tgz|zip  → the whole app as one archive (302 → S3 when configured)
+DELETE /v1/clones/:id              → purge artifacts
+GET    /healthz                    → { ok: true }  (unauthenticated)
+```
+
+Normal product `options` are `{ mode?: "single" | "multi", styling?: "tailwind" | "css", framework?: "next" | "vite" }`.
+`mode` defaults to `"single"`, `styling` defaults to `"tailwind"`, and `framework` defaults to `"next"`. Operational options
+remain `{ verify?, asyncVerify?, maxRoutes?, maxCollection?, captureConcurrency?, validationConcurrency?, viewportConcurrency?, noCache? }`; `noCache` is service-level and
+bypasses the cache. Deprecated aliases (`multiPage`, `humanizeMode`) and dev-only escape
+hatches are still accepted for compatibility, but are not part of the normal product surface.
+`Cache-Control: no-cache` is honored as an alias.
+
+For fast production responses, keep `verify:false` on the delivery job. The service skips
+validation-only full-page screenshots in that mode. When `verify:true`, multi-page validation
+can render routes and viewports concurrently with `validationConcurrency` and
+`viewportConcurrency`; source route capture concurrency is controlled by `captureConcurrency`.
+When running with the DB worker, `asyncVerify:true` persists the clone result first and then
+attaches the verify report afterward while the worker still has the run artifacts. This is the
+current async QA path; a post-hoc verify endpoint would require persisting full capture artifacts,
+not just the generated app bundle.
+
+**Incremental clone (single → multi, for speed).** Clone one URL single-page first
+(fast app back), then POST the **same URL** with `{ mode: "multi" }` — the second call
+reuses the first's entry capture (no re-capture of page 1), crawls + captures only the
+remaining routes, and regenerates the whole site on top (shared chrome / tokens /
+components preserved). The result carries `captureReused: true` when this fired. Backed
+by a persistent per-URL capture cache (`CAPTURE_CACHE_DIR`, default on under
+`local-data/`; staleness bounded by the cache TTL). The two calls have distinct cache
+keys (single vs multi), so neither shadows the other.
+
+## MCP surface (Streamable-HTTP at `/mcp`)
+
+List-then-read so a clone never floods the agent's context:
+
+- `clone_website({ url, options })` → `{ jobId, status }` (returns immediately).
+- `get_clone_status({ jobId })` → `{ status, timings, capture }`.
+- `get_clone_result({ jobId })` → **metadata only** (routes, verify summary, capture, fileCount, totalBytes, bundleUrl).
+- `list_clone_files({ jobId, glob?, route?, cursor?, limit? })` → manifest `[{path,type,bytes,sha256}]`, no content.
+- `read_clone_files({ jobId, paths[], maxBytes? })` → contents for specific files (text inline; binaries as URLs; per-call size budget).
+- `get_clone_bundle({ jobId, format? })` → a download reference `{ url, format, bytes, sha256 }` (not bytes).
+- `list_clones()` / `cancel_clone({ jobId })`.
+
+## Environment reference
+
+| Var | Used by | Default | Notes |
+|---|---|---|---|
+| `PORT` | api | `8787` | |
+| `DATABASE_URL` | api, worker | — | set ⇒ async DB+queue mode |
+| `ARTIFACTS_DIR` | worker, api | `./local-data/artifacts` | local blob root (when not using S3) |
+| `CACHE_STALE_AFTER` | worker | `24h` | cache TTL (`ms/s/m/h/d`; `0` disables) |
+| `HARNESS_DIR` | worker | `./local-data/harness` | per-worker Next/Vite build harness (verify) |
+| `CAPTURE_CACHE_DIR` | worker, api | `./local-data/capture-cache` | per-URL entry-capture cache for the single→multi reuse path (`""` disables) |
+| `VERIFY_TIER` | worker | `stage2` | perceptual gate tier for verify |
+| `PUBLIC_BASE_URL` | api | — | absolute base for MCP-returned URLs |
+| `API_KEYS` | api | — | comma-separated keys; empty = open |
+| `RATE_LIMIT_PER_MINUTE` | api | `0` | per key/IP cap (0 = unlimited) |
+| `SSRF_DISABLE` | api | `false` | turn off the SSRF guard (not recommended) |
+| `SSRF_ALLOW_LOOPBACK` | api | `false` | allow cloning localhost (local dev) |
+| `S3_BUCKET` / `S3_ENDPOINT` / `S3_REGION` / `S3_ACCESS_KEY_ID` / `S3_SECRET_ACCESS_KEY` / `S3_FORCE_PATH_STYLE` / `S3_PUBLIC_URL` | api, worker | — | set `S3_BUCKET` ⇒ object storage |
+
+## Testing
+
+`npm test` runs all workspace suites (node:test via tsx). Tests gate themselves on
+their dependencies: browser tests skip without Chromium (`npx playwright install
+chromium`), Postgres tests use `TEST_DATABASE_URL` or a throwaway local Postgres
+(root only). The compiler's byte-determinism (Gate 6) makes the golden-file tests
+exact.