ditto.site/README.md

# ditto.site

[![CI](https://github.com/ion-design/ditto.site/actions/workflows/ci.yml/badge.svg)](https://github.com/ion-design/ditto.site/actions/workflows/ci.yml)
[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](LICENSE)
[![Node](https://img.shields.io/badge/node-%3E%3D20-brightgreen.svg)](.nvmrc)

ditto.site is a deterministic website compiler. Given a public URL, it compiles the
observed rendered site into a modern TypeScript app, defaulting to Next.js App
Router with an option for Vite React, using
browser-captured evidence: DOM, computed styles, layout boxes, CSS rules, fonts,
assets, source metadata, screenshots, interaction states, and motion specs where
they can be reproduced safely.

The compiler is not an LLM page author. The normal clone path is a pure
generation pipeline over a frozen capture, so the same capture produces
byte-stable output. The generated app uses `ditto` naming for clone-specific
runtime helpers and documentation.

## What It Produces

A generated app is a self-contained project under `generated/app/` during
validation and under `<out>/<site>/app` for delivery. The default framework is
Next.js App Router:

- `src/app/layout.tsx`: root layout, captured language, Next metadata, viewport,
  JSON-LD, and shared multi-route shell.
- `src/app/page.tsx` and route pages: JSX reconstructed from the captured render
  tree.
- `src/app/globals.css`: reset, font faces, design tokens, Tailwind setup when
  enabled.
- `src/app/ditto.css`: clone-specific CSS that still needs stylesheet emission,
  including pseudo-elements, keyframes, raw CSS fallbacks, and non-Tailwind
  fidelity rules.
- `src/app/content.ts` or `content.tsx`: semantic editable data when repeated
  regions are promoted into components or sections.
- `src/app/components/`, `src/app/sections/`, `src/app/svgs/`: generated modules
  split from repeated components, page sections, and inline SVGs.
- `src/app/ditto/`: small generated runtime helpers for recognized interactions,
  accordions, dropdown menus, and reproducible motion.
- `src/app/robots.ts`, `src/app/sitemap.ts`, `src/app/llms.txt/route.ts`, and
  sometimes `src/app/llms-full.txt/route.ts`.
- App Router file-based icons such as `src/app/favicon.ico`, `src/app/icon.png`,
  and `src/app/apple-icon.png` when the source exposes them.
- `public/assets/cloned/`: materialized images, fonts, manifest files, manifest
  icons/screenshots, and other source assets needed by the clone.
- `AGENTS.md` and `ARCHITECTURE.md`: generated documentation for the delivered
  app, derived from clone metadata.

With `--framework=vite` or API option `framework: "vite"`, the generated app is
a Vite React project. Single-page output uses `index.html`, `src/main.tsx`,
`src/page.tsx`, `src/globals.css`, and `src/ditto.css`. Multi-route Vite output
is a Vite multi-page app with one HTML entry per cloned route and route modules
under `src/routes/<routeKey>/`.

Validation builds keep internal `data-cid` attributes so gates can align source
and clone nodes exactly. The exported app strips validation-only ids and only
keeps deterministic `data-ditto-id` anchors where generated CSS or runtime
recipes still need a stable target.

## Quickstart

```bash
npm install
npx playwright install chromium

# Single-page clone, Tailwind output by default.
npm run clone -- https://example.com/

# Explicit product choices.
npm run clone -- https://example.com/ --mode=single --styling=tailwind
npm run clone -- https://example.com/ --mode=single --styling=css
npm run clone -- https://example.com/ --mode=multi --styling=tailwind
npm run clone -- https://example.com/ --mode=multi --styling=css
npm run clone -- https://example.com/ --mode=single --framework=vite
npm run clone -- https://example.com/ --mode=multi --framework=vite

# Deliver into a clean app directory.
npm run clone -- https://example.com/ --out=./output

# Benchmark and validation helpers.
npm run bench -- --tier=easy
npm run bench -- --tier=stage2 --reuse
npm run bench-site
npm run validate-site -- runs/site-example.com/<timestamp>

# Faster multi-page runs on larger sites. Validation is opt-in.
npm run clone -- https://example.com/ --mode=multi --concurrency=5
npm run clone -- https://example.com/ --mode=multi --validate --validate-concurrency=3 --viewport-concurrency=2
npm run validate-site -- runs/site-example.com/<timestamp> --validate-concurrency=3 --viewport-concurrency=2
```

The root scripts forward into the `compiler` workspace. Running the same commands
from `compiler/` also works.

Multi-page generation defaults to the fast no-validation path. For production
delivery, keep first response and QA as separate phases: run single-page first,
expand to multi-page with default CLI behavior or service `verify:false`, then
run strict validation separately. Use service `asyncVerify:true` when a DB worker
should persist the clone first and attach the verify report afterward. Capture
route parallelism is controlled by `--concurrency`; validation route and viewport
parallelism are controlled by `--validate-concurrency` and
`--viewport-concurrency`.

## Architecture

```
URL
  -> capture
  -> normalize render IR
  -> infer assets, fonts, sections, recipes, SEO, metadata
  -> generate app (Next by default, Vite optional)
  -> materialize assets
  -> build and validate
  -> export delivery app
```

### 1. Capture

Capture is Playwright-based. A page is loaded once and resized through the target
viewports, normally `375`, `768`, `1280`, and `1920`, so responsive snapshots are
aligned to the same page state instead of four unrelated reloads.

The in-page walker records:

- DOM tree shape and text.
- Curated computed styles.
- Document-coordinate bounding boxes.
- Per-viewport visibility, scroll dimensions, and page backgrounds.
- Pseudo-element content and styles.
- Inline SVG markup.
- Source attributes that are safe and meaningful, including accessibility
  attributes.
- Head metadata, link tags, JSON-LD, icons, manifest links, alternates, and SEO
  discovery resources.
- CSS, fonts, images, SVGs, videos, manifests, and other linked assets.

Capture also handles common state problems before the snapshot:

- Cookie, consent, newsletter, and modal overlays are dismissed when a safe
  deterministic rule recognizes them.
- Scroll-locked and iframe-backed overlays are removed only when they match the
  blocker patterns.
- Poster-less videos get a representative still when possible.
- Lazy background images and responsive assets are backfilled from CSS, DOM, and
  manifest evidence.
- Interaction capture stamps temporary `data-cid-cap` ids in the browser,
  explores recognized controls, and records state deltas.
- Motion capture extracts declarative CSS keyframes, WAAPI effects, rotating text,
  scroll reveals, and marquee-like tracks when they can be replayed deterministically.

### 2. Normalize

The normalizer merges viewport snapshots into one render IR. Each node receives a
stable pre-order id and carries per-viewport style, box, and visibility data.
Temporary capture ids are kept only as needed for interaction and motion mapping.

The IR is intentionally close to rendered browser facts. Higher-level intent is
inferred later so fidelity gates can always fall back to measured evidence when a
semantic guess is uncertain.

### 3. Infer

Inference is deterministic and local to the capture:

- Sections are detected from rendered structure and visible hierarchy.
- Design tokens are extracted from repeated color, spacing, and type values.
- Semantic color roles are assigned where source evidence and usage make them
  stable enough.
- Primitive roles identify headings, links, buttons, images, icons, nav, badges,
  forms, and related UI pieces.
- Asset and font graphs classify linked files, download them content-addressably,
  rewrite references, and preserve fallbacks.
- Recipe inference recognizes repeated card grids, feature grids, product grids,
  logo clouds, galleries, navs, and other common section patterns.
- Interaction recipes map captured deltas to small runtime helpers rather than
  replaying arbitrary site JavaScript.
- Motion recipes emit reproducible templates for the safe declarative families
  and freeze unsupported motion honestly.
- SEO inventory records source title, description, keywords, canonical URL,
  robots/referrer/theme-color/color-scheme, Open Graph, Twitter cards, icons,
  manifest links and assets, alternates/hreflang, JSON-LD, robots/sitemap links,
  and `llms.txt` or `llms-full.txt` when discoverable.
- Code quality inventory records module organization, generated component split,
  content-model extraction, metadata, and markup hygiene.

### 4. Generate

The generator emits a Next.js App Router app by default, or a Vite React app when
`framework` is `vite`. Tailwind v4 is the default styling mode; plain CSS remains
available.

In Tailwind mode, most exact geometry, typography, display, and spacing becomes
utility classes. Values that do not belong in class names, pseudo-elements,
keyframes, and recipe/runtime selectors stay in `ditto.css`. In CSS mode, shared
semantic classes and per-rule stylesheet emission preserve the same computed
style contract.

The generator also:

- Splits clean page regions into `sections/`.
- Extracts repeated DOM skeletons into `components/`.
- Keeps editable semantic data in `content.ts`, while validation ids and
  per-instance class overrides stay in generated plumbing modules.
- Hoists inline SVGs into `svgs/` where that improves readability.
- Emits `ditto` runtime utilities only for recognized recipes that need them.
- Rewrites same-origin internal links for generated routes.
- Emits framework-appropriate metadata, JSON-LD, robots, sitemap, `llms.txt`,
  icons, web manifests, and manifest assets from the SEO inventory.
- Emits generated `AGENTS.md` and `ARCHITECTURE.md` for the delivered app.

### 5. Multi-Route Cloning

Multi-route mode starts at one entry URL, crawls same-origin links, groups routes
by deterministic URL templates, and applies a CMS/template boundary:

- Singletons are reproduced.
- Pairs are reproduced because there is not enough evidence to collapse them.
- Larger collections are represented by the listing and one representative route.
- The full instance list is recorded as a handoff boundary rather than cloning
  every CMS item.

Each selected route is captured separately. The site generator then emits one app
with shared assets, shared tokens, link rewriting, optional shared header/footer
chrome, and route-level pages. A multi-route job can reuse a prior single-page
entry capture to return the first page quickly and expand later.

### 6. Validate

Validation builds and serves the generated app, captures it with the same walker,
and grades deterministic gates:

- Build and static export success.
- Capture sanity: the source was not an empty shell, bot wall, or polluted overlay
  state.
- Asset and font materialization.
- DOM shape and valid retags.
- Computed-style fidelity.
- Layout boxes, section positions, page dimensions, and responsive behavior.
- Byte determinism from regenerating the same frozen capture.
- Perceptual screenshot similarity.
- Interaction recipe behavior for menus, tabs, accordions, carousels, modals,
  hover, and focus states that were captured.
- Motion reproduction for supported declarative families.
- Site-level link integrity and site determinism for multi-route output.
- Output quality, including componentization, naming, content extraction,
  styling organization, and metadata hygiene.

The gates are intended to be deterministic grading functions. Failures should
produce a reproducible artifact and a narrow compiler improvement, not a manual
one-off patch to an output app.

## SEO And Documentation Layer

ditto.site treats source metadata as part of the clone contract. The current
generator preserves source-provided metadata in the generated framework shell
where possible, materializes linked icons and manifest assets, preserves JSON-LD
with safe script emission, and generates or preserves `llms.txt`.

If a source exposes `llms.txt` or `llms-full.txt`, those files are preserved as
static routes. If not, the generator creates a concise `llms.txt` from captured
route titles, descriptions, and visible content summaries. The generated app also
receives root `AGENTS.md` and `ARCHITECTURE.md` files that explain the app
structure, safe edit zones, `src/app/ditto`, `content.ts`, components, sections,
SVG modules, `ditto.css`, and `ditto-meta.ts`.

SEO inventory metrics are written beside generated artifacts as `seo.json` and
`seo.md` so coverage can be inspected without changing output behavior.

## Service Layer

The hosted service wraps the compiler without changing clone semantics:

```
compiler/            # capture, IR, inference, generation, validation
packages/core/       # compiler adapter and file-map collection
packages/db/         # Drizzle schema, migrations, repository, queue wrapper
packages/storage/    # local and S3/R2 artifact storage
packages/api/        # Hono REST API and MCP server
packages/worker/     # queued clone runner and optional verify harness
packages/test-utils/ # fixture server and integration-test helpers
```

REST accepts a URL and clone options, then returns either an inline result or an
async job id depending on whether a database queue is configured. The MCP server
uses a list-then-read model: agents get job metadata and file manifests first,
then request only the files they need.

See [docs/SERVICE.md](docs/SERVICE.md) and [docs/DEPLOY.md](docs/DEPLOY.md) for
the operational API and deployment details.

## Repository Map

| Path | Purpose |
| --- | --- |
| `compiler/` | deterministic clone compiler |
| `compiler/src/capture/` | Playwright capture, walker, assets, SEO resources, interactions, motion |
| `compiler/src/normalize/` | render IR construction |
| `compiler/src/infer/` | sections, tokens, assets, fonts, recipes, primitives |
| `compiler/src/generate/` | app generation, SEO/docs layer, code quality reports |
| `compiler/src/site/` | multi-route crawl/generate/validate flow |
| `compiler/src/validate/` | fidelity, perceptual, interaction, motion, and determinism gates |
| `compiler/benchmarks/` | benchmark URL lists |
| `packages/` | REST, MCP, queue, storage, and service adapters |
| `docs/SERVICE.md` | service architecture and API reference |
| `docs/DEPLOY.md` | Railway, Neon, and R2 deployment guide |
| `examples/` | benchmark result summaries, composites, motion evidence, and small runnable outputs |

## Deferred Work

ditto.site intentionally does not attempt arbitrary JavaScript replay. It does not
recreate full third-party applications, live personalization, auth, payments,
analytics behavior, or remote iframe internals. It can preserve static scaffolding
around those regions and may emit placeholders when that is the more faithful
self-contained representation.

Video-like animation replay, scroll-scrubbed canvases, WebGL, and finished
entrance animations remain outside the deterministic contract unless a safe,
observable recipe exists. Unsupported motion is frozen rather than shipped as a
broken imitation.

## Contributing

Use `npm run typecheck` and `npm test` before opening a PR. Browser tests require
Chromium; Postgres-backed tests use `TEST_DATABASE_URL` or the local compose
stack. Changes that alter compiler output should include focused fixture or
benchmark evidence.

See [CONTRIBUTING.md](CONTRIBUTING.md), [SECURITY.md](SECURITY.md), and
[CODE_OF_CONDUCT.md](CODE_OF_CONDUCT.md).

## License

[MIT](LICENSE) © ion-design and contributors.