Initial commit

examples/README.md

@@ -0,0 +1,70 @@
+# Clone review — before / after
+
+Side-by-side captures for the deterministic website cloner. Each image is one
+site: **SOURCE (left) vs CLONE (right)**, with a desktop row (1280px, shown at
+50%) on top and a mobile row (375px, 100%) below.
+
+## Results
+
+| Tier | Gates 0–6 | Avg score | Detail |
+|------|:---------:|:---------:|--------|
+| **easy** (24 personal/portfolio/content sites) | **24 / 24** | 99.7 | [easy/RESULTS.md](easy/RESULTS.md) |
+| **medium** (23 SaaS / ecommerce marketing pages) | **16 / 23** | 98.9 | [medium/RESULTS.md](medium/RESULTS.md) |
+| **hard** (32 dense editorial / docs / marketing / ecommerce) | **23 / 32** | 98.8 | [hard/RESULTS.md](hard/RESULTS.md) |
+| **stage2** (25 popup / video / animation pages) | **22 / 25** | 99.1 | [stage2/RESULTS.md](stage2/RESULTS.md) |
+| **sites** (5 multi-page sites, Stage 3) | **47 / 49 routes** | — | [sites/RESULTS.md](sites/RESULTS.md) |
+
+**Stage 2** ("capture the right state") adds two gates beyond 0–6 — **pollution**
+(degenerate / bot-wall / un-dismissed-modal capture) and **perceptual** (tier-
+thresholded screenshot diff) — plus capture-time **overlay/popup dismissal**,
+**animation settling**, and **dynamic-media first-frame**, and a single-load+resize
+capture that keeps content identical across viewports. **19/25 also clear the
+stricter stage-2 bar** (gates 0–6 + pollution + perceptual); browse [`stage2/`](stage2/).
+
+**Stage 3** ("multi-page / whole-site") clones a whole site from one entry URL into
+**one** Next.js app: it crawls routes, collapses CMS collections to one representative
+(jamstack 552→11, 11ty 1085→12 reproduced routes), hoists shared header/footer chrome
+into the layout, rewrites internal links to the clone routes, and adds site-level
+**link-integrity** + **site-determinism** gates. **47/49 routes pass gates 0–6, all 5
+sites green on links + determinism**; browse [`sites/`](sites/).
+
+Gates 0–6 = build, capture, asset/font, DOM, computed-style, layout, determinism.
+Every non-passing medium site still scores 95–99; the misses are documented
+limitations (see below). Screenshots are a human sanity check on top of the
+quantitative gates — browse them in [`easy/`](easy/) and [`medium/`](medium/).
+
+## A few highlights
+
+Pixel-perfect static reconstruction:
+
+![michaelcole](easy/easy-001-michaelcole.me.png)
+
+High-fidelity medium reconstruction:
+
+![medium sample](medium/medium-002-origami.chat.png)
+
+## Known limitations (visible in some medium shots)
+
+A static, deterministic clone reconstructs the captured **static** state. Content
+driven by JS / time / scroll, or embedded from a third party, is **non-
+deterministic frame-to-frame** and cannot be reproduced — the static scaffolding
+(text, layout, materialized images) around it is still correct:
+
+- **Rotating hero text** — shopify, notion (a JS timer swaps the word).
+- **Scroll-scrubbed / autoplay hero video** — mercury (`<video>` frame depends on
+  scroll position; the fallback image is reconstructed, behind the video).
+- **`<iframe>` embeds** — loom (dropped to keep the clone self-contained).
+- **Fancy multi-line headings** (per-word `inline-block` + spacers) stack a few px
+  taller — linear, notion, deel.
+
+Current architecture overview: [`../README.md`](../README.md).
+
+## Generated app shape
+
+The compiler emits a Next.js App Router + TypeScript project per site:
+`src/app/page.tsx` (JSX reconstructed from the DOM), `src/app/ditto.css`
+(clone-specific CSS that cannot be represented cleanly as utilities), and
+`layout.tsx`. Larger clones additionally carry `public/assets/` (materialized
+images/fonts/icons/manifests), generated documentation, and optional
+`src/app/ditto/` runtime helpers. Regenerate any site with
+`cd ../compiler && npm run bench -- --tier=medium`.

examples/easy/RESULTS.md

@@ -0,0 +1,30 @@
+# Easy tier — final results (gates 0-6)
+
+| id | site | score | gates0-6 | note |
+|----|------|------:|:--------:|------|
+| easy-001 | www.michaelcole.me | 100.0 | PASS |  |
+| easy-002 | f.inc | 99.4 | PASS |  |
+| easy-003 | www.arkli.io | 100.0 | PASS |  |
+| easy-004 | gist-quiz.com | 100.0 | PASS |  |
+| easy-005 | brittanychiang.com | 99.7 | PASS |  |
+| easy-006 | leerob.io | 99.9 | PASS |  |
+| easy-007 | paco.me | 99.9 | PASS |  |
+| easy-008 | rauno.me | 100.0 | PASS |  |
+| easy-009 | sive.rs | 99.9 | PASS |  |
+| easy-010 | daringfireball.net | 99.6 | PASS |  |
+| easy-011 | www.julian.com | 99.9 | PASS |  |
+| easy-012 | ianlunn.co.uk | 99.3 | PASS | screenshot: project preview images (fine detail) |
+| easy-013 | adamwathan.me | 100.0 | PASS |  |
+| easy-014 | www.joshwcomeau.com | 99.4 | PASS | hero is a runtime <canvas> animation (not reconstructable) |
+| easy-015 | emilkowal.ski | 100.0 | PASS |  |
+| easy-016 | bradfrost.com | 99.2 | PASS | screenshot: confetti header pattern (sub-pixel) |
+| easy-017 | maggieappleton.com | 98.9 | PASS | screenshot: garden illustrations |
+| easy-018 | www.robinwieruch.de | 100.0 | PASS |  |
+| easy-019 | www.karlsutt.com | 99.7 | PASS |  |
+| easy-020 | www.maxbo.me | 98.5 | PASS | WebGL shader bg not reconstructable (canvas) |
+| easy-021 | www.olaolu.dev | 99.8 | PASS |  |
+| easy-022 | www.taniarascia.com | 99.9 | PASS |  |
+| easy-023 | kentcdodds.com | 99.7 | PASS |  |
+| easy-024 | www.swyx.io | 99.5 | PASS |  |
+
+**24/24 pass gates 0-6**, average 99.7. The 4 screenshot-diff cases all pass gates 0-6 (DOM/style/layout/determinism); they are fine-detail raster or runtime-canvas content.

examples/hard/RESULTS.md

@@ -0,0 +1,38 @@
+# Hard tier — results (gates 0-6)
+
+**23/32 pass gates 0-6**, average 98.8. Hard = dense/heavy editorial, docs, marketing & ecommerce.
+
+| id | site | score | gates0-6 | failing |
+|----|------|------:|:--------:|---------|
+| hard-001 | www.paulgraham.com | 95.8 | FAIL | style,layout |
+| hard-002 | news.ycombinator.com | 99 | PASS | screenshot |
+| hard-003 | en.wikipedia.org/wiki/Cascading_Style_Sheets | 99.1 | FAIL | layout,screenshot |
+| hard-004 | tailwindcss.com | 99.5 | PASS |  |
+| hard-005 | developer.mozilla.org/en-US | 99.2 | PASS | screenshot |
+| hard-006 | stripe.com | 98.5 | FAIL | layout |
+| hard-007 | github.com | 99.5 | PASS |  |
+| hard-008 | www.apple.com | 99.1 | PASS | screenshot |
+| hard-009 | www.engadget.com | 99.4 | PASS | screenshot |
+| hard-010 | www.theregister.com | 99 | FAIL | layout |
+| hard-011 | react.dev | 99.7 | PASS |  |
+| hard-012 | www.netlify.com | 99.6 | PASS |  |
+| hard-013 | www.smashingmagazine.com | 99.6 | PASS |  |
+| hard-014 | css-tricks.com | 99.2 | PASS | screenshot |
+| hard-015 | www.sitepoint.com | 99.6 | PASS |  |
+| hard-016 | dev.to | 99.9 | PASS |  |
+| hard-017 | www.producthunt.com | 99.8 | PASS |  |
+| hard-018 | www.awwwards.com | 97.8 | FAIL | asset_font |
+| hard-019 | medium.com | 99.7 | PASS |  |
+| hard-020 | hackernoon.com | 96.8 | FAIL | layout,screenshot |
+| hard-021 | www.gov.uk | 99.5 | PASS |  |
+| hard-022 | blog.cloudflare.com | 99.9 | PASS |  |
+| hard-023 | www.nasa.gov | 98.4 | PASS | screenshot |
+| hard-024 | www.mozilla.org/en-US | 99.7 | PASS |  |
+| hard-025 | web.dev | 99.8 | PASS |  |
+| hard-026 | www.nike.com | 98.6 | PASS | screenshot |
+| hard-027 | www.zara.com | 99.1 | PASS |  |
+| hard-028 | www.ssense.com | 93.4 | FAIL | layout,screenshot |
+| hard-029 | www.gymshark.com | 97.9 | FAIL | layout,screenshot |
+| hard-030 | www.sephora.com | 99.1 | PASS |  |
+| hard-031 | www.etsy.com | 99.9 | PASS |  |
+| hard-032 | www.uniqlo.com/us/en | 95.3 | FAIL | layout,screenshot |

examples/medium/RESULTS.md

@@ -0,0 +1,29 @@
+# Medium tier — results (gates 0-6)
+
+**16/23 pass gates 0-6**, average 98.9. All non-passing sites score 95+ (medium bar is 80+ with documented limitations).
+
+| id | site | score | gates0-6 | note |
+|----|------|------:|:--------:|------|
+| medium-002 | origami.chat | 99.7 | PASS |  |
+| medium-004 | www.tryskylink.com | 99.9 | PASS |  |
+| medium-005 | linear.app | 98.6 | FAIL | mobile heading wraps taller (inline-block word spans) |
+| medium-006 | www.raycast.com | 99.4 | PASS |  |
+| medium-007 | supabase.com | 99.7 | PASS |  |
+| medium-008 | retool.com | 96.5 | FAIL | 1 of 652 asset refs 404s |
+| medium-009 | www.warp.dev | 99.8 | PASS |  |
+| medium-010 | plaid.com | 99.3 | PASS |  |
+| medium-011 | www.loom.com | 99.5 | PASS |  |
+| medium-012 | webflow.com | 99.5 | PASS |  |
+| medium-013 | www.notion.com/product | 98.7 | FAIL | inline heading height; minor pixel diff |
+| medium-014 | slack.com | 99.6 | PASS |  |
+| medium-015 | www.figma.com | 99.5 | PASS |  |
+| medium-016 | www.attio.com | 99.1 | FAIL | 375 offset + height ~5.6% |
+| medium-017 | www.airtable.com | 99.5 | PASS |  |
+| medium-018 | www.deel.com | 97.5 | FAIL | inline heading height |
+| medium-019 | mercury.com | 95.0 | FAIL | custom-font text width (content-sized elems) |
+| medium-020 | asana.com | 96.7 | FAIL | heavy site; some assets dropped under capture cap |
+| medium-021 | replit.com | 99.9 | PASS |  |
+| medium-022 | vercel.com | 99.8 | PASS |  |
+| medium-023 | www.shopify.com | 99.2 | PASS |  |
+| medium-024 | www.allbirds.com | 99.8 | PASS |  |
+| medium-025 | www.glossier.com | 98.9 | PASS |  |

examples/motion/RESULTS.md

@@ -0,0 +1,64 @@
+# Stage 5 — motion / animation results
+
+Stage 5 reproduces the *motion* (not just the settled frame) behind a deterministic
+**motion gate**, on by default (`--no-motion`). Declarative allowlist + fixed templates;
+freeze anything not confidently reproducible. The clone **replays motion on load**; the
+validator cancels animations to grade the settled frame, so Gates 0–6 + perceptual are
+untouched (no motion regression). The current architecture overview is in the root
+`README.md`.
+
+The motion gate's accuracy contract: **`reproduced + frozen === captured`** — every captured
+animation is either reproduced-and-verified (running in the built clone) or honestly frozen
+(its keyframes weren't capturable, e.g. a cross-origin sheet). Never ship broken motion.
+
+## Fixtures (deterministic)
+
+| fixture | motion | motion gate | gates 0–6 | score |
+|---|---|---|---|---|
+| `fixtures/motion.html` | finite entrance + infinite spin + pulse + card stagger (CSS `@keyframes`) | **CSS 7/7 reproduced & running** | ✅ | 100 |
+| `fixtures/motion2.html` | WAAPI orbit (∞) + WAAPI fade-in + rotating word | **WAAPI 2/2, rotators 1/1** | ✅ | 99.9 |
+
+## Motion benchmark tier (`benchmarks/motion.json`) — `npm run bench -- --tier=motion`
+
+**6 / 6 pass gates 0–6 + Stage-2 + the motion gate, average 99.8, zero failing gates.**
+Real declarative motion is reproduced where present; sites whose only motion is
+canvas/finished-entrance are honestly N/A (frozen), with no regression.
+
+| id | site | score | gates 0–6 | motion gate |
+|---|---|---:|:--:|---|
+| motion-001 | rauno.me | 100 | ✅ | **CSS 7/7 running** (were frozen pre-Stage-5) |
+| motion-002 | sive.rs | 100 | ✅ | N/A (static) |
+| motion-003 | emilkowal.ski | 100 | ✅ | N/A (entrance WAAPI finishes pre-capture) |
+| motion-004 | leerob.io | 99.9 | ✅ | N/A (static) |
+| motion-005 | joshwcomeau.com | 99.5 | ✅ | **CSS 9/11 + scroll-reveals 13/13** (2 CSS frozen; hero canvas out of scope) |
+| motion-006 | framer.com | 99.6 | ✅ | **CSS 16/16 + WAAPI 2/2** (Framer Motion showcase) |
+
+Four declarative families are reproduced & gate-verified: **CSS `@keyframes`, WAAPI,
+rotating text, scroll-triggered reveals**. Visual evidence (timeline filmstrips + webms,
+built from the existing clones with `runner/motionFilmstrip.ts`) lives in
+[`evidence/`](evidence/): CSS spinner/pulse, WAAPI orbit + rotating word, scroll-reveal
+cards fading in, and framer.com.
+
+## No regression (easy tier, motion ON by default)
+
+| site | score | gates 0–6 | motion |
+|---|---|---|---|
+| michaelcole.me | 100 | ✅ | N/A (static) |
+| f.inc | 99.8 | ✅ | CSS 2/2 |
+| arkli.io | 100 | ✅ | N/A (static) |
+| gist-quiz.com | 100 | ✅ | N/A (static) |
+
+The static set is unaffected: where there's no motion the gate is N/A (and no browser is
+launched); where there is, it reproduces — and either way the cancel-before-grade step means
+the base gates measure the same settled frame the static clone always did.
+
+## Limitations (by contract)
+
+- **Scroll-scrubbed** motion (frame = f(scrollY)) and **entrance WAAPI that finishes before
+  capture** are left frozen (not observable as a stable declarative spec).
+- **Canvas/WebGL** is out of scope (frozen, as before).
+- Some heavy SaaS marketing sites (**linear.app**) serve a degenerate JS-shell to headless
+  capture — a capture/bot-wall limit (like warbyparker in stage 2), orthogonal to motion.
+
+Reproduce: serve `compiler/fixtures/` and `npm run clone -- http://127.0.0.1:PORT/motion.html`,
+then `npx tsx src/runner/validateOne.ts <runDir>`; or `npm run bench -- --tier=motion`.

examples/polish/README.md

@@ -0,0 +1,65 @@
+# Polish backlog — landed
+
+Four lower-risk fidelity wins from the stage reviews, each gate-verified on real
+benchmark sites with no regression on the static set.
+
+## 1. `<video>` first-frame still (no longer blank)
+
+A poster-less `<video>` whose stream aborts at snapshot used to render empty. Capture now
+materializes a representative still and points the element's `poster` at it: canvas-draw the
+decoded frame where readable, else **screenshot the element** over the DevTools protocol
+(composited pixels → works for cross-origin / tainted / late-decode videos canvas can't read).
+
+- `video-descript-3panel.png` — **descript.com** hero-video region, three panels: **[1] source**
+  (real site) · **[2] clone before the fix** (blank) · **[3] clone after the fix**. Panel 2 ≠ 1
+  (broken); panel 3 ≈ 1 (fixed). The video is cross-origin/canvas-tainted, so it produced **0
+  stills** before this change.
+- `video-clay-source-vs-clone.png` — **clay.com** video region, **source left / clone right**:
+  the captured still matches the source frame (the only diff is an unrelated cookie banner).
+- `video-descript-captured-still.jpg` — the raw frame the fallback captured for descript.
+
+**Consistency** — re-validated across four hero-video benchmark sites; the fallback fires only
+where there's no real poster, and never regresses gates 0–6:
+
+| site | videos | stills materialized | score | gates 0–6 |
+|------|-------:|--------------------:|------:|:---------:|
+| descript.com | 1 | 1 (canvas-tainted → element shot) | 99.9 | ✅ |
+| clay.com | 2 | 2 | 99.6 | ✅ |
+| squarespace.com | 9 | 9 | 99.1 | ✅ (perceptual-only miss — its documented dynamic-hero near-miss, now with stills not blank) |
+| webflow.com | 7 | 0 (all had real posters) | 99.7 | ✅ |
+
+## 2. `<iframe>` placeholder (no longer dropped)
+
+Iframes were dropped as noise, collapsing the layout the embed occupied. They're now kept as a
+**sized placeholder box** (captured geometry, document-loading attrs stripped → self-contained,
+no remote ref). Invisible tracking/chat iframes are still pruned.
+
+- `iframe-loom-source-vs-clone.png` — **loom.com** (source **left**, clone **right**): the
+  "This is Loom" video iframe (previously dropped) is preserved as a placeholder holding the
+  player's box. DOM match 100%, page-height delta 0.85%, gates 0–6 green (98.4).
+- `fixture-iframe-and-headings.png` — served fixture (source **left**, clone **right**): a
+  YouTube embed + a map embed both render as clean placeholders; no remote URL leaks into the
+  clone.
+
+## 3. Route cap for very large directories
+
+A directory listing of a huge collection (11ty's **882-instance `/authors` index**) is one page
+too heavy to capture/grade in budget. New opt-in `maxCollectionInstances` (CLI `--max-collection`)
+leaves an oversized collection's *listing* as a CMS-handoff (links to it absolutize to origin)
+while still reproducing the representative detail page.
+
+    npx tsx src/site/cloneSite.ts https://www.11ty.dev/ --max-routes=12 --max-collection=500
+
+now **completes** (previously intractable): **11/12 routes pass gates 0–6**, link-integrity +
+site-determinism green; `/authors` is cleanly absent and the lone miss is `/speedlify` (a JS
+dashboard — dynamic, out of the static contract, as before).
+
+## 4. Inline-block multi-line heading — root-caused
+
+The third image (`fixture-iframe-and-headings.png`) also carries three heading variants — plain,
+per-word `inline-block`, and words + zero-width spacers. At system fonts all three reproduce to
+**≤0.01px**, so the inline-formatting reconstruction is already exact — *not* the "deeper
+inline-layout model" gap it was thought to be. The residual few-px drift on real sites is
+**font-metric** wrapping that accumulates on very dense pages (linear and deel pass gates 0–6;
+notion's `/product` misses gate5 by an +8% page-height accumulation — the documented dense-page
+limit, unchanged by this work). Left as a known, now correctly-attributed limit.

examples/sites/RESULTS.md

@@ -0,0 +1,54 @@
+# Stage 3 — multi-page / whole-site results
+
+`clone-site <url>` crawls a site from one entry URL, selects a bounded, deterministic
+route set (singletons reproduce, pairs reproduce, larger CMS-like collections collapse
+to a listing plus one representative),
+captures each route, and generates **one** Next.js App Router app with shared chrome,
+internal links rewritten to the clone routes, and content-addressed shared assets.
+`bench-site` builds + grades it: per-route Gates 0–6 + stage-2 pollution/perceptual,
+plus site-level **link-integrity** and **site-determinism** gates.
+
+The composites in this folder show SOURCE vs CLONE for the first few routes of each
+site at the 1280px viewport.
+
+## Results
+
+| Site | Discovered → reproduced | Routes pass G0–6 | Collections collapsed | Shared chrome | Links | Determinism |
+| --- | --- | --- | --- | --- | --- | --- |
+| overreacted.io | 59 → **2** | **2 / 2** | 1 (58 posts → 1) | header | ✅ | ✅ |
+| brew.sh | 88 → **12** | **12 / 12** | 6 (27 → 6) | footer | ✅ | ✅ |
+| jamstack.org | 552 → **11** | **11 / 11** | 3 (**534** → 3) | — | ✅ | ✅ |
+| gatsbyjs.com | 281 → **12** | **11 / 12** | 2 (62 → 2) | — | ✅ | ✅ |
+| 11ty.dev | 1085 → **12** | 11 / 12 | 4 (**1076** → 4) | — | ✅ | ✅ |
+
+**Totals: 47 / 49 routes pass Gates 0–6; link-integrity and site-determinism green on
+every site.** The reproduction policy is the headline: jamstack's 552 pages and 11ty's
+1085 (incl. an 882-instance `/authors/:id` directory) each reduce to ~11–12 reproduced
+routes — the full instance map is recorded in each run's `site-manifest.json` as the
+CMS-handoff boundary.
+
+## Notes
+
+- **overreacted.io** — a flat root-level blog: 58 `/:slug` posts collapse to the home
+  (listing) + one representative; the shared `<header>` is hoisted into `layout.tsx`
+  (emitted once). Both routes clone at 99.9.
+- **brew.sh** — home + per-year blog archives; the shared footer (`<aside>` + footer)
+  is hoisted; 12/12 at ~99.6.
+- **jamstack.org** — a directory site: `/generators` (377), `/headless-cms` (134),
+  `/glossary` (23) each collapse to listing + representative. 11/11 at 99.9.
+- **gatsbyjs.com** (11/12) and **11ty.dev** (11/12) each miss exactly one route, and
+  both are out-of-reach classes for a static deterministic clone (not multi-page
+  defects): gatsby's `/blog/gatsby-5-typescript` (residual inline-formatting /
+  margin-collapse drift on a content-heavy post — the documented linear/notion/
+  paulgraham class) and 11ty's `/speedlify` (a JS performance dashboard — dynamic
+  content, outside the static-clone contract). The crawl, collection collapse, link
+  rewriting, shared layout, and determinism are exact — a route scores what the same
+  page scores cloned standalone.
+- Two general fixes surfaced here lifted gatsby 4/12 → 11/12 with no single-page
+  regression (easy re-ran 4/4 @ 100): (1) block-level `grid`/`flex` containers are now
+  margin-auto re-centered like plain blocks (a centered `<main class=grid>` was
+  left-aligning its column ~96px off), and (2) the DOM gate credits the compiler's own
+  valid-HTML retags (button/ul → div) as matched instead of docking them.
+
+Reproduce: `cd compiler && npm run bench-site` (add `--regen` to re-grade existing
+captures). Composites: `python3 scripts/make_site_compare.py`.