diff --git a/CHANGELOG.md b/CHANGELOG.md index b9b3714..6dde4f5 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -6,6 +6,38 @@ so minor/patch semantics are not yet guaranteed. ## [Unreleased] +### Changed — Local CLI success output and orientation + +- The `clone-static` CLI now prints a copy-paste-safe success summary to stderr: a + single **quoted** `cd "" && npm install && npm run dev` line (survives terminal + wrapping — no more broken `cd` from a wrapped timestamped path) plus the key + `AGENTS.md` safe-to-edit pointers (`src/app/content.ts`, `src/app/components/`). The + machine-readable `{ "event": "done", ... }` JSON line on stdout is unchanged (now + also carrying `stableApp`). +- Added `--serve` (run `npm install` + `npm run dev` in the generated app) and + `--open` (also launch the browser at the dev URL) so "see it locally" is one flag. +- In the default runs layout, each clone now refreshes a `runs//latest` symlink + pointing at the newest run, giving a stable, timestamp-free path for `cd` and scripts. + +### Added — Docs hub and terminology/secret-hygiene clarifications + +- Added [`docs/README.md`](docs/README.md) as a central documentation index. +- Clarified prominently that ditto.site "cloning" means generating a codebase from a + live URL, **not** `git clone` (no source repo required), in the README and docs hub. +- Added an explicit "API keys are secrets — use `$DITTO_API_KEY`, never inline/commit, + rotate anytime" note beside the key/auth examples in the README and `docs/SERVICE.md`. + +### Added — `ditto` unpack CLI + +- **`packages/cli`** — a zero-dependency `ditto` command-line helper. `npm run + unpack -- ` turns the `files` map returned by + `POST /v1/clones` (or `GET /v1/clones/:id/result`) into a real project tree on + disk: text files written inline, binary assets materialized from inline base64 + or fetched from their reference URL (`$DITTO_API_URL` / `$DITTO_API_KEY`), with + path-traversal guards and pre-write `sha256` integrity checks. Reads from stdin + so a `curl` response can be piped straight in. Documented next to the REST + examples in the README and `docs/SERVICE.md`. + ### Added — Open-source readiness - Added support, release, responsible-use, CODEOWNERS, and gitattributes files. diff --git a/README.md b/README.md index 476cb5a..6335c66 100644 --- a/README.md +++ b/README.md @@ -15,8 +15,14 @@ Router project by default, or Vite React when requested. The compiler is not an LLM page author. It is a capture-to-code pipeline: same frozen capture in, byte-stable app out. +> **"Cloning" here means generating a codebase from a live URL — not `git clone`.** +> You don't need an existing repository, and you don't need the site's source. Point +> ditto.site at a public URL and it writes you a fresh project from what the page +> renders in a browser. + Read the public development and evaluation method in -[docs/METHODOLOGY.md](docs/METHODOLOGY.md). +[docs/METHODOLOGY.md](docs/METHODOLOGY.md). For a map of all the docs, see +[docs/README.md](docs/README.md). ## Usage @@ -35,6 +41,11 @@ curl -sS -X POST "https://api.ditto.site/v1/signup/request" \ The emailed verification link lands on `/api-key?token=...`, which calls `POST /v1/signup/verify` and displays the new `dtto_live_...` key once. +> **Keys are secrets.** Put your key in an environment variable (`export +> DITTO_API_KEY=dtto_live_...`) and reference `$DITTO_API_KEY` in every command — +> never paste the raw key inline, where it lands in shell history, logs, or a chat. +> Don't commit it. You can rotate a key anytime from the dashboard. + ### REST API Start a clone job: @@ -56,20 +67,52 @@ curl -sS -X POST "$DITTO_API_URL/v1/clones" \ }' ``` -The service returns either an inline result or a queued job: +The service returns either a queued job or an inline result. A finished result +is a file map — every generated file keyed by its app-relative path: ```json -{ "jobId": "job_123", "status": "queued" } +{ + "jobId": "job_123", + "status": "succeeded", + "files": { + "package.json": { "type": "text", "content": "{ ... }", "bytes": 812, "sha256": "..." }, + "src/app/page.tsx": { "type": "text", "content": "export default ...", "bytes": 2048, "sha256": "..." }, + "public/assets/logo.png": { "type": "binary", "url": ".../files/public/assets/logo.png", "bytes": 5123, "sha256": "..." } + } +} ``` -Poll and download the generated app: +**Turn that JSON into a project on disk** with the official unpacker — from a +checked-out `ditto.site` repo with dependencies installed, pipe the response +straight in with no temp file: + +```bash +curl -sS -X POST "$DITTO_API_URL/v1/clones" \ + -H "authorization: Bearer $DITTO_API_KEY" \ + -H "content-type: application/json" \ + -d '{"url":"https://example.com/","options":{"mode":"single"}}' \ + | npm run --silent unpack -- - ./out +``` + +`npm run unpack -- ` writes the text files inline and +materializes binary assets (inline base64, or fetched from their `url` using +`$DITTO_API_URL` / `$DITTO_API_KEY`). The CLI package is repo-local until the +npm distribution story is ready, so do not use `npx ditto` yet. See +[`packages/cli`](packages/cli/README.md) for options. + +If you got back a queued job (`{ "jobId": "job_123", "status": "queued" }`), +poll it, then unpack the finished result — or download the whole app as one +archive: ```bash JOB_ID="job_123" +# poll status, then unpack the finished file map curl -sS -H "authorization: Bearer $DITTO_API_KEY" \ - "$DITTO_API_URL/v1/clones/$JOB_ID" + "$DITTO_API_URL/v1/clones/$JOB_ID/result" \ + | npm run --silent unpack -- - ./out +# ...or grab the whole app as a single archive curl -L -H "authorization: Bearer $DITTO_API_KEY" \ "$DITTO_API_URL/v1/clones/$JOB_ID/bundle?format=tgz" \ -o ditto-clone.tgz @@ -139,6 +182,8 @@ src/app/page.tsx, and src/app/ditto.css. ### Local CLI ```bash +# this git clone gets the ditto.site tool itself — the URL you clone into a +# codebase comes later, as the argument to `npm run clone`. git clone https://github.com/ion-design/ditto.site.git cd ditto.site @@ -148,7 +193,17 @@ npx playwright install chromium npm run clone -- https://example.com/ --out=./output ``` -The generated app lands under `output//app`. +The generated app lands under `output//app`. On success the CLI prints a +copy-paste-safe summary — a single quoted `cd … && npm install && npm run dev` +line and pointers to the safe-to-edit files (`src/app/content.ts`, +`src/app/components/`; the app's `AGENTS.md` has the full guide). + +To skip the copy-paste entirely and go straight to a running preview: + +```bash +npm run clone -- https://example.com/ --serve # clone, npm install, npm run dev +npm run clone -- https://example.com/ --open # ...and open the browser too +``` Common local variants: @@ -159,6 +214,10 @@ npm run clone -- https://example.com/ --framework=vite npm run validate-site -- runs/site-example.com/ ``` +Without `--out`, runs land under `runs///` and a stable +`runs//latest` symlink always points at the newest clone, so scripts and +`cd` targets don't depend on the timestamp. + ### Local REST And MCP Service Quick inline mode, with no database: @@ -241,6 +300,7 @@ minting is intentional. | --- | --- | | `compiler/` | deterministic capture, inference, generation, and validation | | `packages/core/` | compiler adapter and file-map helpers | +| `packages/cli/` | `ditto` CLI — unpack a clone result JSON into a project tree | | `packages/api/` | Hono REST API and MCP server | | `packages/db/` | Drizzle schema, migrations, repository, and queue wrapper | | `packages/storage/` | local and S3/R2 artifact storage | diff --git a/compiler/README.md b/compiler/README.md index 30e702d..f37fb3c 100644 --- a/compiler/README.md +++ b/compiler/README.md @@ -16,6 +16,8 @@ npm install npx playwright install chromium npm run clone -- https://example.com/ +npm run clone -- https://example.com/ --serve # then npm install + npm run dev +npm run clone -- https://example.com/ --open # ...and open the browser too npm run clone -- https://example.com/ --mode=multi --styling=tailwind npm run clone -- https://example.com/ --mode=single --framework=vite npm run clone-site -- https://example.com/ @@ -34,6 +36,13 @@ npm run typecheck Root-level scripts forward to these commands, so `npm run clone -- ` works from the repository root too. +("Clone" here = generating a codebase from a live URL, not `git clone`; no source +repo required.) On success the CLI prints a copy-paste-safe summary: a single +quoted `cd … && npm install && npm run dev` line plus safe-to-edit pointers. Pass +`--serve` to run install + dev automatically, or `--open` to also launch the +browser. Without `--out`, a `runs//latest` symlink always points at the +newest run so paths aren't timestamp-fragile. + Multi-page generation defaults to the fast no-validation path. Use `--validate` when the clone command itself should run the full build/render/gates QA pass, or run `validate-site` separately. `--concurrency` controls source route capture; diff --git a/compiler/src/cli.ts b/compiler/src/cli.ts index 198fa6f..ba1a5a5 100755 --- a/compiler/src/cli.ts +++ b/compiler/src/cli.ts @@ -1,11 +1,12 @@ #!/usr/bin/env -S npx tsx import { basename, dirname, join, resolve, sep } from "node:path"; -import { cpSync, existsSync, readdirSync, readFileSync, rmSync, statSync, writeFileSync } from "node:fs"; +import { cpSync, existsSync, readdirSync, readFileSync, rmSync, statSync, symlinkSync, writeFileSync } from "node:fs"; import { captureSite, REQUIRED_VIEWPORTS, SAMPLE_VIEWPORTS, type CaptureResult } from "./capture/capture.js"; import { fileURLToPath } from "node:url"; import { generateAll } from "./generate/pipeline.js"; import { refineSizing } from "./generate/refineSizing.js"; import { writeJSON, writeText, ensureDir, readJSON, fileExists } from "./util/fsx.js"; +import { doneSummary, serveApp } from "./cliSummary.js"; import type { AppFramework } from "./generate/app.js"; export type CloneOptions = { @@ -34,6 +35,9 @@ export type CloneResult = { sourceDir: string; appDir: string; sourceUrl: string; + /** A stable, timestamp-free path to the app (via the `runs//latest` symlink), + * present in runs-layout mode when the symlink could be created. */ + stableAppDir?: string; }; export function siteIdFromUrl(url: string): string { @@ -462,16 +466,34 @@ export async function runClone(opts: CloneOptions): Promise { writeText(join(runDir, "logs", "compiler.log.jsonl"), logEvents.map((e) => JSON.stringify(e)).join("\n") + "\n"); + let stableAppDir: string | undefined; if (out) { // Publish the deliverable to /app; keep working artifacts in .clone. exportApp(appDir, out.appDir); logBoth({ event: "exported", app: out.appDir }); } else { - // latest pointer (runs/ layout, used by --reuse / regen) - writeJSON(join(runsDir, siteId, "latest.json"), { runDir, ts: timestamp() }); + // latest pointer (runs/ layout, used by --reuse / regen) + a `latest` symlink so the + // app has a stable, timestamp-free path that survives copy-paste. + stableAppDir = writeLatestPointer(runsDir, siteId, runDir); } - return { runDir, sourceDir, appDir: out ? out.appDir : appDir, sourceUrl: opts.url }; + return { runDir, sourceDir, appDir: out ? out.appDir : appDir, sourceUrl: opts.url, stableAppDir }; +} + +/** Record the newest run for a site in the runs layout: a `latest.json` breadcrumb (used by + * `--reuse`/regen) and a `latest` symlink → runDir. Returns the stable app path when the + * symlink was created (symlinks can be unavailable, e.g. Windows without privilege — non-fatal). */ +export function writeLatestPointer(runsDir: string, siteId: string, runDir: string): string | undefined { + writeJSON(join(runsDir, siteId, "latest.json"), { runDir, ts: timestamp() }); + const link = join(runsDir, siteId, "latest"); + try { + // Refresh an existing symlink; refuse (via non-recursive rm) to clobber a real directory. + rmSync(link, { force: true }); + symlinkSync(runDir, link, "junction"); + return join(link, "generated", "app"); + } catch { + return undefined; + } } function copySourceRef(from: string, to: string): void { @@ -543,12 +565,22 @@ async function main(): Promise { const args = process.argv.slice(2); const url = args.find((a) => !a.startsWith("--")); if (!url) { - console.error("usage: clone-static [--mode=single|multi] [--styling=tailwind|css] [--framework=next|vite] [--out=]"); + console.error("usage: clone-static [--mode=single|multi] [--styling=tailwind|css] [--framework=next|vite] [--out=] [--serve] [--open]"); process.exit(1); } const mode = parseProductMode(args); const styling = parseProductStyling(args); const framework = parseProductFramework(args); + // --serve installs deps + starts the dev server after cloning; --open also launches the browser. + const open = hasAnyFlag(args, ["--open"]); + const serve = open || hasAnyFlag(args, ["--serve"]); + const finish = async (res: { appDir: string; stableAppDir?: string }) => { + if (serve) { + await serveApp(res.appDir, { open }); + } else { + process.stderr.write(doneSummary({ url, appDir: res.appDir, framework, stableAppDir: res.stableAppDir })); + } + }; const vpArg = firstFlagValue(args, ["--dev-viewports", "--viewports"]); const runsArg = firstFlagValue(args, ["--dev-runs", "--runs"]); // --out=: clean //{app,.clone} layout (default: ./output when bare --out). @@ -604,7 +636,8 @@ async function main(): Promise { tier, log: (e) => console.log(JSON.stringify(e)), }); - console.log(JSON.stringify({ event: "done", runDir: res.runDir, app: res.appDir })); + console.log(JSON.stringify({ event: "done", runDir: res.runDir, app: res.appDir, stableApp: res.stableAppDir })); + await finish(res); return; } @@ -625,7 +658,8 @@ async function main(): Promise { reuseSource, log: (e) => console.log(JSON.stringify(e)), }); - console.log(JSON.stringify({ event: "done", runDir: res.runDir, app: res.appDir })); + console.log(JSON.stringify({ event: "done", runDir: res.runDir, app: res.appDir, stableApp: res.stableAppDir })); + await finish(res); } if (import.meta.url === `file://${process.argv[1]}`) { diff --git a/compiler/src/cliSummary.ts b/compiler/src/cliSummary.ts new file mode 100644 index 0000000..fa56c3e --- /dev/null +++ b/compiler/src/cliSummary.ts @@ -0,0 +1,112 @@ +/** + * Human-facing success output for the `clone-static` CLI. + * + * The CLI streams machine-readable JSON events to stdout; this module renders the + * copy-paste-friendly summary a person actually reads at the end, plus the optional + * `--serve` runner that installs deps and starts the dev server for them. + * + * Why this exists: a bare `runs///generated/app` path pasted into a + * terminal wraps mid-word, so `cd` fails and the follow-up `npm run dev` runs in the + * wrong directory. The preview command here is a single quoted line that survives + * wrapping. + */ +import { spawn } from "node:child_process"; +import { platform } from "node:os"; + +export type DoneSummaryInput = { + url: string; + /** The exact generated app directory for this run. */ + appDir: string; + framework: "next" | "vite"; + /** Stable path (a `runs//latest` symlink target) preferred as the `cd` target when present. */ + stableAppDir?: string; +}; + +/** Double-quote a path for a POSIX shell so spaces and wrapping don't break copy-paste. */ +export function shellQuote(p: string): string { + return `"${p.replace(/(["\\$`])/g, "\\$1")}"`; +} + +/** The success block. The preview command is one quoted line on purpose. */ +export function doneSummary(input: DoneSummaryInput): string { + const root = input.framework === "next" ? "src/app" : "src"; + const cdTarget = input.stableAppDir ?? input.appDir; + const lines: string[] = [ + "", + `✓ Done — cloned ${input.url}`, + "", + " Preview it locally (copy-paste this one line):", + "", + ` cd ${shellQuote(cdTarget)} && npm install && npm run dev`, + "", + " What's safe to edit — full guide in AGENTS.md inside the app:", + ` • page copy & content → ${root}/content.ts`, + ` • components → ${root}/components/`, + "", + " Or re-run with --serve to install deps and start the dev server for you", + " (add --open to launch the browser too).", + ]; + if (input.stableAppDir && input.stableAppDir !== input.appDir) { + lines.push(""); + lines.push(` The path above is a stable pointer to the newest clone. This exact run:`); + lines.push(` ${input.appDir}`); + } + return lines.join("\n") + "\n"; +} + +function openBrowser(url: string): void { + const cmd = platform() === "darwin" ? "open" : platform() === "win32" ? "cmd" : "xdg-open"; + const args = platform() === "win32" ? ["/c", "start", "", url] : [url]; + try { + const child = spawn(cmd, args, { stdio: "ignore", detached: true }); + child.on("error", () => {}); // opening a browser is best-effort + child.unref(); + } catch { + /* best-effort */ + } +} + +function runToCompletion(cmd: string, args: string[], cwd: string): Promise { + return new Promise((resolvePromise, reject) => { + const child = spawn(cmd, args, { cwd, stdio: "inherit", shell: platform() === "win32" }); + child.on("error", reject); + child.on("exit", (code) => + code === 0 ? resolvePromise() : reject(new Error(`\`${cmd} ${args.join(" ")}\` exited with code ${code}`)), + ); + }); +} + +/** + * Run `npm install` then `npm run dev` in the generated app (foreground — `npm run dev` + * keeps running until the user stops it). With `open`, launches the default browser at + * the dev URL once the server prints it. + */ +export async function serveApp(appDir: string, opts: { open: boolean }): Promise { + process.stderr.write(`\n→ Installing dependencies in ${appDir}\n`); + await runToCompletion("npm", ["install"], appDir); + + process.stderr.write(`→ Starting the dev server (Ctrl-C to stop)\n\n`); + await new Promise((resolvePromise, reject) => { + const dev = spawn("npm", ["run", "dev"], { + cwd: appDir, + stdio: opts.open ? ["inherit", "pipe", "inherit"] : "inherit", + shell: platform() === "win32", + }); + if (opts.open && dev.stdout) { + let opened = false; + dev.stdout.on("data", (chunk: Buffer) => { + process.stdout.write(chunk); + if (opened) return; + const m = String(chunk).match(/https?:\/\/(?:localhost|127\.0\.0\.1)(?::\d+)?\S*/i); + if (m) { + opened = true; + openBrowser(m[0]); + } + }); + } + dev.on("error", reject); + dev.on("exit", (code) => + code === 0 || code === null ? resolvePromise() : reject(new Error(`\`npm run dev\` exited with code ${code}`)), + ); + }); +} diff --git a/compiler/src/site/cloneSite.ts b/compiler/src/site/cloneSite.ts index 2803da2..1bf2b3a 100644 --- a/compiler/src/site/cloneSite.ts +++ b/compiler/src/site/cloneSite.ts @@ -21,7 +21,7 @@ import { structurallySimilar } from "./signature.js"; import { generateSiteApp, routeToSegment, routeKey, type RouteArtifact } from "./generateSite.js"; import { detectSharedChrome, chromeSignatureId } from "./sharedLayout.js"; import { validateSite, type SiteReport } from "./validateSite.js"; -import { siteIdFromUrl, namedOutDirs, exportApp } from "../cli.js"; +import { siteIdFromUrl, namedOutDirs, exportApp, writeLatestPointer } from "../cli.js"; import { writeJSON, readJSON, ensureDir, fileExists, writeText } from "../util/fsx.js"; import { seoInventoryToMarkdown } from "../generate/seo.js"; import type { AppFramework } from "../generate/app.js"; @@ -56,6 +56,8 @@ export type CloneSiteResult = { plan: RoutePlan; routes: RouteArtifact[]; siteReport?: SiteReport; + /** Stable, timestamp-free path to the app (via the `runs//latest` symlink), when created. */ + stableAppDir?: string; }; function timestamp(): string { @@ -236,8 +238,10 @@ export async function runCloneSite(opts: CloneSiteOptions): Promise { + it("wraps a path in double quotes so spaces survive copy-paste", () => { + assert.equal(shellQuote("/Users/x/runs/site/20260701/generated/app"), '"/Users/x/runs/site/20260701/generated/app"'); + assert.equal(shellQuote("/Users/x/My Sites/app"), '"/Users/x/My Sites/app"'); + }); + + it("escapes characters the shell would otherwise interpret inside double quotes", () => { + assert.equal(shellQuote('/a/"b"/$c/`d`/e\\f'), '"/a/\\"b\\"/\\$c/\\`d\\`/e\\\\f"'); + }); +}); + +describe("doneSummary", () => { + const base = { url: "https://academyux.com/", appDir: "/runs/academyux.com/20260701-173012/generated/app", framework: "next" as const }; + + it("emits a single quoted preview line (no mid-word wrap on the path)", () => { + const out = doneSummary(base); + const line = out.split("\n").find((l) => l.includes("cd ")); + assert.ok(line, "has a cd line"); + // one line, path is quoted so a wrapping terminal can't split the command + assert.match(line!, /cd "\/runs\/academyux\.com\/20260701-173012\/generated\/app" && npm install && npm run dev/); + }); + + it("points at the Next safe-edit areas from AGENTS.md", () => { + const out = doneSummary(base); + assert.match(out, /src\/app\/content\.ts/); + assert.match(out, /src\/app\/components\//); + assert.match(out, /AGENTS\.md/); + }); + + it("uses the Vite src root when framework is vite", () => { + const out = doneSummary({ ...base, framework: "vite" }); + assert.match(out, /cd .* && npm install && npm run dev/); + assert.match(out, /• page copy & content {2}→ {2}src\/content\.ts/); + assert.match(out, /• components {11}→ {2}src\/components\//); + }); + + it("prefers the stable path as the cd target and still shows the exact run", () => { + const out = doneSummary({ ...base, stableAppDir: "/runs/academyux.com/latest/generated/app" }); + const cdLine = out.split("\n").find((l) => l.includes("cd "))!; + assert.match(cdLine, /cd "\/runs\/academyux\.com\/latest\/generated\/app"/); + assert.match(out, /This exact run/); + assert.match(out, /\/runs\/academyux\.com\/20260701-173012\/generated\/app/); + }); + + it("mentions the --serve / --open shortcut", () => { + assert.match(doneSummary(base), /--serve/); + assert.match(doneSummary(base), /--open/); + }); +}); diff --git a/compiler/test/runsLayout.test.ts b/compiler/test/runsLayout.test.ts new file mode 100644 index 0000000..79b1b57 --- /dev/null +++ b/compiler/test/runsLayout.test.ts @@ -0,0 +1,31 @@ +import { describe, it } from "node:test"; +import assert from "node:assert/strict"; +import { mkdtempSync, mkdirSync, existsSync, readlinkSync, realpathSync } from "node:fs"; +import { tmpdir } from "node:os"; +import { join } from "node:path"; +import { writeLatestPointer } from "../src/cli.js"; + +describe("writeLatestPointer", () => { + it("writes latest.json and a `latest` symlink to the newest run, refreshing it in place", () => { + const runs = mkdtempSync(join(tmpdir(), "ditto-runs-")); + const siteId = "example.com"; + + const run1 = join(runs, siteId, "20260701-100000"); + mkdirSync(join(run1, "generated", "app"), { recursive: true }); + const stable1 = writeLatestPointer(runs, siteId, run1); + + assert.ok(existsSync(join(runs, siteId, "latest.json")), "writes latest.json breadcrumb"); + assert.ok(stable1 && existsSync(stable1), "returned stable app path resolves through the symlink"); + assert.equal(realpathSync(readlinkSync(join(runs, siteId, "latest"))), realpathSync(run1)); + + // A second run must re-point the existing symlink, not throw on the collision. + const run2 = join(runs, siteId, "20260701-200000"); + mkdirSync(join(run2, "generated", "app"), { recursive: true }); + const stable2 = writeLatestPointer(runs, siteId, run2); + + assert.equal(realpathSync(readlinkSync(join(runs, siteId, "latest"))), realpathSync(run2)); + assert.ok(stable2 && existsSync(stable2)); + // The stable path is timestamp-free (goes through `latest`, not the run's timestamp). + assert.match(stable2!, /example\.com\/latest\/generated\/app$/); + }); +}); diff --git a/docs/README.md b/docs/README.md new file mode 100644 index 0000000..d1db372 --- /dev/null +++ b/docs/README.md @@ -0,0 +1,57 @@ +# ditto.site Documentation + +The central index for ditto.site's docs. Start here. + +> **First, the word "clone."** In ditto.site, *cloning* means **generating a +> codebase from a live URL** — it is **not** `git clone`. You do **not** need an +> existing Git repository, and you do **not** need the target site's source code. +> You give ditto.site a public URL; it captures what the page renders in a browser +> and writes you a fresh, runnable project. (The only `git clone` involved is +> optionally cloning *this tool's* repository to run the compiler locally.) + +## Get started + +| I want to… | Go to | +| --- | --- | +| Understand what ditto.site is and see it run | [Project README](../README.md) | +| Call the hosted **REST API** or **MCP** server | [Project README → Usage](../README.md#usage), [SERVICE.md](SERVICE.md) | +| Turn a clone result JSON into files on disk | [Repo-local unpack CLI](../packages/cli/README.md) | +| Run the **compiler locally** from the command line | [compiler/README.md](../compiler/README.md) | +| Deploy the service | [DEPLOY.md](DEPLOY.md) | +| Read the development & evaluation method | [METHODOLOGY.md](METHODOLOGY.md) | +| Understand responsible-use boundaries | [RESPONSIBLE_USE.md](RESPONSIBLE_USE.md) | +| Cut a release | [RELEASING.md](RELEASING.md) | + +## API keys are secrets + +Keys look like `dtto_live_...`. Keep them in an environment variable and reference +it in commands: + +```bash +export DITTO_API_KEY="dtto_live_..." +curl -sS -H "authorization: Bearer $DITTO_API_KEY" "$DITTO_API_URL/v1/clones" +``` + +Never paste a raw key inline (it leaks into shell history, logs, and chat), and +never commit one. Rotate a leaked key anytime from the dashboard. See +[SERVICE.md](SERVICE.md) for the full auth model. + +## The short version of the workflow + +1. **Clone** a URL → `POST /v1/clones` (API) or `npm run clone -- ` (local CLI). +2. **Get the app** → unpack the result JSON with the [repo-local unpack CLI](../packages/cli/README.md), + download the `bundle?format=tgz` archive, or read files from `runs//latest/`. +3. **Preview it** → `cd` into the app and `npm install && npm run dev` (or let the + local CLI do it for you with `--serve` / `--open`). +4. **Edit safely** → each generated app ships an `AGENTS.md` describing what's safe + to change (copy in `src/app/content.ts`, components in `src/app/components/`, etc.). + +## Full doc list + +- [SERVICE.md](SERVICE.md) — REST + MCP service reference (endpoints, options, env vars). +- [DEPLOY.md](DEPLOY.md) — production deployment. +- [METHODOLOGY.md](METHODOLOGY.md) — how the compiler is developed and evaluated. +- [RESPONSIBLE_USE.md](RESPONSIBLE_USE.md) — acceptable-use boundaries. +- [RELEASING.md](RELEASING.md) — release process. +- [../compiler/README.md](../compiler/README.md) — local compiler commands. +- [../packages/cli/README.md](../packages/cli/README.md) — the repo-local unpack CLI. diff --git a/docs/SERVICE.md b/docs/SERVICE.md index 504d429..f4d2cf6 100644 --- a/docs/SERVICE.md +++ b/docs/SERVICE.md @@ -60,6 +60,21 @@ curl -s -X POST localhost:8787/v1/clones -H 'content-type: application/json' \ -d '{"url":"https://example.com/","options":{"mode":"single","styling":"tailwind"}}' | jq '.files | keys' ``` +To turn that `files` map into a project on disk, pipe the response into the +repo-local `ditto` CLI (`packages/cli`) instead of inspecting the JSON by hand: + +```bash +curl -s -X POST localhost:8787/v1/clones -H 'content-type: application/json' \ + -d '{"url":"https://example.com/","options":{"mode":"single"}}' \ + | npm run --silent unpack -- - ./out +# binary assets: set DITTO_API_URL (and DITTO_API_KEY when authenticated) so the +# CLI can fetch each file's reference URL; --no-fetch writes only the text tree. +``` + +The CLI package is intentionally private for now; run this command from a +checked-out `ditto.site` repo with dependencies installed. Do not use +`npx ditto` until this package is published. + ## REST surface ``` @@ -78,6 +93,11 @@ GET /healthz → { ok: true } (unauthenticated) `/v1/clones*` and `/mcp` are authenticated when `API_KEYS` is set or DB-backed keys exist. Use `Authorization: Bearer ` or `x-api-key: `. + +> **Treat keys as secrets.** In any snippet you copy or share, template the key as +> an environment variable (`Authorization: Bearer $DITTO_API_KEY`) rather than an +> inline `dtto_live_...` token — inline keys leak into shell history, logs, and +> pasted transcripts. Never commit a key; rotate a leaked one from the dashboard. Signup routes are intentionally public only when `SIGNUP_ENABLED=true` **and** `DATABASE_URL` is set. Direct `POST /v1/signup` mints a `dtto_live_...` key immediately when `SIGNUP_DIRECT_ENABLED=true`. For public production signup, diff --git a/package-lock.json b/package-lock.json index 822ff09..741b5da 100644 --- a/package-lock.json +++ b/package-lock.json @@ -1878,6 +1878,10 @@ "node": ">= 0.8" } }, + "node_modules/ditto": { + "resolved": "packages/cli", + "link": true + }, "node_modules/drizzle-kit": { "version": "0.31.10", "resolved": "https://registry.npmjs.org/drizzle-kit/-/drizzle-kit-0.31.10.tgz", @@ -3802,6 +3806,17 @@ "typescript": "5.7.3" } }, + "packages/cli": { + "name": "ditto", + "version": "0.1.0", + "license": "MIT", + "bin": { + "ditto": "bin/ditto.mjs" + }, + "engines": { + "node": ">=20" + } + }, "packages/core": { "name": "@cloner/core", "version": "0.1.0", diff --git a/package.json b/package.json index 6be2c4b..b213d9a 100644 --- a/package.json +++ b/package.json @@ -41,7 +41,8 @@ "dev:api": "npm run dev --workspace @cloner/api", "dev:worker": "npm run dev --workspace @cloner/worker", "db:generate": "npm run generate --workspace @cloner/db", - "db:migrate": "npm run migrate --workspace @cloner/db" + "db:migrate": "npm run migrate --workspace @cloner/db", + "unpack": "node packages/cli/bin/ditto.mjs unpack" }, "devDependencies": { "tsx": "4.22.4", diff --git a/packages/cli/README.md b/packages/cli/README.md new file mode 100644 index 0000000..ee8e391 --- /dev/null +++ b/packages/cli/README.md @@ -0,0 +1,50 @@ +# ditto (CLI) + +The official ditto.site command-line helper. Turns a clone **result JSON** — +the giant blob you get back from `POST /v1/clones` or +`GET /v1/clones/:id/result` — into an actual project tree on disk. + +Zero dependencies. Needs Node >= 20. + +This workspace is currently private and repo-local. Run these commands from the +repository root after `npm install`; do not use `npx ditto` until the package is +published. + +## Unpack + +```bash +# from a saved file +npm run unpack -- clone.json ./out + +# straight from curl, no temp file +curl -sS -X POST "$DITTO_API_URL/v1/clones" \ + -H "authorization: Bearer $DITTO_API_KEY" \ + -H "content-type: application/json" \ + -d '{"url":"https://example.com/","options":{"mode":"single"}}' \ + | npm run --silent unpack -- - ./out +``` + +`npm run unpack -- `: + +- writes every text file from its inline `content`, +- materializes binary assets from inline base64 when present, otherwise fetches + them from their reference `url`, +- refuses paths that escape `` and verifies `sha256` when the result + carries it. + +### Binary assets + +Clone results return binaries by reference (`{ "type": "binary", "url": ... }`) +rather than inlining megabytes of base64. To fetch them, the unpacker needs to +know where the API lives: + +| Source | Flag | Env | +| --- | --- | --- | +| Base URL for relative asset URLs | `--base-url ` | `DITTO_API_URL` | +| Bearer key for authenticated APIs | `--api-key ` | `DITTO_API_KEY` | + +Use `--no-fetch` to write only the text tree and list the binaries as skipped. +To grab everything in one shot instead, download the archive directly: +`GET /v1/clones//bundle?format=tgz`. + +Run `npm run unpack -- --help` for the full option list. diff --git a/packages/cli/bin/ditto.mjs b/packages/cli/bin/ditto.mjs new file mode 100755 index 0000000..b91771c --- /dev/null +++ b/packages/cli/bin/ditto.mjs @@ -0,0 +1,253 @@ +#!/usr/bin/env node +// ditto — official command-line helper for ditto.site. +// +// `ditto unpack ` turns the JSON returned by +// `POST /v1/clones` (or `GET /v1/clones/:id/result`) into a real project tree +// on disk: text files are written from their inline `content`, and binary +// assets are materialized from inline base64 or fetched from their reference +// URL. Zero dependencies — just Node >= 20. + +import { createHash } from "node:crypto"; +import { mkdir, readFile, writeFile } from "node:fs/promises"; +import { dirname, join, relative, resolve, sep } from "node:path"; + +const USAGE = `ditto — unpack a ditto.site clone result into a project tree + +Usage: + ditto unpack [options] + +Arguments: + Path to the saved JSON (POST /v1/clones or GET .../result), + or "-" to read the JSON from stdin (e.g. piped from curl). + Directory to write the project into (created if missing). + +Options: + --base-url Base URL used to resolve relative binary asset URLs. + Defaults to $DITTO_API_URL. + --api-key Bearer key sent when fetching binary assets. + Defaults to $DITTO_API_KEY. + --no-fetch Do not fetch binary assets over the network; report them + as skipped instead. Text files are still written. + --quiet Only print the final summary line. + -h, --help Show this help. + +Examples: + npm run unpack -- clone.json ./out + + curl -sS -X POST "$DITTO_API_URL/v1/clones" \\ + -H "authorization: Bearer $DITTO_API_KEY" \\ + -H "content-type: application/json" \\ + -d '{"url":"https://example.com/","options":{"mode":"single"}}' \\ + | npm run --silent unpack -- - ./out + +`; + +/** Print to stderr and exit non-zero. */ +function fail(msg) { + process.stderr.write(`ditto: ${msg}\n`); + process.exit(1); +} + +/** Minimal flag parser: pulls known --flags out, returns { positionals, flags }. */ +function parseArgs(argv) { + const positionals = []; + const flags = { fetch: true, quiet: false }; + for (let i = 0; i < argv.length; i++) { + const a = argv[i]; + switch (a) { + case "-h": + case "--help": + flags.help = true; + break; + case "--no-fetch": + flags.fetch = false; + break; + case "--quiet": + flags.quiet = true; + break; + case "--base-url": + flags.baseUrl = argv[++i]; + break; + case "--api-key": + flags.apiKey = argv[++i]; + break; + default: + if (a.startsWith("--")) fail(`unknown option: ${a}`); + positionals.push(a); + } + } + return { positionals, flags }; +} + +/** Read the whole of stdin as a string. */ +async function readStdin() { + const chunks = []; + for await (const chunk of process.stdin) chunks.push(chunk); + return Buffer.concat(chunks).toString("utf8"); +} + +/** Accept either the full result envelope ({ files: {...} }) or a bare file map. */ +function extractFiles(doc) { + if (doc && typeof doc === "object" && doc.files && typeof doc.files === "object") { + return doc.files; + } + // A bare file map: values look like clone file entries. + if (doc && typeof doc === "object") { + const vals = Object.values(doc); + if (vals.length && vals.every((v) => v && typeof v === "object" && ("content" in v || "url" in v || "type" in v))) { + return doc; + } + } + return null; +} + +/** Resolve `path` under `outDir`, refusing anything that escapes the tree + * (absolute paths, `..`, leading slashes) — clone results are untrusted input. */ +function safeJoin(outDir, path) { + const cleaned = String(path).replace(/^[/\\]+/, ""); + const dest = resolve(outDir, cleaned); + const rel = relative(outDir, dest); + if (rel === "" || rel.startsWith("..") || rel.split(sep).includes("..")) { + throw new Error(`refusing to write outside output dir: ${path}`); + } + return dest; +} + +/** Decode a binary entry's inline bytes, if present (base64 or plain). */ +function inlineBytes(entry) { + if (typeof entry.base64 === "string") return Buffer.from(entry.base64, "base64"); + if (typeof entry.content === "string") { + const enc = entry.encoding || (entry.type === "binary" ? "base64" : "utf8"); + return Buffer.from(entry.content, enc === "base64" ? "base64" : "utf8"); + } + return null; +} + +function sha256(buf) { + return createHash("sha256").update(buf).digest("hex"); +} + +async function fetchBinary(url, baseUrl, apiKey) { + const abs = /^https?:\/\//i.test(url) ? url : baseUrl ? new URL(url, baseUrl).toString() : null; + if (!abs) throw new Error("relative asset URL but no --base-url / $DITTO_API_URL set"); + const headers = apiKey ? { authorization: `Bearer ${apiKey}` } : {}; + const res = await fetch(abs, { headers }); + if (!res.ok) throw new Error(`GET ${abs} -> ${res.status} ${res.statusText}`); + return Buffer.from(await res.arrayBuffer()); +} + +async function unpack(positionals, flags) { + const [input, outDir] = positionals; + if (!input || !outDir) fail("unpack needs and \n\n" + USAGE); + + const raw = input === "-" ? await readStdin() : await readFile(input, "utf8").catch((e) => fail(`cannot read ${input}: ${e.message}`)); + let doc; + try { + doc = JSON.parse(raw); + } catch (e) { + fail(`input is not valid JSON: ${e.message}`); + } + + if (doc && (doc.status === "queued" || doc.jobId) && !doc.files) { + fail( + `this looks like a queued job (${doc.jobId ? `jobId ${doc.jobId}` : doc.status}), not a finished result.\n` + + `Poll GET /v1/clones//result until it has a "files" map, then unpack that.`, + ); + } + + const files = extractFiles(doc); + if (!files) fail(`no "files" map found in the input — is this a clone result JSON?`); + + const outAbs = resolve(outDir); + const baseUrl = flags.baseUrl || process.env.DITTO_API_URL || ""; + const apiKey = flags.apiKey || process.env.DITTO_API_KEY || ""; + const log = (m) => { + if (!flags.quiet) process.stderr.write(m + "\n"); + }; + + let written = 0; + let bytes = 0; + const skipped = []; + + for (const [path, entry] of Object.entries(files)) { + if (!entry || typeof entry !== "object") { + skipped.push({ path, reason: "malformed entry" }); + continue; + } + const isBinary = entry.type === "binary" || entry.kind === "binary"; + let buf; + + if (isBinary) { + buf = inlineBytes(entry); + if (!buf) { + if (!flags.fetch) { + skipped.push({ path, reason: "binary asset (fetch disabled)" }); + continue; + } + if (typeof entry.url !== "string") { + skipped.push({ path, reason: "binary asset with no inline bytes or url" }); + continue; + } + try { + buf = await fetchBinary(entry.url, baseUrl, apiKey); + } catch (e) { + skipped.push({ path, reason: `could not fetch asset: ${e.message}` }); + continue; + } + } + } else { + buf = Buffer.from(typeof entry.content === "string" ? entry.content : "", "utf8"); + } + + if (typeof entry.sha256 === "string" && entry.sha256 && sha256(buf) !== entry.sha256) { + log(` ! ${path}: sha256 mismatch`); + fail(`${path} failed sha256 integrity check`); + } + + const dest = safeJoin(outAbs, path); + await mkdir(dirname(dest), { recursive: true }); + await writeFile(dest, buf); + written++; + bytes += buf.length; + log(` + ${path}`); + } + + const kb = (bytes / 1024).toFixed(1); + log(""); + process.stdout.write(`Wrote ${written} file${written === 1 ? "" : "s"} (${kb} KB) to ${relative(process.cwd(), outAbs) || "."}\n`); + + if (skipped.length) { + process.stderr.write(`\n${skipped.length} file${skipped.length === 1 ? "" : "s"} skipped:\n`); + for (const s of skipped) process.stderr.write(` - ${s.path}: ${s.reason}\n`); + const anyAsset = skipped.some((s) => /asset|binary/.test(s.reason)); + if (anyAsset) { + process.stderr.write( + `\nTip: binary assets are referenced by URL. Set $DITTO_API_URL (and $DITTO_API_KEY if\n` + + `the API is authenticated) so ditto can fetch them, or download the whole app in one\n` + + `shot with GET /v1/clones//bundle?format=tgz.\n`, + ); + } + } +} + +async function main() { + const argv = process.argv.slice(2); + const { positionals, flags } = parseArgs(argv); + const command = positionals.shift(); + + if (flags.help || !command) { + process.stdout.write(USAGE); + // No command at all is a misuse (exit 1); an explicit --help is success. + process.exit(flags.help ? 0 : 1); + } + + switch (command) { + case "unpack": + await unpack(positionals, flags); + break; + default: + fail(`unknown command: ${command}\n\n${USAGE}`); + } +} + +main().catch((e) => fail(e?.stack || String(e))); diff --git a/packages/cli/package.json b/packages/cli/package.json new file mode 100644 index 0000000..3b260b5 --- /dev/null +++ b/packages/cli/package.json @@ -0,0 +1,30 @@ +{ + "name": "ditto", + "version": "0.1.0", + "private": true, + "description": "Official ditto.site command-line helper: unpack a clone JSON result into a project tree on disk.", + "license": "MIT", + "author": "ion-design", + "repository": { + "type": "git", + "url": "https://github.com/ion-design/ditto.site.git", + "directory": "packages/cli" + }, + "bugs": { + "url": "https://github.com/ion-design/ditto.site/issues" + }, + "homepage": "https://github.com/ion-design/ditto.site#readme", + "type": "module", + "bin": { + "ditto": "./bin/ditto.mjs" + }, + "files": [ + "bin" + ], + "engines": { + "node": ">=20" + }, + "scripts": { + "test": "node --test test/*.test.mjs" + } +} diff --git a/packages/cli/test/unpack.test.mjs b/packages/cli/test/unpack.test.mjs new file mode 100644 index 0000000..95b4767 --- /dev/null +++ b/packages/cli/test/unpack.test.mjs @@ -0,0 +1,186 @@ +import test from "node:test"; +import assert from "node:assert/strict"; +import { spawn } from "node:child_process"; +import { createHash } from "node:crypto"; +import { mkdtemp, readFile, rm, stat, writeFile } from "node:fs/promises"; +import { tmpdir } from "node:os"; +import { fileURLToPath } from "node:url"; +import { dirname, join } from "node:path"; + +const BIN = fileURLToPath(new URL("../bin/ditto.mjs", import.meta.url)); + +function sha256(s) { + return createHash("sha256").update(Buffer.from(s)).digest("hex"); +} + +/** Run the CLI. `stdin` (optional) is written to the child's stdin. */ +function run(args, { stdin, env } = {}) { + return new Promise((resolvePromise) => { + const child = spawn(process.execPath, [BIN, ...args], { + env: { ...process.env, ...env }, + }); + let out = ""; + let err = ""; + child.stdout.on("data", (d) => (out += d)); + child.stderr.on("data", (d) => (err += d)); + child.on("close", (code) => resolvePromise({ code, out, err })); + if (stdin !== undefined) child.stdin.end(stdin); + else child.stdin.end(); + }); +} + +async function withTmp(fn) { + const dir = await mkdtemp(join(tmpdir(), "ditto-cli-")); + try { + return await fn(dir); + } finally { + await rm(dir, { recursive: true, force: true }); + } +} + +test("unpack: writes a text file tree from a result envelope", async () => { + await withTmp(async (dir) => { + const content = "export default function Page() { return null; }\n"; + const doc = { + jobId: "job_1", + files: { + "package.json": { type: "text", content: '{"name":"x"}\n', bytes: 13, sha256: sha256('{"name":"x"}\n') }, + "src/app/page.tsx": { type: "text", content, bytes: content.length, sha256: sha256(content) }, + }, + }; + const jsonPath = join(dir, "clone.json"); + const out = join(dir, "out"); + await writeFile(jsonPath, JSON.stringify(doc)); + + const res = await run(["unpack", jsonPath, out]); + assert.equal(res.code, 0, res.err); + assert.equal(await readFile(join(out, "package.json"), "utf8"), '{"name":"x"}\n'); + assert.equal(await readFile(join(out, "src/app/page.tsx"), "utf8"), content); + assert.match(res.out, /Wrote 2 files/); + }); +}); + +test("unpack: reads JSON from stdin with '-'", async () => { + await withTmp(async (dir) => { + const doc = { files: { "a.txt": { type: "text", content: "hi" } } }; + const out = join(dir, "out"); + const res = await run(["unpack", "-", out], { stdin: JSON.stringify(doc) }); + assert.equal(res.code, 0, res.err); + assert.equal(await readFile(join(out, "a.txt"), "utf8"), "hi"); + }); +}); + +test("unpack: materializes inline base64 binary assets", async () => { + await withTmp(async (dir) => { + const bytes = Buffer.from([0x89, 0x50, 0x4e, 0x47, 0x0d, 0x0a]); + const doc = { + files: { + "public/logo.png": { + type: "binary", + content: bytes.toString("base64"), + encoding: "base64", + bytes: bytes.length, + sha256: createHash("sha256").update(bytes).digest("hex"), + }, + }, + }; + const out = join(dir, "out"); + const res = await run(["unpack", "-", out], { stdin: JSON.stringify(doc) }); + assert.equal(res.code, 0, res.err); + const written = await readFile(join(out, "public/logo.png")); + assert.deepEqual([...written], [...bytes]); + }); +}); + +test("unpack: fetches binary assets by URL and resolves relative URLs against base", async () => { + await withTmp(async (dir) => { + const asset = Buffer.from("PNGDATA"); + const { createServer } = await import("node:http"); + const server = createServer((req, res) => { + if (req.url === "/v1/clones/job_1/files/public/img.png" && req.headers.authorization === "Bearer k") { + res.writeHead(200); + res.end(asset); + } else { + res.writeHead(404); + res.end("no"); + } + }); + await new Promise((r) => server.listen(0, r)); + const port = server.address().port; + try { + const doc = { + files: { + "public/img.png": { + type: "binary", + url: "/v1/clones/job_1/files/public/img.png", + bytes: asset.length, + sha256: createHash("sha256").update(asset).digest("hex"), + }, + }, + }; + const out = join(dir, "out"); + const res = await run(["unpack", "-", out], { + stdin: JSON.stringify(doc), + env: { DITTO_API_URL: `http://127.0.0.1:${port}`, DITTO_API_KEY: "k" }, + }); + assert.equal(res.code, 0, res.err); + assert.deepEqual([...(await readFile(join(out, "public/img.png")))], [...asset]); + } finally { + server.close(); + } + }); +}); + +test("unpack: --no-fetch skips remote binaries but still writes text", async () => { + await withTmp(async (dir) => { + const doc = { + files: { + "index.html": { type: "text", content: "

hi

" }, + "public/img.png": { type: "binary", url: "/v1/clones/x/files/public/img.png" }, + }, + }; + const out = join(dir, "out"); + const res = await run(["unpack", "-", out, "--no-fetch"], { stdin: JSON.stringify(doc) }); + assert.equal(res.code, 0, res.err); + assert.equal(await readFile(join(out, "index.html"), "utf8"), "

hi

"); + await assert.rejects(stat(join(out, "public/img.png"))); + assert.match(res.err, /skipped/); + }); +}); + +test("unpack: refuses path traversal", async () => { + await withTmp(async (dir) => { + const doc = { files: { "../escape.txt": { type: "text", content: "x" } } }; + const out = join(dir, "out"); + const res = await run(["unpack", "-", out], { stdin: JSON.stringify(doc) }); + assert.notEqual(res.code, 0); + assert.match(res.err, /outside output dir/); + }); +}); + +test("unpack: fails clearly on a queued job with no files", async () => { + await withTmp(async (dir) => { + const doc = { jobId: "job_9", status: "queued" }; + const out = join(dir, "out"); + const res = await run(["unpack", "-", out], { stdin: JSON.stringify(doc) }); + assert.notEqual(res.code, 0); + assert.match(res.err, /queued job/); + }); +}); + +test("unpack: reports sha256 mismatch as a failure", async () => { + await withTmp(async (dir) => { + const doc = { files: { "a.txt": { type: "text", content: "hi", sha256: "deadbeef" } } }; + const out = join(dir, "out"); + const res = await run(["unpack", "-", out], { stdin: JSON.stringify(doc) }); + assert.notEqual(res.code, 0); + assert.match(res.err, /integrity check/); + await assert.rejects(stat(join(out, "a.txt"))); + }); +}); + +test("help: exits 0 and prints usage", async () => { + const res = await run(["--help"]); + assert.equal(res.code, 0); + assert.match(res.out, /ditto unpack/); +});