diff --git a/CHANGELOG.md b/CHANGELOG.md index 2577993..6dde4f5 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -29,13 +29,14 @@ so minor/patch semantics are not yet guaranteed. ### Added — `ditto` unpack CLI -- **`packages/cli`** — a zero-dependency `ditto` command-line helper. `ditto unpack - ` turns the `files` map returned by `POST /v1/clones` (or - `GET /v1/clones/:id/result`) into a real project tree on disk: text files written - inline, binary assets materialized from inline base64 or fetched from their - reference URL (`$DITTO_API_URL` / `$DITTO_API_KEY`), with path-traversal guards - and `sha256` integrity checks. Reads from stdin so a `curl` response can be piped - straight in. Documented next to the REST examples in the README and `docs/SERVICE.md`. +- **`packages/cli`** — a zero-dependency `ditto` command-line helper. `npm run + unpack -- ` turns the `files` map returned by + `POST /v1/clones` (or `GET /v1/clones/:id/result`) into a real project tree on + disk: text files written inline, binary assets materialized from inline base64 + or fetched from their reference URL (`$DITTO_API_URL` / `$DITTO_API_KEY`), with + path-traversal guards and pre-write `sha256` integrity checks. Reads from stdin + so a `curl` response can be piped straight in. Documented next to the REST + examples in the README and `docs/SERVICE.md`. ### Added — Open-source readiness diff --git a/README.md b/README.md index bfc7abe..6335c66 100644 --- a/README.md +++ b/README.md @@ -82,20 +82,22 @@ is a file map — every generated file keyed by its app-relative path: } ``` -**Turn that JSON into a project on disk** with the official unpacker — pipe the -response straight in, no temp file: +**Turn that JSON into a project on disk** with the official unpacker — from a +checked-out `ditto.site` repo with dependencies installed, pipe the response +straight in with no temp file: ```bash curl -sS -X POST "$DITTO_API_URL/v1/clones" \ -H "authorization: Bearer $DITTO_API_KEY" \ -H "content-type: application/json" \ -d '{"url":"https://example.com/","options":{"mode":"single"}}' \ - | npx ditto unpack - ./out + | npm run --silent unpack -- - ./out ``` -`ditto unpack ` writes the text files inline and +`npm run unpack -- ` writes the text files inline and materializes binary assets (inline base64, or fetched from their `url` using -`$DITTO_API_URL` / `$DITTO_API_KEY`). See +`$DITTO_API_URL` / `$DITTO_API_KEY`). The CLI package is repo-local until the +npm distribution story is ready, so do not use `npx ditto` yet. See [`packages/cli`](packages/cli/README.md) for options. If you got back a queued job (`{ "jobId": "job_123", "status": "queued" }`), @@ -108,7 +110,7 @@ JOB_ID="job_123" # poll status, then unpack the finished file map curl -sS -H "authorization: Bearer $DITTO_API_KEY" \ "$DITTO_API_URL/v1/clones/$JOB_ID/result" \ - | npx ditto unpack - ./out + | npm run --silent unpack -- - ./out # ...or grab the whole app as a single archive curl -L -H "authorization: Bearer $DITTO_API_KEY" \ diff --git a/docs/README.md b/docs/README.md index 353189f..d1db372 100644 --- a/docs/README.md +++ b/docs/README.md @@ -15,7 +15,7 @@ The central index for ditto.site's docs. Start here. | --- | --- | | Understand what ditto.site is and see it run | [Project README](../README.md) | | Call the hosted **REST API** or **MCP** server | [Project README → Usage](../README.md#usage), [SERVICE.md](SERVICE.md) | -| Turn a clone result JSON into files on disk | [`ditto unpack` CLI](../packages/cli/README.md) | +| Turn a clone result JSON into files on disk | [Repo-local unpack CLI](../packages/cli/README.md) | | Run the **compiler locally** from the command line | [compiler/README.md](../compiler/README.md) | | Deploy the service | [DEPLOY.md](DEPLOY.md) | | Read the development & evaluation method | [METHODOLOGY.md](METHODOLOGY.md) | @@ -39,7 +39,7 @@ never commit one. Rotate a leaked key anytime from the dashboard. See ## The short version of the workflow 1. **Clone** a URL → `POST /v1/clones` (API) or `npm run clone -- ` (local CLI). -2. **Get the app** → unpack the result JSON with [`ditto unpack`](../packages/cli/README.md), +2. **Get the app** → unpack the result JSON with the [repo-local unpack CLI](../packages/cli/README.md), download the `bundle?format=tgz` archive, or read files from `runs//latest/`. 3. **Preview it** → `cd` into the app and `npm install && npm run dev` (or let the local CLI do it for you with `--serve` / `--open`). @@ -54,4 +54,4 @@ never commit one. Rotate a leaked key anytime from the dashboard. See - [RESPONSIBLE_USE.md](RESPONSIBLE_USE.md) — acceptable-use boundaries. - [RELEASING.md](RELEASING.md) — release process. - [../compiler/README.md](../compiler/README.md) — local compiler commands. -- [../packages/cli/README.md](../packages/cli/README.md) — the `ditto unpack` CLI. +- [../packages/cli/README.md](../packages/cli/README.md) — the repo-local unpack CLI. diff --git a/docs/SERVICE.md b/docs/SERVICE.md index 01e0c5d..f4d2cf6 100644 --- a/docs/SERVICE.md +++ b/docs/SERVICE.md @@ -61,16 +61,20 @@ curl -s -X POST localhost:8787/v1/clones -H 'content-type: application/json' \ ``` To turn that `files` map into a project on disk, pipe the response into the -`ditto` CLI (`packages/cli`) instead of inspecting the JSON by hand: +repo-local `ditto` CLI (`packages/cli`) instead of inspecting the JSON by hand: ```bash curl -s -X POST localhost:8787/v1/clones -H 'content-type: application/json' \ -d '{"url":"https://example.com/","options":{"mode":"single"}}' \ - | npx ditto unpack - ./out + | npm run --silent unpack -- - ./out # binary assets: set DITTO_API_URL (and DITTO_API_KEY when authenticated) so the # CLI can fetch each file's reference URL; --no-fetch writes only the text tree. ``` +The CLI package is intentionally private for now; run this command from a +checked-out `ditto.site` repo with dependencies installed. Do not use +`npx ditto` until this package is published. + ## REST surface ``` diff --git a/packages/cli/README.md b/packages/cli/README.md index 1757a5b..ee8e391 100644 --- a/packages/cli/README.md +++ b/packages/cli/README.md @@ -6,21 +6,25 @@ the giant blob you get back from `POST /v1/clones` or Zero dependencies. Needs Node >= 20. +This workspace is currently private and repo-local. Run these commands from the +repository root after `npm install`; do not use `npx ditto` until the package is +published. + ## Unpack ```bash # from a saved file -ditto unpack clone.json ./out +npm run unpack -- clone.json ./out # straight from curl, no temp file curl -sS -X POST "$DITTO_API_URL/v1/clones" \ -H "authorization: Bearer $DITTO_API_KEY" \ -H "content-type: application/json" \ -d '{"url":"https://example.com/","options":{"mode":"single"}}' \ - | ditto unpack - ./out + | npm run --silent unpack -- - ./out ``` -`ditto unpack `: +`npm run unpack -- `: - writes every text file from its inline `content`, - materializes binary assets from inline base64 when present, otherwise fetches @@ -31,8 +35,8 @@ curl -sS -X POST "$DITTO_API_URL/v1/clones" \ ### Binary assets Clone results return binaries by reference (`{ "type": "binary", "url": ... }`) -rather than inlining megabytes of base64. To fetch them, `ditto` needs to know -where the API lives: +rather than inlining megabytes of base64. To fetch them, the unpacker needs to +know where the API lives: | Source | Flag | Env | | --- | --- | --- | @@ -43,4 +47,4 @@ Use `--no-fetch` to write only the text tree and list the binaries as skipped. To grab everything in one shot instead, download the archive directly: `GET /v1/clones//bundle?format=tgz`. -Run `ditto --help` for the full option list. +Run `npm run unpack -- --help` for the full option list. diff --git a/packages/cli/bin/ditto.mjs b/packages/cli/bin/ditto.mjs index 5cf4a40..b91771c 100755 --- a/packages/cli/bin/ditto.mjs +++ b/packages/cli/bin/ditto.mjs @@ -32,13 +32,14 @@ Options: -h, --help Show this help. Examples: + npm run unpack -- clone.json ./out + curl -sS -X POST "$DITTO_API_URL/v1/clones" \\ -H "authorization: Bearer $DITTO_API_KEY" \\ -H "content-type: application/json" \\ -d '{"url":"https://example.com/","options":{"mode":"single"}}' \\ - | ditto unpack - ./out + | npm run --silent unpack -- - ./out - ditto unpack clone.json ./out `; /** Print to stderr and exit non-zero. */ @@ -166,7 +167,6 @@ async function unpack(positionals, flags) { let written = 0; let bytes = 0; - let hashMismatch = 0; const skipped = []; for (const [path, entry] of Object.entries(files)) { @@ -199,18 +199,17 @@ async function unpack(positionals, flags) { buf = Buffer.from(typeof entry.content === "string" ? entry.content : "", "utf8"); } + if (typeof entry.sha256 === "string" && entry.sha256 && sha256(buf) !== entry.sha256) { + log(` ! ${path}: sha256 mismatch`); + fail(`${path} failed sha256 integrity check`); + } + const dest = safeJoin(outAbs, path); await mkdir(dirname(dest), { recursive: true }); await writeFile(dest, buf); written++; bytes += buf.length; - - if (typeof entry.sha256 === "string" && entry.sha256 && sha256(buf) !== entry.sha256) { - hashMismatch++; - log(` ! ${path}: sha256 mismatch`); - } else { - log(` + ${path}`); - } + log(` + ${path}`); } const kb = (bytes / 1024).toFixed(1); @@ -229,7 +228,6 @@ async function unpack(positionals, flags) { ); } } - if (hashMismatch) fail(`${hashMismatch} file(s) failed sha256 integrity check`); } async function main() { diff --git a/packages/cli/test/unpack.test.mjs b/packages/cli/test/unpack.test.mjs index b8c6fb0..95b4767 100644 --- a/packages/cli/test/unpack.test.mjs +++ b/packages/cli/test/unpack.test.mjs @@ -175,6 +175,7 @@ test("unpack: reports sha256 mismatch as a failure", async () => { const res = await run(["unpack", "-", out], { stdin: JSON.stringify(doc) }); assert.notEqual(res.code, 0); assert.match(res.err, /integrity check/); + await assert.rejects(stat(join(out, "a.txt"))); }); });