Add ditto unpack CLI to turn clone result JSON into a project tree
The REST API returns a clone as a giant `files` map JSON blob with no
first-party way to use it — users were left staring at kilobytes of
escaped JSON with no path from "you got JSON" to "here's a project".
Add packages/cli: a zero-dependency `ditto` CLI whose `unpack` command
walks the files{} map from POST /v1/clones (or GET .../result) and
writes a real project tree to disk:
- text files written from inline content;
- binary assets materialized from inline base64, else fetched from their
reference URL using $DITTO_API_URL / $DITTO_API_KEY (--no-fetch writes
just the text tree and lists skipped assets);
- path-traversal guards (clone results are untrusted) and sha256
integrity checks;
- reads JSON from stdin ("-") so a curl response pipes straight in;
- clear errors for queued jobs with no files yet, and a tip pointing at
the /bundle endpoint when assets can't be fetched.
Document the pipe-from-curl one-liner prominently next to the REST
examples in README.md and docs/SERVICE.md, add a package README, a
Repository Map row, a root `unpack` script, and a CHANGELOG entry.
Covered by 9 tests (text tree, stdin, inline base64, live URL fetch with
auth + relative-URL resolution, --no-fetch, traversal refusal, queued-job
detection, sha256 mismatch).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
co-authored by
Claude Opus 4.8
parent
4de6e244a0
commit
638a0516c5
@@ -6,6 +6,16 @@ so minor/patch semantics are not yet guaranteed.
|
|||||||
|
|
||||||
## [Unreleased]
|
## [Unreleased]
|
||||||
|
|
||||||
|
### Added — `ditto` unpack CLI
|
||||||
|
|
||||||
|
- **`packages/cli`** — a zero-dependency `ditto` command-line helper. `ditto unpack
|
||||||
|
<clone.json|-> <out-dir>` turns the `files` map returned by `POST /v1/clones` (or
|
||||||
|
`GET /v1/clones/:id/result`) into a real project tree on disk: text files written
|
||||||
|
inline, binary assets materialized from inline base64 or fetched from their
|
||||||
|
reference URL (`$DITTO_API_URL` / `$DITTO_API_KEY`), with path-traversal guards
|
||||||
|
and `sha256` integrity checks. Reads from stdin so a `curl` response can be piped
|
||||||
|
straight in. Documented next to the REST examples in the README and `docs/SERVICE.md`.
|
||||||
|
|
||||||
### Added — Open-source readiness
|
### Added — Open-source readiness
|
||||||
|
|
||||||
- Added support, release, responsible-use, CODEOWNERS, and gitattributes files.
|
- Added support, release, responsible-use, CODEOWNERS, and gitattributes files.
|
||||||
|
|||||||
@@ -56,20 +56,50 @@ curl -sS -X POST "$DITTO_API_URL/v1/clones" \
|
|||||||
}'
|
}'
|
||||||
```
|
```
|
||||||
|
|
||||||
The service returns either an inline result or a queued job:
|
The service returns either a queued job or an inline result. A finished result
|
||||||
|
is a file map — every generated file keyed by its app-relative path:
|
||||||
|
|
||||||
```json
|
```json
|
||||||
{ "jobId": "job_123", "status": "queued" }
|
{
|
||||||
|
"jobId": "job_123",
|
||||||
|
"status": "succeeded",
|
||||||
|
"files": {
|
||||||
|
"package.json": { "type": "text", "content": "{ ... }", "bytes": 812, "sha256": "..." },
|
||||||
|
"src/app/page.tsx": { "type": "text", "content": "export default ...", "bytes": 2048, "sha256": "..." },
|
||||||
|
"public/assets/logo.png": { "type": "binary", "url": ".../files/public/assets/logo.png", "bytes": 5123, "sha256": "..." }
|
||||||
|
}
|
||||||
|
}
|
||||||
```
|
```
|
||||||
|
|
||||||
Poll and download the generated app:
|
**Turn that JSON into a project on disk** with the official unpacker — pipe the
|
||||||
|
response straight in, no temp file:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
curl -sS -X POST "$DITTO_API_URL/v1/clones" \
|
||||||
|
-H "authorization: Bearer $DITTO_API_KEY" \
|
||||||
|
-H "content-type: application/json" \
|
||||||
|
-d '{"url":"https://example.com/","options":{"mode":"single"}}' \
|
||||||
|
| npx ditto unpack - ./out
|
||||||
|
```
|
||||||
|
|
||||||
|
`ditto unpack <clone.json|-> <out-dir>` writes the text files inline and
|
||||||
|
materializes binary assets (inline base64, or fetched from their `url` using
|
||||||
|
`$DITTO_API_URL` / `$DITTO_API_KEY`). See
|
||||||
|
[`packages/cli`](packages/cli/README.md) for options.
|
||||||
|
|
||||||
|
If you got back a queued job (`{ "jobId": "job_123", "status": "queued" }`),
|
||||||
|
poll it, then unpack the finished result — or download the whole app as one
|
||||||
|
archive:
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
JOB_ID="job_123"
|
JOB_ID="job_123"
|
||||||
|
|
||||||
|
# poll status, then unpack the finished file map
|
||||||
curl -sS -H "authorization: Bearer $DITTO_API_KEY" \
|
curl -sS -H "authorization: Bearer $DITTO_API_KEY" \
|
||||||
"$DITTO_API_URL/v1/clones/$JOB_ID"
|
"$DITTO_API_URL/v1/clones/$JOB_ID/result" \
|
||||||
|
| npx ditto unpack - ./out
|
||||||
|
|
||||||
|
# ...or grab the whole app as a single archive
|
||||||
curl -L -H "authorization: Bearer $DITTO_API_KEY" \
|
curl -L -H "authorization: Bearer $DITTO_API_KEY" \
|
||||||
"$DITTO_API_URL/v1/clones/$JOB_ID/bundle?format=tgz" \
|
"$DITTO_API_URL/v1/clones/$JOB_ID/bundle?format=tgz" \
|
||||||
-o ditto-clone.tgz
|
-o ditto-clone.tgz
|
||||||
@@ -241,6 +271,7 @@ minting is intentional.
|
|||||||
| --- | --- |
|
| --- | --- |
|
||||||
| `compiler/` | deterministic capture, inference, generation, and validation |
|
| `compiler/` | deterministic capture, inference, generation, and validation |
|
||||||
| `packages/core/` | compiler adapter and file-map helpers |
|
| `packages/core/` | compiler adapter and file-map helpers |
|
||||||
|
| `packages/cli/` | `ditto` CLI — unpack a clone result JSON into a project tree |
|
||||||
| `packages/api/` | Hono REST API and MCP server |
|
| `packages/api/` | Hono REST API and MCP server |
|
||||||
| `packages/db/` | Drizzle schema, migrations, repository, and queue wrapper |
|
| `packages/db/` | Drizzle schema, migrations, repository, and queue wrapper |
|
||||||
| `packages/storage/` | local and S3/R2 artifact storage |
|
| `packages/storage/` | local and S3/R2 artifact storage |
|
||||||
|
|||||||
@@ -60,6 +60,17 @@ curl -s -X POST localhost:8787/v1/clones -H 'content-type: application/json' \
|
|||||||
-d '{"url":"https://example.com/","options":{"mode":"single","styling":"tailwind"}}' | jq '.files | keys'
|
-d '{"url":"https://example.com/","options":{"mode":"single","styling":"tailwind"}}' | jq '.files | keys'
|
||||||
```
|
```
|
||||||
|
|
||||||
|
To turn that `files` map into a project on disk, pipe the response into the
|
||||||
|
`ditto` CLI (`packages/cli`) instead of inspecting the JSON by hand:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
curl -s -X POST localhost:8787/v1/clones -H 'content-type: application/json' \
|
||||||
|
-d '{"url":"https://example.com/","options":{"mode":"single"}}' \
|
||||||
|
| npx ditto unpack - ./out
|
||||||
|
# binary assets: set DITTO_API_URL (and DITTO_API_KEY when authenticated) so the
|
||||||
|
# CLI can fetch each file's reference URL; --no-fetch writes only the text tree.
|
||||||
|
```
|
||||||
|
|
||||||
## REST surface
|
## REST surface
|
||||||
|
|
||||||
```
|
```
|
||||||
|
|||||||
Generated
+15
@@ -1878,6 +1878,10 @@
|
|||||||
"node": ">= 0.8"
|
"node": ">= 0.8"
|
||||||
}
|
}
|
||||||
},
|
},
|
||||||
|
"node_modules/ditto": {
|
||||||
|
"resolved": "packages/cli",
|
||||||
|
"link": true
|
||||||
|
},
|
||||||
"node_modules/drizzle-kit": {
|
"node_modules/drizzle-kit": {
|
||||||
"version": "0.31.10",
|
"version": "0.31.10",
|
||||||
"resolved": "https://registry.npmjs.org/drizzle-kit/-/drizzle-kit-0.31.10.tgz",
|
"resolved": "https://registry.npmjs.org/drizzle-kit/-/drizzle-kit-0.31.10.tgz",
|
||||||
@@ -3802,6 +3806,17 @@
|
|||||||
"typescript": "5.7.3"
|
"typescript": "5.7.3"
|
||||||
}
|
}
|
||||||
},
|
},
|
||||||
|
"packages/cli": {
|
||||||
|
"name": "ditto",
|
||||||
|
"version": "0.1.0",
|
||||||
|
"license": "MIT",
|
||||||
|
"bin": {
|
||||||
|
"ditto": "bin/ditto.mjs"
|
||||||
|
},
|
||||||
|
"engines": {
|
||||||
|
"node": ">=20"
|
||||||
|
}
|
||||||
|
},
|
||||||
"packages/core": {
|
"packages/core": {
|
||||||
"name": "@cloner/core",
|
"name": "@cloner/core",
|
||||||
"version": "0.1.0",
|
"version": "0.1.0",
|
||||||
|
|||||||
+2
-1
@@ -41,7 +41,8 @@
|
|||||||
"dev:api": "npm run dev --workspace @cloner/api",
|
"dev:api": "npm run dev --workspace @cloner/api",
|
||||||
"dev:worker": "npm run dev --workspace @cloner/worker",
|
"dev:worker": "npm run dev --workspace @cloner/worker",
|
||||||
"db:generate": "npm run generate --workspace @cloner/db",
|
"db:generate": "npm run generate --workspace @cloner/db",
|
||||||
"db:migrate": "npm run migrate --workspace @cloner/db"
|
"db:migrate": "npm run migrate --workspace @cloner/db",
|
||||||
|
"unpack": "node packages/cli/bin/ditto.mjs unpack"
|
||||||
},
|
},
|
||||||
"devDependencies": {
|
"devDependencies": {
|
||||||
"tsx": "4.22.4",
|
"tsx": "4.22.4",
|
||||||
|
|||||||
@@ -0,0 +1,46 @@
|
|||||||
|
# ditto (CLI)
|
||||||
|
|
||||||
|
The official ditto.site command-line helper. Turns a clone **result JSON** —
|
||||||
|
the giant blob you get back from `POST /v1/clones` or
|
||||||
|
`GET /v1/clones/:id/result` — into an actual project tree on disk.
|
||||||
|
|
||||||
|
Zero dependencies. Needs Node >= 20.
|
||||||
|
|
||||||
|
## Unpack
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# from a saved file
|
||||||
|
ditto unpack clone.json ./out
|
||||||
|
|
||||||
|
# straight from curl, no temp file
|
||||||
|
curl -sS -X POST "$DITTO_API_URL/v1/clones" \
|
||||||
|
-H "authorization: Bearer $DITTO_API_KEY" \
|
||||||
|
-H "content-type: application/json" \
|
||||||
|
-d '{"url":"https://example.com/","options":{"mode":"single"}}' \
|
||||||
|
| ditto unpack - ./out
|
||||||
|
```
|
||||||
|
|
||||||
|
`ditto unpack <clone.json|-> <out-dir>`:
|
||||||
|
|
||||||
|
- writes every text file from its inline `content`,
|
||||||
|
- materializes binary assets from inline base64 when present, otherwise fetches
|
||||||
|
them from their reference `url`,
|
||||||
|
- refuses paths that escape `<out-dir>` and verifies `sha256` when the result
|
||||||
|
carries it.
|
||||||
|
|
||||||
|
### Binary assets
|
||||||
|
|
||||||
|
Clone results return binaries by reference (`{ "type": "binary", "url": ... }`)
|
||||||
|
rather than inlining megabytes of base64. To fetch them, `ditto` needs to know
|
||||||
|
where the API lives:
|
||||||
|
|
||||||
|
| Source | Flag | Env |
|
||||||
|
| --- | --- | --- |
|
||||||
|
| Base URL for relative asset URLs | `--base-url <url>` | `DITTO_API_URL` |
|
||||||
|
| Bearer key for authenticated APIs | `--api-key <key>` | `DITTO_API_KEY` |
|
||||||
|
|
||||||
|
Use `--no-fetch` to write only the text tree and list the binaries as skipped.
|
||||||
|
To grab everything in one shot instead, download the archive directly:
|
||||||
|
`GET /v1/clones/<id>/bundle?format=tgz`.
|
||||||
|
|
||||||
|
Run `ditto --help` for the full option list.
|
||||||
Executable
+255
@@ -0,0 +1,255 @@
|
|||||||
|
#!/usr/bin/env node
|
||||||
|
// ditto — official command-line helper for ditto.site.
|
||||||
|
//
|
||||||
|
// `ditto unpack <clone.json> <out-dir>` turns the JSON returned by
|
||||||
|
// `POST /v1/clones` (or `GET /v1/clones/:id/result`) into a real project tree
|
||||||
|
// on disk: text files are written from their inline `content`, and binary
|
||||||
|
// assets are materialized from inline base64 or fetched from their reference
|
||||||
|
// URL. Zero dependencies — just Node >= 20.
|
||||||
|
|
||||||
|
import { createHash } from "node:crypto";
|
||||||
|
import { mkdir, readFile, writeFile } from "node:fs/promises";
|
||||||
|
import { dirname, join, relative, resolve, sep } from "node:path";
|
||||||
|
|
||||||
|
const USAGE = `ditto — unpack a ditto.site clone result into a project tree
|
||||||
|
|
||||||
|
Usage:
|
||||||
|
ditto unpack <clone.json|-> <out-dir> [options]
|
||||||
|
|
||||||
|
Arguments:
|
||||||
|
<clone.json> Path to the saved JSON (POST /v1/clones or GET .../result),
|
||||||
|
or "-" to read the JSON from stdin (e.g. piped from curl).
|
||||||
|
<out-dir> Directory to write the project into (created if missing).
|
||||||
|
|
||||||
|
Options:
|
||||||
|
--base-url <url> Base URL used to resolve relative binary asset URLs.
|
||||||
|
Defaults to $DITTO_API_URL.
|
||||||
|
--api-key <key> Bearer key sent when fetching binary assets.
|
||||||
|
Defaults to $DITTO_API_KEY.
|
||||||
|
--no-fetch Do not fetch binary assets over the network; report them
|
||||||
|
as skipped instead. Text files are still written.
|
||||||
|
--quiet Only print the final summary line.
|
||||||
|
-h, --help Show this help.
|
||||||
|
|
||||||
|
Examples:
|
||||||
|
curl -sS -X POST "$DITTO_API_URL/v1/clones" \\
|
||||||
|
-H "authorization: Bearer $DITTO_API_KEY" \\
|
||||||
|
-H "content-type: application/json" \\
|
||||||
|
-d '{"url":"https://example.com/","options":{"mode":"single"}}' \\
|
||||||
|
| ditto unpack - ./out
|
||||||
|
|
||||||
|
ditto unpack clone.json ./out
|
||||||
|
`;
|
||||||
|
|
||||||
|
/** Print to stderr and exit non-zero. */
|
||||||
|
function fail(msg) {
|
||||||
|
process.stderr.write(`ditto: ${msg}\n`);
|
||||||
|
process.exit(1);
|
||||||
|
}
|
||||||
|
|
||||||
|
/** Minimal flag parser: pulls known --flags out, returns { positionals, flags }. */
|
||||||
|
function parseArgs(argv) {
|
||||||
|
const positionals = [];
|
||||||
|
const flags = { fetch: true, quiet: false };
|
||||||
|
for (let i = 0; i < argv.length; i++) {
|
||||||
|
const a = argv[i];
|
||||||
|
switch (a) {
|
||||||
|
case "-h":
|
||||||
|
case "--help":
|
||||||
|
flags.help = true;
|
||||||
|
break;
|
||||||
|
case "--no-fetch":
|
||||||
|
flags.fetch = false;
|
||||||
|
break;
|
||||||
|
case "--quiet":
|
||||||
|
flags.quiet = true;
|
||||||
|
break;
|
||||||
|
case "--base-url":
|
||||||
|
flags.baseUrl = argv[++i];
|
||||||
|
break;
|
||||||
|
case "--api-key":
|
||||||
|
flags.apiKey = argv[++i];
|
||||||
|
break;
|
||||||
|
default:
|
||||||
|
if (a.startsWith("--")) fail(`unknown option: ${a}`);
|
||||||
|
positionals.push(a);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return { positionals, flags };
|
||||||
|
}
|
||||||
|
|
||||||
|
/** Read the whole of stdin as a string. */
|
||||||
|
async function readStdin() {
|
||||||
|
const chunks = [];
|
||||||
|
for await (const chunk of process.stdin) chunks.push(chunk);
|
||||||
|
return Buffer.concat(chunks).toString("utf8");
|
||||||
|
}
|
||||||
|
|
||||||
|
/** Accept either the full result envelope ({ files: {...} }) or a bare file map. */
|
||||||
|
function extractFiles(doc) {
|
||||||
|
if (doc && typeof doc === "object" && doc.files && typeof doc.files === "object") {
|
||||||
|
return doc.files;
|
||||||
|
}
|
||||||
|
// A bare file map: values look like clone file entries.
|
||||||
|
if (doc && typeof doc === "object") {
|
||||||
|
const vals = Object.values(doc);
|
||||||
|
if (vals.length && vals.every((v) => v && typeof v === "object" && ("content" in v || "url" in v || "type" in v))) {
|
||||||
|
return doc;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return null;
|
||||||
|
}
|
||||||
|
|
||||||
|
/** Resolve `path` under `outDir`, refusing anything that escapes the tree
|
||||||
|
* (absolute paths, `..`, leading slashes) — clone results are untrusted input. */
|
||||||
|
function safeJoin(outDir, path) {
|
||||||
|
const cleaned = String(path).replace(/^[/\\]+/, "");
|
||||||
|
const dest = resolve(outDir, cleaned);
|
||||||
|
const rel = relative(outDir, dest);
|
||||||
|
if (rel === "" || rel.startsWith("..") || rel.split(sep).includes("..")) {
|
||||||
|
throw new Error(`refusing to write outside output dir: ${path}`);
|
||||||
|
}
|
||||||
|
return dest;
|
||||||
|
}
|
||||||
|
|
||||||
|
/** Decode a binary entry's inline bytes, if present (base64 or plain). */
|
||||||
|
function inlineBytes(entry) {
|
||||||
|
if (typeof entry.base64 === "string") return Buffer.from(entry.base64, "base64");
|
||||||
|
if (typeof entry.content === "string") {
|
||||||
|
const enc = entry.encoding || (entry.type === "binary" ? "base64" : "utf8");
|
||||||
|
return Buffer.from(entry.content, enc === "base64" ? "base64" : "utf8");
|
||||||
|
}
|
||||||
|
return null;
|
||||||
|
}
|
||||||
|
|
||||||
|
function sha256(buf) {
|
||||||
|
return createHash("sha256").update(buf).digest("hex");
|
||||||
|
}
|
||||||
|
|
||||||
|
async function fetchBinary(url, baseUrl, apiKey) {
|
||||||
|
const abs = /^https?:\/\//i.test(url) ? url : baseUrl ? new URL(url, baseUrl).toString() : null;
|
||||||
|
if (!abs) throw new Error("relative asset URL but no --base-url / $DITTO_API_URL set");
|
||||||
|
const headers = apiKey ? { authorization: `Bearer ${apiKey}` } : {};
|
||||||
|
const res = await fetch(abs, { headers });
|
||||||
|
if (!res.ok) throw new Error(`GET ${abs} -> ${res.status} ${res.statusText}`);
|
||||||
|
return Buffer.from(await res.arrayBuffer());
|
||||||
|
}
|
||||||
|
|
||||||
|
async function unpack(positionals, flags) {
|
||||||
|
const [input, outDir] = positionals;
|
||||||
|
if (!input || !outDir) fail("unpack needs <clone.json|-> and <out-dir>\n\n" + USAGE);
|
||||||
|
|
||||||
|
const raw = input === "-" ? await readStdin() : await readFile(input, "utf8").catch((e) => fail(`cannot read ${input}: ${e.message}`));
|
||||||
|
let doc;
|
||||||
|
try {
|
||||||
|
doc = JSON.parse(raw);
|
||||||
|
} catch (e) {
|
||||||
|
fail(`input is not valid JSON: ${e.message}`);
|
||||||
|
}
|
||||||
|
|
||||||
|
if (doc && (doc.status === "queued" || doc.jobId) && !doc.files) {
|
||||||
|
fail(
|
||||||
|
`this looks like a queued job (${doc.jobId ? `jobId ${doc.jobId}` : doc.status}), not a finished result.\n` +
|
||||||
|
`Poll GET /v1/clones/<id>/result until it has a "files" map, then unpack that.`,
|
||||||
|
);
|
||||||
|
}
|
||||||
|
|
||||||
|
const files = extractFiles(doc);
|
||||||
|
if (!files) fail(`no "files" map found in the input — is this a clone result JSON?`);
|
||||||
|
|
||||||
|
const outAbs = resolve(outDir);
|
||||||
|
const baseUrl = flags.baseUrl || process.env.DITTO_API_URL || "";
|
||||||
|
const apiKey = flags.apiKey || process.env.DITTO_API_KEY || "";
|
||||||
|
const log = (m) => {
|
||||||
|
if (!flags.quiet) process.stderr.write(m + "\n");
|
||||||
|
};
|
||||||
|
|
||||||
|
let written = 0;
|
||||||
|
let bytes = 0;
|
||||||
|
let hashMismatch = 0;
|
||||||
|
const skipped = [];
|
||||||
|
|
||||||
|
for (const [path, entry] of Object.entries(files)) {
|
||||||
|
if (!entry || typeof entry !== "object") {
|
||||||
|
skipped.push({ path, reason: "malformed entry" });
|
||||||
|
continue;
|
||||||
|
}
|
||||||
|
const isBinary = entry.type === "binary" || entry.kind === "binary";
|
||||||
|
let buf;
|
||||||
|
|
||||||
|
if (isBinary) {
|
||||||
|
buf = inlineBytes(entry);
|
||||||
|
if (!buf) {
|
||||||
|
if (!flags.fetch) {
|
||||||
|
skipped.push({ path, reason: "binary asset (fetch disabled)" });
|
||||||
|
continue;
|
||||||
|
}
|
||||||
|
if (typeof entry.url !== "string") {
|
||||||
|
skipped.push({ path, reason: "binary asset with no inline bytes or url" });
|
||||||
|
continue;
|
||||||
|
}
|
||||||
|
try {
|
||||||
|
buf = await fetchBinary(entry.url, baseUrl, apiKey);
|
||||||
|
} catch (e) {
|
||||||
|
skipped.push({ path, reason: `could not fetch asset: ${e.message}` });
|
||||||
|
continue;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
} else {
|
||||||
|
buf = Buffer.from(typeof entry.content === "string" ? entry.content : "", "utf8");
|
||||||
|
}
|
||||||
|
|
||||||
|
const dest = safeJoin(outAbs, path);
|
||||||
|
await mkdir(dirname(dest), { recursive: true });
|
||||||
|
await writeFile(dest, buf);
|
||||||
|
written++;
|
||||||
|
bytes += buf.length;
|
||||||
|
|
||||||
|
if (typeof entry.sha256 === "string" && entry.sha256 && sha256(buf) !== entry.sha256) {
|
||||||
|
hashMismatch++;
|
||||||
|
log(` ! ${path}: sha256 mismatch`);
|
||||||
|
} else {
|
||||||
|
log(` + ${path}`);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
const kb = (bytes / 1024).toFixed(1);
|
||||||
|
log("");
|
||||||
|
process.stdout.write(`Wrote ${written} file${written === 1 ? "" : "s"} (${kb} KB) to ${relative(process.cwd(), outAbs) || "."}\n`);
|
||||||
|
|
||||||
|
if (skipped.length) {
|
||||||
|
process.stderr.write(`\n${skipped.length} file${skipped.length === 1 ? "" : "s"} skipped:\n`);
|
||||||
|
for (const s of skipped) process.stderr.write(` - ${s.path}: ${s.reason}\n`);
|
||||||
|
const anyAsset = skipped.some((s) => /asset|binary/.test(s.reason));
|
||||||
|
if (anyAsset) {
|
||||||
|
process.stderr.write(
|
||||||
|
`\nTip: binary assets are referenced by URL. Set $DITTO_API_URL (and $DITTO_API_KEY if\n` +
|
||||||
|
`the API is authenticated) so ditto can fetch them, or download the whole app in one\n` +
|
||||||
|
`shot with GET /v1/clones/<id>/bundle?format=tgz.\n`,
|
||||||
|
);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if (hashMismatch) fail(`${hashMismatch} file(s) failed sha256 integrity check`);
|
||||||
|
}
|
||||||
|
|
||||||
|
async function main() {
|
||||||
|
const argv = process.argv.slice(2);
|
||||||
|
const { positionals, flags } = parseArgs(argv);
|
||||||
|
const command = positionals.shift();
|
||||||
|
|
||||||
|
if (flags.help || !command) {
|
||||||
|
process.stdout.write(USAGE);
|
||||||
|
// No command at all is a misuse (exit 1); an explicit --help is success.
|
||||||
|
process.exit(flags.help ? 0 : 1);
|
||||||
|
}
|
||||||
|
|
||||||
|
switch (command) {
|
||||||
|
case "unpack":
|
||||||
|
await unpack(positionals, flags);
|
||||||
|
break;
|
||||||
|
default:
|
||||||
|
fail(`unknown command: ${command}\n\n${USAGE}`);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
main().catch((e) => fail(e?.stack || String(e)));
|
||||||
@@ -0,0 +1,30 @@
|
|||||||
|
{
|
||||||
|
"name": "ditto",
|
||||||
|
"version": "0.1.0",
|
||||||
|
"private": true,
|
||||||
|
"description": "Official ditto.site command-line helper: unpack a clone JSON result into a project tree on disk.",
|
||||||
|
"license": "MIT",
|
||||||
|
"author": "ion-design",
|
||||||
|
"repository": {
|
||||||
|
"type": "git",
|
||||||
|
"url": "https://github.com/ion-design/ditto.site.git",
|
||||||
|
"directory": "packages/cli"
|
||||||
|
},
|
||||||
|
"bugs": {
|
||||||
|
"url": "https://github.com/ion-design/ditto.site/issues"
|
||||||
|
},
|
||||||
|
"homepage": "https://github.com/ion-design/ditto.site#readme",
|
||||||
|
"type": "module",
|
||||||
|
"bin": {
|
||||||
|
"ditto": "./bin/ditto.mjs"
|
||||||
|
},
|
||||||
|
"files": [
|
||||||
|
"bin"
|
||||||
|
],
|
||||||
|
"engines": {
|
||||||
|
"node": ">=20"
|
||||||
|
},
|
||||||
|
"scripts": {
|
||||||
|
"test": "node --test test/*.test.mjs"
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -0,0 +1,185 @@
|
|||||||
|
import test from "node:test";
|
||||||
|
import assert from "node:assert/strict";
|
||||||
|
import { spawn } from "node:child_process";
|
||||||
|
import { createHash } from "node:crypto";
|
||||||
|
import { mkdtemp, readFile, rm, stat, writeFile } from "node:fs/promises";
|
||||||
|
import { tmpdir } from "node:os";
|
||||||
|
import { fileURLToPath } from "node:url";
|
||||||
|
import { dirname, join } from "node:path";
|
||||||
|
|
||||||
|
const BIN = fileURLToPath(new URL("../bin/ditto.mjs", import.meta.url));
|
||||||
|
|
||||||
|
function sha256(s) {
|
||||||
|
return createHash("sha256").update(Buffer.from(s)).digest("hex");
|
||||||
|
}
|
||||||
|
|
||||||
|
/** Run the CLI. `stdin` (optional) is written to the child's stdin. */
|
||||||
|
function run(args, { stdin, env } = {}) {
|
||||||
|
return new Promise((resolvePromise) => {
|
||||||
|
const child = spawn(process.execPath, [BIN, ...args], {
|
||||||
|
env: { ...process.env, ...env },
|
||||||
|
});
|
||||||
|
let out = "";
|
||||||
|
let err = "";
|
||||||
|
child.stdout.on("data", (d) => (out += d));
|
||||||
|
child.stderr.on("data", (d) => (err += d));
|
||||||
|
child.on("close", (code) => resolvePromise({ code, out, err }));
|
||||||
|
if (stdin !== undefined) child.stdin.end(stdin);
|
||||||
|
else child.stdin.end();
|
||||||
|
});
|
||||||
|
}
|
||||||
|
|
||||||
|
async function withTmp(fn) {
|
||||||
|
const dir = await mkdtemp(join(tmpdir(), "ditto-cli-"));
|
||||||
|
try {
|
||||||
|
return await fn(dir);
|
||||||
|
} finally {
|
||||||
|
await rm(dir, { recursive: true, force: true });
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
test("unpack: writes a text file tree from a result envelope", async () => {
|
||||||
|
await withTmp(async (dir) => {
|
||||||
|
const content = "export default function Page() { return null; }\n";
|
||||||
|
const doc = {
|
||||||
|
jobId: "job_1",
|
||||||
|
files: {
|
||||||
|
"package.json": { type: "text", content: '{"name":"x"}\n', bytes: 13, sha256: sha256('{"name":"x"}\n') },
|
||||||
|
"src/app/page.tsx": { type: "text", content, bytes: content.length, sha256: sha256(content) },
|
||||||
|
},
|
||||||
|
};
|
||||||
|
const jsonPath = join(dir, "clone.json");
|
||||||
|
const out = join(dir, "out");
|
||||||
|
await writeFile(jsonPath, JSON.stringify(doc));
|
||||||
|
|
||||||
|
const res = await run(["unpack", jsonPath, out]);
|
||||||
|
assert.equal(res.code, 0, res.err);
|
||||||
|
assert.equal(await readFile(join(out, "package.json"), "utf8"), '{"name":"x"}\n');
|
||||||
|
assert.equal(await readFile(join(out, "src/app/page.tsx"), "utf8"), content);
|
||||||
|
assert.match(res.out, /Wrote 2 files/);
|
||||||
|
});
|
||||||
|
});
|
||||||
|
|
||||||
|
test("unpack: reads JSON from stdin with '-'", async () => {
|
||||||
|
await withTmp(async (dir) => {
|
||||||
|
const doc = { files: { "a.txt": { type: "text", content: "hi" } } };
|
||||||
|
const out = join(dir, "out");
|
||||||
|
const res = await run(["unpack", "-", out], { stdin: JSON.stringify(doc) });
|
||||||
|
assert.equal(res.code, 0, res.err);
|
||||||
|
assert.equal(await readFile(join(out, "a.txt"), "utf8"), "hi");
|
||||||
|
});
|
||||||
|
});
|
||||||
|
|
||||||
|
test("unpack: materializes inline base64 binary assets", async () => {
|
||||||
|
await withTmp(async (dir) => {
|
||||||
|
const bytes = Buffer.from([0x89, 0x50, 0x4e, 0x47, 0x0d, 0x0a]);
|
||||||
|
const doc = {
|
||||||
|
files: {
|
||||||
|
"public/logo.png": {
|
||||||
|
type: "binary",
|
||||||
|
content: bytes.toString("base64"),
|
||||||
|
encoding: "base64",
|
||||||
|
bytes: bytes.length,
|
||||||
|
sha256: createHash("sha256").update(bytes).digest("hex"),
|
||||||
|
},
|
||||||
|
},
|
||||||
|
};
|
||||||
|
const out = join(dir, "out");
|
||||||
|
const res = await run(["unpack", "-", out], { stdin: JSON.stringify(doc) });
|
||||||
|
assert.equal(res.code, 0, res.err);
|
||||||
|
const written = await readFile(join(out, "public/logo.png"));
|
||||||
|
assert.deepEqual([...written], [...bytes]);
|
||||||
|
});
|
||||||
|
});
|
||||||
|
|
||||||
|
test("unpack: fetches binary assets by URL and resolves relative URLs against base", async () => {
|
||||||
|
await withTmp(async (dir) => {
|
||||||
|
const asset = Buffer.from("PNGDATA");
|
||||||
|
const { createServer } = await import("node:http");
|
||||||
|
const server = createServer((req, res) => {
|
||||||
|
if (req.url === "/v1/clones/job_1/files/public/img.png" && req.headers.authorization === "Bearer k") {
|
||||||
|
res.writeHead(200);
|
||||||
|
res.end(asset);
|
||||||
|
} else {
|
||||||
|
res.writeHead(404);
|
||||||
|
res.end("no");
|
||||||
|
}
|
||||||
|
});
|
||||||
|
await new Promise((r) => server.listen(0, r));
|
||||||
|
const port = server.address().port;
|
||||||
|
try {
|
||||||
|
const doc = {
|
||||||
|
files: {
|
||||||
|
"public/img.png": {
|
||||||
|
type: "binary",
|
||||||
|
url: "/v1/clones/job_1/files/public/img.png",
|
||||||
|
bytes: asset.length,
|
||||||
|
sha256: createHash("sha256").update(asset).digest("hex"),
|
||||||
|
},
|
||||||
|
},
|
||||||
|
};
|
||||||
|
const out = join(dir, "out");
|
||||||
|
const res = await run(["unpack", "-", out], {
|
||||||
|
stdin: JSON.stringify(doc),
|
||||||
|
env: { DITTO_API_URL: `http://127.0.0.1:${port}`, DITTO_API_KEY: "k" },
|
||||||
|
});
|
||||||
|
assert.equal(res.code, 0, res.err);
|
||||||
|
assert.deepEqual([...(await readFile(join(out, "public/img.png")))], [...asset]);
|
||||||
|
} finally {
|
||||||
|
server.close();
|
||||||
|
}
|
||||||
|
});
|
||||||
|
});
|
||||||
|
|
||||||
|
test("unpack: --no-fetch skips remote binaries but still writes text", async () => {
|
||||||
|
await withTmp(async (dir) => {
|
||||||
|
const doc = {
|
||||||
|
files: {
|
||||||
|
"index.html": { type: "text", content: "<h1>hi</h1>" },
|
||||||
|
"public/img.png": { type: "binary", url: "/v1/clones/x/files/public/img.png" },
|
||||||
|
},
|
||||||
|
};
|
||||||
|
const out = join(dir, "out");
|
||||||
|
const res = await run(["unpack", "-", out, "--no-fetch"], { stdin: JSON.stringify(doc) });
|
||||||
|
assert.equal(res.code, 0, res.err);
|
||||||
|
assert.equal(await readFile(join(out, "index.html"), "utf8"), "<h1>hi</h1>");
|
||||||
|
await assert.rejects(stat(join(out, "public/img.png")));
|
||||||
|
assert.match(res.err, /skipped/);
|
||||||
|
});
|
||||||
|
});
|
||||||
|
|
||||||
|
test("unpack: refuses path traversal", async () => {
|
||||||
|
await withTmp(async (dir) => {
|
||||||
|
const doc = { files: { "../escape.txt": { type: "text", content: "x" } } };
|
||||||
|
const out = join(dir, "out");
|
||||||
|
const res = await run(["unpack", "-", out], { stdin: JSON.stringify(doc) });
|
||||||
|
assert.notEqual(res.code, 0);
|
||||||
|
assert.match(res.err, /outside output dir/);
|
||||||
|
});
|
||||||
|
});
|
||||||
|
|
||||||
|
test("unpack: fails clearly on a queued job with no files", async () => {
|
||||||
|
await withTmp(async (dir) => {
|
||||||
|
const doc = { jobId: "job_9", status: "queued" };
|
||||||
|
const out = join(dir, "out");
|
||||||
|
const res = await run(["unpack", "-", out], { stdin: JSON.stringify(doc) });
|
||||||
|
assert.notEqual(res.code, 0);
|
||||||
|
assert.match(res.err, /queued job/);
|
||||||
|
});
|
||||||
|
});
|
||||||
|
|
||||||
|
test("unpack: reports sha256 mismatch as a failure", async () => {
|
||||||
|
await withTmp(async (dir) => {
|
||||||
|
const doc = { files: { "a.txt": { type: "text", content: "hi", sha256: "deadbeef" } } };
|
||||||
|
const out = join(dir, "out");
|
||||||
|
const res = await run(["unpack", "-", out], { stdin: JSON.stringify(doc) });
|
||||||
|
assert.notEqual(res.code, 0);
|
||||||
|
assert.match(res.err, /integrity check/);
|
||||||
|
});
|
||||||
|
});
|
||||||
|
|
||||||
|
test("help: exits 0 and prints usage", async () => {
|
||||||
|
const res = await run(["--help"]);
|
||||||
|
assert.equal(res.code, 0);
|
||||||
|
assert.match(res.out, /ditto unpack/);
|
||||||
|
});
|
||||||
Reference in New Issue
Block a user