Samraaj BathandClaude Fable 5
1ad0f28f48
The 2026-07-06 prod outage was a missed manual migration: the 0002 job_events migration never ran against prod, so the worker crash-looped on every job. Make the deploy pipeline own it: - railway.json sets `npm run db:migrate` as the pre-deploy command. Railway runs it with the service's env before starting the new deployment; a failed migration fails the deploy and the old version keeps serving. - runMigrations() now serializes on a Postgres advisory lock so api and worker pre-deploys firing off the same push can't race Drizzle's journal writes. - DEPLOY.md: document the automatic path; keep the manual command for first-time setup. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
104 lines
4.2 KiB
Markdown
104 lines
4.2 KiB
Markdown
# Deploy — Railway + Neon + R2
|
||
|
||
The service is two long-lived processes (an **api** and a **worker**) plus managed
|
||
Postgres and S3-compatible storage. The recommended OSS-friendly stack:
|
||
|
||
- **Railway** — hosts the `api` and `worker` services (one Railway service each,
|
||
built from the Dockerfiles in [`docker/`](../docker)).
|
||
- **Neon** — managed Postgres (the database *and* the pg-boss queue).
|
||
- **Cloudflare R2** (or AWS S3) — blob storage for binary assets + bundles. R2 has
|
||
no egress fees, friendly for forks/self-host.
|
||
|
||
## 1. Database (Neon)
|
||
|
||
1. Create a Neon project; copy the pooled connection string.
|
||
2. Migrations run **automatically before every Railway deploy**: the repo's
|
||
`railway.json` sets `npm run db:migrate` as the pre-deploy command, which
|
||
Railway executes (with the service's env, so `DATABASE_URL` is present)
|
||
before starting the new deployment. Re-running is a no-op for anything
|
||
already applied, and concurrent runs (api + worker deploying off the same
|
||
push) serialize on a Postgres advisory lock. A failed migration fails the
|
||
deploy — the old version keeps serving.
|
||
|
||
To apply migrations manually (first-time setup, or outside Railway):
|
||
```bash
|
||
DATABASE_URL='postgres://...neon.../ditto_site?sslmode=require' npm run db:migrate
|
||
```
|
||
(Alternatively, apply a single migration by piping its SQL through
|
||
`psql`/the Neon CLI — but prefer `db:migrate`, which also records the
|
||
journal entry so future runs stay consistent.)
|
||
|
||
Current migrations: `0000` core tables, `0001` signup tokens,
|
||
`0002` `job_events` (structured clone-progress events behind
|
||
`GET /v1/clones/:id/events`).
|
||
|
||
## 2. Blob storage (R2 / S3)
|
||
|
||
Create a bucket (e.g. `ditto-site-artifacts`) and an access key. Note the S3 API
|
||
endpoint (R2: `https://<accountid>.r2.cloudflarestorage.com`).
|
||
|
||
## 3. Railway services
|
||
|
||
Create two services in one Railway project, both deploying this repo:
|
||
|
||
| Service | Dockerfile | Scaling |
|
||
|---|---|---|
|
||
| `api` | `docker/api.Dockerfile` | light; autoscale on CPU |
|
||
| `worker` | `docker/worker.Dockerfile` | scale by queue depth (Playwright image) |
|
||
|
||
Set these variables on **both** services:
|
||
|
||
```
|
||
DATABASE_URL=postgres://...neon.../ditto_site?sslmode=require
|
||
S3_BUCKET=ditto-site-artifacts
|
||
S3_ENDPOINT=https://<accountid>.r2.cloudflarestorage.com # omit for AWS S3
|
||
S3_REGION=auto # 'auto' for R2
|
||
S3_ACCESS_KEY_ID=…
|
||
S3_SECRET_ACCESS_KEY=…
|
||
```
|
||
|
||
API-only:
|
||
|
||
```
|
||
PORT=8787
|
||
PUBLIC_BASE_URL=https://api.yourdomain.com # so MCP-returned URLs are absolute
|
||
API_KEYS=key_live_xxx,key_live_yyy # require auth
|
||
RATE_LIMIT_PER_MINUTE=60
|
||
SIGNUP_ENABLED=true # optional: public API-key minting
|
||
SIGNUP_RATE_LIMIT_PER_HOUR=3
|
||
DEFAULT_SIGNUP_KEY_RATE_LIMIT=30
|
||
SIGNUP_DIRECT_ENABLED=false # recommended once verified email is configured
|
||
RESEND_API_KEY=re_...
|
||
SIGNUP_FROM_EMAIL=Ditto <hello@ditto.site> # must be a verified Resend sender/domain
|
||
SIGNUP_VERIFY_URL=https://www.ditto.site/api-key
|
||
SIGNUP_TOKEN_TTL_MINUTES=30
|
||
SIGNUP_CORS_ORIGINS=https://ditto.site,https://www.ditto.site # browser origins allowed to call signup routes
|
||
# SSRF is on by default; do NOT set SSRF_ALLOW_LOOPBACK in prod.
|
||
```
|
||
|
||
Worker-only:
|
||
|
||
```
|
||
CACHE_STALE_AFTER=24h
|
||
ARTIFACTS_DIR=/data/artifacts # only used if S3 is not configured
|
||
HARNESS_DIR=/data/harness # isolated build harness for verify
|
||
```
|
||
|
||
## 4. Scaling notes
|
||
|
||
- **Capture workers** parallelize freely (each clone = one headless Chromium,
|
||
~0.5–1 GB peak); scale horizontally by queue depth.
|
||
- **Verify** is the heavy step (`next build`). Each worker container has its own
|
||
filesystem, so its `HARNESS_DIR` is naturally isolated; for multiple workers on
|
||
one host, give each a distinct `HARNESS_DIR`.
|
||
- Per-job wall-clock caps and retries are handled by pg-boss; a `compilerVersion`
|
||
bump invalidates the cache automatically.
|
||
|
||
## 5. CI
|
||
|
||
[`.github/workflows/ci.yml`](../.github/workflows/ci.yml) typechecks and runs the
|
||
test suite on every push/PR with a Postgres service container + Chromium installed,
|
||
so the DB and browser-gated tests run in CI. Heavy real-site benchmarks remain a
|
||
manual/nightly concern; benchmark lists live in
|
||
[`compiler/benchmarks/`](../compiler/benchmarks).
|