ADR 0022: Leave Railway for Render + 1Password brokerCopy Anchor Link

Numbering and status (2026-10-25). Renumbered from 0016 to 0022 when #232 merged: ADR 0016 is "approved stack required" and the runtime decision is ADR 0017 "Render runtime" (Samir, 2026-10-24 01:36 EDT: "Render is the way"); 0018-0021 are taken by other open branches. This file is the Railway-exit record and the broker pilot. The current plan, cutover order (Construct first), costs and the Needs Samir list are in docs/runbooks/render-cutover/README.md, which supersedes the D0-D2 order below.

• Status: proposed (draft PR; nothing deployed, no account created, no DNS changed)
• Date: 2026-10-22 (revised 2026-10-22 22:00 EDT with Samir's decisions: Render, Observatory staging-only, Interspace last; recompressed 2026-10-22 22:43 EDT to 2-3 days: Samir, "move faster ... crank this out in the next 2-3 days or less", while the Render Pro account is being created)
• Decided by: Samir (owner). Host decided: Render. The pilot settles only the broker question.
• Relates to: ADR 0014 (Exo Secure: secrets from 1Password at boot), ADR 0015 (key rotation, on the stacked rotation PR), Construct's own Render plan (construct PR #406), runbook render-pilot-exo-staging.md

ContextCopy Anchor Link

Railway disclosed on 2026-10-21 that an attacker read decrypted customer environment variables and paused deploys (disclosure). ADR 0014 already moves EXO's secrets into 1Password, but on Railway the one bootstrap token (OP_SERVICE_ACCOUNT_TOKEN) still sits in the same env store that leaked, and Railway has no workload identity (OIDC) to replace it. Every Interspace app is hosted there: EXO, arch-router, Observatory, Cosmograph, Universe, Construct (which also serves Interspace at interspace.ventures), Hadron/Dropship and Moonshot.

The goal Samir asked for: leave Railway, and ideally reach a host where no long-lived secret is stored at all, using the 1Password Credential Broker (workload OIDC token in, 1Password Environment variables out).

What is true today (2026-10-22):

So a Render service can probably not get a token 1Password will accept. That is unverified and is pilot test #1. It is cheap to settle: when an admin creates a custom OIDC integration, 1Password shows the expected audience. If it is not one of Render's four, the broker cannot run on Render.

DecisionCopy Anchor Link

1. Render is the host (decided by Samir, 2026-10-22). Every app that stays with Interspace moves from Railway to Render, all staging environments first (D0), then the productions (D1-D2), then Interspace (D2-D3), in 2-3 days (timeline below). Render because it is the fastest exit: Blueprints in git, PR previews, health-checked zero-downtime deploys, Docker and native Node, Pro-plan OIDC for AWS, OpenAI and Anthropic, and Construct already has a Render plan (#406). For the record: the broker fits Fly.io or Cloud Run better, because they mint OIDC tokens for any audience.
2. Observatory moves staging only; it has no production cutover. Observatory production (observatory.bio, Railway) is retired after Samir hands the code to his dad. See "Observatory" below. Nothing is deleted without Samir's explicit yes.
3. Ship on the fallback, not the broker. Each app environment gets one read-only 1Password service account token scoped to that app's vault for that environment (OP_SERVICE_ACCOUNT_TOKEN, Render sync: false). That is ADR 0014 unchanged, moved to a host that has not just leaked its env store, with tokens minted fresh (never copied from Railway).
4. The broker is a pilot item, not a dependency. The loader learns EXO_SECURE_SOURCE=broker|auto now (src/lib/exo-secure/broker.ts), so trying it later is a config change. auto tries broker, then service account, then file (development) or env. Nobody buys 1Password Business until pilot test #1 says Render can present the right token.
5. Blueprints are the only way services are defined. render.yaml holds no secret values, only sync: false declarations. EXO Verify's new render-blueprint check fails the gate if one does.
6. The pilot is EXO staging, not Observatory staging. EXO staging already has the Exo Secure loader (#225) and its Blueprint (this PR), so it can deploy on D0; Observatory staging would first need a loader-port PR in another repo, which puts it on the critical path of a three-day staging window. The stakes stay low: the pilot runs on exo-staging.onrender.com while staging.exo.now keeps pointing at Railway until go/no-go. Runbook: render-pilot-exo-staging.md.

Pushback, stated plainlyCopy Anchor Link

• Render does not deliver "no stored secret". With the broker unavailable, each service still stores one long-lived secret. It is read-only and scoped to one vault, and Render is not the host that just leaked, but it is the same shape of risk we had on Railway. Moving hosts is not the security fix; rotation is.
• Rotation matters more than the move. Every value that was on Railway is compromised. Copying them to Render unrotated carries the breach with us. Rotate into 1Password first (ADR 0015 and the secrets inventory), then point Render at the vault.
• Paying for a preview. Business at $8.99/user/mo is cheap, but it buys a public-preview broker whose custom-OIDC path is beta, for a team of one. Do not upgrade on spec.
• Two to three days is very tight. Render is not the slow part. Secret rotation (every value must be new before its vault is filled), the Universe Postgres move, DNS, and the repo gap for Moonshot, Hadron and Dropship (parallel track below) are. The plan holds by: staging all on D0 behind onrender.com URLs, TTLs at 300 s from D0, no app's 7-day rollback window ever blocking the next cutover, "red smoke test stops that app, not the line", and production on a weekend morning (lowest traffic, but slower vendor support). The real risk is rotation: an app whose vault is not filled with new values simply waits; it never ships with the leaked ones.
• Render does remove some keys entirely. Its managed OIDC for OpenAI and Anthropic means an app that calls those APIs directly can drop OPENAI_API_KEY / ANTHROPIC_API_KEY from its vault once its client supports workload identity. That is a real win the broker was never going to give us.

Alternatives consideredCopy Anchor Link

CostCopy Anchor Link

Render Pro workspace $25/mo once, shared by every app, plus compute per service. Plans: starter 0.5 CPU / 512 MB $7, standard 1 CPU / 2 GB $25. Sizes below are estimates, not benchmarks; arch-router at 1 CPU / 2 GB especially (CPU only; Render has no GPUs). Pro includes 15 custom domains, enough for every hostname below.

Plus, outside Render: PR previews at $7 each while open (manual generation, so near zero); Neon for Universe (Free or Launch; usage-based); 1Password stays Families (no change) unless the broker pilot passes, then Business at $8.99/user/mo for one seat. Render Key Value for Hadron's Valkey only if it turns out to be needed (price to check). Observatory production keeps its Railway bill until it is retired.

Break-even and margin:

• Against Railway: Railway's actual bill could not be pulled (the Railway API token was invalidated after the incident). Fill in last month's invoice here before approving; if it was under ~$215/mo, Render costs more and we are paying for the exit, not saving.
• Against revenue: Cosmograph is the only app with prices set ($9.99/yr or $29.99 lifetime). Its $35.13/mo ($422/yr) breaks even at about 43 annual members or 15 lifetime sales a year, before payment fees. Every other app is pre-revenue; their hosting is pure cost.
• Cheapest honest variant: all staging on starter, prod on standard only for EXO and arch-router, the rest starter until load says otherwise: about $145/mo. Starter (512 MB) is tight for a Next build server; watch memory in the pilot before choosing it for production.

Migration orderCopy Anchor Link

Staging first, all of it, on D0; then production; Interspace last. Record-by-record DNS: render-cutover/dns-changes.md. Vaults and items: render-cutover/vaults-and-items.md. D0 commands: render-cutover/commands-d0.md. Samir's batched yes list: render-cutover/approval-list.md.

Staging (D0)

1. EXO staging (pilot, first) — root render.yaml in this PR; runbook docs/runbooks/render-pilot-exo-staging.md. Proves Blueprint, 1Password service-account boot, health check and previews on exo-staging.onrender.com; pilot test #1 is recorded but no longer gates the rest. Go for the others after two healthy deploys (about an hour), then staging.exo.now flips the same day. The staging gateway's *_STAGING_ORIGINs move to Render URLs as each app below lands.
2. arch-router — Docker web service (Convex calls it from outside Render, so not a private service). llama-server is a C++ binary, so its key arrives through an op run wrapper entrypoint (child-process env only, ADR 0014's documented exception); that entrypoint and its Blueprint entry are a small follow-up PR, not in this one. One instance shared by staging and production: staging Convex's ARCH_ROUTER_URL moves on D0, production Convex's with EXO production (D1). The Railway instance keeps serving production until then.
3. Cosmograph staging — Neon staging branch stays; new role password.
4. Universe staging — Railway Postgres staging -> Neon staging branch (below).
5. Construct staging — per construct PR #406 and tonight's render.yaml PR; staging.construct.page and staging.construct.codes flip on D0.
6. Observatory staging — with its loader-port and render.yaml PR (opening tonight). No DNS change: it sits behind staging.exo.now/observatory. Staging only; see "Observatory".

Production (D1-D2)

7. EXO production (D1) — exo.now apex and www. Production Convex ARCH_ROUTER_URL -> Render.
8. Construct production (D1) — construct.codes, www.construct.codes, construct.page. It must not take over interspace.ventures yet: Railway's Construct production keeps running and serving Interspace (and, unless Samir moves them on D2, structured.glass, foundation.financial, bumblebee.nyc).
9. Cosmograph production (D2) — cosmograph.space (+ a new www).
10. Universe production (D2) — Railway Postgres -> Neon production, then release.universe.audio (new, the canary) and universe.audio (today an old Replit build at Google Frontend, not Railway).

Last (D2-D3, ~06:00 ET)

11. Interspace — interspace.ventures + www, served by Construct production on Render. Zero-downtime cutover below.

Parallel track (not on the critical path): Moonshot, Hadron and Dropship need git repos first. See "Parallel track".

Timeline (2-3 days)Copy Anchor Link

D0 is the day the Render account exists: tonight (Thu 2026-10-22) or Fri 2026-10-23. Dates below assume D0 = Fri 10-23; if it lands later every row slides by the same amount. Times are ET.

Rules: one hostname group at a time, each with Samir's yes. A red smoke test stops that app and rolls it back; it does not stop apps that are already green or unrelated. Interspace never goes while any other production cutover is red. An app whose vault does not yet hold freshly rotated values waits; it never ships with Railway's values.

ObservatoryCopy Anchor Link

• Staging only on Render. observatory-staging moves on D0 with its Exo Secure loader port. It never gets a Render production service, an Observatory Production vault, or a production service account.
• No production cutover. Observatory production (observatory.bio, Railway) keeps running on Railway until Samir hands the code to his dad, then it is retired. No observatory.bio DNS record changes as part of this ADR.
• Nothing is deleted without Samir's explicit yes. That covers the Railway Observatory production project, its database credential, the observatory.bio records (including _railway-verify), and the Railway Observatory staging project after its 7-day window. Retirement is a separate step Samir starts, after the handoff.
• Until retirement, production still runs on the leaked host. Its Railway env holds exposed values. Recommendation: rotate its provider keys in place (OpenAI, OpenRouter, Resend, intake secret) so the exposed ones stop working, even though the app is leaving. Samir's call.
• Cost drops to one starter instance on Render ($7/mo plus the Pro share).

What stays in ConvexCopy Anchor Link

Nothing about Convex moves. Convex functions, data, file storage, Exo Key (Exo ID) and Convex Auth, and every Convex-side secret stay in the Convex deployment env exactly as ADR 0014's migration table says: BIRD_API_KEY, EXO_ID_PRIVATE_KEY, AUTH_PRIVATE_KEY, AUTH_*_CLIENT_SECRET, ARCH_ROUTER_KEY, ARTIFICIAL_ANALYSIS_API_KEY, EXO_AI_BROKER_TOKEN, EXO_INTAKE_SECRET_*, EXO_VERIFY_TOKEN (verifier side), EXO_GATEWAY_CLIENT_SECRET_SHA256. Convex pushes keep running from GitHub Actions, never from a Render build. The only Convex edits the move needs are non-secret URLs: ARCH_ROUTER_URL and any *_STAGING_ORIGIN or app origin Convex calls.

Postgres and NeonCopy Anchor Link

Neon is the Postgres home: it is host-independent, branches per environment, and Cosmograph already uses it. No Render Postgres.

DNS cutover at SpaceshipCopy Anchor Link

All six domains are hosted at Spaceship: exo.now, observatory.bio, cosmograph.space, construct.page, interspace.ventures, moonshot.supply. Today: exo.now apex A 69.46.46.38 (Railway), www CNAME vlc5968k.up.railway.app, staging CNAME jai6l6x5.up.railway.app; _railway-verify TXT on exo.now, staging.exo.now and observatory.bio.

Hostnames that move: staging.exo.now, exo.now + www, cosmograph.space, construct.page, interspace.ventures (last). observatory.bio does not move (Observatory is staging only); moonshot.supply waits for the parallel track.

Per hostname, with Samir's approval at each flip:

1. T-24 h: record the current value of every record you will touch (screenshot or export), then lower its TTL to 300 s.
2. Add the domain in Render (dashboard service -> Settings -> Custom Domains, or POST /v1/services/{serviceId}/custom-domains). Render shows the target.
3. Check CAA and AAAA: if the domain has CAA records, add issue and issuewild for letsencrypt.org and pki.goog; remove any AAAA records (Render is IPv4) (render.com/docs/custom-domains). Adding an apex in Render also adds www as a redirect to it, and the reverse.
4. Flip at Spaceship:
   • Subdomain (staging, www): replace the Railway CNAME with CNAME <service>.onrender.com.
   • Apex: use ALIAS/ANAME <service>.onrender.com if Spaceship offers it for the zone; otherwise replace A 69.46.46.38 with A 216.24.57.1.
5. Verify in Render (POST /v1/services/{serviceId}/custom-domains/{id}/verify), wait for the certificate, then run the smoke tests against the real hostname.
6. Leave _railway-verify TXT records in place during the soak (harmless; needed for rollback), delete them when the Railway project is deleted.
7. Raise TTLs back to 3600 s after 24 h green.

Interspace (interspace.ventures): zero-downtime cutover, lastCopy Anchor Link

Interspace (interspace.ventures) is served by the root Construct app (not services/interspace, which is staging-only) and must stay up. The rule: both origins serve the site for the whole flip, so whichever answer a resolver holds, it gets a working page.

1. D0 (Samir's yes): export every interspace.ventures record. Lower the TTL of the apex and www records to 300 s, at least 24 h before the flip. Touch nothing else: MX, SPF, DKIM, DMARC and site-verification TXT records stay exactly as they are.
2. D1-D2: Construct production is live on Render and construct.codes has been green for at least 12 h. Add interspace.ventures (Render adds www) as a custom domain on the Render Construct production service. Add CAA entries for letsencrypt.org and pki.goog if the zone has CAA. Remove AAAA records only at flip time, and only if any exist.
3. Verify before flipping: on the onrender.com URL, check the Interspace routes Construct serves for that host, e.g. curl -fsS -H 'Host: interspace.ventures' https://<construct-prod>.onrender.com/ and the same for the key pages and /api/health. Compare with the live Railway page (status, title, canonical links, assets). Construct's host routing must accept the onrender.com host or the Host override; if it does not, stop and fix that before the flip.
4. D2 or D3, ~06:00 ET, flip: apex to ALIAS/ANAME <construct-prod>.onrender.com (or A 216.24.57.1), www to CNAME <construct-prod>.onrender.com. Keep Railway's Construct running and serving interspace.ventures throughout. Immediately press Verify in Render and flush the Google and Cloudflare public DNS caches.
5. Watch for 30 minutes: curl the real hostname every minute from two networks; check the certificate issuer and expiry; check /api/health.
6. Known gap, unverified: Render documents issuing the TLS certificate only after DNS verifies, so a resolver that picks up the new record in the first minutes may see a TLS error or a 502. On D0, ask Render support whether they can pre-issue the certificate (as they do for wildcard domains via an _acme-challenge CNAME). If they cannot, the 300 s TTL and an early-morning flip keep the window to minutes; if the certificate is not issued within 10 minutes, put the records back. (interspace.ventures already has an _acme-challenge TXT of unknown origin; a Render _acme-challenge CNAME would need it removed first.)
7. After 24 h green: raise TTLs to 3600 s. Railway's Construct stays paused, not deleted, for 7 days.

RollbackCopy Anchor Link

• Every service gets a 7-day rollback window after its own cutover, staging and production alike: the Railway service is paused, not deleted, env untouched (it is all compromised and rotated anyway, so it is only useful for the rollback itself). The window runs in the background and does not block the next app's cutover.
• To roll back a hostname: set the Spaceship record back to its recorded Railway value (A 69.46.46.38, CNAME vlc5968k.up.railway.app / jai6l6x5.up.railway.app, and so on), resume the Railway service, and wait out the 300 s TTL. If the rotated secrets differ, point the Railway service at the same 1Password vault (it reads the same items through its own, freshly minted read-only token) rather than pasting values.
• universe.audio rolls back to its current Replit origin (A 34.111.179.208, replit-verify TXT kept), not to Railway, which never served it. Keep that Replit deployment untouched until Universe production has been green for 7 days.
• Rolling back one app does not roll back the others. The exception is arch-router: if it rolls back, set both Convex deployments' ARCH_ROUTER_URL back to the Railway instance.
• Interspace rolls back the same way, and stays rollback-ready longest because Railway's Construct keeps serving it until the day after its flip.
• A Render deploy that fails its health check never takes traffic; the previous deploy keeps serving. Production also fails closed when 1Password cannot be read.
• Nothing is deleted without Samir's explicit yes: no Railway project, database, credential or DNS record, after 7 green days or ever. Observatory production is not on this clock at all; it is retired only after the handoff (see "Observatory").

Parallel track: repos for Moonshot, Hadron and DropshipCopy Anchor Link

Render builds only from a git repo. Three apps have none it can use, so they run alongside the main week and never block it:

Samir must OK creating each repo (name, visibility). Nobody creates a repo, a Render service or a DNS record for these on their own. Each moves to Render staging whenever its repo exists, and stays on Railway until then; moonshot.supply does not change in this ADR's week.

How each app receives its secretsCopy Anchor Link

One rule everywhere: the host stores one read-only, vault-scoped bootstrap token per app per environment, as Render sync: false. Nothing else secret lives in Render env, in render.yaml, or in a build. Builds need no secret (and on Docker builds there is no OIDC anyway); never declare a Docker ARG for a secret. PR previews inherit staging's token and vault, never production's.

If pilot test #1 ever passes, that app's bootstrap changes from OP_SERVICE_ACCOUNT_TOKEN to OP_INTEGRATION_KEY + OP_WORKLOAD_ID + EXO_SECURE_ENVIRONMENT_ID with EXO_SECURE_SOURCE=broker (or auto), and its secrets move from vault items to a 1Password Environment. OP_INTEGRATION_KEY is still a stored secret, but it is useless without a fresh token from the host's issuer that matches 1Password's trust policy.

1Password for AgentsCopy Anchor Link

1Password for Agents, launched 2026-10-22, lets a personal AI agent ask to use a login from Samir's vault; he approves, and 1Password fills the sign-in form without the agent ever seeing the password. It is free on Individual and Families, so it costs nothing to turn on. It is about people's sign-ins, not servers: it does not hand secrets to a running workload, so it replaces neither the service-account token nor the broker. Where it helps here is the migration itself: the agent doing Render, Spaceship and Neon dashboard steps can sign in with Samir's approval instead of a saved password, which shrinks what an agent session could leak. Treat it as an operator-side improvement, separate from this decision.

ConsequencesCopy Anchor Link

• Railway leaves the critical path in 2-3 days from D0 (all staging on D0, productions D1-D2, Interspace D2-D3, last). Railway projects are deleted, with Samir's yes, no sooner than 7 days after each service's cutover. Observatory production stays on Railway until it is retired after the handoff.
• Hosting costs about $217/mo at the sizes above (about $145/mo lean).
• The stored-secret risk shrinks (one read-only, vault-scoped, freshly minted token per app env) but does not reach zero on Render.
• EXO_SECURE_SOURCE gains broker and auto; the default stays env until each app's cutover sets auto.
• EXO Verify fails any render.yaml that holds a secret value; OP_INTEGRATION_KEY, OP_WORKLOAD_ID, EXO_SECURE_ENVIRONMENT_ID and EXO_SECURE_OIDC_TOKEN_FILE join the platform-env allow list (the first is a bootstrap secret, the rest are identifiers).

Open items for the ownerCopy Anchor Link

One batched, numbered list with a link per item: render-cutover/approval-list.md. In short: rotate the exposed keys before reconnecting anything; Render GitHub App and API key; the 10 vaults and their read-only service accounts; merging #225, #227 and the per-repo render.yaml PRs; Convex device approvals; the DNS changes by day (D0 TTLs + staging, D1 EXO and Construct, D2 Cosmograph and Universe, D2-D3 Interspace); the extra Construct domains; Render's custom-domain allowance; the Moonshot and Hadron repos; and, later, each deletion (Railway after its 7 days, Observatory production after the handoff). Still open beyond that list: Railway's last invoice (break-even), and 1Password Business only if pilot test #1 ever passes.