# 0022: Leave Railway for Render + 1Password broker https://exo.now/docs/adr/0022-leave-railway-for-render-and-1password-broker > Status: proposed (draft PR; nothing deployed, no account created, no DNS changed) · Date: 2026-10-22 (revised 2026-10-22 22:00 EDT with Samir's decisions: Render, Observatory # ADR 0022: Leave Railway for Render + 1Password broker [#adr-0022-leave-railway-for-render--1password-broker] > **Numbering and status (2026-10-25).** Renumbered from 0016 to **0022** when #232 merged: ADR > 0016 is "approved stack required" and the runtime decision is **ADR 0017 "Render runtime"** > (Samir, 2026-10-24 01:36 EDT: "Render is the way"); 0018-0021 are taken by other open branches. > This file is the Railway-exit record and the broker pilot. The current plan, cutover > order (Construct first), costs and the Needs Samir list are in > [`docs/runbooks/render-cutover/README.md`](https://github.com/Interspace-Ventures/exo/blob/staging/docs/runbooks/render-cutover/README.md), which supersedes > the D0-D2 order below. * Status: proposed (draft PR; nothing deployed, no account created, no DNS changed) * Date: 2026-10-22 (revised 2026-10-22 22:00 EDT with Samir's decisions: Render, Observatory staging-only, Interspace last; **recompressed 2026-10-22 22:43 EDT** to 2-3 days: Samir, "move faster ... crank this out in the next 2-3 days or less", while the Render Pro account is being created) * Decided by: Samir (owner). Host decided: Render. The pilot settles only the broker question. * Relates to: [ADR 0014](https://exo.now/docs/adr/0014-exo-secure-secrets-from-1password) (Exo Secure: secrets from 1Password at boot), ADR 0015 (key rotation, on the stacked rotation PR), Construct's own Render plan (construct PR #406), runbook [render-pilot-exo-staging.md](https://github.com/Interspace-Ventures/exo/blob/staging/docs/runbooks/render-pilot-exo-staging.md) ## Context [#context] Railway disclosed on 2026-10-21 that an attacker read decrypted customer environment variables and paused deploys ([disclosure](https://blog.railway.com/p/security-disclosure-wednesday-october-21st)). ADR 0014 already moves EXO's secrets into 1Password, but on Railway the one bootstrap token (`OP_SERVICE_ACCOUNT_TOKEN`) still sits in the same env store that leaked, and Railway has no workload identity (OIDC) to replace it. Every Interspace app is hosted there: EXO, arch-router, Observatory, Cosmograph, Universe, Construct (which also serves Interspace at `interspace.ventures`), Hadron/Dropship and Moonshot. The goal Samir asked for: leave Railway, and ideally reach a host where no long-lived secret is stored at all, using the **1Password Credential Broker** (workload OIDC token in, 1Password Environment variables out). What is true today (2026-10-22): | Fact | Source | | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------- | | Credential Broker is a public preview for **1Password Business only**. Custom OIDC workflows are beta and **JS SDK only**. | [1password.dev/brokered-access](https://www.1password.dev/brokered-access), [custom workflow](https://www.1password.dev/brokered-access/custom-workflow) | | A broker workload still holds `OP_INTEGRATION_KEY` (secret), `OP_WORKLOAD_ID` and an Environment ID. The admin's integration fixes the expected audience; the workload must present a token for it. SDK 0.5.0 asks for it via `oidcFetcher(audience)`. | same; `@1password/sdk` 0.5.0 types (`ClientConfiguration.oidcFetcher`, `WorkloadDetails`) | | Samir is on 1Password **Families**. Business is **$8.99/user/mo**. Families service accounts: 1,000 reads/hour per token, 1,000/day per account (Teams 5,000/day, Business 50,000/day). | 1Password pricing and service-account limits | | Render OIDC needs a **Pro** workspace ($25/mo) plus compute ($7 for 0.5 CPU/512 MB, $25 for 1 CPU/2 GB). | [render.com/pricing](https://render.com/pricing), [platform features by plan](https://render.com/docs/platform-features-by-plan) | | Render mints tokens only for **provider audiences**: AWS STS (`sts.amazonaws.com`, via `AWS_ROLE_ARN` -> `AWS_WEB_IDENTITY_TOKEN_FILE`), OpenAI (`api.openai.com`), Anthropic (`api.anthropic.com`), GCP. Issuer `https://oidc.render.com/{tea-…}`. **No OIDC at build time for Dockerfile builds.** | [render.com/docs/oidc](https://render.com/docs/oidc) | | Fly.io and Cloud Run mint OIDC tokens for **any** audience. | Fly.io `/.fly/api` OIDC tokens; Cloud Run metadata server identity tokens | | 1Password for Agents (launched 2026-10-22) is consumer sign-in approval for personal agents; free on Individual and Families. It does not deliver workload secrets. | [1Password blog](https://1password.com/blog/1password-for-ai-agents) | So a Render service can probably **not** get a token 1Password will accept. That is unverified and is pilot test #1. It is cheap to settle: when an admin creates a custom OIDC integration, 1Password shows the expected audience. If it is not one of Render's four, the broker cannot run on Render. ## Decision [#decision] 1. **Render is the host (decided by Samir, 2026-10-22).** Every app that stays with Interspace moves from Railway to Render, all staging environments first (D0), then the productions (D1-D2), then Interspace (D2-D3), in **2-3 days** (timeline below). Render because it is the fastest exit: Blueprints in git, PR previews, health-checked zero-downtime deploys, Docker and native Node, Pro-plan OIDC for AWS, OpenAI and Anthropic, and Construct already has a Render plan (#406). For the record: the broker fits Fly.io or Cloud Run better, because they mint OIDC tokens for any audience. 2. **Observatory moves staging only; it has no production cutover.** Observatory production (`observatory.bio`, Railway) is retired after Samir hands the code to his dad. See "Observatory" below. Nothing is deleted without Samir's explicit yes. 3. **Ship on the fallback, not the broker.** Each app environment gets one **read-only 1Password service account token** scoped to that app's vault for that environment (`OP_SERVICE_ACCOUNT_TOKEN`, Render `sync: false`). That is ADR 0014 unchanged, moved to a host that has not just leaked its env store, with tokens minted fresh (never copied from Railway). 4. **The broker is a pilot item, not a dependency.** The loader learns `EXO_SECURE_SOURCE=broker|auto` now (`src/lib/exo-secure/broker.ts`), so trying it later is a config change. `auto` tries broker, then service account, then file (development) or env. Nobody buys 1Password Business until pilot test #1 says Render can present the right token. 5. **Blueprints are the only way services are defined.** `render.yaml` holds no secret values, only `sync: false` declarations. EXO Verify's new `render-blueprint` check fails the gate if one does. 6. **The pilot is EXO staging, not Observatory staging.** EXO staging already has the Exo Secure loader (#225) and its Blueprint (this PR), so it can deploy on D0; Observatory staging would first need a loader-port PR in another repo, which puts it on the critical path of a three-day staging window. The stakes stay low: the pilot runs on `exo-staging.onrender.com` while `staging.exo.now` keeps pointing at Railway until go/no-go. Runbook: [render-pilot-exo-staging.md](https://github.com/Interspace-Ventures/exo/blob/staging/docs/runbooks/render-pilot-exo-staging.md). ### Pushback, stated plainly [#pushback-stated-plainly] * **Render does not deliver "no stored secret".** With the broker unavailable, each service still stores one long-lived secret. It is read-only and scoped to one vault, and Render is not the host that just leaked, but it is the same shape of risk we had on Railway. Moving hosts is not the security fix; rotation is. * **Rotation matters more than the move.** Every value that was on Railway is compromised. Copying them to Render unrotated carries the breach with us. Rotate into 1Password first (ADR 0015 and the secrets inventory), then point Render at the vault. * **Paying for a preview.** Business at $8.99/user/mo is cheap, but it buys a public-preview broker whose custom-OIDC path is beta, for a team of one. Do not upgrade on spec. * **Two to three days is very tight.** Render is not the slow part. Secret rotation (every value must be new before its vault is filled), the Universe Postgres move, DNS, and the repo gap for Moonshot, Hadron and Dropship (parallel track below) are. The plan holds by: staging all on D0 behind `onrender.com` URLs, TTLs at 300 s from D0, no app's 7-day rollback window ever blocking the next cutover, "red smoke test stops that app, not the line", and production on a weekend morning (lowest traffic, but slower vendor support). The real risk is rotation: an app whose vault is not filled with new values simply waits; it never ships with the leaked ones. * **Render does remove some keys entirely.** Its managed OIDC for OpenAI and Anthropic means an app that calls those APIs directly can drop `OPENAI_API_KEY` / `ANTHROPIC_API_KEY` from its vault once its client supports workload identity. That is a real win the broker was never going to give us. ### Alternatives considered [#alternatives-considered] | Option | Verdict | | ------------------------------------------- | ------------------------------------------------------------------------------------------------------------------ | | Stay on Railway, rotate, harden | Rejected. No workload identity; the bootstrap token stays in the store that leaked; deploys were paused. | | **Render + service-account token (chosen)** | Fastest exit, Blueprints, previews. One read-only token per app env. Broker probably impossible (audience). | | Render + Credential Broker | Blocked on audience (pilot #1) and Business plan. Kept behind `EXO_SECURE_SOURCE=broker`. | | Fly.io + Credential Broker | Not chosen. More ops (`fly.toml`, machines); no Blueprint-style previews. | | Cloud Run + Credential Broker | Not chosen. Heaviest migration (GCP project, IAM, Artifact Registry). | | Vercel | Fine for Next front ends, wrong for arch-router (long-running CPU model) and Express APIs; secrets still env vars. | | Cloudflare (Workers/Containers) | Rewrites for Node-specific servers; not a one-week move. | ## Cost [#cost] Render Pro workspace **$25/mo** once, shared by every app, plus compute per service. Plans: `starter` 0.5 CPU / 512 MB **$7**, `standard` 1 CPU / 2 GB **$25**. Sizes below are estimates, not benchmarks; arch-router at 1 CPU / 2 GB especially (CPU only; Render has no GPUs). Pro includes 15 custom domains, enough for every hostname below. | App | Staging | Production | Compute / mo | + Pro share (1/8) | Per app / mo | | ------------------------ | --------------------- | ----------------------------------------------------- | ------------ | ----------------- | ----------------------- | | EXO | standard $25 | standard $25 | $50 | $3.13 | **$53.13** | | arch-router | (one shared instance) | standard $25 | $25 | $3.13 | **$28.13** | | Observatory | starter $7 | none on Render (retired on Railway after handoff) | $7 | $3.13 | **$10.13** | | Cosmograph | starter $7 | standard $25 | $32 | $3.13 | **$35.13** | | Universe | starter $7 | standard $25 (RC env dropped; PR previews replace it) | $32 | $3.13 | **$35.13** | | Construct (+ Interspace) | starter $7 | standard $25 | $32 | $3.13 | **$35.13** | | Hadron (incl. Dropship) | starter $7 | not yet | $7 | $3.13 | **$10.13** | | Moonshot | starter $7 | not yet | $7 | $3.13 | **$10.13** | | **Total** | | | **$192** | **$25** | **$217/mo ($2,604/yr)** | Plus, outside Render: PR previews at $7 each while open (manual generation, so near zero); Neon for Universe (Free or Launch; usage-based); 1Password stays Families (no change) unless the broker pilot passes, then Business at $8.99/user/mo for one seat. Render Key Value for Hadron's Valkey only if it turns out to be needed (price to check). Observatory production keeps its Railway bill until it is retired. Break-even and margin: * **Against Railway:** Railway's actual bill could not be pulled (the Railway API token was invalidated after the incident). Fill in last month's invoice here before approving; if it was under \~$215/mo, Render costs more and we are paying for the exit, not saving. * **Against revenue:** Cosmograph is the only app with prices set ($9.99/yr or $29.99 lifetime). Its $35.13/mo ($422/yr) breaks even at about **43 annual members** or **15 lifetime sales a year**, before payment fees. Every other app is pre-revenue; their hosting is pure cost. * **Cheapest honest variant:** all staging on `starter`, prod on `standard` only for EXO and arch-router, the rest `starter` until load says otherwise: about **$145/mo**. Starter (512 MB) is tight for a Next build server; watch memory in the pilot before choosing it for production. ## Migration order [#migration-order] Staging first, all of it, on D0; then production; Interspace last. Record-by-record DNS: [render-cutover/dns-changes.md](https://github.com/Interspace-Ventures/exo/blob/staging/docs/runbooks/render-cutover/dns-changes.md). Vaults and items: [render-cutover/vaults-and-items.md](https://github.com/Interspace-Ventures/exo/blob/staging/docs/runbooks/render-cutover/vaults-and-items.md). D0 commands: [render-cutover/commands-d0.md](https://github.com/Interspace-Ventures/exo/blob/staging/docs/runbooks/render-cutover/commands-d0.md). Samir's batched yes list: [render-cutover/approval-list.md](https://github.com/Interspace-Ventures/exo/blob/staging/docs/runbooks/render-cutover/approval-list.md). **Staging (D0)** 1. **EXO staging (pilot, first)** — root `render.yaml` in this PR; runbook `docs/runbooks/render-pilot-exo-staging.md`. Proves Blueprint, 1Password service-account boot, health check and previews on `exo-staging.onrender.com`; pilot test #1 is recorded but no longer gates the rest. Go for the others after two healthy deploys (about an hour), then `staging.exo.now` flips the same day. The staging gateway's `*_STAGING_ORIGIN`s move to Render URLs as each app below lands. 2. **arch-router** — Docker web service (Convex calls it from outside Render, so not a private service). `llama-server` is a C++ binary, so its key arrives through an `op run` wrapper entrypoint (child-process env only, ADR 0014's documented exception); that entrypoint and its Blueprint entry are a small follow-up PR, not in this one. One instance shared by staging and production: staging Convex's `ARCH_ROUTER_URL` moves on D0, production Convex's with EXO production (D1). The Railway instance keeps serving production until then. 3. **Cosmograph staging** — Neon staging branch stays; new role password. 4. **Universe staging** — Railway Postgres staging -> Neon `staging` branch (below). 5. **Construct staging** — per construct PR #406 and tonight's render.yaml PR; `staging.construct.page` and `staging.construct.codes` flip on D0. 6. **Observatory staging** — with its loader-port and render.yaml PR (opening tonight). No DNS change: it sits behind `staging.exo.now/observatory`. Staging only; see "Observatory". **Production (D1-D2)** 7. **EXO production (D1)** — `exo.now` apex and `www`. Production Convex `ARCH_ROUTER_URL` -> Render. 8. **Construct production (D1)** — `construct.codes`, `www.construct.codes`, `construct.page`. **It must not take over `interspace.ventures` yet**: Railway's Construct production keeps running and serving Interspace (and, unless Samir moves them on D2, `structured.glass`, `foundation.financial`, `bumblebee.nyc`). 9. **Cosmograph production (D2)** — `cosmograph.space` (+ a new `www`). 10. **Universe production (D2)** — Railway Postgres -> Neon `production`, then `release.universe.audio` (new, the canary) and `universe.audio` (today an old Replit build at Google Frontend, not Railway). **Last (D2-D3, \~06:00 ET)** 11. **Interspace** — `interspace.ventures` + `www`, served by Construct production on Render. Zero-downtime cutover below. **Parallel track (not on the critical path):** Moonshot, Hadron and Dropship need git repos first. See "Parallel track". ## Timeline (2-3 days) [#timeline-2-3-days] D0 is the day the Render account exists: **tonight (Thu 2026-10-22) or Fri 2026-10-23**. Dates below assume D0 = Fri 10-23; if it lands later every row slides by the same amount. Times are ET. | Day | Date | What | | ------------ | ------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | D0 | Thu night / Fri 10-23 | Samir's approval items 1-10: rotate exposed keys; Render GitHub App; Render API key; 10 vaults + 10 read-only service accounts; merge #225, #227 and the per-repo render.yaml PRs into `staging`. **All staging on Render:** EXO staging pilot first, then arch-router (staging Convex URL), Cosmograph, Universe (Neon `staging`), Construct, Observatory staging. TTLs to 300 s on every production record that moves. Flip `staging.exo.now`, `staging.construct.page`, `staging.construct.codes`. Railway staging paused, not deleted. Ask Render support about early certificate issuance for `interspace.ventures`. | | D1 | Sat 10-24, morning | **EXO production** (`exo.now`, `www`; production Convex `ARCH_ROUTER_URL`). **Construct production** (`construct.codes`, `www`, `construct.page`), **not** `interspace.ventures`. Interspace checked on the Render Construct service via its `onrender.com` URL and a `Host` header, DNS untouched. | | D2 | Sun 10-25, morning | **Cosmograph production** (`cosmograph.space`). **Universe production** (Neon `production`, `release.universe.audio`, then `universe.audio`). Optionally Construct's other domains. | | D2-D3 | Sun 10-25 or Mon 10-26, \~06:00 | **Interspace** (`interspace.ventures`, `www`), last, zero downtime. Sunday only if Construct production has been green 12 h and every other cutover is green; otherwise Monday. | | +7 days each | from Fri 10-30 to \~Tue 11-03 | Each Railway service stays paused 7 days after its own cutover, then is deleted only with Samir's yes. Runs in the background; never blocks the next cutover. | | parallel | from D0 | Repos for Moonshot, Hadron (and Dropship if it does not fold into Hadron); their staging moves whenever a repo exists. | Rules: one hostname group at a time, each with Samir's yes. A red smoke test stops that app and rolls it back; it does not stop apps that are already green or unrelated. Interspace never goes while any other production cutover is red. An app whose vault does not yet hold freshly rotated values waits; it never ships with Railway's values. ## Observatory [#observatory] * **Staging only on Render.** `observatory-staging` moves on D0 with its Exo Secure loader port. It never gets a Render production service, an `Observatory Production` vault, or a production service account. * **No production cutover.** Observatory production (`observatory.bio`, Railway) keeps running on Railway until Samir hands the code to his dad, then it is retired. No `observatory.bio` DNS record changes as part of this ADR. * **Nothing is deleted without Samir's explicit yes.** That covers the Railway Observatory production project, its database credential, the `observatory.bio` records (including `_railway-verify`), and the Railway Observatory staging project after its 7-day window. Retirement is a separate step Samir starts, after the handoff. * **Until retirement, production still runs on the leaked host.** Its Railway env holds exposed values. Recommendation: rotate its provider keys in place (OpenAI, OpenRouter, Resend, intake secret) so the exposed ones stop working, even though the app is leaving. Samir's call. * Cost drops to one `starter` instance on Render ($7/mo plus the Pro share). ## What stays in Convex [#what-stays-in-convex] Nothing about Convex moves. Convex functions, data, file storage, Exo Key (Exo ID) and Convex Auth, and every Convex-side secret stay in the Convex deployment env exactly as ADR 0014's migration table says: `BIRD_API_KEY`, `EXO_ID_PRIVATE_KEY`, `AUTH_PRIVATE_KEY`, `AUTH_*_CLIENT_SECRET`, `ARCH_ROUTER_KEY`, `ARTIFICIAL_ANALYSIS_API_KEY`, `EXO_AI_BROKER_TOKEN`, `EXO_INTAKE_SECRET_*`, `EXO_VERIFY_TOKEN` (verifier side), `EXO_GATEWAY_CLIENT_SECRET_SHA256`. Convex pushes keep running from GitHub Actions, never from a Render build. The only Convex edits the move needs are non-secret URLs: `ARCH_ROUTER_URL` and any `*_STAGING_ORIGIN` or app origin Convex calls. ## Postgres and Neon [#postgres-and-neon] Neon is the Postgres home: it is host-independent, branches per environment, and Cosmograph already uses it. No Render Postgres. | Database | Today | Move | | ------------------------------ | --------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | Cosmograph prod + staging | Neon | No data move. Reset both role passwords (exposed), store the new `DATABASE_URL`s in `Cosmograph Production` / `Cosmograph Staging`. If Neon IP allow is on, add Render's Virginia outbound IPs. | | Universe staging + prod (+ RC) | Railway Postgres | New Neon project with `staging` and `production` branches. Freeze writes; `pg_dump --format=custom` from Railway (temporary TCP proxy or `railway connect`), `pg_restore --no-owner` into Neon; compare row counts per table; switch `DATABASE_URL` in the vault; unfreeze. RC is dropped. | | Observatory | `DATABASE_URL` set on Railway (legacy data backend) | Staging: confirm the Convex backend is live and Postgres unread; if so do not migrate. Production stays on Railway until retirement. Delete the credential only with Samir's yes. | | Construct | Legacy Postgres/Payload vars on Railway | Not migrated (Convex is the content authority). Delete after confirming production no longer reads them. | | Old `exobase` Neon project | Neon, Payload content | Delete (deletion already pending Samir) or reset its password. | | Hadron/Dropship Valkey | Railway | If it is only a cache, drop it; otherwise Render Key Value. Decide at that step. | ## DNS cutover at Spaceship [#dns-cutover-at-spaceship] All six domains are hosted at Spaceship: exo.now, observatory.bio, cosmograph.space, construct.page, interspace.ventures, moonshot.supply. Today: `exo.now` apex A `69.46.46.38` (Railway), `www` CNAME `vlc5968k.up.railway.app`, `staging` CNAME `jai6l6x5.up.railway.app`; `_railway-verify` TXT on exo.now, staging.exo.now and observatory.bio. Hostnames that move: `staging.exo.now`, `exo.now` + `www`, `cosmograph.space`, `construct.page`, `interspace.ventures` (last). `observatory.bio` does **not** move (Observatory is staging only); `moonshot.supply` waits for the parallel track. Per hostname, with Samir's approval at each flip: 1. **T-24 h:** record the current value of every record you will touch (screenshot or export), then lower its TTL to 300 s. 2. **Add the domain in Render** (dashboard service -> Settings -> Custom Domains, or `POST /v1/services/{serviceId}/custom-domains`). Render shows the target. 3. **Check CAA and AAAA:** if the domain has CAA records, add `issue` and `issuewild` for `letsencrypt.org` and `pki.goog`; remove any AAAA records (Render is IPv4) ([render.com/docs/custom-domains](https://render.com/docs/custom-domains)). Adding an apex in Render also adds `www` as a redirect to it, and the reverse. 4. **Flip at Spaceship:** * Subdomain (`staging`, `www`): replace the Railway CNAME with CNAME `.onrender.com`. * Apex: use ALIAS/ANAME `.onrender.com` if Spaceship offers it for the zone; otherwise replace A `69.46.46.38` with A `216.24.57.1`. 5. **Verify** in Render (`POST /v1/services/{serviceId}/custom-domains/{id}/verify`), wait for the certificate, then run the smoke tests against the real hostname. 6. **Leave `_railway-verify` TXT records** in place during the soak (harmless; needed for rollback), delete them when the Railway project is deleted. 7. Raise TTLs back to 3600 s after 24 h green. ### Interspace (`interspace.ventures`): zero-downtime cutover, last [#interspace-interspaceventures-zero-downtime-cutover-last] Interspace (`interspace.ventures`) is served by the root Construct app (not `services/interspace`, which is staging-only) and must stay up. The rule: both origins serve the site for the whole flip, so whichever answer a resolver holds, it gets a working page. 1. **D0 (Samir's yes):** export every `interspace.ventures` record. Lower the TTL of the apex and `www` records to 300 s, at least 24 h before the flip. Touch nothing else: MX, SPF, DKIM, DMARC and site-verification TXT records stay exactly as they are. 2. **D1-D2:** Construct production is live on Render and `construct.codes` has been green for at least 12 h. Add `interspace.ventures` (Render adds `www`) as a custom domain on the Render Construct production service. Add CAA entries for `letsencrypt.org` and `pki.goog` if the zone has CAA. Remove AAAA records only at flip time, and only if any exist. 3. **Verify before flipping:** on the `onrender.com` URL, check the Interspace routes Construct serves for that host, e.g. `curl -fsS -H 'Host: interspace.ventures' https://.onrender.com/` and the same for the key pages and `/api/health`. Compare with the live Railway page (status, title, canonical links, assets). Construct's host routing must accept the `onrender.com` host or the `Host` override; if it does not, stop and fix that before the flip. 4. **D2 or D3, \~06:00 ET, flip:** apex to ALIAS/ANAME `.onrender.com` (or A `216.24.57.1`), `www` to CNAME `.onrender.com`. Keep Railway's Construct running and serving `interspace.ventures` throughout. Immediately press Verify in Render and flush the Google and Cloudflare public DNS caches. 5. **Watch for 30 minutes:** `curl` the real hostname every minute from two networks; check the certificate issuer and expiry; check `/api/health`. 6. **Known gap, unverified:** Render documents issuing the TLS certificate only after DNS verifies, so a resolver that picks up the new record in the first minutes may see a TLS error or a 502. On D0, ask Render support whether they can pre-issue the certificate (as they do for wildcard domains via an `_acme-challenge` CNAME). If they cannot, the 300 s TTL and an early-morning flip keep the window to minutes; if the certificate is not issued within 10 minutes, put the records back. (`interspace.ventures` already has an `_acme-challenge` TXT of unknown origin; a Render `_acme-challenge` CNAME would need it removed first.) 7. **After 24 h green:** raise TTLs to 3600 s. Railway's Construct stays paused, not deleted, for 7 days. ## Rollback [#rollback] * **Every service** gets a **7-day rollback window** after its own cutover, staging and production alike: the Railway service is **paused, not deleted**, env untouched (it is all compromised and rotated anyway, so it is only useful for the rollback itself). The window runs in the background and does **not** block the next app's cutover. * To roll back a hostname: set the Spaceship record back to its recorded Railway value (A `69.46.46.38`, CNAME `vlc5968k.up.railway.app` / `jai6l6x5.up.railway.app`, and so on), resume the Railway service, and wait out the 300 s TTL. If the rotated secrets differ, point the Railway service at the same 1Password vault (it reads the same items through its own, freshly minted read-only token) rather than pasting values. * `universe.audio` rolls back to its current Replit origin (A `34.111.179.208`, `replit-verify` TXT kept), not to Railway, which never served it. Keep that Replit deployment untouched until Universe production has been green for 7 days. * Rolling back one app does not roll back the others. The exception is arch-router: if it rolls back, set both Convex deployments' `ARCH_ROUTER_URL` back to the Railway instance. * Interspace rolls back the same way, and stays rollback-ready longest because Railway's Construct keeps serving it until the day after its flip. * A Render deploy that fails its health check never takes traffic; the previous deploy keeps serving. Production also fails closed when 1Password cannot be read. * **Nothing is deleted without Samir's explicit yes**: no Railway project, database, credential or DNS record, after 7 green days or ever. Observatory production is not on this clock at all; it is retired only after the handoff (see "Observatory"). ## Parallel track: repos for Moonshot, Hadron and Dropship [#parallel-track-repos-for-moonshot-hadron-and-dropship] Render builds only from a git repo. Three apps have none it can use, so they run alongside the main week and never block it: | App | Today | Needs | | -------- | ----------------------------------------- | ----------------------------------------------------------------------------------------------------------- | | Moonshot | a local folder deployed with `railway up` | a new GitHub repo in `Interspace-Ventures`, the folder pushed to it (after a secret scan), then a Blueprint | | Hadron | no GitHub repo | a new GitHub repo, then a Blueprint | | Dropship | code moving into Hadron | no repo of its own if the move into Hadron lands first; otherwise its own repo until it does | **Samir must OK creating each repo** (name, visibility). Nobody creates a repo, a Render service or a DNS record for these on their own. Each moves to Render staging whenever its repo exists, and stays on Railway until then; `moonshot.supply` does not change in this ADR's week. ## How each app receives its secrets [#how-each-app-receives-its-secrets] One rule everywhere: the host stores one read-only, vault-scoped bootstrap token per app per environment, as Render `sync: false`. Nothing else secret lives in Render env, in `render.yaml`, or in a build. Builds need no secret (and on Docker builds there is no OIDC anyway); never declare a Docker `ARG` for a secret. PR previews inherit staging's token and vault, never production's. | App | Vaults (read-only service account each) | How the process gets values | | --------------------------------- | ---------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------- | | EXO | `Exo Staging`, `Exo Production` | `src/lib/exo-secure` in-process at boot, `EXO_SECURE_SOURCE=auto` (resolves to the service account on Render) | | arch-router | `Arch Router` | entrypoint `op run -- /opt/llama/llama-server …` with `LLAMA_API_KEY=op://Arch Router/llama/credential`; values exist only in the child process | | Observatory | `Observatory Staging` only (production stays on Railway until retired) | copy of the Exo Secure loader (port PR in the observatory repo, D0); OpenAI key is a candidate for Render's managed OpenAI OIDC later | | Cosmograph | `Cosmograph Staging`, `Cosmograph Production` | loader copy in `api-server`, or `op run` wrapper if the server is not easily patched | | Universe | `Universe Staging`, `Universe Production` | loader copy or `op run` wrapper | | Construct (+ Interspace) | `Construct Staging`, `Construct Production` | per #406: loader port, one token per env | | Hadron (incl. Dropship), Moonshot | `Hadron Staging`, `Moonshot Staging` | `op run` wrapper until they adopt the loader; needs a repo first (parallel track) | | Convex (all apps) | n/a | Convex deployment env, unchanged | If pilot test #1 ever passes, that app's bootstrap changes from `OP_SERVICE_ACCOUNT_TOKEN` to `OP_INTEGRATION_KEY` + `OP_WORKLOAD_ID` + `EXO_SECURE_ENVIRONMENT_ID` with `EXO_SECURE_SOURCE=broker` (or `auto`), and its secrets move from vault items to a 1Password Environment. `OP_INTEGRATION_KEY` is still a stored secret, but it is useless without a fresh token from the host's issuer that matches 1Password's trust policy. ## 1Password for Agents [#1password-for-agents] 1Password for Agents, launched 2026-10-22, lets a personal AI agent ask to use a login from Samir's vault; he approves, and 1Password fills the sign-in form without the agent ever seeing the password. It is free on Individual and Families, so it costs nothing to turn on. It is about people's sign-ins, not servers: it does not hand secrets to a running workload, so it replaces neither the service-account token nor the broker. Where it helps here is the migration itself: the agent doing Render, Spaceship and Neon dashboard steps can sign in with Samir's approval instead of a saved password, which shrinks what an agent session could leak. Treat it as an operator-side improvement, separate from this decision. ## Consequences [#consequences] * Railway leaves the critical path in 2-3 days from D0 (all staging on D0, productions D1-D2, Interspace D2-D3, last). Railway projects are deleted, with Samir's yes, no sooner than 7 days after each service's cutover. Observatory production stays on Railway until it is retired after the handoff. * Hosting costs about $217/mo at the sizes above (about $145/mo lean). * The stored-secret risk shrinks (one read-only, vault-scoped, freshly minted token per app env) but does not reach zero on Render. * `EXO_SECURE_SOURCE` gains `broker` and `auto`; the default stays `env` until each app's cutover sets `auto`. * EXO Verify fails any `render.yaml` that holds a secret value; `OP_INTEGRATION_KEY`, `OP_WORKLOAD_ID`, `EXO_SECURE_ENVIRONMENT_ID` and `EXO_SECURE_OIDC_TOKEN_FILE` join the platform-env allow list (the first is a bootstrap secret, the rest are identifiers). ## Open items for the owner [#open-items-for-the-owner] One batched, numbered list with a link per item: [render-cutover/approval-list.md](https://github.com/Interspace-Ventures/exo/blob/staging/docs/runbooks/render-cutover/approval-list.md). In short: rotate the exposed keys before reconnecting anything; Render GitHub App and API key; the 10 vaults and their read-only service accounts; merging #225, #227 and the per-repo render.yaml PRs; Convex device approvals; the DNS changes by day (D0 TTLs + staging, D1 EXO and Construct, D2 Cosmograph and Universe, D2-D3 Interspace); the extra Construct domains; Render's custom-domain allowance; the Moonshot and Hadron repos; and, later, each deletion (Railway after its 7 days, Observatory production after the handoff). Still open beyond that list: Railway's last invoice (break-even), and 1Password Business only if pilot test #1 ever passes.