# 0014: Exo Secure loads secrets from 1Password at boot; no secrets in platform env vars https://exo.now/docs/adr/0014-exo-secure-secrets-from-1password > Status: proposed (draft PR; owner decision pending on the open items below) · Date: 2026-10-21 # ADR 0014: Exo Secure loads secrets from 1Password at boot; no secrets in platform env vars [#adr-0014-exo-secure-loads-secrets-from-1password-at-boot-no-secrets-in-platform-env-vars] * Status: proposed (draft PR; owner decision pending on the open items below) * Date: 2026-10-21 * Decided by: Samir (owner), standing rule set 2026-10-21 * Relates to: [ADR 0004](https://exo.now/docs/adr/0004-exo-id-central-identity-provider) (Exo ID issuer and JWKS), [ADR 0005](https://exo.now/docs/adr/0005-exo-verify-verification-service) (EXO Verify), roadmap item "Exo Secure" ## Context [#context] On 2026-10-21 Railway disclosed that an unauthorized party accessed systems that store customer environment variables, obtained the decrypted contents of some of them, and appears to have been hunting for AI provider keys; Railway recommends rolling every variable and paused all deployments ([Railway disclosure](https://blog.railway.com/p/security-disclosure-wednesday-october-21st)). Every EXO secret that lives in a platform env store (Railway, Vercel, Convex) is exposed to a breach of that platform, and an env var also leaks through crash reports, child processes, `env` dumps and build logs. Separately, staging.exo.now proves the gateway to five upstream Apps (Exo, Moonshot, Hadron, Observatory, Universe) with one shared `STAGING_GATEWAY_SECRET` in the `x-exo-gateway-secret` header, so one leaked App env lets anyone impersonate the gateway to all of them (security review item M1). Standing rule from the owner: **Exo Secure must never store secrets as platform env vars.** The only exception is a single bootstrap token per app. ## Decision [#decision] ### 1. One read-only 1Password service account per app environment [#1-one-read-only-1password-service-account-per-app-environment] * Each app environment gets its own vault and one **read-only** service account scoped to only that vault: `Exo Staging` and `Exo Production` for EXO itself; every managed app follows the same pattern (`Moonshot Staging`, …). * The platform holds only `OP_SERVICE_ACCOUNT_TOKEN` (the bootstrap token) plus non-secret config: `EXO_SECURE_SOURCE=1password` and `EXO_SECURE_VAULT=`. * Secrets are referenced as `op:////`. References are not secret; they live in code (`src/lib/exo-secure/manifest.ts`), so the list of what an app needs is reviewed like code. ### 2. Resolve at boot into memory, never back into `process.env` [#2-resolve-at-boot-into-memory-never-back-into-processenv] * `src/instrumentation.ts` `register()` calls `loadSecrets(EXO_SECRET_MANIFEST)` once, before the server takes traffic. It uses the official [@1password/sdk](https://github.com/1Password/onepassword-sdk-js) (MIT, v0.5.0) with one batched `secrets.resolveAll` call. * Values go into an in-memory store on `globalThis` (a registered symbol, so the Next proxy bundle and route bundles share it). Server code reads them with `secret(name)` / `requireSecret(name)`. Nothing is written to `process.env`, so env dumps and child processes never carry them. * **Fail closed in production.** With `EXO_SECURE_SOURCE=1password` in production, a missing token, an unreachable vault, or any missing required secret throws in `register()`: the new deploy never becomes healthy and the previous one stays live. Staging and development log a warning instead. * After loading, the server warns (by name, never value) about any managed secret that is still also set as a platform variable, so leftovers get deleted. * **Migration mode.** `EXO_SECURE_SOURCE` defaults to `env`, which reads `process.env` exactly as today and warns in staging and production. Merging this ADR therefore changes no behaviour until an environment is switched to `1password`. * **Local development** uses either `op run --env-file=.env.op -- pnpm dev` (resolved values exist only in that child process; default `env` source) or `EXO_SECURE_SOURCE=file` with a gitignored `.env.secrets.local` (the `.env.*` rule already ignores it). `file` is refused in production. ### 3. Bird and other Convex-only secrets: Convex env as the interim home [#3-bird-and-other-convex-only-secrets-convex-env-as-the-interim-home] Bird SMS already runs only in Convex (`convex/lib/bird.ts`, `convex/communications.ts` `sendSms` internal mutation and `deliverSms` internal action). No Next code reads `BIRD_*` or calls the Bird API; `tests/int/exo-secure-boundaries.int.spec.ts` now fails if that ever changes. Both functions are registered `internal`, so only Convex code (phone sign-in in `convex/phoneAuth.ts`, or a deploy-key caller) can trigger a send; they need no entry in the public-write allowlist of `tests/int/convex-auth-guards.int.spec.ts`. When Bird email arrives it follows the same shape. Convex env is also a platform env store. Options: | Option | What it means | Verdict | | ------------------------------ | ------------------------------------------------------------------------------------------ | --------------------------------- | | A. Convex env (interim) | Bird, Exo ID signing keys, OAuth client secrets, AI keys stay in the Convex deployment env | **Recommended now** | | B. Convex pulls from 1Password | A `"use node"` internal action resolves from an `Exo Convex` vault and caches per isolate | Later, for the highest-value keys | Why A now: the breach was Railway's, and Convex is a separate vendor and blast radius; Convex functions run inside Convex, so these secrets never touch Railway or the Next server; and B still needs an `OP_SERVICE_ACCOUNT_TOKEN` in Convex env (the bootstrap exception moves, it does not disappear), adds a 1Password round trip to Node-action cold starts, cannot serve the default (V8) runtime that most EXO functions use, and spends service-account rate limit (Teams: 1,000 reads per hour per token; Business: 10,000; [rate limits](https://developer.1password.com/docs/service-accounts/rate-limits/)). Revisit B for `EXO_ID_PRIVATE_KEY` and `AUTH_PRIVATE_KEY` when Convex supports a secret-manager integration or if Convex's posture changes. Run the new platform-env check against `npx convex env list` output with an explicit allowlist to keep that store reviewed. ### 4. Per-app Exo Key gateway tokens replace `STAGING_GATEWAY_SECRET` (M1) [#4-per-app-exo-key-gateway-tokens-replace-staging_gateway_secret-m1] * The gateway asks Exo Key for a token per upstream App: `POST /exo-key/v1/gateway-token {"app": ""}` with `Authorization: Bearer ` (in the `Exo Staging` vault). Convex stores only `EXO_GATEWAY_CLIENT_SECRET_SHA256`, which is not a secret. The route answers 404 until that hash is set, so it ships dark. * The token is an RS256 JWT signed by the existing Exo ID issuer key (`convex/lib/gatewayToken.ts`, same `kid` as access tokens): `iss` = the Exo ID issuer, `aud` = `exo-gateway:`, `sub` = `exo-gateway`, `token_use` = `gateway`, random `jti`, 5-minute lifetime. EXO caches one per App until a minute before expiry and forwards it in `x-exo-gateway-token`. * Each App verifies against the public JWKS (`/exo-key/.well-known/jwks.json`) with `verifyGatewayToken` (`src/platform/delivery/gatewayTokens.ts`). A token minted for one App fails at every other App, and an ordinary Exo ID access token is never accepted (no `token_use`). Apps hold no gateway secret at all. * Flag `EXO_GATEWAY_TOKENS` on the EXO service: `off` (default; shared header only, today's behaviour), `dual` (both headers), `only` (token only; a missing token fails closed with 503). Caller-supplied copies of either header are always stripped. Per-app migration (each step is reversible by flipping the flag back): 1. Generate the gateway client secret (48+ random bytes) into `Exo Staging/staging-gateway/exo-key-client-secret`; set its SHA-256 (base64url) as `EXO_GATEWAY_CLIENT_SECRET_SHA256` in the staging Convex deployment. 2. Move the current shared value into `Exo Staging/staging-gateway/legacy-shared-secret`, switch the EXO staging service to `EXO_SECURE_SOURCE=1password`, delete `STAGING_GATEWAY_SECRET` from it. 3. Set `EXO_GATEWAY_TOKENS=dual` on EXO staging. Verify with a request through staging.exo.now/\{app} that both headers reach the upstream. 4. For each App (Moonshot, Hadron, Observatory, Universe; then Construct, Rampart, Cosmograph, Vector, Dropship as they adopt the gateway): ship `verifyGatewayRequest` with `acceptLegacy: true`, deploy, confirm requests verify as `token`; then set `acceptLegacy: false` and delete `STAGING_GATEWAY_SECRET` from that App's service. 5. When every App is on tokens only, set `EXO_GATEWAY_TOKENS=only`, delete the legacy item from the vault, and remove the legacy code path. ### 5. EXO Verify enforces it [#5-exo-verify-enforces-it] `exo-verify security` gains a built-in `platform-env` check (`packages/exo-verify/lib/level1/platform-env.mjs`, plain Node, no git, no network). Given a variables export (`railway variables --json`, a `[{name,value}]` list, or dotenv; file path or stdin) it fails on values that look like secrets: known prefixes (`sk-`, `sk_live_`, `ghp_`, `github_pat_`, `xox[bpas]-`, `re_`, `bk_`, `AKIA`, `ops_` other than `OP_SERVICE_ACCOUNT_TOKEN`, Stripe `whsec_`, Google `AIza`, Convex deploy keys), JWT and PEM shapes, JSON carrying private JWK members, high-entropy strings, and names ending `_KEY`, `_SECRET`, `_TOKEN`, `_PASSWORD`. An allowlist (`OP_SERVICE_ACCOUNT_TOKEN` plus explicit non-secret config) is the only escape; `op://` references and Railway `${{ }}` references pass. Findings name the variable and the reason, never the value. No export provided means a skip (advisory), so builds without one stay green. Standalone: `pnpm verify:platform-env `. ## Threat model [#threat-model] | Threat | Before | After | | -------------------------------------------------- | ---------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Platform env store breach (Railway, 2026-10-21) | Every secret on the service leaks | Only `OP_SERVICE_ACCOUNT_TOKEN` leaks: read-only, one vault, revocable in one click | | Env dump, crash report, child process inherits env | Secrets included | Secrets not in `process.env` | | One App's env leaks the gateway secret | Impersonate the gateway to all five Apps | No App holds a gateway secret; a token is good for one App for at most 5 minutes | | Stolen gateway token | n/a | Valid only for its App, expires in 5 minutes | | Stolen bootstrap token | n/a | Attacker can read that one vault until revoked; it cannot write or see other vaults. Detect via 1Password activity log and the service account's usage report | | 1Password outage at boot | n/a | Production deploy fails closed; the running deploy keeps serving from memory | | Code exfiltrates memory (RCE in the server) | Secrets readable | Still readable: Exo Secure does not defend against code execution in the app | ## Rotation [#rotation] * Rotate a secret by editing its 1Password item, then restarting (redeploying) the app; no platform change is needed. Restart is required because values are read once at boot. * Rotate the bootstrap token by creating a new service-account token, setting it on the platform, redeploying, then revoking the old one. * Gateway tokens rotate themselves every 5 minutes. Rotating the signing key follows the existing Exo ID JWKS procedure (publish the new key first; Apps fetch JWKS by `kid`). * The 2026-10-21 Railway advice to roll every variable is a separate, owner-approved action; this ADR does not rotate or revoke anything. ## Blast radius [#blast-radius] One service account per app environment, read-only, one vault. A leak of `Exo Staging`'s token exposes only EXO staging's few secrets, never production, never another app. Today a Railway leak exposes every variable on every service in the project. ## Cost and unit economics [#cost-and-unit-economics] * 1Password service accounts come with the 1Password plan; there is no per-service-account fee in the plan Exo already uses for people (owner brief, 2026-10-21; the roadmap's 2026-10-11 survey lists Business at $8.99 per user per month). Not re-verified today: 1Password's Business pricing page did not render for the check. * Rate limits are the real constraint ([1Password rate limits](https://developer.1password.com/docs/service-accounts/rate-limits/)): Teams 1,000 reads per hour per token and 5,000 requests per day per account; Business 10,000 reads per hour per token and 50,000 per day per account. Exo Secure makes one batched read per boot, so even 100 deploys and restarts a day across every app is far inside Teams limits. * Cost per unit: $0 marginal per app, per deploy and per request. Gateway tokens cost one Convex HTTP action per App every \~4 minutes while traffic flows (about 360 calls per App per day), well inside the Convex plan's included function calls. The platform-env check adds well under a second to the gate. * Margin note: Exo Secure is not a cost driver, so it is not a pricing metric; include it in every plan, unmetered. If Exo later hosts vaults for customers, cost scales per human seat on 1Password, so price per seat, never per app or per secret. ## Migration table [#migration-table] Every secret-shaped variable the exo repo reads today (grep of `process.env` and `app.env` across `src/`, `convex/`, `packages/`, `scripts/`, plus `.env.example`), with its destination. Owner is Samir (Interspace) for every row unless noted; the provider column is who issues the value. | Variable | Read by (today) | Current location | Destination | Owner / issuer | | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------- | --------------------------- | | `TWENTY_API_KEY` | `src/platform/crm/twenty.ts` (Next) | Railway env (EXO service) | 1Password app vault `twenty-crm/credential`; read via `secret()` | Samir / Twenty | | `STAGING_GATEWAY_SECRET` | `src/platform/delivery/stagingGateway.ts` (Next proxy); each upstream App | Railway env on EXO staging and on Moonshot, Hadron, Observatory, Universe | 1Password `Exo Staging/staging-gateway/legacy-shared-secret`, then retired (section 4) | Samir | | `EXO_GATEWAY_CLIENT_SECRET` (new) | `src/platform/delivery/gatewayTokens.ts` | none | 1Password `Exo Staging/staging-gateway/exo-key-client-secret` | Samir | | `EXO_GATEWAY_CLIENT_SECRET_SHA256` (new) | `convex/lib/gatewayToken.ts` | none | Convex env (not a secret: a hash) | Samir | | `OP_SERVICE_ACCOUNT_TOKEN` (new) | `src/lib/exo-secure` | none | Railway env: the one bootstrap exception | Samir / 1Password | | `BIRD_API_KEY` | `convex/lib/bird.ts` | Convex env | Convex env (interim, section 3); never Railway | Samir / Bird | | `EXO_ID_PRIVATE_KEY` | `convex/lib/exoIdCrypto.ts` | Convex env | Convex env (interim; candidate for option B) | Samir | | `AUTH_PRIVATE_KEY` | `convex/convex.config.ts` (Convex Auth) | Convex env | Convex env (interim; candidate for option B) | Samir | | `AUTH_GOOGLE_CLIENT_SECRET` | `convex/convex.config.ts` | Convex env | Convex env (interim) | Samir / Google Cloud | | `AUTH_GITHUB_CLIENT_SECRET` | `convex/convex.config.ts` | Convex env | Convex env (interim) | Samir / GitHub | | `ARCH_ROUTER_KEY` | `convex/ai/assessors.ts` | Convex env | Convex env (interim) | Samir | | `ARTIFICIAL_ANALYSIS_API_KEY` | `convex/ai/benchmarks.ts` | Convex env | Convex env (interim) | Samir / Artificial Analysis | | `EXO_AI_BROKER_TOKEN` | `convex/aiOpenAICompat.ts`, `convex/lib/callerAuth.ts` (legacy path); `packages/exo-verify` review | Convex env; CI env for review | Convex env (interim); retire the legacy shared token in favour of Exo Key tokens | Samir | | `EXO_INTAKE_SECRET_UNIVERSE` / `_COSMOGRAPH` / `_OBSERVATORY` | `convex/lib/intakeProtocol.ts` | Convex env; each App's own env | Convex env (interim); App side moves to that App's 1Password vault | Samir | | `EXO_VERIFY_TOKEN` | `convex/verifyHttp.ts`; `packages/exo-verify` report posting; Dockerfile build `ARG` | Convex env; Railway build variable | Convex env (verifier side). Build side: open item (a build-time token cannot come from a runtime vault); replace with a short-lived Exo Key token | Samir | | `EXO_STATE_MIGRATION_TOKEN` | `scripts/*-exo-state-staging.ts` (operator) | operator shell | `op run` from the operator's 1Password; never a platform variable | Samir | | `RESEND_API_KEY`, `CRON_SECRET`, `ENTIRE_API_KEY`, `ENTIRE_WEBHOOK_SECRET`, `GITHUB_APP_PRIVATE_KEY`, `GITHUB_WEBHOOK_SECRET`, `GITHUB_TOKEN`, `RAILWAY_API_TOKEN`, `TYPESAFE_API_KEY` | not read by code today (listed in `.env.example` only) | possibly Railway env (unverified) | Delete from Railway if present; any future use goes in the 1Password app vault via the manifest | Samir | | `AUTH_GOOGLE_CLIENT_ID`, `AUTH_GITHUB_CLIENT_ID`, `AUTH_JWKS`, `EXO_ID_JWKS` | Convex | Convex env | not a secret (public); keep `EXO_ID_JWKS` public-only (the check flags private JWK members) | Samir | | `EXO_CONVEX_URL`, `NEXT_PUBLIC_CONVEX_URL`, `NEXT_PUBLIC_SERVER_URL`, `EXO_CONVEX_SITE_URL`, `CONVEX_SITE_URL`, `EXO_KEY_SITE_URL`, `EXO_ID_ISSUER`, `EXO_ID_AUTHORIZE_URL`, `EXO_KEY_DEVICE_URL`, `EXO_DOCS_URL`, `ARCH_ROUTER_URL`, `TWENTY_API_URL`, `BIRD_API_HOST` | various | platform env | not a secret (URLs) | n/a | | `*_STAGING_ORIGIN` (9), `EXO_STAGING_HOSTS`, `EXO_CONTROL_PLANE_HOSTS`, `EXO_DEPLOYMENT_ENVIRONMENT`, `EXO_DELIVERY_STAGE`, `EXO_IDENTITY_PLANE`, `EXO_SERVICE_PROVISIONING_ENABLED`, `EXO_WORK_REGISTER_ENVIRONMENT`, `EXO_MCP_ALLOWED_ORIGINS`, `EXO_PROTECTED_RESOURCES`, `EXO_AI_MODEL`, `EXO_AI_ASSESSOR`, `TWENTY_INTERSPACE_OPPORTUNITY_STAGE`, `BIRD_SMS_FROM`, `EMAIL_*`, `DROPSHIP_RAILWAY_*`, `EXO_SECURE_SOURCE`, `EXO_SECURE_VAULT`, `EXO_GATEWAY_TOKENS` | various | platform env | not a secret (config and flags) | n/a | | `EXO_STAGING_GRANTS` | `src/proxy.ts` | Railway env | not a credential (user ids and scopes); keep as config, review as access policy | Samir | ## Consequences [#consequences] * EXO's Railway service ends with one secret-shaped variable. Every other managed app adopts the same module (copy `src/lib/exo-secure`) and the same gate check. * A boot now depends on 1Password being reachable; a failed deploy stays on the previous version. * Secrets change only on restart. Hot reload of secrets is out of scope. * Open items for the owner: create the vaults and service accounts; choose Teams or Business (rate limits); approve option A for Convex; decide how the build-time `EXO_VERIFY_TOKEN` is replaced.