choir documentation

Getting started is in the top-level README. Everything else is here, indexed by task.

Each page is also pulled into the crate that implements it with #![doc = include_str!], and the release gate runs cargo doc -D warnings, so a Rust example here that stops compiling fails the build.

API documentation →

choir docs builds the book with rustdoc inside it at /api/. From a checkout: cargo run -p choir-cli -- docs --open. The comment above is a marker choir docs replaces with a link to the node (D76).

Start here

Operating a node

Runbooks:

• Private single-node beta runbook: network hold, TLS proxy, backups, staging promotion, go-live receipts.
• Restoring a node from a backup: ordering rule, and the secrets a backup never holds.
• Canonical-node flip runbook: supervised install, protected-ref gates.

Using a node

Reference

Which of these is generated

The gate fails when a generated file differs from its source. Regenerate with cargo run -p choir-cli --example gen-surface.

Everything else on this page is hand-written.

Why choir exists

The problem

Several coding agents on one repository collide over the turn, not the code: each needs to know the tip of main, and the answer is only true until another agent pushes. On a forge built for people, a human arbitrates by merging and rebasing, so every agent queues behind that person.

Merging is solved. The bottleneck is ordering: deciding which concurrent attempt came first, without a person.

The mechanism

A change is a signed operation appended to a hash-chained log.

• One writer thread per repository decides the order. It is the only thing that appends, so “what happened first” has exactly one answer.
• Every operation carries its author’s signature over (workspace, payload); every entry carries the hash of the previous one.
• Refs, workspaces, changes, reviews and approvals are folds over that log. Nothing is stored twice, so nothing can disagree with the order.

A git push joins the same order: the pre-receive hook turns each ref update into a signed operation with git’s old object id as the compare-and-swap precondition. The git client is unmodified.

Three consequences

A race is a compare-and-swap. The later of two pushes to one ref gets a rejection naming the head it lost to. Fetch, integrate, submit again. No lock is held.

A conflict is a committed value. A strategy that cannot resolve returns a conflict, which is appended like any other state and built on. Nothing silently picks a side.

Undo is arithmetic. The state at operation 4,110 is a replay, not a restore. A bad landing is subtracted by a compensating operation, and the history stays readable.

Who it is for

Someone running coding agents against a repository, in three roles:

If one agent works on your repository at a time, you do not have this problem.

What it is not

Not a GitHub replacement. No organizations, marketplace, actions, or wiki. The surface is repositories, changes, reviews, and the log.

You do not have to move. The bridge mirrors an upstream GitHub repository and runs the sequencer beside it. Upstream stays canonical; there is no dual-write.

Next: Architecture or The CLI and HTTP API.

Architecture

Many agents work on one repository; a single-writer sequencer puts every change in one total order; merge conflicts are values, not errors. This page maps the layers and the crate for each.

The model

A change is a signed operation appended to an append-only log. One writer thread per repository decides the order, stamps a sequence number and appends. Refs, the review queue and workspace ownership are folds over that log, recomputed rather than stored. Undo is a function of position; two agents racing one ref get a compare-and-swap.

Layers

Layer numbers match code comments and DECISIONS.md.

Beside the stack:

Tools: choir-cli (the choir binary and the surface table every generated document renders from), choir-bridge (forge follower, D21), choir-demo (narrated walkthrough), choir-spike (Phase-0 gate binary). choir-fs holds the atomic-write and lock primitives the binaries share. choir-guards holds source-scanning tripwires (D77).

What one operation does

  agent
    │  choir submit / choir batch          (choir-cli)
    ▼
  POST /api/submit                          (choir-node)
    │  authenticate            --auth-file
    │  authorize               --acl-file            D29
    │  meter                   --rate-limit-api      D33
    │  bound the body          --quota-push-bytes    D37
    ▼
  fairness queue                            (choir-sequencer::fairness)
    │  one bounded window per actor, served round-robin
    ▼
  THE SINGLE WRITER                         (choir-sequencer)
    │  verify the signature                          (choir-identity)  L8
    │  run the admission policy                      (choir-view)
    │  compare-and-swap the ref it touches
    │  stamp seq, append                             (choir-oplog)     L1
    │  record the decision     --journal
    ▼
  the view is refolded                      (choir-view)
    │  refs, workspaces, reviews, provenance
    ▼
  ref landed → webhook        --hooks-file            D32

• The order is decided in one place. Fairness decides who is asked next; the writer stamps seq alone and appends in that order.
• Checks before the writer are advisory. The actor key the fairness queue buckets on is a claim; the signature is verified on the writer thread.

Important

Nothing in front of the writer may become load-bearing for authorization.

A conflict is a value

TreeEntry::Conflict in choir-view is a valid commit: hashed, signed, appended, and buildable on. The merge queue never blocks; a conflicting change is evicted from the speculative train as a conflict and the queue keeps moving. choir-merge is a pipeline (trivial, line, Mergiraf) where each strategy may decline.

Replay purity

Replaying the log from zero must produce exactly the served state (D40). So:

1. Derived data is never a durability barrier. Request log, decision journal and lag log are one-way records.
2. Some projections are folded, not persisted. The per-user workspace tally behind --quota-workspaces is rebuilt from the log on restart.
3. Accounts and credentials sit outside the log (D36). Revocation is deletion.

What the format versions are for

FORMAT_VERSION appears in choir-store, choir-oplog and choir-view.

• Hashes are self-describing: a codec byte names the function, so a migration adds a codec.
• Fields needed later exist from day one: OpEntry::witnesses has always been present and empty (D16, D67).
• Adding an operation variant is one-way for readers.

Where to go next

Running a node

One binary serves two protocols on one port: git smart-HTTP for clone, fetch and push, and the platform API for signed operations. This page is how to start it and what each policy file means.

Using a node: docs/using/cli.md. The policy behind each flag: docs/operating/authorization.md, docs/operating/limits.md, docs/operating/webhooks.md, docs/operating/observability.md, docs/operating/transports.md.

Default port 8417. A repository is sequenced only when it carries a pre-receive hook: create with choir repo create or --create; a bare repository that arrives any other way is adopted and hooked at the next start. With no arguments the binary uses ./repos and port 8417; configured invocations must supply both <repo-root> and <port> before any flags.

Getting the binaries

One command, no Rust toolchain and no OpenSSL headers. macOS and Linux, x86-64 and arm64:

curl -fsSL https://github.com/deadcaf3/choir/releases/latest/download/choir-node-installer.sh | sh
curl -fsSL https://github.com/deadcaf3/choir/releases/latest/download/choir-cli-installer.sh | sh

Take both: every command on this page is choir, and choir node serve execs the daemon. The four binaries land in $CARGO_HOME/bin (~/.cargo/bin by default) as choir-node, choir-ssh, choir and choir-mcp; each release publishes a SHA256 per archive and a sha256.sum.

cargo binstall choir-node choir-cli fetches the same archives. From a git checkout, needing only a Rust toolchain:

cargo install --git https://github.com/deadcaf3/choir choir-cli choir-node

Neither package pulls in choir-actor, the one crate that needs this workspace’s LIBSQLITE3_FLAGS.

From source: the toolchain and headers the README lists, then:

cargo build --release -p choir-node -p choir-cli
export PATH="$PWD/target/release:$PATH"

choir node status and choir --version report the built commit from a stamp the release workflow sets.

One command

choir host

It prints the URL people use and, with --invite, one invite link:

choir host --repo me/thing.git --invite Ada

The two public modes take two commands. Obtaining a certificate is privileged and choir host does not run sudo: it prints the line to paste and exits 3. Paste it, run the same choir host again, and it continues.

$ choir host --domain node.example

  ok  state           /home/you/.choir — credential, key, trusted keys

  next: a certificate for node.example has to be issued as root.
  certbot writes /etc/letsencrypt, and the renewal hook that keeps
  this working for the next two years lives there too. Paste this:

    sudo choir node tls node.example --user you --port 8417
    sudo ufw allow 8417/tcp  &&  sudo ufw allow 80/tcp

  then: choir host --domain node.example

Dry-run the certificate step without spending a Let’s Encrypt rate-limit slot:

sudo choir node tls node.example --user "$(id -un)" --dry-run

Why a public bind needs a certificate at all

choir-node refuses a non-loopback bind without TLS (invariant 9). There is no flag to soften it.

--public, and why the address is ugly

choir host --public uses an sslip.io name built from the box’s address, 203-0-113-7.sslip.io for 203.0.113.7, so a certificate can be issued with no DNS. Replace it any time:

choir host --domain the-name-you-bought.example

sslip.io and nip.io share one Let’s Encrypt rate-limit pool. If issuance fails for that reason, --public-name <name> takes any name pointing at this box.

What it did

Every step is a command you could run yourself:

Two things it prints but never runs:

• the firewall. ufw or firewalld, opening the serving port, and port 80 when HTTP-01 is the challenge (renewals rebind it every ~60 days).
• linger. Without loginctl enable-linger, systemd stops a --user unit at logout. choir host stops with the line to paste; --yes accepts a node that dies at logout.

TLS, and what happens every 60 days

The daemon terminates TLS itself with --tls-cert and --tls-key. An nginx or Caddy front is a valid topology; it is not required.

The daemon reads its certificate once, at bind. The certbot deploy hook choir node tls installs re-projects the pair into ~/.choir/tls/ and restarts the unit. Renewal runs on certbot’s timer.

Root for certbot, an unprivileged account for the node; the node’s copy of the pair is a copy for that reason.

The ACME account is registered without an email. sudo certbot renew --dry-run is the manual check; choir doctor reports the expiry date.

Checking it

choir doctor

On a hosting machine it adds six rows: bind address, TLS on, certificate expiry, linger, unit loaded and running, and whether the public URL answers (asked from the box, so it proves name and certificate, not the firewall).

Uninstalling

choir node uninstall

Removes the unit and stops the node. Keeps ~/.choir: keys, repositories and the op log. If a renewal hook is installed it prints the two sudo lines that remove it and the certificate.

In a container

Dockerfile at the repository root builds from source and runs the daemon as an unprivileged user with /var/lib/choir as a volume. Loopback mode and bring-your-own-certificate mode; no ACME client in the daemon. No compose file: the volume and the port are one flag each.

By hand

choir init                                  # the same layout, without a service manager
choir node serve                            # runs here, in this terminal
choir node serve -- --reviewers-file ~/.choir/reviewers   # any daemon flag, after `--`

choir node serve derives root, port, credential and trusted-key file from what choir init wrote, then execs the daemon. Three files switch behaviour by existing:

Supervised, step by step:

choir init                      # mint ~/.choir: credential, key, trusted keys, config
choir node install              # hand it to launchd (macOS) or systemd (Linux)
choir node status               # health, the commit serving, sequencer position
choir repo create me/thing.git  # a repository on the running node
choir repo url me/thing.git     # the clone URL, and the git config to go with it
choir node logs                 # the tail of the daemon log
# choir node restart | stop | uninstall   -- uninstall keeps ~/.choir

choir node install writes a unit that runs choir node serve. Daemon flags go after -- and are recorded in the unit:

choir node install -- --acl-file ~/.choir/acl --rate-limit-api 60

The daemon can be run directly:

choir-node /tmp/choir-repos 8417 \
  --create owner/demo.git \
  --auth-file ~/.choir/auth \
  --keys-file ~/.choir/keys

Useful flags: --bind, --tls-cert / --tls-key, --acl-file <file> (required before a second credential), --request-log <file> and --rate-limit-api / --rate-limit-git (also required before a second credential), --quota-push-bytes / --quota-workspaces, --api-body-limit, --batch-limit, --ready-min-free-bytes, --read-only-browser, --journal <file>, --require-assignment, --protected-refs <file>, --require-review, --reviewer-conflict-graph <file> with --reviewer-conflict-distance <hops>, --review-retention <count>, and --review-lapse-after-secs <seconds>. Authenticated operations endpoints: /healthz, /readyz, /metrics. Flag reference: crates/choir-node/src/main.rs module docs, or AGENTS.md.

File formats (all mode 0600)

Hot-reload: trusted keys, channel bindings, push-certificate signers, reviewers, and ACL grants take effect on the next request.

Warning

Without --acl-file, every credential reaches every repository. A second user:token line can clone every repo, push to any unprotected ref and provision workspaces anywhere. Do not issue a second credential without an ACL. choir host writes one.

Behind a TLS proxy

For the private beta, bind 127.0.0.1 and terminate TLS at a hardened reverse proxy. Do not use the legacy direct-TLS installer. The beta service renderer requires an ACL, protected-ref review policy, scoped operations, operator-issued auth, and the read-only browser.

docs/private-beta-runbook.md covers the network hold, renderers, backups, CI packaging, staging promotion, monitoring, rollback and go-live receipts.

scripts/flip/RUNBOOK.md is the operator’s own dogfood procedure; not the page to start from.

Authorization

Five separate questions:

Important

Without --acl-file, every credential reaches every repository.

Per-repository authorization (D29)

Three whitespace-separated columns, an optional fourth, # comments:

# <user>   <repo|*|@node>   <level>    [until=<unix seconds>]
alice      owner/demo       write
bob        owner/demo       read
bob        owner/notes      write
carol      *                read
dave       @node            auditor
erin       owner/demo       propose
frank      owner/demo       write      until=1788000000

Each level adds to the one above:

propose admits one thing (D60): a push to refs/for/<branch>/<user>/<topic>, which opens a review (D53). Every other ref is refused. The pusher’s own name is a required segment. The grant is checked at the smart-HTTP boundary and again when the pre-receive hook reports the refs.

The operator’s credential usually wants:

myself     *      write
myself     @node  write

* never covers @node. @node auditor reads /api/log and /api/ref-agreement; @node write is needed for ops naming no repository, such as key bindings. Vouching (D65) needs only @node auditor.

A grant that ends (D66)

The fourth column is a deadline in unix seconds. The table is dated on every request; no restart.

A deadline lapses downward. Pair a permanent read with a write that ends:

frank      owner/demo       read
frank      owner/demo       write      until=1788000000

A single expiring grant leaves the holder with a 404.

• A deadline in the past parses and never matches; startup says acl enabled (7 grants, 1 expired).
• until= is absolute: date -v+90d +%s on macOS, date -d '+90 days' +%s on GNU.
• The node’s clock decides.

own may carry a deadline; when it lapses the landing gate returns to the approval-weight rule (D42).

Fail closed: anything not granted is refused. An unreadable repository answers 404, never 403. The flag requires --auth-file. A malformed file refuses to start; a malformed edit keeps the previous table and complains.

/api/view, /api/reviews and the browser page are narrowed to readable repositories. Node-wide sections (ref-state attestation, key bindings, vouch graph, telemetry) need @node auditor; the log head and build stamp reach everyone. A review you were assigned to reaches you on any repository.

Repository ownership (D42)

With --protected-refs and --require-review, a protected ref needs approval weight 2 from two distinct operators. own changes the question:

myself     owner/demo       own

On a protected ref of an owned repository, one owner’s assent is necessary and sufficient: the owner lands it, or the owner approved a review naming that exact (ref, commit).

• An owner’s key is equivalent to the repositories they own.
• An owner submitting directly must have their key bound in --keys-file (<channel> <64-hex>). Approving a review does not need this.
• own is granted in the file you write; self-service cannot issue it.

The file is re-read per landing. An unreadable or malformed file refuses the landing.

Landing a review with the reason it was allowed (D43)

Submit is a ref move with the gate’s answer attached:

{"Submit": {"review": "r1", "name": "demo.git:refs/heads/main",
            "commit": {...}, "prev": {...},
            "authorization": {"format_version": 1,
                              "basis": {"OwnerApproved": {"owner": "myself"}},
                              "approvers": [{...}]}}}

Basis is OwnerLanded, OwnerApproved, or ApprovalWeight {required, met}.

The authorization is never a client’s to assert. Post a Submit; on mismatch the rejection’s expected field is the gate’s record. Sign that verbatim and post again:

curl -u "$USER" -X POST https://<HOST>/api/submit -d "$FIRST_ATTEMPT" \
  | jq -r .expected          # the authorization the gate produced

• this landing cannot name its approvers: an approving channel has no key binding (or two). Bind the reviewer’s key and merge again.
• is not gated on this node: unprotected ref, or no --require-review. Use SetRef.

Archiving a review discards its verdicts; the entry bytes remain the answer to who approved a landing.

Rotating a key without breaking anything (D44)

An approval is credited against the key live when the verdict was cast.

choir revoke <api> <node-key-file> <old-pubkey-hex> "laptop lost"
choir bind   <api> <node-key-file> <operator> <new-pubkey-hex> [channel]

Both take the public key hex choir key prints and are node-signed.

• An approval cast by an already-revoked key cannot be credited; the reviewer needs a fresh key and verdict.
• choir log --verify checks signatures as of their own position. It fetches revocation positions from /api/view, so it needs API access.

Warning

Keep revoked keys in the trusted-keys file. The log stores a key id, never the public key, so deleting the line makes every entry that key signed permanently unverifiable.

That applies to ed25519 keys only. A passkey-signed entry carries its own credential key (D45); choir log --verify counts those as intact but unanchored.

Review retention is opt-in: --review-retention N archives completed reviews when more than N remain live. --review-lapse-after-secs is invalid without it.

Issuing a credential without editing a file (D36)

--accounts-file <path> turns on invite-only self-service; issued credentials are added to what the auth and ACL files say.

Passkeys are --passkeys. Without it, POST /api/accounts/passkey and /account answer 503.

choir-node ./repos 8417 --auth-file ~/.choir/auth --acl-file ~/.choir/acl \
  --keys-file ~/.choir/keys --accounts-file ./repos/.choir/accounts.json \
  --ssh-handoff ./repos/.choir/ssh-handoff \
  --ssh-authorized-keys ./repos/.choir/authorized_keys

The console (D72)

/people, for @node write: the queue of people asking for access, a button to let one in, and a form that mints an invite link.

A stranger can ask for access on the front page and keeps the link it gives them; granting turns that link into their invite. Each request costs a proof of work; the queue caps at 64.

A POST whose Origin names another site is refused.

The same three by hand

Mint an invite, as @node write:

curl -u "$OPERATOR" -X POST https://<HOST>/api/accounts/invite \
  -d '{"user":"bob","grants":["owner/demo read","owner/notes write"]}'

The invite names nobody (D75); the username is theirs to pick. Send {"user":"buildbot"} when the name must be exact.

The response carries invite, an id:secret pair, once. It is presented as basic auth to one endpoint:

curl -u "<INVITE>" -X POST https://<HOST>/api/accounts/redeem \
  -d "{\"ssh_key\":\"$(cat ~/.ssh/id_ed25519.pub)\"}"

That answers with the clone token and registers the key for SSH. Invites are single use and expire in a day (expires_in_secs).

In a browser the link enrols a passkey and opens a session. A token for git is minted from /account (POST /account/token), one per account.

GET /api/accounts lists accounts, live invites and pending requests. POST /api/accounts/request/grant with {"request_id":"ask-...","grants":[...]} answers one; POST /api/accounts/request/decline drops one; POST /api/accounts/revoke with {"user":"bob"} deletes an account, its grants and its authorized_keys line. Revocation is deletion.

Caution

@node can never be issued by self-service. The flag requires both --auth-file and --acl-file.

• An issued grant may carry a deadline: {"grants":["owner/demo write until=1788000000"]}.
• The generated authorized_keys is generated. Point sshd at it (AuthorizedKeysFile /path/to/repos/.choir/authorized_keys, see Git over SSH) and never edit it.

Publishing a repository to everybody (D78)

One line in the ACL opens one repository to readers with no account:

@anon    owner/project.git    read

@anon is the reader who presented no credential. An unauthenticated browse or fetch is evaluated under that principal; every check after the gate is unchanged. No flag.

It cannot be authenticated as. Account names are ASCII letters, digits, -, _ and ., so @ is unspellable. A user called anon is an ordinary account.

Three grants it will not take, refused at parse time:

What opens: the browse surface for that repository and the read half of git smart-HTTP, so git clone works with no credentials. What does not: /api/view, /api/log, /reviews, and git-receive-pack. An unpublished repository answers exactly as one that does not exist.

Pair with --site-repo owner/project to make that repository the front page.

Rate limits, quotas and fairness

The flagged ones require --auth-file and share three exemptions.

Request log and rate limiting (D33)

cargo run -p choir-node -- /tmp/choir-repos 8417 \
  --auth-file ~/.choir/auth \
  --acl-file ~/.choir/acl \
  --request-log ~/.choir/requests.jsonl \
  --rate-limit-api 600 \
  --rate-limit-git 120

The request log is one JSON object per served request:

{"format_version":1,"at_unix_ms":1755100000000,"user":"alice","method":"GET","path":"/api/view","us":812,"status":200,"bytes":4310}

Refusals are recorded too: a 401 as anon (never the attempted username), a 429 against the charged user, a failed response with "status":0. The path is truncated at ?; headers and bodies never reach the file.

Rotation: past --request-log-max-bytes (32 MiB default) the file becomes <path>.1 and a fresh one starts. Two generations kept. Writes are unbuffered and unsynced.

Rate limiting is a token bucket per user per class, in memory, requests per minute with one minute’s burst. Over-limit answers 429 with Retry-After in seconds. The browser pages count against the API bucket.

Exempt:

With --auth-file but no --acl-file, only the hook callback is exempt.

Per-user quotas (D37)

cargo run -p choir-node -- /tmp/choir-repos 8417 \
  --auth-file ~/.choir/auth \
  --acl-file ~/.choir/acl \
  --quota-push-bytes 268435456 \
  --quota-workspaces 20

--quota-push-bytes bounds one git request body. Over it: 413 with both numbers. Checked before git http-backend spawns, so a refused push runs no hook. The body is drained so the client reads the 413.

--quota-workspaces bounds workspaces per user: 403 with quota_exceeded. The count is folded out of the op log at startup.

• Checked on POST /api/workspace, not POST /api/submit.
• Read outside the provisioning lock, so two simultaneous creations at the ceiling can both pass.

Fairness at the sequencer’s door

Each actor holds at most 512 ops awaiting a decision (twice the writer’s 256-op batch), served round-robin. The total order is untouched: the writer stamps seq alone.

Over the bound, a submission is rejected naming the count and the limit; retry when one of your own ops completes.

• The actor key is a claim, verified later on the writer thread. It bounds an honest flooder.
• The wait is unmeasured. decision_latency starts when the writer picks an op up.

Important

Nothing here may become load-bearing for authorization.

The bound has no flag.

Webhooks (D32)

The node’s only outbound request to an address somebody else chose. It fires when a named ref moves and is best-effort; a receiver that may not miss a ref polls GET /api/log?from=N.

--hooks-file, one subscription per line, # comments:

# <repo:refname pattern>        <url>                        <secret>   [allow-private]
owner/demo:refs/heads/main      https://ci.example/choir      <SECRET>
owner/demo:refs/heads/*         https://ci.example/branches   <SECRET>
owner/notes:refs/tags/*         http://127.0.0.1:9000/hook    <SECRET>   allow-private

Patterns follow --protected-refs: trailing * is a prefix, else exact, matched against <repo>:<refname>. Re-read on mtime change. Needs --keys-file. Delivery records go to <repo-root>/.choir/hooks.jsonl.

The body:

{"format_version":1,"event":"ref-landed","repo":"owner/demo","ref":"refs/heads/main",
 "ref_key":"owner/demo:refs/heads/main","old":"<git oid>","new":"<git oid>","seq":41,
 "entry":"<entry hash>","actor":"<channel>","key_id":"<signing key id>"}

old is null for a created ref, new null for a deleted one. entry is unique per event; discard repeats on it.

Verify the secret. The delivery carries X-Choir-Hook-Secret: <secret>. It is a bearer secret, so give each subscription its own (openssl rand -hex 32), keep the file 0600, and use https for any non-loopback target; the node refuses to send the secret in clear.

Best-effort, never silent. Three attempts per delivery; every attempt, refusal and dropped event is a line in hooks.jsonl. A bounded queue on its own thread drops events rather than delaying op admission.

Targets are vetted. Loopback, private, carrier-NAT, link-local (including 169.254.169.254), unique-local and unspecified addresses are refused unless the line ends in allow-private. It connects to the vetted address and follows no redirects.

Observability and repair

All three are derived data: unsynced writes, dropped under load.

Warning

Keep the op log. None of these substitutes for it.

Metrics and what can be alerted on (/metrics)

Prometheus text format, authenticated.

Gauges: choir_ready, choir_log_verified, choir_sequencer_live, choir_storage_writable, choir_free_disk_bytes, choir_ref_disagreements, choir_process_start_time_seconds.

Counters: choir_requests_total, choir_requests_unauthorized_total (401 and 403), choir_requests_throttled_total (429), choir_requests_failed_total (5xx and unfinished responses), choir_request_duration_microseconds_total. Counters increment whether or not --request-log is on. Every scrape is one request behind.

The rules themselves

scripts/flip/choir-alerts.rules.yml: choir-node-state off the gauges, choir-node-traffic off the counters. Every window and rate is a starting point. The latency rule is a mean.

every_metric_these_alert_rules_name_is_one_the_node_exports in crates/choir-node/tests/limits.rs checks every choir_* name the rules read is still exported.

The alerts a node cannot source

Of the nine critical alerts in docs/private-beta-runbook.md, five come from this endpoint: readiness (choir_ready), durability (choir_sequencer_live), disk (choir_free_disk_bytes), request spikes and latency (the counters), restart loops (choir_process_start_time_seconds).

Decision journal (--journal)

Accepted and refused ops are both 200 on POST /api/submit; the journal records the decision:

cargo run -p choir-node -- /tmp/choir-repos 8417 \
  --keys-file ~/.choir/keys \
  --journal ~/.choir/decisions.jsonl

{"format_version":1,"kind":"decision","actor_id":"8f3a…","workspace":"op/agent","op_type":"SetRef","decision":"accepted","reject_reason":null,"seq":41,"parent":"…","decision_latency_us":812}

Every record carries kind:

Derived data (Architecture): written on its own thread, dropped rather than stalling the writer. The flag gates construction too.

Repairing a log (choir repair)

FileLog::open truncates a torn final record automatically, after saving the bytes to <log>.torn-<offset>. Everything else is explicit:

cargo run -p choir-cli -- repair ~/.choir/repos/.choir/ops.jsonl --verify

Damage anywhere but the tail is refused: the tool prints restore-from-backup steps and exits 1.

Taking a node with you (--export, D61)

cargo run -p choir-node -- --export ~/.choir/repos /tmp/choir-export
cargo run -p choir-node -- --verify-export /tmp/choir-export
cargo run -p choir-node -- --import /tmp/choir-export /srv/new-root

Offline, on the node’s machine. Writes ops.jsonl, one repos/<owner>/<name>.git.bundle per repository, and manifest.json with its own format_version. A never-pushed repository is listed without a bundle.

--verify-export requires every ref the log names to be in a bundle at the same oid. Extra refs in a bundle are reported as ahead; a log naming a commit no bundle holds is refused.

An export is secret-free by construction: the output is walked and refused if it holds anything named auth or ending .key or .pem. The signing key stays with the node (Restoring from a backup). Policy files stay behind; the manifest records that.

--import verifies, then refuses a root holding a log or any named repository. It places files; the daemon adopts them. Settle a restore by accepting a write.

scripts/flip/pull_backup.sh is the disaster-recovery path over ssh; --export is the format tool beside it.

Transports and the browser surface

All four sit behind the same auth wall and --acl-file grants; a repository you hold no grant on answers 404 on all four.

Browser surface

/ with a credential serves the repository index and one read-only page per repository: refs, the review queue, the latest ref-state attestation, workspaces, and sequencer health.

Exception: the bare address (D57). GET / with no credential gets a static front page: what a node is, the commands to use one, and that it is invite-only. A wrong credential still gets 401.

Pages are server-rendered from /api/view, cached by view sequence, and revalidated with an ETag.

Browser writes exist in one place (D39): a verdict or comment on a review page, and passkey enrolment on /account, signed with a key that never leaves the device.

The client half is /static/webauthn.js. Its two pages are served script-src 'self'; every other page, including /r/, is default-src 'none'.

The pages that are not repositories

A text/html request not naming a .git path gets the sign-in page as the body of its 401 (D74); git, curl and API clients get a bare 401 with WWW-Authenticate. Repositories granted to @anon are the exception (D78).

crates/choir-node/tests/it/routes.rs holds the expected status of every route for three readers and crawls the surface.

The other half of the site (D76)

The node is at the apex; the book is on a docs. subdomain. Each is told where the other is at run time.

Point the node at the book. One line, no restart:

mkdir -p <root>/.choir
printf 'https://<docs-host>\n' > <root>/.choir/docs-url

<root> is the repositories directory. The value must be absolute http:// or https://. Read per request.

Point the book at the node. Two repository variables, read by .github/workflows/pages.yml:

Also enter the domain under Settings → Pages and add a DNS CNAME for <docs-host> pointing at the Pages host.

The workflow passes CHOIR_DOCS_REPO_BASE, which repoints links to files outside docs/ at the commit being published.

Repository browsing

/r/ lists the repositories your credential may read:

Same read grant a clone needs. Content pages revalidate on the commit oid.

Files over 512 KiB are described, binaries are named, diffs truncate at 2,000 lines. Revision arithmetic, traversal and option-shaped input are refused at the router.

A review page shows the target ref, reviewers and verdicts, approval weight, slashing, and a three-dot diff. Comments are signed PostComment operations (D38).

Git compatibility path

choir repo url owner/repo.git      # prints the URL, and the git config for the credential
git clone http://127.0.0.1:8417/owner/repo.git
git -C repo config credential.helper '!choir git-credential ~/.choir/auth'

git push origin HEAD:main

The credential is not in the URL. Pushes are CAS-sequenced: on rejection, fetch, rebase or merge, push again.

Warning

Never force-push over a sequencer rejection. See The contribution workflow.

Git over SSH (D31)

The host’s sshd serves git@host:owner/repo.git through a forced command that hands each connection to choir-ssh.

Start the node with a handoff file, which carries the daemon’s address and loopback secret:

choir-node <repo-root> 8417 --auth-file <auth-file> --keys-file <keys-file> \
  --acl-file <acl-file> --ssh-handoff <handoff-file>

One authorized_keys line per registered key:

command="/usr/local/bin/choir-ssh --root <repo-root> --user <choir-user> --acl-file <acl-file> --handoff <handoff-file> --git-binary /usr/bin/git",restrict ssh-ed25519 AAAA... <user>@<host>

--user is the choir username for that key. restrict turns off pty, agent, port and X11 forwarding. Set --git-binary explicitly.

git clone ssh://<ssh-account>@<SERVER_IP>/owner/demo.git
git clone <ssh-account>@<SERVER_IP>:owner/demo.git

The shim serves:

• exactly git-upload-pack '<repo>' and git-receive-pack '<repo>', one argument, never a shell. Anything else is refused.
• owner/repo or owner/repo.git, two ASCII segments, none starting with a dot.
• the same --acl-file as HTTP: read to fetch, write to push. Omit --acl-file to use the daemon’s.
• pushes through the repository’s pre-receive hook, sequenced like HTTPS.

Caution

A shim installed without --handoff serves fetches and refuses pushes.

Limits:

• the handoff file holds the loopback secret at 0600, so the SSH account and the daemon must be the same uid.
• one line per key; revocation is deleting the line. With --accounts-file the node writes those lines (above); point AuthorizedKeysFile at the generated file.
• sshd stays the operator’s.

The choir CLI and the node’s HTTP API

The complete surface, rendered from one table shared with the CLI’s --help, AGENTS.md, /llms.txt and /api/schema. A staleness test fails the gate when they differ.

Authentication and exit codes

Auth on the CLI is flags, not env:

choir --auth-file ~/.choir/auth --auth-user choir <command> ...

Exit codes: 0 accepted, 1 rejected (JSON body printed, see ERRORS.md), 2 usage.

The signed-operation API is the primary agent path: it carries actor identity and batches many operations behind one durability barrier. git push remains the compatibility and bulk-transfer path.

Signed-operation CLI and API (primary agent path)

HTTP endpoints

The choir CLI

getting started

• choir host [--domain <name> | --public [--ip <addr>] | --public-name <name>] [--port <n>] [--repo <owner/name.git>] [--invite <name>] [--state <dir>] [--yes] [--dry-run] [--foreground] [-- <daemon flags>]
  take this machine from nothing to a running node and print its URL; bare binds loopback, –domain issues a Let’s Encrypt certificate for a name you own, –public uses a magic-DNS name over this box’s address, –foreground execs the daemon instead of installing a unit
• choir init [<state-dir>] [--port <n>] [--force]
  set up a node’s layout on this machine: repository root, credential at 0600, an actor key the node trusts, and .choir/config; refuses to overwrite what exists
• choir key <key-file> [name]
  mint a key and print the line the operator registers; pass your channel name to print the bound form
• choir git-credential <auth-file> [--auth-user <name>] get|store|erase
  git credential helper: hands git your token on stdin so it never lives in a remote URL; configure with git config credential.helper '!choir git-credential <auth-file>'
• choir join <link> | <api> <invite-file> <key-file> [--user <name>] [--channel <name>] [--key-file <path>] [--ssh-key <path>] [--token-file <path>]
  redeem an invite link and set this machine up: actor key at ~/.choir/agent.key, token at ~/.choir/auth (0600), a git credential helper for that node, and the node URL in ~/.choir/config; –user names the account when the invite left it open, asked on the terminal otherwise; the three-argument form takes the invite from a file, answers JSON and touches neither git nor your home directory
• choir docs [--open]
  build the book from docs/ with the API documentation inside it at book/api/; needs a checkout and mdbook, and names the install command if it is missing
• choir skill install [--into <dir>]
  install the choir agent skill (default .claude/skills), rendered from this binary’s own surface table; re-run after upgrading

changing code

• choir workspace <api> <owner/repo> <name> [--base <git-oid> --owner <channel> --key-file <path> --change <id> --idempotency-key <key>] [--path <prefix>]...
  provision a CoW workspace; advanced flags owner-sign an exact base and stable change, and each –path owner-signs a subtree
• choir checkpoint <api> <key-file> <channel> <change-id> <workspace-id> <git-oid>
  publish an immutable change revision after committing and pushing its git object
• choir propose [reviewer]... [--key-file <path>] [--channel <name>] [--api <url>] [--repo <owner/repo>] [--remote <name>] [--onto <branch>] [--change <id>] [--path <prefix>]...
  create a change, push its commits and request review, with no arguments; run from a git checkout, with the key and channel from ~/.choir, every value overridable by flag; re-running after an amend updates the same proposal; a leading <key-file> <channel> pair is still accepted
• choir workspace-archive <api> <key-file> <channel> <owner/repo> <name> <change-id> <idempotency-key>
  owner-sign and recoverably archive a bound workspace; exact retries are idempotent
• choir submit <api> <key-file> <channel> '<op-json>'
  sign and submit one raw operation
• choir batch <api> <key-file> <channel> <ops-file>
  sign and submit many operations as one batch, the primary path for agent workloads; one op per line, - reads stdin, one result line per op
• choir intent <api> <key-file> <channel> <subject> <kind> '<body>'
  publish a task spec or plan so other agents can see intent
• choir state <api> <channel>
  list what you owe and what you are waiting on; every row carries the command that answers it and its risk

review

• choir review <api> <key-file> <channel> <id> <git-oid> [--ref <repo:ref>] [reviewer]...
  request review on a commit; name no reviewers and the node draws them
• choir verdict <api> <key-file> <reviewer> <id> approve|request-changes [note]
  answer a review you were assigned
• choir comment <api> <key-file> <channel> <review-id> <comment-id> '<body>'
  say something on a review; append-only, and the comment id is your retry identity
• choir viewed <api> <key-file> <viewer> <review-id>
  record that you read a review; first read only, resubmitting is refused
• choir slash <api> <node-key-file> <id> <reviewer> '<reason>'
  invalidate one reviewer’s approval; operator-only and never moves a ref
• choir abandon <api> <node-key-file> <id>
  archive a stale incomplete review as lapsed, settling it unapproved; operator-only and never moves a ref
• choir reviews <api> <reviewer>
  your pending review queue

checks

• choir check <api> <key-file> <channel> <git-oid> <name> passed|failed|running|errored [evidence] [--ref <repo:ref>]
  report one automated check’s outcome on a commit; any runner or person can report by signing, and the node never runs the check
• choir checks <api> <git-oid>
  every check reported on a commit, and one verdict; exits 0 passed, 1 failed or unreported, 3 still running, 4 could not be run

trust

• choir witness <api> <key-file> <channel>
  cosign the node’s current ref-state attestation (D67); the snapshot id is read from the view, and the node may not witness its own
• choir vouch <api> <key-file> <channel> <subject> [note]
  vouch for another operator; both ends need a key bound in the log, and it authorizes nothing on its own
• choir unvouch <api> <key-file> <channel> <subject> '<reason>'
  withdraw a vouch; both ops stay in the log, and vouching again starts a fresh clock

reading the node

• choir schema <api>
  print this node’s machine-readable API description and its live capabilities
• choir log <api> [--from <n>] [--verify] [--keys <file>]
  read log entries from a cursor; –verify checks continuity, recomputes every hash and verifies the signatures whose keys you hold
• choir appeal <api> <attempt-id>
  appeal a rejected newcomer attempt for operator adjudication; never grants privilege
• choir profile <api> <channel>
  what the log records about one actor: keys and their age, changes owned, verdicts given, checks reported
• choir search <api> <term> [--in files|code|commits] [--repo owner/name] [--rev R] [--limit N]
  find a literal term across every repository you may read
• choir triage <api>
  every review and change in a bucket (landed, awaiting verdicts, changes requested, approved awaiting landing), most actionable first, capped, with truncation marked in-band
• choir funnel <api>
  the contribution funnel from admission to first verdict and the steepest drop between stages; counts what this credential may read, and reports an unmeasured stage as null
• choir view <api> [--limit <n>] [--offset <n>]
  read the materialized view, its ref-state attestation and the node’s health counters; map-shaped sections page 200 rows at a time, with <section>_omitted and paging.next

operating a node

• choir invite <api> <name> <owner/repo> [read|write]
  mint an invite and print the one link to send; the same thing the /people page does
• choir asks <api>
  who has asked for access and is waiting on an answer (D72)
• choir grant <api> <request-id> <owner/repo> [read|write]
  let one of them in; the link they already hold becomes their invite
• choir decline <api> <request-id>
  drop a pending request; their link then reads as never valid
• choir runner <config-file>
  drive one workspace lifecycle step for an orchestrator; JSON request on stdin, JSON result on stdout
• choir bind <api> <node-key-file> <operator> <key-hex> [channel]
  record in the log that a key belongs to an operator; operator-only and never moves a ref
• choir revoke <api> <node-key-file> <key-hex> '<reason>'
  withdraw a key binding; terminal, and the attribution row survives
• choir acl render <api> <acl-file>
  rewrite an ACL file’s trailing comments to name the person behind each handle; grants are copied through unchanged
• choir repo create <api> <owner/repo.git>
  create a repository on a running node, sequenced from its first push; needs a node-wide write grant, answers 409 when it already exists, and prints the clone URL
• choir repo list <api>
  the repositories on a node this credential can read, one per line; an ACL narrows the list rather than refusing it
• choir repo url <api> <owner/repo.git>
  the clone URL for a repository, and the one line of git configuration that makes pushing work; the credential is never put in the URL
• choir node serve [--state <dir>] [--port <n>] [--create <owner/repo.git>] [-- <daemon flags>]
  run the node in this terminal, deriving root, credential and trusted keys from what choir init wrote; execs the daemon so signals and the exit code reach the real process
• choir node install [--state <dir>] [--port <n>] [-- <daemon flags>]
  hand the node to launchd (macOS) or a systemd user unit (Linux) so it survives logout, crash and reboot; the unit runs choir node serve
• choir node tls <domain> --user <account> [--port <n>] [--dry-run | --staging]
  obtain a Let’s Encrypt certificate for this node and wire up renewal: certbot, a deploy hook that re-projects the pair and restarts the node, and the marker node serve reads; the only command here that expects root, and --user is required
• choir node stop
  stop the supervised node for this boot, leaving the unit in place; node uninstall is the one that ends it
• choir node restart
  reload the unit and start it again, which is how a rebuilt binary reaches the running node
• choir node uninstall
  stop the node and remove its unit; the state directory, with the keys, repositories and op log, is kept
• choir node logs [<lines>] [--state <dir>]
  the tail of the node’s log; defaults to the last 30 lines
• choir node status [<api>]
  health, the commit serving, the sequencer’s position and its p99 against the 100 ms gate, and how much this credential can see
• choir doctor [<api>] [--state <dir>]
  check everything the other commands assume: the binaries shelled out to, the auth file and its mode, and whether a node answers; each failure prints the fix; on a hosting machine it adds bind address, TLS, certificate expiry, linger, unit state and whether the public URL answers
• choir backup verify <backup-dir>
  whether a backup can be restored from: the four files, the manifest checksum, the hash chain, the policy archive and every git bundle, refusing a backup that carries a key or credential; every check local
• choir backup restore <backup-dir> <target-root>
  turn a backup back into a node and prove it by accepting a real push: reads and refuses before writing, unbundles git objects before the first boot, rehearses on a port it picks; exit 3 means a secret only you can supply is missing
• choir repair <log-file> --verify | --truncate-tail
  inspect a stopped node’s op log, or repair a tail that was still being written; --verify changes nothing, --truncate-tail quarantines the partial record before cutting, and damage anywhere but the tail is refused

Most commands take the node’s URL first. Put node = <url> in .choir/config, in the working directory or any parent, and it is filled in when omitted. choir <command> --help prints one command’s spec.

Exit codes: 0 accepted, 1 the node rejected (its JSON error body is printed), 2 usage error.

Live surface on a running node: GET /llms.txt. Sync verification: SYNC.md / GET /sync.md.

MCP adapter

A synchronous stdio adapter mapping generated tools onto the same HTTP endpoints; no second implementation, no session state.

choir-mcp http://127.0.0.1:8417 --auth-file ~/.choir/auth --auth-user choir

Serves the legacy handshakes and the stateless 2026-07-28 request path. Tool order and schemas come from crates/choir-cli/src/surface.rs.

The contribution workflow

One change, from an empty workspace to a landed ref. Every step is a signed operation in the node’s total order.

choir propose does steps 1, 3 and 4 from a git checkout. choir state answers “what next” from the node’s view.

Minimal day-one loop

API=http://127.0.0.1:8417
A=(--auth-file "$HOME/.choir/auth" --auth-user choir)
OWNER=myop/agent
CHANGE=change-1
WORKSPACE=owner/demo/agent-a

# 1. Confirm the node
choir "${A[@]}" view "$API"

# 2. Exact-base CoW workspace and stable change
choir "${A[@]}" workspace "$API" owner/demo agent-a \
  --base "$(git rev-parse HEAD)" --owner "$OWNER" --change "$CHANGE" \
  --idempotency-key request-1

# 3. Publish intent. After editing, commit and push before checkpointing.
choir "${A[@]}" intent "$API" "$HOME/.choir/agent.key" "$OWNER" "$CHANGE" task 'ship feature X'
git push origin HEAD:refs/heads/agent-a
choir "${A[@]}" checkpoint "$API" "$HOME/.choir/agent.key" "$OWNER" \
  "$CHANGE" "$WORKSPACE" "$(git rev-parse HEAD)"

# 4. Request review with no reviewer names so the node draws them
choir "${A[@]}" review "$API" "$HOME/.choir/agent.key" "$OWNER" rev-1 "$(git rev-parse HEAD)" \
  --ref owner/demo.git:refs/heads/main

# 5. Drawn reviewers answer
choir "${A[@]}" reviews "$API" otherop/reviewer
choir "${A[@]}" verdict "$API" "$HOME/.choir/other.key" otherop/reviewer rev-1 approve

Review rules

• Name no reviewers on choir review; the node draws them. Self-picked lists may be refused under --require-assignment or protected refs.
• Channel names are operator/agent. Same-operator agents cannot review each other.
• Register keys with choir key ~/.choir/agent.key myop/agent >> ~/.choir/keys.
• Protected landing with --require-review needs approval weight 2 (two operators), unless somebody holds own, in which case one owner’s assent lands it (D42). scripts/flip/RUNBOOK.md enables the gates.
• choir slash invalidates a bad approval and requires re-review; it never rewrites a landed ref.
• An optional reviewer conflict graph excludes operators within a hop distance of the requester; it fails closed by leaving the review unassigned.
• choir view reports T3 concentration with exact counts; unknown attribution makes the status indeterminate.
• choir view reports view_growth; total_authoritative_view covers workspaces, refs, reviews and provenance and excludes runtime projections.
• choir view reports newcomer_harm when the two 0600 audit files are enabled. A rejected newcomer can choir appeal <api> <attempt-id>; an appeal never grants privilege.
• Prefer POST /api/submit-batch for several ops (one durability barrier).

Forge follower and speculative GitHub queue: choir-bridge (cargo doc -p choir-bridge). Utility modes: --pubkey, app-debug, post-status, calibrate, harvest (D27, offline). Queue mode can run the advisory D23 detector; it never changes the landing condition. Grant only the bridge permission model; queue --land alone needs contents write.

Agent templates

Snippets for Claude Code, Codex and Cursor, plus a tested Claude Code WorktreeCreate/WorktreeRemove adapter: templates/README.md.

source templates/choir.env.sh   # sets CHOIR_API; optional user/token/key
# then install the harness snippet listed in templates/README.md

Private single-node beta runbook

Status and launch hold

Keep production DNS unpublished and the production firewall closed to non-operator addresses until every go-live receipt below is attached to the release record. Reach staging over the operator VPN or an allowlist.

The beta application is the server-rendered UI in choir-node. No separate frontend.

Product boundary

Enabled:

• Authenticated dashboard, repository browser, commits, diffs, review pages.
• Git smart HTTP clone and push through the reverse proxy.
• Signed CLI operations, operator-issued credentials, repository ACLs, scoped operations, assigned reviewers, protected-ref review gates.
• Authenticated health, readiness and Prometheus metrics endpoints.
• Invite-only self-service accounts (D36); the manifest carries accounts=enabled.
• Sign-in on the node’s own page (D71, D74): username and password plus a passkey button; the session is an opaque in-memory token. A first password sign-in with no passkey lands on /account. Non-browser clients meet 401 with WWW-Authenticate. Passkeys are --passkeys; the renderer refuses passkeys=enabled without accounts=enabled. Browser writes stay off.

Out of scope: anonymous access, browser mutations, SSH, webhooks, the bridge, a separate SPA, a consumer login.

Objectives: 99.5 percent monthly availability, one-hour RPO, four-hour RTO. Ceiling: 25 beta users, 10 concurrent sessions. Per user: 512 MiB per push, eight workspaces, 120 API and 60 Git requests per minute. Raise caps only after a recorded staging load test.

Host and network

1. One supported Linux VM with twice the forecast data, encrypted storage, inode and disk monitoring, and a system account such as choir with /usr/sbin/nologin.

2. Install the checksummed artifact under a versioned release directory and point /opt/choir/current at it. Never build on the production host.

3. Populate the service user’s state directory, outside this repository, with auth, keys, reviewers, protected-refs, repos.list, acl, the three adjudication/audit JSONL files, and an exact copy of scripts/flip/private-beta.manifest. Directory 0700, files 0600.

4. Render the service with scripts/flip/render_private_beta_service.sh, which refuses missing ACL, scope, review gates, policy or manifest. Install as a system unit after review. The node binds 127.0.0.1 with the read-only browser, request and decision logging, limits, quotas, the readiness disk floor and systemd hardening.

5. Render the TLS proxy with scripts/flip/render_beta_nginx.sh <domain> <node-port> <tls-cert> <tls-key>; check with nginx -t before install. It redirects HTTP to HTTPS, sets HSTS and security headers, preserves Authorization, caps bodies at 1 MiB, and gives Git a 512 MiB streaming route with buffering off.

   Pass --behind-tls-proxy. Without it every absolute URL, including the join link, is written http.

   The proxy handles no client address (D59); pre-auth limiting is the node’s node-wide ceiling. Diagnose from the node’s --request-log.

   Warning

   Do not add a per-address limit zone and do not restore the proxy access log.

   After install, send one failing request over TLS and one failed TLS handshake, then confirm no file under the proxy’s log directory names an address.

Expose only 80 and 443 through the allowlist. Confirm the node port is unreachable off loopback with ss -lntp on the host and a connection attempt from another machine.

Warning

Do not use the legacy direct-TLS installer for this beta.

Access and policy

Create credentials on an operator workstation and deliver through the approved secret-sharing system. Grant each beta user only named repositories in acl, no wildcard. Keep an operator credential with the node-wide audit grant and the ownership a recovery needs. Test a denied and an allowed repository from the public host.

Register every signing key in keys and every protected ref in protected-refs. The reviewer pool needs two eligible operators with registered keys; read is enough to review (D55). The service always enables required assignment, review and scope; a missing or malformed policy stops startup.

The browser must return 403 for /account and /api/prepare, and review pages must show no mutation control.

Backup and recovery

Run pull_backup.sh hourly to encrypted off-host storage. It publishes only after checksum, append-only prefix, format version, sequence, parent-chain and recomputed-hash verification. Contents and the secrets you supply: Restoring from a backup.

Store the node identity key separately in the approved secret store. Retain 24 hourly, 14 daily and 8 weekly generations. Run verify_backup.sh daily; alert if the last backup is older than 90 minutes or verification fails.

Before launch and once per release, restore onto a clean host with restore_from_backup.sh, supplying the recovered node key and a new operator auth file. Finish within four hours and prove the log chain, policy files, ref attestation, GUI, clone and a canary push. A second operator must rehearse once from this runbook.

Delivery and rollback

CI uses Rust 1.97.1: formatting, workspace tests, Clippy and rustdoc with warnings denied, the Phase-0 spike, generated-file freshness, RustSec audit. build_private_beta_artifact.sh builds with --locked, stamps the commit, and emits SHA-256 checksums, CycloneDX SBOMs and a versioned artifact.

private-beta-release.yml is manual: it repeats the gate, deploys to staging, and runs the smoke tests. Production promotion is a separate protected environment approval.

smoke_private_beta.sh issues receipt 3:

smoke_private_beta.sh <https-base> <auth-file> <owner/repo.git> \
  [--push-canary] [--denied <owner/repo.git>]

Anonymously, /healthz, /readyz, /metrics, /api/schema, /api/view, the repository page and a git fetch must answer 401. With credentials it repeats the reachable ones, sends one byte over the API body ceiling expecting 413, and clones. --denied covers the denied half of the ACL. smoke_script::a_node_serving_anonymously_fails_the_smoke_script runs it against a real node over TLS.

Read five receipt-3 items off the node: scope policy, review policy, quotas, request logging, and oversized git requests (512 MiB ceiling). The suite covers all five locally; the hostname adds only that the proxy does not alter them.

deploy_private_beta.sh installs into a new release directory, switches the symlink atomically, and restores the old target if the restart fails. rollback_private_beta.sh is the one-command rollback. Rehearse both in staging, recording the active commit before and after.

Monitoring and deliberate alert tests

Probe /healthz, /readyz and /metrics with an operator credential. Readiness checks log format and chain, sequencer durability, a real storage write-and-sync, free disk, and repository/ref agreement.

Warning

Never make these Choir routes anonymous.

Nine alerts are critical; Observability names each. Revise the unmeasured thresholds in scripts/flip/choir-alerts.rules.yml after the first fortnight.

Trigger each alert in staging before launch: an invalid credential for 401, a staging-only low rate limit for 429, the impossible readiness disk floor, a stopped backup timer, and the supervisor fixture for a durability exit.

Caution

Do not fill a production filesystem to test disk alerts.

The BETA-0n tests

All run in the ordinary suite.

BETA-04 refuses * alone; @node stays legitimate for the operator credential, and * never matches @node. render_private_beta_service.sh calls validate_beta_acl.sh first.

Go-live receipts

Production stays network-closed until the release record contains:

1. The BETA-0n tests and the full CI gate green for the artifact commit.
2. Host-local and remote evidence that the node listens only on loopback and all access crosses the TLS proxy.
3. Public-hostname smoke receipts: authentication, allowed and denied ACL access, scope and review policy, quotas, request logging, oversized API and Git requests, GUI, clone, canary push.
4. An off-host backup and clean-host restore within the RPO and RTO, with policy equivalence and the node identity recovered from the secret store.
5. CI rejection, staging-first promotion, every critical alert, atomic deployment and one-command rollback each exercised.

Inviting somebody (D57)

Mint an invite as the operator; the username and grants are frozen:

curl -u <operator> -X POST \
  -d '{"user":"<their-name>","grants":["<owner>/<repo>.git write"]}' \
  https://<host>/api/accounts/invite

Send the response’s join_url and nothing else.

The request log records a redemption attempt under the invite id as user.

Runbook: restoring a node from a backup

Turn a backup written by scripts/flip/pull_backup.sh into a node.

First confirm the backup can be restored from:

choir backup verify ~/choir-backup

Every check is local; it opens no connection. Exit 0 means restorable, 1 means not. Warnings (no attestation, no ACL) are not failures.

Then restore:

choir backup restore ~/choir-backup /srv/choir-repos

It exits 0 only once the restored node has accepted a write. Expect it to stop the first time; supply what it asks for and re-run into the same root, which is resumed.

Exit codes

What a backup does not contain

A backup carries the operation log, node.fingerprint, the ref attestation, one git bundle per repository, acl, repos.list, the review policy and adjudication files, and the private-beta manifest. It never carries a secret, so three things are yours.

1. The node’s signing key

node.key signed every git-derived op. The backup carries only node.fingerprint, its public hash.

(a) You still have the key. Put it back, then re-run:

install -m 600 /path/to/your/held/node.key /srv/choir-repos/.choir/node.key

(b) The key is gone. Drop the fingerprint, then re-run. The daemon mints a new key, the re-run names the seq the seam falls at, historical signatures still verify, and holders of the old fingerprint should be told.

rm /srv/choir-repos/.choir/node.fingerprint

Keep the node key where the node host’s disk failure cannot reach it.

Where it is kept, and how to put it there (D70)

Keep the base64 of the raw 32 bytes in a Keychain secure note, both halves by hand in Keychain Access, because the security CLI puts the secret in argv.

Escrow, once, from the host holding the key. The base64 lands in scrollback, so use a window you will close:

ssh <choir-user>@<SERVER_IP> 'base64 < ~/.choir/repos/.choir/node.key'

Keychain Access, File, New Secure Note Item. Name it for the fingerprint, choir node key 1e-.... Paste, save, close the window.

Retrieval, at restore time. Decode through the clipboard:

tmp=$(mktemp)
pbpaste | base64 -d > "$tmp"
if [ "$(wc -c < "$tmp")" -eq 32 ]; then
  install -m 600 "$tmp" /srv/choir-repos/.choir/node.key && echo "key installed"
else
  echo "REFUSED: clipboard decoded to $(wc -c < "$tmp") bytes, not 32"
fi
rm -f "$tmp"

Install only on 32 bytes. Then re-run the restore, which refuses a key that mismatches node.fingerprint.

Recorded gap: the second operator in receipt 4 of docs/private-beta-runbook.md cannot reach a personal Keychain.

2. Auth tokens

printf '<operator>:%s\n' "$(openssl rand -hex 32)" > /srv/choir-repos/.choir/auth
chmod 600 /srv/choir-repos/.choir/auth

Mint new tokens rather than reusing old ones. The acl travels in the backup; the tokens it grades do not. The rehearsal username must match an operator ACL entry owning at least one restored repository, or the canary push is refused.

3. TLS material

Certificate and key are yours. Restore TLS at the reverse proxy after the loopback rehearsal passes.

The ordering rule

Git objects go in before the daemon starts, never after.

Startup reconciliation compares the log against each repo. A ref naming a commit the repo lacks is repaired with a compensating op: a restored node started against empty repos retracts your ref state.

The script unbundles first and boots second, and treats any choir: retracted line on that first start as a failure. If you see it, start again into a clean root.

Caution

A repo restored from a bundle has no pre-receive hook. The daemon adopts repos named by --create at startup, installing the hook and re-pointing gpg.ssh.allowedSignersFile. List every repo in --create.

repos.list is in the backup for the same reason: without it a restore serves only the default repo and reconciliation retracts the rest.

What the restore proves before exiting 0

1. Format, sequence, parent chain and recomputed hashes verify to the end.
2. keys, reviewers and repos.list are present and no secret is. The other six policy files are named individually when absent.
3. Every repo in repos.list has a bundle.
4. The target root holds no log; an existing one is never overwritten.
5. The node boots, replays, and retracts nothing.
6. The served view matches the D25 ref attestation refs.snapshot, when present.
7. A real git push over HTTP lands through http-backend, the pre-receive hook, the sequencer and the log.
8. The appended entry’s parent is the head served before the push.
9. The backup is a byte-exact prefix of the restored log.

After it exits 0

The canary ref refs/heads/restore-canary-<unix> stays as evidence. Delete it when done:

git push <node-url>/<repo> :refs/heads/restore-canary-<unix>

Render the hardened service with scripts/flip/render_private_beta_service.sh. Re-point the off-host backup job if the host moved.

Pull a backup from the restored node before trusting it:

./choirctl pull-backup          # still shell: it ssh's to the node host
choir backup verify ~/choir-backup

Troubleshooting

Symptoms and fixes. ERRORS.md, generated from the node’s rejection table, is the authority on any code field.

Rejection code table: ERRORS.md.