Module journal 

A structured, append-only decision journal: what the writer decided, why, and how long it took, one JSON object per line.

Derived data, and the consequences of saying so

The journal is not the op log and is never authoritative. Nothing reads it back to reconstruct state, no hash covers it, no signature commits to it. Two rules follow, and both are load-bearing:

• Recording never blocks and never fails a submission. Journal::record takes &self, cannot return an error, and FileJournal drops a record rather than stall the writer. A full disk must slow nothing and refuse nothing.
• Its corruption cannot reach the log. A truncated final line is a truncated final line. Since no one replays it, there is nothing to repair and no state to lose.

Why the I/O is on another thread

crate::Sequencer’s writer thread is the single appender, and choir-sequencer/tests/concurrency.rs asserts p99 decision latency under 100 ms. A synchronous write per decision would put a disk on that path and make the assertion measure the filesystem. So the writer only pushes onto a channel; FileJournal drains it from its own thread and does every write and flush there.

The trade is explicit: a crash loses whatever is still in the channel. For derived data that is the right side to be wrong on.

Examples

use choir_sequencer::journal::{Event, Journal, MemJournal};

let journal = MemJournal::new();
journal.record(Event::WindowResize {
    from: 20,
    to: 10,
    cause: "combined build failed".to_string(),
});
let lines = journal.lines();
assert_eq!(lines.len(), 1);
assert!(lines[0].contains("\"window_resize\""));
assert!(lines[0].contains("\"cause\":\"combined build failed\""));

The operator’s guide to this journal and the other records the node keeps:

Observability and repair

Record	Question it answers	Flag
Request log	what was asked of the node	--request-log (D33, in docs/operating/limits.md)
Decision journal	what the sequencer decided, and why	--journal
Torn-tail sidecar	what was being written when the node died	automatic

All three are derived data: unsynced writes, dropped under load.

[!WARNING] Keep the op log. None of these substitutes for it.

Metrics and what can be alerted on (/metrics)

Prometheus text format, authenticated.

Gauges: choir_ready, choir_log_verified, choir_sequencer_live, choir_storage_writable, choir_free_disk_bytes, choir_ref_disagreements, choir_process_start_time_seconds.

Counters: choir_requests_total, choir_requests_unauthorized_total (401 and 403), choir_requests_throttled_total (429), choir_requests_failed_total (5xx and unfinished responses), choir_request_duration_microseconds_total. Counters increment whether or not --request-log is on. Every scrape is one request behind.

The rules themselves

scripts/flip/choir-alerts.rules.yml: choir-node-state off the gauges, choir-node-traffic off the counters. Every window and rate is a starting point. The latency rule is a mean.

every_metric_these_alert_rules_name_is_one_the_node_exports in crates/choir-node/tests/limits.rs checks every choir_* name the rules read is still exported.

The alerts a node cannot source

Of the nine critical alerts in docs/private-beta-runbook.md, five come from this endpoint: readiness (choir_ready), durability (choir_sequencer_live), disk (choir_free_disk_bytes), request spikes and latency (the counters), restart loops (choir_process_start_time_seconds).

Alert	Where it comes from
Inode exhaustion	the host’s own exporter
Certificate expiry inside 21 days	the reverse proxy
Backup age beyond 90 minutes	the pull timer, on the host that pulls
Staging promotion failures	the deployment path

Decision journal (--journal)

Accepted and refused ops are both 200 on POST /api/submit; the journal records the decision:

cargo run -p choir-node -- /tmp/choir-repos 8417 \
  --keys-file ~/.choir/keys \
  --journal ~/.choir/decisions.jsonl

{"format_version":1,"kind":"decision","actor_id":"8f3a…","workspace":"op/agent","op_type":"SetRef","decision":"accepted","reject_reason":null,"seq":41,"parent":"…","decision_latency_us":812}

Every record carries kind:

kind	What it records
decision	every accept and refusal, with author, op type, reason, and dequeue-to-decision time
queue_depth	commands drained per writer wake-up
window_resize	the speculative merge window moving, with its cause
cas_failure	a lost compare-and-swap, separate from its rejection

Derived data (Architecture): written on its own thread, dropped rather than stalling the writer. The flag gates construction too.

Repairing a log (choir repair)

FileLog::open truncates a torn final record automatically, after saving the bytes to <log>.torn-<offset>. Everything else is explicit:

cargo run -p choir-cli -- repair ~/.choir/repos/.choir/ops.jsonl --verify

Mode	What it does	Exit
--verify	Walks the chain, reports the first bad record. Read-only.	0 usable, 1 damaged
--truncate-tail	Only for a torn final record: quarantines, truncates, syncs.	0 repaired, 1 refused
neither, or both	Usage error.	2

Damage anywhere but the tail is refused: the tool prints restore-from-backup steps and exits 1.

Taking a node with you (--export, D61)

cargo run -p choir-node -- --export ~/.choir/repos /tmp/choir-export
cargo run -p choir-node -- --verify-export /tmp/choir-export
cargo run -p choir-node -- --import /tmp/choir-export /srv/new-root

Offline, on the node’s machine. Writes ops.jsonl, one repos/<owner>/<name>.git.bundle per repository, and manifest.json with its own format_version. A never-pushed repository is listed without a bundle.

--verify-export requires every ref the log names to be in a bundle at the same oid. Extra refs in a bundle are reported as ahead; a log naming a commit no bundle holds is refused.

An export is secret-free by construction: the output is walked and refused if it holds anything named auth or ending .key or .pem. The signing key stays with the node (Restoring from a backup). Policy files stay behind; the manifest records that.

--import verifies, then refuses a root holding a log or any named repository. It places files; the daemon adopts them. Settle a restore by accepting a write.

scripts/flip/pull_backup.sh is the disaster-recovery path over ssh; --export is the format tool beside it.

Structs

FileJournal

A JSONL file written from its own thread.

MemJournal

An in-memory journal, for tests and for the doctest above.

NullJournal

A journal that discards everything.

Enums

Event

One journalled event.

Constants

FORMAT_VERSION

Journal record format. Bumped only for an incompatible change; new fields are additive, exactly as the op log’s own rule (invariant 1).

Traits

Journal

Somewhere decision events go.