Skip to main content

Module limits

Module limits 

Source
Expand description

Request accounting and admission control (D33): who did what, and how much of it they may do.

The node authenticates (crate::AuthTable) and authorizes per repository (crate::acl), and until this module existed it did nothing else. A second credential holder was invisible in the record — an incident left no trace of which credential caused it — and unbounded in consumption, because nothing counted requests at all.

Two pieces, deliberately separate:

  • RequestLog writes one JSON object per served request. It is an observation about this node, not part of the ordered history anyone replays, so it lives beside the op log rather than in it — the same reasoning that keeps lag.jsonl separate.
  • RateLimiter holds one token bucket per (user, Class) in memory and answers “may this request proceed, and if not, when should the caller come back”.

§What never reaches the log

No header, no request body, no query string. Access::start truncates the path at ? before the struct is even built, so a token smuggled into a query parameter cannot reach the file by any later path. This is the property that lets the file be handed to whoever is running an incident: a token that reaches a log is a token that has to be rotated.

No client address either, and not by omission: Access has no field that could hold one, so there is no formatting decision anywhere that could begin writing one (D59). That is also why this file, rather than a proxy access log, is the record the deployment keeps.

§Lock discipline

Both locks are leaf locks held for arithmetic or one write_all, never across a socket write. Access::finish is called after Request::respond has returned, so the log mutex is acquired when the response is already on the wire and a slow reader cannot hold it. The rate check runs on the request’s own thread after authentication, never on the accept loop, so a full bucket map cannot delay accept.

§Examples

use choir_node::limits::{Class, RateLimiter};
use std::num::NonZeroU32;

// Two API requests a minute, git unlimited.
let limiter = RateLimiter::new(NonZeroU32::new(2), None);
assert!(limiter.check("alice", Class::Api).is_none());
assert!(limiter.check("alice", Class::Api).is_none());

// The third is refused, and says how many seconds to wait.
let retry = limiter.check("alice", Class::Api).expect("third is refused");
assert!(retry >= 1);

// Buckets are per user, and an unset ceiling limits nothing.
assert!(limiter.check("bob", Class::Api).is_none());
assert!(limiter.check("alice", Class::Git).is_none());

The operator’s guide to the ceilings this module enforces:

§Rate limits, quotas and fairness

BoundFlagBounds
Requests per minute--rate-limit-api, --rate-limit-git (D33)how often one user may ask
Bytes in one push--quota-push-bytes (D37)how large one git request may be
Workspaces held at once--quota-workspaces (D37)how much disk one user may hold
Ops awaiting a decisionnone, deliberatelyhow far one actor may get ahead of the writer

The flagged ones require --auth-file and share three exemptions.

§Request log and rate limiting (D33)

cargo run -p choir-node -- /tmp/choir-repos 8417 \
  --auth-file ~/.choir/auth \
  --acl-file ~/.choir/acl \
  --request-log ~/.choir/requests.jsonl \
  --rate-limit-api 600 \
  --rate-limit-git 120

The request log is one JSON object per served request:

{"format_version":1,"at_unix_ms":1755100000000,"user":"alice","method":"GET","path":"/api/view","us":812,"status":200,"bytes":4310}

Refusals are recorded too: a 401 as anon (never the attempted username), a 429 against the charged user, a failed response with "status":0. The path is truncated at ?; headers and bodies never reach the file.

Rotation: past --request-log-max-bytes (32 MiB default) the file becomes <path>.1 and a fresh one starts. Two generations kept. Writes are unbuffered and unsynced.

Rate limiting is a token bucket per user per class, in memory, requests per minute with one minute’s burst. Over-limit answers 429 with Retry-After in seconds. The browser pages count against the API bucket.

Exempt:

ExemptWhy
The loopback hook callbackA push of N refs makes N /api/git-update calls; throttling one fails the push halfway.
Any holder of an @node grantAlready total authority; throttling the one actor who can repair the node is worse than the flood.
Every request on a node with no --auth-fileNo per-user identity; the daemon refuses the flags.

With --auth-file but no --acl-file, only the hook callback is exempt.

§Per-user quotas (D37)

cargo run -p choir-node -- /tmp/choir-repos 8417 \
  --auth-file ~/.choir/auth \
  --acl-file ~/.choir/acl \
  --quota-push-bytes 268435456 \
  --quota-workspaces 20

--quota-push-bytes bounds one git request body. Over it: 413 with both numbers. Checked before git http-backend spawns, so a refused push runs no hook. The body is drained so the client reads the 413.

--quota-workspaces bounds workspaces per user: 403 with quota_exceeded. The count is folded out of the op log at startup.

  • Checked on POST /api/workspace, not POST /api/submit.
  • Read outside the provisioning lock, so two simultaneous creations at the ceiling can both pass.

§Fairness at the sequencer’s door

Each actor holds at most 512 ops awaiting a decision (twice the writer’s 256-op batch), served round-robin. The total order is untouched: the writer stamps seq alone.

Over the bound, a submission is rejected naming the count and the limit; retry when one of your own ops completes.

  • The actor key is a claim, verified later on the writer thread. It bounds an honest flooder.
  • The wait is unmeasured. decision_latency starts when the writer picks an op up.

[!IMPORTANT] Nothing here may become load-bearing for authorization.

The bound has no flag.

Structs§

Access
One request in flight: what it was, and when it started.
Counters
The running totals behind the counters in /metrics.
CountersSnapshot
One reading of Counters, taken field by field.
PublicLimiter
Admission control for the routes that answer before a credential is checked (the invite link and the public landing page).
RateLimiter
Per-user token buckets, one per (user, Class).
RequestLog
An append-only, size-bounded record of every request the node served.

Enums§

Class
Which ceiling a request is charged against.

Constants§

DEFAULT_LOG_MAX_BYTES
The default rotation threshold: 32 MiB, so the two generations this keeps cost at most 64 MiB of disk.

Functions§

class_of
The class a URL is charged to, mirroring the accept loop’s own routing order (the API prefix is matched before the git fallback, so a path that contains both spellings lands in the same bucket the router will actually use).