Operations

The first tool to reach for is ursulactl. It covers the raft-aware verbs operators run most often: drain, status, readiness gates. This page covers everything around it: the metrics shape ursulactl reads, the admin endpoints it (and your custom tooling) can call, and the operational policies ursulactl does not encode (backups, log levels). Deployment and process lifecycle belong to the platform, Helm and OpenTofu.

Tooling map

Surface	When to reach for it
ursulactl	Day-2 cluster ops over HTTP: drain, restart, status, wait-ready.
/__ursula/metrics and the /__ursula/raft/... admin endpoints	Custom tooling. ursulactl uses these underneath. The surface is small and stable enough to script directly.

There is no Prometheus scrape and no general-purpose orchestrator yet.

Metrics

curl http://127.0.0.1:4437/__ursula/metrics | jq .

The JSON snapshot covers per-core mailbox depth, append/read counters, latency histograms, per-group leader and last_applied, hot/cold bytes, cold-flush backlog, WAL physical/reclaim/recovery work, WAL disk-pressure admission, HTTP status counters, live-read watchers, and cold-write admission state. Start here when triaging slowness, lag, or 503s. GET /__ursula/ready provides the process-level readiness signal; ursulactl status is a friendlier read of the leadership-related fields across the whole cluster.

Admin endpoints

These are the primitives ursulactl and any custom operator tooling builds on. They live on the admin plane (server.admin_listen, loopback 127.0.0.1:4438 by default), not the public client plane. Nodes expose no cluster-mutation surface on the network. Reach them over a tunnel to the admin port (see ursulactl's operation providers). The examples below assume a forward from local 4438 to the node's admin plane. Each call is local to one node. To act on every group, loop over the IDs in metrics.

# Force a cold flush for one stream (skip the timer)
curl -X POST http://127.0.0.1:4438/__ursula/flush-cold/demo/hello

# Trigger a Raft snapshot for one group
curl -X POST http://127.0.0.1:4438/__ursula/raft/42/snapshot

# Purge log entries below the last snapshot index for one group
curl -X POST http://127.0.0.1:4438/__ursula/raft/42/purge

# Add a learner (non-voting replica) to one group without blocking on catch-up
curl -X POST "http://127.0.0.1:4438/__ursula/raft/42/learners/4?addr=http://node-4:4437&blocking=false"

# Hand leadership of one group to another voter (used by `ursulactl drain`)
curl -X POST http://127.0.0.1:4438/__ursula/raft/42/leader/transfer/2

# Replace the voter set of one group (call on the group's current leader)
curl -X POST "http://127.0.0.1:4438/__ursula/raft/42/membership?voters=1,2,3"

# Put this node into maintenance drain / lift it again
curl -X POST   http://127.0.0.1:4438/__ursula/leadership-shed/maintenance
curl -X DELETE http://127.0.0.1:4438/__ursula/leadership-shed/maintenance

The leader-transfer endpoint refuses with 409 Conflict if the receiving node isn't the current leader and 400 Bad Request if the target isn't a voter. This is why ursulactl is the safer way to chain these calls: it consults metrics first.

The membership endpoint has the same leader-only contract: it answers 409 Conflict (with the current leader's id in the body) when the receiving node does not lead the group, and 400 Bad Request without a non-empty voters query parameter. On success it returns the log index at which the new voter set was committed. It replaces the whole voter set in one call, so include every node that should remain a voter.

Maintenance drain

POST /__ursula/leadership-shed/maintenance marks the node as draining: it stops campaigning and accepting transfers, and the background leadership balancer migrates its current leaderships to eligible peers one transfer per tick until the mark is lifted with DELETE. Both verbs return the node's leadership-shed state so tooling can confirm the transition took effect:

{
  "bits": 1,
  "state": "maintenance-drain",
  "should_accept_transfer": false,
  "should_campaign": false,
  "should_shed_current_leaders": true
}

ursulactl drain wraps exactly this pair: mark first so the node stops re-acquiring groups, then transfer leaderships away explicitly instead of waiting for the balancer. The mark stays set until ursulactl undrain clears it. If a rollout ever dies mid-way, a node can be left refusing leadership. Run ursulactl undrain (or DELETE the mark directly) to recover. Peers read each other's shed state (GET /__ursula/leadership-shed on the cluster-plane listener) when deciding which nodes are eligible to pick up leaderships, so a drained node stops attracting groups cluster-wide, not just locally.

Rejoining a wiped node

Rebuild a wiped voter through committed membership: remove it from each affected voter set, attach it as a blocking learner with its current address, and promote it only after snapshot and log catch-up. Use ursulactl repair-restarted-voter for the cluster-wide, resumable sequence. A process-local log-reversion trigger is not a durable recovery contract and is not exposed by Ursula's admin API.

POST /__ursula/cluster-probe also exists on the client plane, but it is machinery rather than an operator surface: nodes send each other heartbeat-sized bodies through it to verify cluster-plane egress, and it bypasses ingress admission.

Usage export

GET /__ursula/usage returns committed counters and current gauges from local replicated state, aggregated by bucket across this node's Raft groups:

{
  "version": 1,
  "write_unit_bytes": 10240,
  "buckets": {
    "tenant-a": {
      "committed_append_bytes": 24576,
      "committed_records": 3,
      "committed_write_units": 4,
      "retained_bytes": 24576,
      "stream_count": 1
    }
  }
}

The raw byte and record counters are pricing-neutral facts. committed_write_units is a convenience derived meter: every committed, non-deduplicated create or append contributes at least one unit and rounds its canonical payload up independently by write_unit_bytes. Consumers must validate both version and write_unit_bytes before interpreting it; the unit is data in the contract, not encoded into the counter name. The same unit is persisted in Raft snapshots, and a build refuses to restore a snapshot carrying a different unit. Changing it therefore requires a new contract version and an explicit state migration, never a configuration edit. Ursula does not attach currency, a price, or an account owner.

Counters are monotonic and survive snapshots, stream deletion, and bucket purge. retained_bytes and stream_count are current-state gauges and may decrease. A node serves local applied state, so replicated consumers should poll every voter and use their own completeness policy rather than treating one response as a cluster-wide linearizable read.

Tenant offboarding (bucket purge)

DELETE /__ursula/purge/{bucket} (admin plane) erases one tenant's complete bucket domain: every stream in every Raft group, the bucket and quota, cold chunks and indexes, external payloads, and any uncommitted object-store staging orphan below {bucket}/. The replicated purge also installs a permanent bucket-erasure fence in every group. It retains only the bucket's aggregate usage counters, because deleting a counter before an asynchronous meter observes it would erase committed billable work. The response is a completion report an operator can attest to:

curl -X DELETE http://127.0.0.1:4438/__ursula/purge/tenant-a
# {"bucket":"tenant-a","removed_streams":12,"groups_with_streams":[0,3,5],"cold_gc_entries_reclaimed":12,"cold_gc_pending_entries":0,"cold_gc_complete":true,"cold_gc_error":null,"bucket_prefix_absent":true}

Semantics:

• Idempotent and resumable. Re-purging an erased bucket returns the same report shape with zero removals. A purge interrupted mid-way converges on re-run: the permanent fence and stream removal are replicated commands per group, while cold reclamation and the complete bucket-prefix erasure are repeatable object-store operations.
• Permanently fenced. Commands linearized after purge cannot create the bucket or append to its streams. The fence survives Raft snapshots and restore; data-plane access continues to conceal the erased namespace as 404 Not Found instead of recreating it.
• Isolated. Other buckets' streams, offsets, checksums, snapshots, and object-store prefixes are untouched. Packs written by legacy versions may contain several buckets, so purge first rewrites their live slices into bucket-scoped objects and reports incomplete until that compatibility debt reaches zero.
• Accounting residue. The retained entry contains only aggregate counters and zero-valued content gauges, not stream data. Ursula does not currently provide an acknowledgement protocol for erasing it after an external meter persists the final delta. Use opaque bucket IDs if retaining the identifier itself would violate an erasure requirement.
• Physical absence proof. cold_gc_entries_reclaimed is diagnostic only. After a second replicated purge proves every group's bucket-specific GC queue is empty, Ursula recursively deletes the complete {bucket}/ erasure domain and lists the authoritative object store to prove it contains no files. Only cold_gc_complete=true, cold_gc_pending_entries=0, bucket_prefix_absent=true, and no error constitute the proof. If object storage is unavailable or still lists a file, the response remains incomplete and a later idempotent request retries the proof.
• Backups. A purged tenant may persist inside backups taken before the purge until those backups age out; restore replays whatever the manifest contains. Deleting old backup sets is part of completing an offboarding with erasure obligations.

Cleaning S3

Cold data lives under the storage.cold.root prefix. Benchmark and test runs should use a date-stamped root so cleanup is one prefix delete with standard S3 tooling:

aws s3 rm "s3://my-ursula-bucket/ursula-test-20260518T000000Z" --recursive

There is no automatic retention policy. Application checkpoints and explicit stream-retention advances control which cold objects become eligible for GC.

Backup and disaster recovery

ursulactl provides a verifiable whole-cluster backup and restore workflow:

# Create: one MessagePack snapshot object per raft group + manifest.json,
# into a local directory or s3://bucket/prefix.
ursulactl backup-create --config cluster.yaml --location s3://backups/ursula/2026-07-25

# Verify: manifest completeness, byte sizes, BLAKE3 checksums, and deep
# state-machine validation of every snapshot. Touches no cluster.
ursulactl backup-verify --location s3://backups/ursula/2026-07-25

# Restore: into a FRESH, EMPTY cluster with the same raft group count.
ursulactl restore --config new-cluster.yaml --location s3://backups/ursula/2026-07-25

The recovery contract:

• Consistency boundary. Each group export is the deterministic state-machine snapshot the raft snapshot path persists: internally consistent per group while writes continue. Cross-group consistency is not promised; the boundary is per stream. Acknowledged writes in the exporting replica's applied state are included whether or not they were cold-flushed, so RPO is bounded by export time, not by the cold-flush interval. backup-create asks every node and keeps the reply with the highest group commit index.
• Fresh identity on restore. Restore replays each snapshot as one replicated write on the target cluster, which keeps its own raft membership and log identity — nothing from the source cluster's raft metadata is reused. Groups that already hold buckets refuse the import with 409; group-count mismatches fail closed before the first import. Group commit indexes and per-stream append counters restart under the new identity; stream bytes, offsets, record coordinates, attrs, close/TTL state, published snapshots, retention floors, and producer dedup states are preserved exactly.
• Cold objects are part of the backup set. Snapshots reference cold-tier chunks by object key; point the restored cluster at the same (or a copied) cold-store namespace. backup-verify validates snapshots but does not dereference cold objects.
• Format compatibility. The backup format is versioned (format_version in the manifest) independently from the server binary; tools refuse newer formats and servers refuse imports they cannot validate. Within v0.x, restore into the same minor version is the supported path.
• RTO is dominated by transferring and importing the group snapshots; each import is a single replicated write per group.

The recovery drill — write, snapshot, retain, close; back up; destroy the cluster; restore; verify checksums, coordinates, and boundaries; continue appending — runs in CI (backup_restore_drill_preserves_streams_and_allows_continued_appends).

What you can additionally rely on:

• Quorum durability. Acknowledged writes survive as long as a majority of voters survives. Three voters across AZs tolerate any single-AZ outage.
• Cold-tier durability. Once flushed to S3, a chunk inherits S3-grade durability. The unflushed window is bounded by the flush interval (seconds by default).
• Versioned Raft WAL migration. The current WAL automatically migrates the legacy unversioned journal to checksummed v1 under its exclusive lock and retains journal.bin.v0.bak for rollback. Other v0.x state formats do not carry a general compatibility promise; stage upgrades and verify backup restore before removing the retained WAL backup.

Node-level loss: replace the host with the same node_id and cluster config, and it rehydrates from peers. Use ursulactl wait-ready afterwards to confirm the replacement is voting and caught up before declaring the recovery done. Total-cluster loss: restore the latest verified backup into a fresh cluster as above.

Logs

RUST_LOG=ursula=info,ursula_runtime=info,ursula_raft=info is the baseline. Bump to debug for one crate when chasing a subsystem:

RUST_LOG=ursula_raft=debug ./target/release/ursula server ...

debug is verbose under sustained load, so redirect to a file.