Operations

The first tool to reach for is ursulactl. It covers the raft-aware verbs operators run most often: rolling restart, status, readiness gate. This page covers everything around it: the metrics shape ursulactl reads, the admin endpoints it (and your custom tooling) can call, the SSH-side lifecycle that still lives in Python, and the operational policies ursulactl does not encode (backups, log levels, S3 cleanup).

Tooling map

Surface	When to reach for it
ursulactl	Day-2 cluster ops over HTTP: restart, status, wait-ready.
scripts/ursula_ec2.py	SSH/IAM/EC2 lifecycle: push binaries, write systemd units, S3 cleanup, drive the benchmark client.
/__ursula/metrics and the /__ursula/raft/... admin endpoints	Custom tooling. ursulactl uses these underneath; the surface is small and stable enough to script directly.

There is no Prometheus scrape and no general-purpose orchestrator yet.

Metrics

curl http://127.0.0.1:4437/__ursula/metrics | jq .

The JSON snapshot covers per-core mailbox depth, append/read counters, latency histograms, per-group leader and last_applied, hot/cold bytes, cold-flush backlog, HTTP status counters, live-read watchers, and cold-write admission state. Start here when triaging slowness, lag, or 503s. ursulactl status is a friendlier read of the leadership-related fields across the whole cluster.

Admin endpoints

These are the primitives ursulactl and any custom operator tooling builds on. They live on the admin plane (server.admin_listen, loopback 127.0.0.1:4438 by default), not the public client plane — nodes expose no cluster-mutation surface on the network. Reach them over a tunnel to the admin port (see ursulactl's operation providers); the examples below assume a forward from local 4438 to the node's admin plane. Each call is local to one node; to act on every group, loop over the IDs in metrics.

# Force a cold flush for one stream (skip the timer)
curl -X POST http://127.0.0.1:4438/__ursula/flush-cold/demo/hello

# Trigger a Raft snapshot for one group
curl -X POST http://127.0.0.1:4438/__ursula/raft/42/snapshot

# Purge log entries below the last snapshot index for one group
curl -X POST http://127.0.0.1:4438/__ursula/raft/42/purge

# Add a learner (non-voting replica) to one group
curl -X POST http://127.0.0.1:4438/__ursula/raft/42/learners/4

# Hand leadership of one group to another voter (used by `ursulactl restart`)
curl -X POST http://127.0.0.1:4438/__ursula/raft/42/leader/transfer/2

# Replace the voter set of one group (call on the group's current leader)
curl -X POST "http://127.0.0.1:4438/__ursula/raft/42/membership?voters=1,2,3"

# Put this node into maintenance drain / lift it again
curl -X POST   http://127.0.0.1:4438/__ursula/leadership-shed/maintenance
curl -X DELETE http://127.0.0.1:4438/__ursula/leadership-shed/maintenance

# Permit the leader to accept one log revert from a wiped, rejoining node
curl -X POST http://127.0.0.1:4438/__ursula/raft/42/nodes/4/allow-next-revert

The leader-transfer endpoint refuses with 409 Conflict if the receiving node isn't the current leader and 400 Bad Request if the target isn't a voter. This is why ursulactl is the safer way to chain these calls: it consults metrics first.

The membership endpoint has the same leader-only contract: it answers 409 Conflict (with the current leader's id in the body) when the receiving node does not lead the group, and 400 Bad Request without a non-empty voters query parameter. On success it returns the log index at which the new voter set was committed. It replaces the whole voter set in one call, so include every node that should remain a voter.

Maintenance drain

POST /__ursula/leadership-shed/maintenance marks the node as draining: it stops campaigning and accepting transfers, and the background leadership balancer migrates its current leaderships to eligible peers one transfer per tick until the mark is lifted with DELETE. Both verbs return the node's leadership-shed state so tooling can confirm the transition took effect:

{
  "bits": 1,
  "state": "maintenance-drain",
  "should_accept_transfer": false,
  "should_campaign": false,
  "should_shed_current_leaders": true
}

ursulactl restart wraps its drain step in exactly this pair: mark first so the node stops re-acquiring groups, then transfer leaderships away explicitly instead of waiting for the balancer, and clear after the node is back and caught up (or on any abort path). If a rollout ever dies between the two, a node can be left refusing leadership; DELETE the mark by hand to recover. Peers read each other's shed state (GET /__ursula/leadership-shed on the cluster-plane listener) when deciding which nodes are eligible to pick up leaderships, so a drained node stops attracting groups cluster-wide, not just locally.

Rejoining a wiped node

A node that lost its Raft log (disk replaced, data dir wiped) rejoins with a shorter log than the leader has recorded for it, which Raft treats as a fault. POST /__ursula/raft/{group}/nodes/{node}/allow-next-revert tells the group's leader to accept exactly one such log revert from that node and re-replicate from scratch. It is a per-group, one-shot permission: loop over all group IDs when re-admitting a fully wiped node, and expect the leader to answer 500 if the underlying Raft rejects the request.

POST /__ursula/cluster-probe also exists on the client plane, but it is machinery rather than an operator surface: nodes send each other heartbeat-sized bodies through it to verify cluster-plane egress, and it bypasses ingress admission.

SSH-side lifecycle

scripts/ursula_ec2.py drives a static EC2 manifest (instance IDs, IPs, port, binary path, cold env, peer list) over SSH. It complements ursulactl: push binaries and start daemons here, then switch to ursulactl for raft-aware verbs.

# Push a fresh binary to every server in the manifest
python3 scripts/ursula_ec2.py --config cluster.json upload-binary \
  --target servers --local ./target/release/ursula --remote /tmp/ursula

# Bring daemons up, then gate on readiness via ursulactl
python3 scripts/ursula_ec2.py --config cluster.json start
ursulactl wait-ready --config cluster.json --expected-groups 256

# Stop the cluster (kills the pid recorded in the configured pid file)
python3 scripts/ursula_ec2.py --config cluster.json stop

# Drive the benchmark workload from the configured client host
python3 scripts/ursula_ec2.py --config cluster.json perf-many \
  --processes 4 --bucket-prefix benchcmp-mp

perf-many rotates entrypoints across service nodes by default; use --target-mode first only to reproduce a single-ingress run. stop kills only the recorded PID rather than running pkill, because broad patterns can match the SSH cleanup command itself. If a pid file is stale, kill by hand.

The Python script still exposes status and wait-ready, kept around for environments where ursulactl isn't deployed yet. They print the same numbers but via SSH-curl, so prefer ursulactl when you have a choice. Verbs scheduled to migrate to ursulactl once SSH transport lands: upload-binary, install-binary, install-service, install-chaos-agent, install-faultd, deploy-chaos. AWS deployment scaffolding (IAM / EC2 lifecycle / security groups) stays in Python permanently.

Cleaning S3

python3 scripts/ursula_ec2.py --config cluster.json cleanup-s3 \
  --root ursula-test-20260518T000000Z

Deletes everything under one storage.cold.root prefix in the manifest's bucket. Run after benchmark sweeps.

Backup and disaster recovery

What you can rely on:

• Quorum durability. Acknowledged writes survive as long as a majority of voters survives. Three voters across AZs tolerate any single-AZ outage.
• Cold-tier durability. Once flushed to S3, a chunk inherits S3-grade durability. The unflushed window is bounded by the flush interval (seconds by default).
• No on-disk migration. v0.x does not promise stable on-disk formats. The runtime won't refuse to start on stale data, but it won't migrate either. Cross-version upgrades currently mean rebuild + replay from external truth.

Node-level loss: replace the host with the same node_id and cluster config; it rehydrates from peers. Use ursulactl wait-ready afterwards to confirm the replacement is voting and caught up before declaring the recovery done. Total-cluster loss: the durable data on S3 is what you have, and there is no tooling yet to materialize a fresh cluster from those objects.

Logs

RUST_LOG=ursula=info,ursula_runtime=info,ursula_raft=info is the baseline. Bump to debug for one crate when chasing a subsystem:

RUST_LOG=ursula_raft=debug ./target/release/ursula ...

debug is verbose under sustained load; redirect to a file.