ursulactl

ursulactl manages the logical state of a running cluster over Ursula's admin and metrics HTTP APIs: which node leads which Raft groups, whether a node is caught up, and how a restarted voter is rebuilt through committed membership. It executes nothing on hosts. Physical lifecycle belongs to whatever owns the process: Helm and the StatefulSet controller on Kubernetes, systemd on hosts, OpenTofu for the infrastructure underneath. The verbs encode the safety properties an operator otherwise has to remember manually:

• before a node goes down, transfer every Raft group it leads to a healthy successor (drain)
• after it comes back, refuse to move on until last_applied_index has caught up to peers' committed_index (wait)
• abort rather than corner a group with no leader

A safe rolling restart wraps the platform's restart in these verbs, one node at a time. On Kubernetes:

ursulactl drain --config manifest.json --node 3
ursulactl prepare-restart --config manifest.json --node 3
kubectl delete pod ursula-2        # the platform restarts the pod
ursulactl repair-restarted-voter --config manifest.json --node 3
ursulactl wait --config manifest.json --node 3
ursulactl finish-prepared-restart --config manifest.json --node 3
ursulactl verify-cluster --config manifest.json

On bare metal the restart in the middle is systemctl restart ursula on the host. The drain planning and readiness logic is exercised under deterministic simulation.

Verbs

Verb	Effect
drain --node N	Mark the node draining and transfer away every leadership it holds. The mark persists (the node attracts no leaderships) until undrain. --dry-run prints the transfer plan
prepare-restart --node N	After drain, quiesce the target and pin every group leader to one survivor while the platform replaces it
repair-restarted-voter --node N	For each unready group, durably detach the replacement, attach it as a blocking learner, then promote it after catch-up. Safe to resume after interruption
finish-prepared-restart --node N	Release the target and survivor maintenance fences after repair and readiness checks
undrain --node N	Clear the drain mark so the node may hold leaderships again
wait --node N	Block until the node is a voter in every group and within --lag-tolerance of peers. Progress-gated: a node that keeps advancing is never timed out
status	Per-node group counts and leadership distribution
wait-ready	Block until every node reports the expected group count and every group has a leader
verify-cluster	Require every configured voter to be present and caught up in two consecutive samples before the next rollout step

Mutating verbs exit 0 on success and 2 on an abort (drain timeout, no safe transfer target, catch-up stall). Anything else is a configuration or transport error with a single-line, machine-greppable message.

The admin plane

Nodes carry no cluster-mutation surface on the network. The mutating operator endpoints (raft snapshot/purge/membership/learners/leader-transfer, maintenance drain, cold-flush trigger) plus metrics are served on a separate admin plane bound to server.admin_listen, which defaults to loopback (127.0.0.1:4438). The public client plane (:4437) serves only stream traffic and read-only metrics.

status and wait-ready prefer each node's http_url because read-only metrics are available on the client plane, and fall back to admin_url. Mutating verbs always use admin_url.

ursulactl is a plain HTTP client and opens no tunnels itself. When the admin plane is not directly reachable, bring your own forward and point the manifest's admin_url at it. On Kubernetes, kubectl port-forward reaches the loopback-bound plane inside each pod:

kubectl port-forward pod/ursula-0 5441:4438 &
kubectl port-forward pod/ursula-1 5442:4438 &
kubectl port-forward pod/ursula-2 5443:4438 &

[[nodes]]
id = 1
admin_url = "http://127.0.0.1:5441"

[[nodes]]
id = 2
admin_url = "http://127.0.0.1:5442"

[[nodes]]
id = 3
admin_url = "http://127.0.0.1:5443"

On bare metal the same shape works with ssh -N -L 5441:127.0.0.1:4438 admin@node1 per node.

When to use ursulactl vs. the other surfaces

Task	Tool
Day-2 logical operations: drain, observe, gate on readiness	ursulactl
Deployment, restarts, upgrades, topology	Helm and OpenTofu (systemd on bare metal)
Custom operator tooling	The admin-plane HTTP endpoints, reached over your own tunnel

Install

Build from the workspace alongside the server:

cargo build --release -p ursula-ctl --bin ursulactl

The binary lands at target/release/ursulactl. Drop it on your control machine. It does not need to run on the Ursula hosts themselves.

Manifest format

Every verb accepts --config <path>. The manifest is TOML, JSON, or YAML (chosen by file extension, sniffed when read from stdin with -) and lists the cluster's nodes.

Prefer generating the manifest from whatever already knows the topology instead of writing it by hand. On Kubernetes the Helm chart renders one into its ConfigMap, and the URLs in it are in-cluster DNS, so pipe it to ursulactl running where those names resolve (the server image contains ursulactl):

kubectl get configmap ursula -o jsonpath='{.data.cluster-manifest\.json}' \
  | ursulactl status --config -

For infrastructure provisioned by OpenTofu, emit the manifest as a stack output or generated file (the same pattern as deploy/eks's generated-values.yaml) rather than teaching ursulactl to read state files. tofu output -json <name> | ursulactl status --config - composes the same way.

Per-node fields, all optional except id:

• admin_port (default 4438) or an explicit admin_url: the admin plane to reach, directly or through your forward.
• host: address shown in reports. Falls back to the admin URL's host.
• http_url: optional client-plane URL used by status and wait-ready for read-only metrics. Metrics fall back to admin_url when omitted.

Restarting raft-memory nodes

On clusters running the volatile raft.wal.backend = "memory", a restarted node has no prior Raft log. prepare-restart quiesces the target and pins every group leader to one surviving voter. After the platform replacement starts, repair-restarted-voter commits a survivor-only voter set for each unready group, adds the target as a blocking learner, and promotes it only after catch-up. Those membership transitions survive leader changes, process restarts, and a partially completed multi-group repair.

Do not bypass the detach, learner, and promotion sequence with OpenRaft's process-local log-reversion trigger. It cannot provide durable progress across hundreds of independent groups. The rollout controller leaves the target drained and releases survivor fences on failure, so a later attempt can safely resume the committed membership state.

The rebuild after an amnesiac restart installs snapshots for every group and can take 10+ minutes. wait is progress-gated, so no timeout tuning is needed: a rebuild that keeps advancing is never timed out, and --stall-timeout-secs (default 90) only aborts a node that stops making progress.

status

Per-node summary of Raft group count and leadership distribution, sourced from every node's /__ursula/metrics. Nodes whose metrics fail are reported with metrics unavailable — … rather than aborting the report, because status is meant to surface partial cluster health.

ursulactl status --config cluster.json

Sample output:

node 1 (10.0.0.1): groups=4 leaders={1: 2, 2: 2}
node 2 (10.0.0.2): groups=4 leaders={1: 2, 2: 2}
node 3 (10.0.0.3): groups=4 leaders={1: 2, 2: 2}

leaders={…} is the count of groups each node is leading from this reporter's perspective. Healthy clusters report the same distribution from every node.

wait-ready

Block until every node reports --expected-groups Raft groups, each with a leader. Useful in CI / scripts after a deploy or a config change.

ursulactl wait-ready --config cluster.json --expected-groups 4

Exits non-zero with a one-line reason if the timeout passes (cluster not ready after 120s: node 3 has 1 group(s) without a leader).

Underlying HTTP surface

For custom tooling, every verb maps onto a small set of HTTP endpoints on each node:

Verb	Endpoint
status, wait-ready, wait	GET /__ursula/metrics
drain / undrain (maintenance guard)	POST /__ursula/leadership-shed/maintenance, DELETE to clear
drain (transfer step)	POST /__ursula/raft/{raft_group_id}/leader/transfer/{node_id}
repair-restarted-voter	POST /__ursula/raft/{raft_group_id}/membership, then POST /__ursula/raft/{raft_group_id}/learners/{node_id}, then membership promotion

The transfer endpoint refuses with 409 Conflict if the receiving node isn't the current leader of the group, and 400 if the target node isn't a voter. ursulactl uses this to refuse to attempt a transfer it cannot reason about.