ursulactl

ursulactl is the first interface to reach for once a cluster is running. Its verbs encode the safety properties an operator otherwise has to remember manually:

• before restarting a node, transfer every Raft group it leads to a healthy successor;
• after a restart, refuse to move on until last_applied_index has caught up to peers' committed_index;
• abort the rollout rather than corner a group with no leader.

The same orchestration is exercised under deterministic simulation, so failure modes like a leader transfer timing out, the target node entering a crash loop, or a partition during drain produce a clean abort instead of a half-rolled cluster.

The admin plane

Nodes carry no cluster-mutation surface on the network. The mutating operator endpoints (raft snapshot/purge/membership/learners/leader-transfer/allow-next-revert, maintenance drain, cold-flush trigger) plus metrics are served on a separate admin plane bound to server.admin_listen — loopback (127.0.0.1:4438) by default. The public client plane (:4437) serves only stream traffic and read-only metrics.

ursulactl sends every request to the admin plane. Because that plane is loopback-bound, ursulactl reaches it through an operation provider — a per-node tunnel:

• direct (default): hit admin_url as configured. Use when it is already reachable — a node bound to a private interface with a security group in front, an outer tunnel you manage, or a local node.
• forward (--admin-forward-cmd): ursulactl picks a free local port, runs your command to forward it to the node's admin plane, and talks to the local end. SSH, SSM, and kubectl port-forward all fit this one shape; the command is rendered with {local_port}, {admin_port}, {admin_host}, {host}, {node_id}, {name} and run under sh -c, staying in the foreground for the tunnel's life (ursulactl tears it down when done).

# SSH
ursulactl status --config cluster.json \
  --admin-forward-cmd 'ssh -N -L {local_port}:127.0.0.1:{admin_port} ec2-user@{host}'

# AWS SSM (no inbound SSH needed)
ursulactl status --config cluster.json \
  --admin-forward-cmd 'aws ssm start-session --target {name} --document-name AWS-StartPortForwardingSessionToRemoteHost --parameters host=127.0.0.1,portNumber={admin_port},localPortNumber={local_port}'

# Kubernetes
ursulactl status --config cluster.json \
  --admin-forward-cmd 'kubectl port-forward pod/{name} {local_port}:{admin_port}'

When to use ursulactl vs. the other surfaces

Task	Tool
Day-2 cluster operations: restart, observe, gate on readiness	ursulactl
Push binaries, write systemd units, EC2 Instance Connect, S3 cleanup	scripts/ursula_ec2.py
Custom operator tooling	The admin-plane HTTP endpoints, reached over your own tunnel

Install

Build from the workspace alongside the server:

cargo build --release -p ursula-ctl --bin ursulactl

The binary lands at target/release/ursulactl. Drop it on your control machine; it does not need to run on the Ursula hosts themselves.

Manifest format

Most verbs accept --config <path>, a JSON manifest describing the cluster. The shape is compatible with scripts/ursula_ec2.py's cluster.json (the same nodes array is reused), so you can point both tools at the same file. Minimum fields:

{
  "nodes": [
    { "id": 1, "http_url": "http://10.0.0.1:4437", "host": "10.0.0.1" },
    { "id": 2, "http_url": "http://10.0.0.2:4437", "host": "10.0.0.2" },
    { "id": 3, "http_url": "http://10.0.0.3:4437", "host": "10.0.0.3" }
  ]
}

Legacy public_ip / private_ip / http_port fields from the EC2 helper are accepted; if http_url is missing it is synthesised from those plus the port. host is the value substituted into --restart-cmd templates and defaults to the URL's host string.

Each node's admin endpoint defaults to http://{host}:4438; override it with an explicit admin_url, or set admin_port to change only the port. Under --admin-forward-cmd ursulactl rewrites this to the local forwarded port automatically, so the manifest value only needs to name the in-pod/on-host admin port.

restart

Safe rolling restart. Per target node, in manifest order (or --only order):

1. Snapshot every node's /__ursula/metrics.
2. Mark the target's maintenance drain so it stops campaigning and re-acquiring leaderships mid-drain.
3. For every Raft group the target leads, pick the most-caught-up voter as successor and call POST /__ursula/raft/{group}/leader/transfer/{successor}.
4. Poll until the target leads zero groups, or abort on drain timeout.
5. Run --restart-cmd with {node_id} / {host} / {http_url} / {name} substituted.
6. Poll until the target is back as a voter in every group and its last_applied_index is within --lag-tolerance of peers' committed_index, or abort on readiness timeout.
7. Clear the maintenance drain and move to the next node only on success. Any abort clears the drain mark on the current target and stops the rollout.

ursulactl restart \
  --config cluster.json \
  --restart-cmd 'ssh ec2-user@{host} sudo systemctl restart ursula-chaos.service' \
  --drain-timeout-secs 60 \
  --ready-timeout-secs 180 \
  --lag-tolerance 16

--restart-cmd runs in sh -c on the control machine. Anything that can take a templated host argument fits: SSH, AWS SSM, kubectl exec, nomad alloc exec. Add --dry-run to print the drain plan and skip the actual transfers.

Restrict the rollout with --only:

ursulactl restart --config cluster.json --restart-cmd '...' --only 2,3

Restarting raft-memory nodes

On clusters running the volatile raft.wal.backend = "memory", a restarted node rejoins with an empty log, and group leaders refuse that log reversion unless it was explicitly permitted. Two flags matter:

• --allow-empty-raft-rejoin: before running the restart command, ask the current leader of every group to accept one empty-log rejoin from the target. Without it the target never catches up and the readiness step times out. Do not use it on persistent-backend clusters, where an unexpected empty rejoin means data loss that deserves to be refused.
• --ready-timeout-secs: an amnesiac node rebuilds every group from snapshots, which for large groups takes 10+ minutes, not the default two. Size the timeout for a full rebuild.

The permission must land on the node that is leader at that moment; the endpoint answers 409 Conflict from any other node (ursulactl resolves leadership per group and handles this). A node that is the membership initializer for some groups additionally refuses to start at all after losing its volatile log (the bootstrap-marker guard) — that recovery is an explicit operator reset on the host, outside ursulactl's reach.

status

Per-node summary of Raft group count and leadership distribution, sourced from every node's /__ursula/metrics. Nodes whose metrics fail are reported with metrics unavailable — … rather than aborting the report, because status is meant to surface partial cluster health.

ursulactl status --config cluster.json

Sample output:

node 1 (10.0.0.1): groups=4 leaders={1: 2, 2: 2}
node 2 (10.0.0.2): groups=4 leaders={1: 2, 2: 2}
node 3 (10.0.0.3): groups=4 leaders={1: 2, 2: 2}

leaders={…} is the count of groups each node is leading from this reporter's perspective. Healthy clusters report the same distribution from every node.

wait-ready

Block until every node reports --expected-groups Raft groups, each with a leader. Useful in CI / scripts after start or a config change.

ursulactl wait-ready --config cluster.json --expected-groups 4

Exits non-zero with a one-line reason if the timeout passes (cluster not ready after 120s: node 3 has 1 group(s) without a leader).

Exit codes

Code	Meaning
0	All target nodes finished successfully (restart), readiness reached (wait-ready), or status report rendered (status).
2	restart: at least one node aborted (drain timeout, restart command non-zero, or readiness timeout). Subsequent nodes were not touched.
Non-zero (other)	Configuration or transport error. The error message is single-line and machine-greppable.

Roadmap: verbs not yet migrated

These still live in scripts/ursula_ec2.py because they require SSH or AWS APIs and benefit from the same EC2 Instance Connect plumbing that the script already does:

• upload-binary, install-binary: push and stage release artefacts.
• install-service, install-chaos-agent, install-faultd, deploy-chaos: write/refresh systemd units.
• cleanup-s3, perf / perf-many: AWS S3 and benchmark drivers.

They will move to ursulactl once a NodeProvider-backed SSH transport lands; the AWS deployment scaffolding (IAM / EC2 lifecycle / security groups) stays in Python permanently.

Underlying HTTP surface

For custom tooling, every verb maps onto a small set of admin-plane HTTP endpoints on each node (reached over your own tunnel to admin_listen):

Verb	Endpoint
status, wait-ready	GET /__ursula/metrics
restart (maintenance guard)	POST /__ursula/leadership-shed/maintenance before the drain, DELETE once the node is back or the rollout aborts
restart (drain step)	POST /__ursula/raft/{raft_group_id}/leader/transfer/{node_id}
restart (readiness step)	GET /__ursula/metrics polled across all peers

The transfer endpoint refuses with 409 Conflict if the receiving node isn't the current leader of the group, and 400 if the target node isn't a voter. ursulactl uses this to refuse to attempt a transfer it cannot reason about.