Ursulav0.2.0

Deploy a cluster

Production-style Ursula runs as a static-membership Raft cluster: a fixed list of nodes with stable IDs, gRPC peer-to-peer Raft traffic, and a shared S3 cold backend. The migration benchmark target is three voting nodes across availability zones.

For Kubernetes, use the Helm chart. It maps the same static-membership model onto StatefulSet pod ordinals, stable peer DNS, and per-pod PVCs.

Topology

Each node needs:

  • a stable Raft raft.node_id
  • a client/API bind address in server.listen
  • optional separate cluster/Raft bind address in server.cluster_listen when you need to isolate cluster traffic from client/API traffic
  • peer URLs in [[raft.peers]] that every peer can reach over gRPC (including itself)
  • the same raft.wal.backend mode on every peer
  • a one-time membership initializer on the first start (raft.init_membership_per_group = true)
  • a shared cold backend (memory for single-host smoke, S3 or an S3-compatible store for production)

Without server.cluster_listen, server.listen serves client/API and Raft/cluster routes, so each [[raft.peers]].url should point at that node's server.listen address. With server.cluster_listen, server.listen serves client/API routes and server.cluster_listen serves Raft/cluster routes, so each [[raft.peers]].url must point at that node's cluster listener address and port. The same [[raft.peers]].url is also used for HTTP leader redirects today, so clients and gateways must be able to reach that URL unless separate redirect URL configuration is added in the future.

Cluster config file

The Ursula config file is shared across all nodes. Only raft.node_id differs per host, or override it with --node-id.

[server]
listen = "0.0.0.0:4437"

[runtime]
core_count = 16

[raft]
node_id = 1
group_count = 256
init_membership_per_group = true

[raft.wal]
backend = "disk"
path = "/var/lib/ursula"

[[raft.peers]]
node_id = 1
url = "http://10.0.0.1:4437"

[[raft.peers]]
node_id = 2
url = "http://10.0.0.2:4437"

[[raft.peers]]
node_id = 3
url = "http://10.0.0.3:4437"

Same file on each host; change raft.node_id or start with --node-id 2 / --node-id 3.

Start node 1

ursula --config /etc/ursula/ursula.toml --node-id 1

raft.init_membership_per_group = true only needs to be present on the very first start of a fresh cluster. After that, Ursula remembers membership. Flip it to false (or remove it) for subsequent starts.

Start nodes 2 and 3

Use the same config file and change the node ID:

ursula --config /etc/ursula/ursula.toml --node-id 2
ursula --config /etc/ursula/ursula.toml --node-id 3

Use raft.wal.backend = "memory" instead of the disk settings if you want a non-durable test cluster.

Verify with ursulactl

Once daemons are up on every node, point ursulactl at a one-file manifest of the cluster and block until each Raft group has elected a leader:

cat > cluster-manifest.json <<'JSON'
{
  "nodes": [
    {"id": 1, "http_url": "http://10.0.0.1:4437", "host": "10.0.0.1"},
    {"id": 2, "http_url": "http://10.0.0.2:4437", "host": "10.0.0.2"},
    {"id": 3, "http_url": "http://10.0.0.3:4437", "host": "10.0.0.3"}
  ]
}
JSON

ursulactl wait-ready --config cluster-manifest.json --expected-groups 256
ursulactl status      --config cluster-manifest.json

wait-ready returns non-zero with a single-line reason if the timeout elapses. status prints one line per node showing the raft group count and per-leader group counts as observed from that node's metrics. Healthy clusters report the same distribution from every reporter.

If you don't have ursulactl handy, the raw metrics endpoint works as a fallback:

for host in 10.0.0.1 10.0.0.2 10.0.0.3; do
  curl -s "http://$host:4437/__ursula/metrics" | jq '.raft_groups | length'
done

Storage exclusivity

Pick one storage backend per cluster.

  • raft.wal.backend = "memory" - fast, volatile. Survives no restart
  • raft.wal.backend = "disk" with raft.wal.path = "DIR" - durable OpenRaft log mode under DIR/raft-log (recommended for clusters)

Cold storage

A shared object store is recommended for any multi-node deployment because peers need to read each other's flushed chunks. Configure it in the Ursula config file on every node:

[storage.cold]
backend = "s3"
root = "ursula-prod-20260518"

[storage.cold.s3]
bucket = "my-ursula-bucket"
region = "us-east-1"

See configure S3 for canonical [storage.cold] and [storage.cold.s3] keys.

Operating the cluster

The first tool for day-2 work is ursulactl:

  • ursulactl restart: drain-aware rolling restart with applied-index catch-up gates.
  • ursulactl status: leadership distribution per node.
  • ursulactl wait-ready: gate scripts on group + leader counts.

Use scripts/ursula_ec2.py for the SSH/AWS-side plumbing: pushing binaries, writing systemd units, EC2 Instance Connect, S3 cleanup. See operations for the full split.

The HTTP admin surface underneath both tools is small and stable enough to script against directly:

  • GET /__ursula/metrics: per-node JSON snapshot.
  • POST /__ursula/raft/{group_id}/snapshot: manually trigger a Raft snapshot.
  • POST /__ursula/raft/{group_id}/purge: purge stale log entries.
  • POST /__ursula/raft/{group_id}/learners/{node_id}: add a learner (non-voting) replica.
  • POST /__ursula/raft/{group_id}/leader/transfer/{node_id}: hand off leadership to another voter (the primitive ursulactl restart builds on).
  • POST /__ursula/flush-cold/{bucket}/{stream}: force a cold flush for one stream.

Limits in the current build

  • Membership changes (adding/removing voters after bootstrap) are not yet exposed as a routine workflow. Learners can be added via the admin endpoint above.
  • ursulactl restart packages the safe rolling restart loop (drain → restart-cmd → wait-ready); version-upgrade tooling that diffs binaries before restart is not yet packaged.
  • There is no zero-downtime config reload. Restart the process to pick up config-file or CLI-override changes.