Troubleshooting

Start at /__ursula/metrics. Most operational symptoms have a clear signal in the JSON snapshot.

Diagnostic surface

curl http://NODE:4437/__ursula/metrics | jq .

Useful jq selectors:

What you need	jq filter
Per-group leader / term	`.raft_groups[]
Hot bytes per group	`.raft_groups[]
Cold backpressure events	.cold_backpressure_events_total
Per-core mailbox depth	`.cores[]
Live-read watcher counts	`.cores[]
HTTP error counters	.http.responses_by_status
WAL disk pressure	{available: .wal_available_bytes, minimum: .wal_min_available_bytes, pressure: .wal_disk_pressure}
WAL reclaim/restart	{physical: .wal_physical_bytes, reclaims: .wal_reclaims, recovered_bytes: .wal_recovery_bytes, recovery_ns: .wal_recovery_ns}

GET /__ursula/ready is the process readiness probe: it returns 503 while the disk-WAL guard is active. Ursula does not expose /healthz, /readyz, Prometheus /metrics, or /cluster/status; use the JSON snapshot for detailed diagnosis and ursulactl wait-ready for cluster-wide Raft readiness.

Startup failures

raft.wal.path is required when WAL backend is 'disk'

Set raft.wal.path when raft.wal.backend = "disk". Use raft.wal.backend = "memory" for volatile tests.

raft.node_id must be non-zero

Static Raft mode requires a unique raft.node_id in the config file or a --node-id override.

raft.peers must include this node id

The local raft.node_id must appear in [[raft.peers]]. The peer list always includes self.

Multi-peer Raft with a memory WAL is volatile

Use raft.wal.backend = "disk" for a real cluster. Memory WAL remains available for development, benchmarks, and chaos tests, but more than one configured peer requires raft.wal.allow_volatile_multi_peer = true so the loss-on-restart contract is explicit.

storage.cold.s3.bucket is required when cold backend is 's3'

Set storage.cold.s3.bucket, or set storage.cold.backend = "none" if you don't want cold storage.

Port already in use

server.listen is taken. Public HTTP and inter-node gRPC share the same port unless server.cluster_listen is configured separately.

I/O error on raft.wal.path

The directory must be writable and owned by exactly one Ursula process. Each core journal holds an advisory journal.bin.lock containing the owner PID. Do not remove it while that process is alive; stop the conflicting process or fix the volume mount. A stale lock file is harmless because ownership is enforced by the OS lock, not the file's existence.

Bootstrap and cluster join

A fresh cluster never elects leaders

/__ursula/metrics on each node should show non-zero leader_id and a stable current_term. If groups stay leaderless:

• First start must include raft.init_membership_per_group = true (or raft.init_membership = true) on every voting node.
• Every peer URL must be reachable from every node.
• All peers must use the same raft.wal.backend mode. Mismatch causes silent join failures.

Replacing or restarting a node leaves it leaderless

Restart with the same raft.node_id and peer list. A fresh raft.wal.path means rehydrating from peers, which can be slow on cold-storage-only history.

Write path

503 with Retry-After on append

Inspect the JSON error. WalDiskPressure means the WAL filesystem is below raft.wal.min_available_size (or its free-space stat failed); the node also becomes unready and attempts to yield its leaders. Free space or expand the volume. Admission resumes only at raft.wal.resume_available_size, preventing probe flapping.

Other errors indicate cold-write backpressure for that group. Either the hot ring exceeds storage.cold.max_hot_size_per_group (cold flush isn't keeping up), or the cold backend is slow or erroring out.

Raise the per-group ceiling, lower storage.cold.flush_interval, raise storage.cold.flush_max_concurrency, or fix the cold backend.

409 Conflict with producer-expected-seq / producer-received-seq

Out-of-order producer headers. Response tells you what was expected:

• producer-expected-seq: N is the next allowed sequence
• producer-received-seq: M is what the client sent

Common causes:

• Producer restart without bumping Producer-Epoch. Bump every restart.
• Two writers share the same Producer-Id. Use distinct IDs.
• A retry skipped a sequence number. Producer sequences must be contiguous within an epoch.

409 Conflict from Stream-Seq

The supplied Stream-Seq is not lexicographically greater than the last accepted value. Re-read and re-derive, or use producer dedup instead.

404 Not Found on POST /{bucket}/{stream}

The stream hasn't been created. Create with PUT /{bucket}/{stream} first. There is no implicit creation on POST.

400 Bad Request on JSON appends

JSON streams are parsed and normalized on the server. Malformed JSON or empty arrays without allow_empty_array are rejected. Send valid JSON or switch to application/octet-stream.

Read path

Stream-Up-To-Date: false

More committed data exists past this response. Page forward with Stream-Next-Offset. Not an error.

SSE drops after 30-60 s idle

A proxy or load balancer is closing idle TCP. Raise its idle timeout or enable TCP keepalive on the proxy-to-Ursula leg. Ursula doesn't emit SSE keepalive comments.

SSE on a binary stream returns base64 text

Expected. SSE wire format is text-only, so binary stream data events carry raw base64 text and the Stream-Sse-Data-Encoding: base64 header signals it. See binary SSE.

Reads stall for hundreds of ms for cold offsets

The first read of a cold offset triggers an S3 GetObject range read. It runs off the actor turn so it doesn't block other commands, but the request still pays S3 latency.

Replication and consensus

A group has no leader (leader_id == 0)

Either an election is in progress or quorum is lost for that group. Only streams hashed to that group are affected.

• Multiple nodes briefly claim leader: election is flapping. Check peer reachability and CPU pressure.
• No node claims leader: fewer than n/2+1 voters are reachable. Restore reachability.

One follower lags behind

State-machine apply runs on the follower's owner core. If CPU is pinned, last_applied trails. Check:

• Per-core mailbox depth (sustained queueing means saturation).
• Disk pressure if raft.wal.backend = "disk" (fsync latency).
• A pathological hot stream forcing constant cold flushes (hot_bytes_total per group).

Manually trigger a snapshot

These are admin-plane endpoints (server.admin_listen, loopback :4438 by default), reached over a tunnel (see ursulactl's operation providers). The examples assume a forward from local 4438 to the node's admin plane.

curl -X POST http://127.0.0.1:4438/__ursula/raft/{group_id}/snapshot
curl -X POST http://127.0.0.1:4438/__ursula/raft/{group_id}/purge

Purge only after the snapshot replicates. Otherwise a slow follower may need log entries you just dropped.

Cold flush issues

Writes succeed but data never reaches S3

• storage.cold.backend is none, or set to memory when you expected s3.
• IAM permissions missing. Required: s3:GetObject, s3:PutObject, s3:ListBucket, s3:DeleteObject.
• storage.cold.s3.endpoint unreachable.
• Cold flush worker stalled (cold_backpressure_events_total climbing).

If hot bytes grow unbounded, you'll eventually hit storage.cold.max_hot_size_per_group and writes start returning 503. Fix the backend before that.

Still stuck?

Open a GitHub issue with:

• /__ursula/metrics from every node (with timestamps)
• Config file and CLI overrides per node (redact credentials)
• Last ~200 log lines at RUST_LOG=ursula_runtime=debug,ursula_raft=debug,ursula=info