Overview

The problem

If you want to build a collaborative app, a real-time AI agent, or anything that needs to push state changes to multiple clients the moment they happen, you need a durable stream. Something you can append to, read back from any point, and subscribe to for live updates.

The obvious place to build this is on top of object storage. S3 is cheap, durable, and infinitely scalable. But if you've tried to build a low-latency write path on S3, you've probably hit two problems that no amount of clever engineering can fully hide.

The first is that S3 has no append operation. Objects are immutable. Every write creates a new object, which means you either pay per-PUT at high frequency or you batch writes and accept the latency that comes with waiting for a batch to fill. The second is that S3's conditional writes are optimistic. Under concurrent writers, they degrade into retry storms where most attempts conflict, back off, and try again. Latency grows exponentially with contention. (This post from Chroma explains the mechanics well.)

S2 addresses the latency problem by writing to S3 Express One Zone, which gets them under 50ms. But Express storage costs 7x more than Standard, and the optimistic concurrency model is still there underneath. WarpStream goes the other direction: batch everything to S3 Standard and keep costs low, but accept 250ms+ p50 latency as a consequence. Both are making the best of what the S3 write path allows.

The fact that multiple teams are independently building the same thing tells you something: "append to a stream, read it back, subscribe for updates" is a general-purpose primitive, not an implementation detail of any one product.

How Ursula works

Ursula sidesteps this trade-off by not using S3 for the write path at all.

The interface

The API implements the Durable Streams protocol, an open specification (MIT licensed) published by Electric. It defines the primitive that keeps coming up: create a stream, append to it, read from any offset, and tail it in real time, with exactly-once semantics. The whole thing runs over plain HTTP with SSE for live subscriptions, so there is no custom binary protocol and no mandatory client library.

But a protocol only defines the interface. The hard part is what sits underneath, specifically the write path. "Append this record durably based on S3" is easy to say. Making it fast, consistent, and cheap is where the architectural choices live.

The write path

Instead, writes go to a cluster of nodes that coordinate through a consensus algorithm. One node acts as the leader and sequences every append into a replicated log. Once a majority of nodes have persisted the entry, it's committed and the client gets back a confirmed offset. There are no CAS races, no retry storms, and no batching delays. Appending is just appending.

Each node keeps the recent portion of every stream in memory, so reads serve from local state with no network round-trip. A background process flushes older data to S3 Standard for long-term retention. When a client reads an offset that has left the hot window, the read transparently pulls from S3. From the client's perspective, there's just one stream with one API. The hot-to-cold boundary is invisible.

                Clients
                   │
            ┌──────┴──────┐
            │  HTTP API   │
            │  /{b}/{s}   │
            └──────┬──────┘
                   │
      ┌────────────┼────────────┐
      v            v            v
 ┌─────────┐  ┌─────────┐  ┌─────────┐
 │ Node A  │  │ Node B  │  │ Node C  │
 │ leader  │  │follower │  │follower │
 ├─────────┤  ├─────────┤  ├─────────┤
 │ OpenRaft│  │ OpenRaft│  │ OpenRaft│
 ├─────────┤  ├─────────┤  ├─────────┤
 │  state  │  │  state  │  │  state  │
 │ machine │  │ machine │  │ machine │
 ├─────────┤  ├─────────┤  ├─────────┤
 │ RocksDB │  │ RocksDB │  │ RocksDB │
 │  (hot)  │  │  (hot)  │  │  (hot)  │
 └────┬────┘  └────┬────┘  └────┬────┘
      │            │            │
      └────────────┼────────────┘
                   v
            ┌─────────────┐
            │ Cold storage│
            │  (fs / S3)  │
            └─────────────┘

The whole thing is written in async Rust.

Why this matters now

A year ago, most applications could get away without a durable stream. State lived in a database, updates happened through polling or webhooks, and if something got lost in transit, the user refreshed the page. That's changing fast, for a few reasons.

The first is AI agents. When an agent takes an action on behalf of a user, other agents and the user both need to see it immediately and be able to replay the full history. Traditional request-response APIs don't support subscribing to a feed of changes. WebSockets are real-time but ephemeral. What agents need is something in between: a persistent, ordered log they can write to and read from, ideally over a protocol they already understand. HTTP and SSE fit that bill, and the Durable Streams protocol is exactly this interface.

Collaborative applications also have gone from niche to default. The Notion/Figma model, where multiple users see each other's changes in real time, is now what users expect from any tool that involves shared state. Every team building one of these apps ends up building their own sync layer on top of WebSockets, CRDTs, or a message queue. A durable stream is the primitive that makes all of those layers simpler, because it gives you a single source of truth that supports both catch-up reads and live tailing.