How we built real-time audit logging without slowing down the API

Every request that touches Gatehive gets logged. Every login, every permission change, every access grant. Customers rely on that trail being complete and immediate, especially when they pull it during an actual audit.

The problem is that writing a log entry synchronously, on the same request that's doing the real work, adds latency to every single API call. At scale, that latency compounds fast. We had to get logging out of the critical path without losing the "real-time" part customers actually need.

The naive approach, and why it broke

The first version did exactly what you'd expect: write the audit event to the database in the same transaction as the action itself. Simple, consistent, easy to reason about.

It also meant every API call now waited on two writes instead of one. Under normal load it was fine. Under a bulk operation, like provisioning fifty new users at once, response times doubled.

Pull quote:

Consistency is easy to get right. Consistency without cost is the actual problem.

Decoupling the write

The fix was moving audit logging off the request path entirely, using an event queue. When an action happens, we publish an event and return immediately. A separate consumer picks up the event and writes it to the audit store.

• The API call returns as soon as the action completes, not after logging finishes

• A queue absorbs bursts, so a spike in activity doesn't create a spike in latency

• The consumer scales independently, so we can add capacity for logging without touching the API layer at all

This got response times back to baseline, even under bulk operations.

The tradeoff nobody likes to admit

Decoupling introduces a small window where an action has happened but hasn't been logged yet. For most systems, that's an acceptable tradeoff. For an audit trail, it's the one thing customers explicitly can't tolerate: a gap between something happening and it being recorded.

We closed that gap with two guarantees instead of one:

• At-least-once delivery on the queue, so an event is never silently dropped

• A reconciliation job that compares actions against logged events on a short interval, and backfills anything missing before it's ever visible to a customer

In practice, the window is small enough that no customer has ever seen a gap. But we built the system assuming they eventually would, and designed the reconciliation step so that even then, nothing gets lost.

What this means for search

Decoupling the write also let us optimize the audit store differently than the primary database. Audit data is written once and read constantly, in large ranges, often filtered by actor or resource. That's a different access pattern than the transactional data it started as, so once it lived in its own store, we could index it specifically for search and export speed. That's part of what made the faster audit search from last month's release possible.

The short version

Real-time doesn't have to mean synchronous. It means the customer sees the event quickly, not that the system has to pay for it on every request. Separating those two lets you scale one without punishing the other.