Keeping analytics off the hot path with Kafka

Every time a shopper opens a size guide in Fitly, taps a size, or accepts a recommendation, we want to know about it — that is what makes the analytics dashboard worth having. But a storefront is the worst possible place to do bookkeeping. The shopper is mid-decision, the page is someone else's, and nothing we do there is allowed to be slow.

So the events do not go into the database when they happen. They go into Apache Kafka, and something else writes them down later. This post is about why — and the reason is less about speed than it sounds. Fitly is SaaS, so the real risk is not a slow request. It is one busy storefront taking the database down for every other merchant on the platform.

What Kafka actually is

Kafka is usually described as a message queue, which undersells it. It is closer to an append-only log that many readers can share. A producer appends a message to a named log — a topic — and that is the end of the producer's job. Consumers read forward through the log at their own pace, each tracking its own position.

Three properties follow from that design, and all three matter here:

• Writing is cheap and constant-time. Appending to the end of a log costs the same whether the log holds a thousand messages or a billion.
• Reading is decoupled from writing. A slow consumer does not slow the producer down. It just falls further behind, and catches up later.
• A topic is split into partitions. Each partition is an independent log that can be read in parallel, so throughput scales with partition count rather than being capped by one writer.

That last point is also the one constraint: ordering is guaranteed within a partition, not across the topic. Which key you choose decides what stays ordered.

The problem: a write on the hot path

The obvious way to record a widget event is to insert a row. The storefront calls our endpoint, the endpoint runs an INSERT, and the response comes back when Postgres says the row is committed.

That works fine until it doesn't. The shopper's request now depends on our database being fast, reachable, and not busy — and a single product page can easily emit several events in a few seconds: the widget rendered, the button was clicked, a size was picked, then another size, then another.

The deeper problem is that the two sides have nothing to do with each other. Recording an event is our concern. Waiting for it is the shopper's. Kafka is how we stop those being the same thing.

Now multiply that by every store

Fitly is SaaS: one deployment serves every merchant who installs it. The traffic we have to survive is therefore never a single store's traffic — it is the sum of all of them, and that sum grows on three axes at once. More merchants install the app. Each merchant sends more shoppers. Each shopper emits several events per session. Those multiply rather than add.

A few hundred active storefronts is not a big number for a Shopify app, and it does not take many shoppers on each before we are fielding thousands of concurrent events. The peaks do not politely take turns, either. Stores in the same region get busy at the same hours, and on the days that matter most — a launch, a seasonal sale, Black Friday — every one of them peaks together.

This is where writing on the hot path stops being merely slow and starts being dangerous. Every concurrent insert needs a database connection, and a connection pool is a small fixed number — ours is twenty. Two thousand simultaneous events do not become two thousand parallel writes. They become a queue for twenty connections, and under a sustained spike that queue grows faster than it drains. Request timeouts follow, then retries, which add load to a database that is already the bottleneck.

What turns that slowdown into an outage is what else lives in the same database. The merchant admin — the chart builder, the assignment screens, the analytics dashboard — reads and writes the same Postgres instance. Storefront events exhausting the pool would not merely delay analytics; they would starve the app the merchant is actually trying to use. The failure arrives sideways: the admin crawls because strangers are browsing someone else's shop.

That is the scenario Kafka exists to prevent here. Not “inserts are slow”, but one tenant's traffic spike degrading every other tenant, through a resource they all share.

What the pipeline looks like

The storefront widget posts to an endpoint on our NestJS service that answers 202 Accepted — not 200 OK. The distinction is honest rather than pedantic: the event has been accepted for processing, and it has not been processed yet.

Behind that endpoint, the service does one thing. It serialises the event and emits it to a widget.events topic, keyed by shop domain, and returns immediately — the emit is deliberately not awaited, so the response does not wait on the broker either. If Kafka were unreachable, the failure is logged and the shopper still gets their answer. Analytics is not worth degrading a product page over.

Keying by shop is the decision that makes the ordering constraint work for us instead of against us. All events from one store land in the same partition, in order, which is what the size-confusion analysis needs — it reconstructs sequences like S → M → S within a session. Across different stores, order is irrelevant, so we get parallelism for free. The topic runs six partitions by default, and the consumer processes those partitions concurrently.

Batching is where the performance actually comes from

Offloading the write to a queue moves the work; it does not reduce it. The gain comes from what the consumer is now free to do, which is stop working one row at a time.

Our consumer reads whole batches rather than individual messages, and for each batch it:

• parses every message in the batch into memory first, skipping malformed ones rather than letting one bad record stall the partition;
• resolves the foreign keys once for the entire batch — shop domains, chart UUIDs and Shopify product IDs all have to become internal integer IDs, and doing that per row would mean three lookups per event instead of three lookups per batch;
• writes the rows in multi-row INSERT statements, flushing every five thousand rows.

Ten thousand events that would have been ten thousand round trips become a handful. This is the part worth stealing even if you never use Kafka: the queue's real value is that it accumulates work, and work in bulk is dramatically cheaper than the same work in pieces.

There is a fallback for the awkward case. If a bulk insert fails, one bad row would otherwise take five thousand good ones down with it, so the consumer retries the batch row by row, logs the individual failures, and keeps the rest. Slow, but it only happens when something is already wrong.

What this changes about scaling

The useful shift is that a traffic spike stops being a database problem and becomes a queue-depth problem — and queue depth is a number that goes up and comes back down on its own. When a store runs a campaign, its events pile up in the topic and the consumer works through the backlog at whatever rate the database is comfortable with. Nothing is dropped, nothing times out, and no shopper waits.

Crucially, the database now sees a workload it can predict. Instead of thousands of tiny concurrent inserts competing for twenty connections, it sees one consumer, holding one connection at a time, submitting large batched writes. The load stops being a function of how many shoppers happen to be browsing and becomes a function of how fast we choose to drain the log. That is the whole trick: the busiest tenant can no longer set the pace for everyone else, because no tenant is talking to the database at all.

It also makes the two halves independently scalable. Ingest capacity is the web service; write capacity is the consumer. They can be sized separately, and the consumer can be restarted, deployed, or briefly taken down without the storefront endpoint noticing, because messages wait in the log until someone reads them.

The topic is not a database, though, and we do not treat it as one. Messages are retained for a day and capped at 500 MB per partition — long enough to absorb a backlog or a bad deploy, not long enough to pretend the log is storage. Postgres remains the source of truth.

Where the events end up

The consumer writes into two append-only tables: one for widget events (views, engagements, accepted recommendations) and one for size selections, which keeps the pick order within a session. A nightly aggregation job rolls those into daily snapshots, and the merchant-facing dashboard reads the snapshots rather than the raw logs.

So there are three stages, each doing what it is good at: Kafka absorbs and orders, Postgres stores in bulk, and the aggregation job turns millions of rows into the handful of numbers a merchant actually looks at — return-rate trend, per-chart performance, and the per-product confusion score.

Would we do it again?

Kafka is not free. It is another service to run, another thing to monitor, and it makes the system eventually consistent: an event is recorded a moment after it happened, not at the instant it happened. For a dashboard measured in days, that is a non-issue. For something that needed to be correct to the second, it would be the wrong trade.

We would do it again here, for one reason. In a multi-tenant product the thing you are really protecting is not latency, it is isolation — the guarantee that one merchant's good day cannot become every other merchant's bad one. Kafka is how we keep the busiest storefront on the platform from being able to reach the database that everyone shares. Everything else in this post is a consequence of that.