All posts

The price that changed twice

By Event Horizon Data
engineeringecommerce

We already read change data capture from Postgres and MySQL, log based, without adding query load to the database a store depends on for checkout. That part worked. What we hadn't looked at closely was what happens to that stream once it lands.

Every change arrived as one more row in an append only log, which is correct for an order or a page view. It is the wrong shape for a product's price, or a customer's address, or a stock count. Those don't accumulate. They change, and the only honest answer to "what is it now" is the latest one, not a history that grows every time someone edits a field.

The fix is one flag. Name the column that identifies a record, a SKU say, and every later change to that key collapses to its current state automatically, the same way every other derived view on this platform works: a materialized view that fires on insert, never a scheduled job. A delete is recorded as a delete, not silently dropped.

The bug it surfaced

While wiring the joined-dashboard side of this up, we mutation tested the part that keeps a Top-N breakdown honest, the code that decides which rows survive a "top 5" cut. It assumed every joined table had a time column, because every joined table always had, until now. A product's current state has no time column at all. The first real test we ran, breaking revenue down by product category, threw UNKNOWN_IDENTIFIER.

Forcing the old code back and rerunning the same test reproduced the exact error before we trusted the fix. That's the headline use case for this feature, and it would have failed on the first real chart anyone tried to build with it.

What we tested

We set a product's price to $9.99, then repriced it to $12.99, both changes going through the same stream as any other. Joined against its orders, the dashboard reads $12.99, and the order count isn't doubled by the two price versions sitting in the change history underneath. Breaking a chart down by a joined table's own columns, category or name, not only the join key, is the ordinary case now, in the no-code builder and in SQL both.

What it costs

We measured it before writing this. On a single ClickHouse node, breaking a 500,000-row orders table down by a joined product's category took 233 milliseconds at 10,000 products and 345 at a million, a hundredfold jump in products moving the answer by about a third. ClickHouse builds the product side into a hash table once and probes it from the orders side. The join, not the lookup, sets the floor.