I did not set out to build a data platform. I set out to stop assembling one.

I had worked on analytics before, and doing it end to end was miserable in a specific way: nothing on the market solved the whole path simply. You picked an ingestion tool, a warehouse, a transformation layer and a dashboard product, and then you spent your time on the seams between them. Sometimes the pieces did not even talk to each other. The work that produced insight was a fraction of the work that kept the assembly standing.

So I started building the thing I wanted. What follows is the part nobody puts in a product blog: the three things I was sure of, and what each one cost me when I finally measured it.

"A database scales in a cluster, and the answers come back in under a second"

This was the first one, and it is the most comfortable belief in the industry, because it moves the problem somewhere else. If the answers are slow, add nodes.

What I learned is that the cluster is not the question. The question is what the query has to read. A distributed engine can spread a scan over more machines and still be reading a hundred times more data than it needs to, and then you are paying for a cluster to do the wrong work faster. Sub-second is a property of the storage layout and the query, not of the number of machines under it.

The corollary is uncomfortable for anyone who has ever sized a cluster to fix a latency problem, myself included: I had been treating hardware as an answer to a design question.

"OLAP cubes solve performance, because they are cached"

The second one. Precompute the aggregates, keep them warm, serve them fast. It works, and I built it.

What it does not tell you upfront is the cost of being wrong about the grain. A cube is an answer to the questions you thought of when you defined it. The moment somebody asks for a breakdown you did not precompute, you are either back on the raw data with all the latency you were avoiding, or you are widening the cube and paying for it in maintenance, storage and refresh time. The cache is not free, it is prepaid, and you pay for every combination whether anyone asks for it or not.

I ended up deleting a whole hierarchy of precomputed tiers and replacing it with a single coarse one, maintained by the engine itself rather than by a job I had to babysit. Fewer answers precomputed, and the ones that remain are actually used.

"ClickHouse will do billions of rows in milliseconds"

The third. It is the reason a lot of people arrive at columnar engines, and it is true in the way that "a car does 200 kilometres an hour" is true. It is true about the engine, on the right road, in the right gear.

Reading billions of rows in milliseconds requires the rows to be laid out so the engine can skip almost all of them, and it requires the values to be typed rather than parsed. I found out how much that second half costs by doing it wrong first. Serving a query by converting a JSON column to a string and parsing it took 106 seconds; the same query against pre-classified typed columns took 4 seconds over 40 million rows, both with FINAL. That is not a tuning difference. That is a different product.

The engine was never the thing that made it fast. What made it fast was giving the engine something it could skip.

The storage arc, and what killed each step

Underneath those three beliefs there is a sequence, and each step ended because something measurable said it had to.

EAV first. One row per attribute. It is the schema you reach for when you do not know what your customers will send you, and it is beautifully flexible right up until the moment you have to reassemble an event from its parts at scale. It did not scale, and I abandoned it.

Then a wide table per tenant. Real typed columns, one physical table per customer per dataset. Fast, clean, and it fails in a way you do not see coming: the ceiling is not row count, it is table count. Every table costs disk and, far worse, costs maintenance throughput. A few thousand of them and the platform spends its life keeping its own catalogue alive.

Now one shared table with typed maps. Every tenant's events in the same table, ordered by tenant first so a customer's query never reads another's storage, with the metrics and dimensions pre-classified into typed map columns and a JSON column as the fallback for everything else. One table, whatever the number of customers.

Three schemas in three years, and I did not change any of them because I read something. I changed them because something I had built stopped answering.

The part I find hardest to admit

When I decided to move the analytics engine to a columnar database, I wrote down the reasons. Then, because the migration was big enough to be frightening, I measured them.

Every single one failed.

The scaling problem I blamed on the job model turned out to be a configuration value sized wrong: corrected, the old engine handled far more than I was asking of it. The native ingestion I was going to gain removed no services and would have added one. The throughput ceiling I was migrating away from was not the database at all, which was running at a little over one percent of its capacity, but a single application process saturating one CPU core. And the dictionary layer I was sure I would need was slower than the join I already had, on top of costing gigabytes of memory the join did not need.

Four arguments, four refutations. And I did the migration anyway, because the measuring turned up different reasons that were better than the ones I started with: query latency, storage size, how the catalogue behaved under churn, and how many scans could run at once.

That is the whole lesson, and it took me years to learn it properly. The belief you should measure first is the one you are most sure about, because it is the one you have stopped questioning, and it is therefore the one quietly shaping every decision downstream of it.

Was it worth it

When somebody asks me what building this has been like, the honest answer is short and not very marketable: it has been hell. Years of rethinking an architecture, three storage models, and a recurring discovery that the thing I was most confident about was the thing costing me the most.

I would do it again, and not out of stubbornness. The reason is that the alternative was the assembly I started out trying to escape, and every one of those failed assumptions is now a property of the product rather than a lesson I keep in my head. The tiers are maintained by the engine because I got tired of babysitting a job. The columns are typed because I measured what parsing cost. The table is shared and ordered by tenant because I watched the per-tenant model hit its ceiling.

None of that would be in there if I had been right the first time.