Closing the gap between the event data organizations already collect and the timely, trustworthy insight they struggle to get out of it.
Modern organizations instrument everything. Every product click, pipeline run, transaction, and sensor reading is an event, and storage is cheap enough that most teams keep all of it “for later.” Yet the distance between collecting that data and understanding it has never been wider.
The industry’s answer is the modern data stack: assemble an ingestion tool, a cloud warehouse, a transformation layer, and a business-intelligence front end, then hire the data engineers and analysts to operate it. That path works, but it is out of reach for exactly the teams that need answers most: agencies and consultancies serving many clients, operations teams, and lean product companies with no dedicated data function. They inherit the tools’ complexity without the team to absorb it.
Event Horizon Data collapses that stack into a single self-service platform. Data flows in through drop-in SDKs, a language-agnostic collection agent, change-data-capture, and webhooks. It lands in a columnar warehouse engineered for low per-tenant cost and sub-second reads. And it becomes drag-and-drop dashboards, no-SQL exploration, and data mining computed in the warehouse itself (anomaly detection, forecasting, driver analysis, cohorts) that a non-analyst can drive. It is multi-tenant and white-label by design, so an agency can put its own brand in front of its clients.
This paper frames the problem, weighs the realistic solutions, and explains why an integrated, self-service, columnar approach is the right answer for organizations that need decisions rather than a data department.
Two decades of cheap storage and easy telemetry produced a habit: instrument first, ask questions later. It is a reasonable habit (you cannot analyze what you did not record) but it quietly accumulates a debt. Recording an event is nearly free; making that event trustworthy, joined, and legible months later is not.
The tooling that grew up to service this debt (managed connectors, cloud warehouses, transformation frameworks, and BI dashboards) is genuinely excellent. It is also built around an assumption that is invisible until you lack it: a team. Someone models the tables. Someone maintains the pipelines. Someone writes the query behind the chart. The tools are “self-service” in the sense that a data professional can serve themselves; they are not self-service for the marketer, the account manager, or the founder who actually has the question.
So the organizations with the sharpest need and the least infrastructure end up in one of two places. Either they buy the powerful stack and watch it sit half-configured because nobody owns it, or they never start, and decisions keep getting made on gut feel while the data they paid to collect goes cold in a bucket. The result is the same: dashboards that lag reality, numbers nobody quite trusts, and analysis that waits in a queue, or never happens.
“Turn raw events into timely, trustworthy insight” sounds like one problem. It is really five, each with its own failure mode, and a team without a data function tends to hit all of them.
Getting events reliably from many sources into a warehouse (typed, time-normalized, and each event counted exactly once) is a distributed-systems problem that teams routinely underestimate. Retries duplicate rows. A clock in the wrong timezone shifts a day’s worth of data. A dropped connection loses the tail of a batch nobody notices until a total is wrong. Change-data-capture, webhooks, file drops, and agents each fail differently.
Row-oriented databases buckle under analytical scans; cloud warehouses meter every one of those scans, so the bill scales with curiosity. And getting the economics per tenant right (cheap to store, cheap to isolate, cheap to query) is a design problem most teams solve badly on the first try, then live with.
Most BI tools assume a person writes the query, models the join, and knows which aggregation is correct. That person is the bottleneck. When “self-service analytics” means “self-service for people who already know SQL,” the marketer with the actual question is still filing a ticket.
A human notices an anomaly after it has done its damage, builds a forecast in a spreadsheet that is stale by Monday, and rarely gets to ask why a number moved. Anomaly detection, forecasting, and driver analysis usually live in a separate machine-learning effort that most teams never staff.
Every hand-off, ingest to transform to model to dashboard, is a place for a number to drift. When the stack is five vendors, nobody can answer the most corrosive question in analytics: why do these two dashboards disagree? Trust, once lost, is expensive to rebuild.
There is no shortage of tools. The question is which shape of solution fits an organization without a data team. Four approaches are worth weighing plainly.
Best-in-class connectors, a cloud warehouse, a transformation framework, and a BI tool. Unbounded flexibility and the industry’s deepest ecosystem, but it needs a data engineer and an analyst, several recurring vendor bills, and months to stand up and keep healthy.
Right when you already have a data team. Wrong when you don’t.Fast for product funnels, retention, and event taxonomies. But they are narrow by construction: bound to an event model, not a general warehouse, thin on custom modeling, and metered so that success, more volume, is what raises the price.
Great for one narrow job; not a home for all your data.Solves the dashboard layer cleanly and embeds well. But it still assumes you bring the warehouse, the pipelines that fill it, and the person who models the data underneath: the hard three-quarters of the problem.
Solves the last mile; leaves the first three.Ingestion, warehouse, exploration, and data mining as one product: drivable without SQL, multi-tenant, one bill, one coherent system. Fewer degrees of freedom than assembling your own stack, and that is the point: the trade is à-la-carte flexibility for a system a non-specialist can actually run, and whose numbers are defensible end to end because nothing crosses a vendor seam.
The fit for a team that needs decisions, not a data department.The trade-off is real and worth stating: integration versus flexibility. If you have data engineers, keep your degrees of freedom: approach (a) is built for you. If you do not, every degree of freedom is a degree of maintenance you cannot afford, and integration is not a compromise but the whole value.
For organizations without a data function, and especially for agencies reselling insight to their own clients, we recommend approach (d): an integrated, self-service platform on a columnar warehouse, with data mining computed in the warehouse rather than bolted on. Event Horizon Data is one embodiment of that recommendation; the architecture below is described as a template for what to look for, not a feature list.
Data enters through whatever the source already speaks: a drop-in JavaScript/TypeScript SDK (the same code in a browser or on a server); a language-agnostic collection agent that tails files, watches directories, and receives syslog over TLS with mutual authentication, delivering exactly once across crashes and shipping as a single self-contained binary; change-data-capture from Postgres, MySQL, and MongoDB; webhooks; scheduled pulls; and file transfer. Behind them, a distributed ingestion cluster, coordinated for leader-election and fed through a durable log, handles ordering, retries, and deduplication so the customer never has to.
Every event lands in a single shared columnar table, ordered so that each tenant’s rows sit together and a query skips every other tenant’s data before it reads a byte. Each event carries its attributes as typed, semi-structured columns on the row itself (an immutable point-in-time fact, not a surrogate key pointing at a mutable lookup) so relabeling a category tomorrow can never silently rewrite last year’s numbers. Low-cardinality attributes dictionary-compress by factors of tens to hundreds, holding the marginal storage cost per tenant to a few tens of bytes a row. Partitioning is by month and retention is a policy, not a cron job. Queries serve directly from that table with time-bucketing applied at read time, so there is no fragile hierarchy of pre-aggregates to drift out of sync; a hot field is promoted to a fully-typed column when it earns it, and onboarding a new tenant with its own schema is a row, not a migration. Because one table holds every tenant, the model scales to thousands of tenants without the file-and-metadata explosion that sinks a table-per-tenant design, and it is cluster-ready, sharded by a hash of the tenant, so it also scales out horizontally.
A drag-and-drop dashboard builder and a no-SQL explorer (filters, aggregations, drill-through) let a non-analyst ask and answer questions directly. On top sits data mining that runs in the warehouse, not in a separate pipeline: anomaly detection that flags a number the moment it leaves its expected band, forecasting that draws the trend forward, driver analysis that explains which dimension moved the metric, and cohort/retention over first-touch behavior. Add target and threshold lines, alerts, scheduled PDF reports, and shareable embeds, and the platform becomes a planning tool, not just a rear-view mirror.
For the minority who do write SQL, that no-SQL default is a floor, not a ceiling. A governed SQL Explorer and an HTTP connector for notebooks (Python, R, Jupyter) let a power user query their own data directly, both riding the same rails. You write ordinary SQL against your buckets as logical tables, blind to how the warehouse physically stores those rows, so the storage model stays ours to make faster without breaking a saved query. Your text never executes as written: it is parsed, checked against a default-deny allowlist, and reconstructed server-side into a query locked to your tenant, so a SQL surface adds no new path across the isolation boundary. Every query is cost-checked before it runs, the way BigQuery’s dry-run estimates a query, and one that would scan past your plan’s budget is refused up front with the estimate shown, so no single query, and no noisy neighbour, can run a shared warehouse into the ground. The notebook connector authenticates with a revocable, tenant-scoped token, and a saved SQL query drops onto a dashboard as an ordinary widget. SQL is an addition here, never a prerequisite.
Tenant isolation is enforced at a single fail-closed checkpoint through which every warehouse read passes: a query that cannot name the tenant it is working for does not run. On the shared table, that checkpoint injects the tenant predicate itself (isolation is a property of every query, not of a table name a caller might forget) and warehouse access uses least-privilege identities. On top of that boundary, an agency’s brand (name, logo, colors) carries all the way through dashboards, embeds, transactional emails, and PDF exports, so the agency’s clients see the agency, not the platform.
Architecture is a claim; measurement is evidence. On a single modest host, a small cluster of ingestion workers sustains roughly 4,400 events per second with no backlog at a 44-millisecond p99, and delivers about 7,400 per second with zero data loss: integrity verified two independent ways, by reconciling the data table against the audit log row for row and by a shutdown test proving no buffered event is lost. Push past the sustained rate and the excess is draining backlog, not lost data; with four workers on the host, acceptance climbs to close to 9,000 events per second. The bottleneck throughout is ingestion compute, not the warehouse. Separately, on a single modest warehouse node (about four cores and six gigabytes of memory), over a 20-million-row event table, time-filtered dashboard queries return in about 30 milliseconds, distinct counts in 186, and full-history scans in around a second, with no memory exhaustion up to sixteen heavy queries at once.