All posts

SQL and notebooks, without a warehouse to run

By Ricardo Rodrigues
productsqlanalytics

The first thing we ever wrote down about Event Horizon Data is that the last mile should not need an analyst. That has not changed. Filters, aggregations, drill-through, cohorts, forecasts, driver analysis, all of it drag-and-drop, is still the default, and still the point.

But a product is used by a team, and a team is not one kind of person. Some of your people do write SQL, and some of them live in a notebook rather than a dashboard. Asking them to give that up is the wrong trade. So we built two things for them, and built both on rails.

Write SQL against your tables, not our warehouse

There is now a SQL editor inside Explore. You write ordinary SQL, joins, CTEs, subqueries, window functions, against your buckets as if each were a table. What you do not see, and do not need to, is how the warehouse actually stores those rows underneath. That is deliberate, and it is the whole design:

  • Your text never runs as-is. Every query is parsed, checked against a default-deny allowlist, and rebuilt on the server before it touches the database. The version that executes is one we constructed from your query's meaning, locked to your tenant. A query that cannot be expressed safely does not run, it is rejected with a reason.
  • You are blind to our schema, on purpose. A bucket is a table to you. How it is partitioned, tiered, and stored is ours to change without breaking a single query you have saved. You get a stable surface; we keep the freedom to make it faster underneath.

Every query is cost-checked before it runs

Anyone who has run a warehouse has watched one careless query scan a fortune. We do what BigQuery's dry-run does, and we do it for every query, every time: before a query runs, we estimate how much it would read and compare it against your plan's budget. One that would read too much is refused up front, with the estimate shown, so you can narrow it, not after the bill arrives. On top of that a query may join at most four of your tables, and its execution time, memory and join size are all capped, with ad-hoc SQL running at a lower priority than dashboards. We are precise about this rather than absolute: the estimate counts rows READ, and a join does work proportional to the product of its sides, so what those caps buy is that a heavy query degrades itself and yields, instead of taking the warehouse down with it.

Saved a query you like? Save it, and drop it straight onto a dashboard as a widget. A SQL query and a drag-and-drop widget are the same kind of object here.

A connector for your notebook

The second thing is an HTTP API, so the notebook people can stay in the notebook. Point Python, R, or Jupyter at one endpoint, send SQL, get rows back as JSON or CSV. It runs through the exact same rails as the in-app editor: same parsing, same rebuild, same cost gate.

Authentication is a revocable token you generate yourself, optionally with an expiry date. The tenant is taken from the token, never from the request body, so a token for one account cannot ask for another account's data. There is a test that proves this, and it fails our build if that ever stops being true.

Why on rails matters

It would have been faster to hand power users a raw connection and wish them luck. We didn't, for the same reason the whole product exists: the hard, unglamorous three-quarters of analytics is the part that keeps you safe and keeps you fast when you are not the only tenant on the box. Tenant isolation, cost control, and a schema you can change without breaking anyone are not features you bolt on after a breach or a surprise bill. They are the rails, and we would rather build them once, for everyone, than leave them to each query.

If you want to see how the isolation holds up to an attacker rather than take our word for it, we red-teamed our own boundary and published the result. That is on the security page. The longer architectural argument is in the whitepaper.