Real-Time Analytics Platform
Sub-second dashboards over 12 billion events
A growth-stage SaaS analytics client
A growth-stage SaaS had a great analytics product, on yesterday's data. Customers were happy until enterprise buyers started asking why the dashboard lagged the production system by six hours. The existing pipeline was Postgres + nightly dbt, and tacking real-time onto it would have meant rewriting half the warehouse. I designed and shipped a streaming layer alongside the warehouse: events flow through Kafka into ClickHouse for hot queries, with the warehouse continuing to serve the long-tail historical work. The dashboard now renders sub-second over the last 90 days, and the team kept their dbt investment intact.
Representative architecture study. Patterns, trade-offs and approach reflect engagements I take on; specific metrics and clients are illustrative.
Results
What changed, in numbers
The metrics the engagement is measured by.
<30s
Freshness
from 6 hours
780ms
p95 Latency
from 8.4 seconds
$1.4M
Enterprise ARR
unblocked in 60 days
-22%
Warehouse Cost
via hot/cold split
Challenge
What was broken
The product depended on freshness the architecture couldn't deliver. Every customer interview surfaced the same complaint: 'Why is yesterday's data the freshest you have?' Postgres was already groaning under 12 billion events, and the nightly dbt run was on its third hour. Two enterprise deals had stalled in procurement specifically over data freshness SLAs.
Solution
The shape of the fix
A two-tier analytics architecture, ClickHouse for the hot 90-day window, Postgres + dbt for everything older, with a query router and reconciliation pipeline that kept the two stores honest. The product team kept their dbt models, the customers got real-time, and the warehouse stopped being on fire.
Approach
How I tackled it
The concrete moves that took the project from broken to shipped.
Stood up a Kafka + ClickHouse hot path for the last 90 days of events, with materialized views computed at ingest
Kept the existing Postgres + dbt warehouse for historical and finance-critical queries, no rewrite needed
Built a query router so the dashboard transparently picked hot vs. cold storage based on time range and aggregation
Wired ClickHouse projections and pre-aggregations so p95 dashboard queries dropped under 800ms even on 90-day windows
Shipped a dual-write reconciliation job that validated hot-path totals against the warehouse hourly, catching schema drift before customers did
Built tenant-isolation and row-level security into the ClickHouse layer to keep the SOC 2 boundary clean
Outcomes
What shipped, and what it changed
Measured results from the engagement, told as a story rather than a scoreboard.
Reduced dashboard data freshness from 6 hours to under 30 seconds end to end
Drove p95 dashboard query latency from 8.4s to 780ms on the same 90-day windows
Closed two stalled enterprise deals worth a combined $1.4M ARR within 60 days of launch
Cut warehouse cost by 22% by moving high-frequency queries off the hot path
Held SOC 2 controls intact through the migration with zero scope expansion
Stack
Technologies used
Linked entries open the technology page with related studies, playbooks, and notes.
Services
How I helped
The specific services involved in this engagement. Each links to a deeper breakdown.
Lessons
What I would tell the next team
The takeaways I carry into every similar engagement.
Don't rewrite the warehouse to add real-time, run a hot path next to it and route by time range
Reconciliation jobs are the only thing that catches the silent drift between two storage systems
ClickHouse projections do most of the work, get the projection design right before tuning anything else
Related
Other studies you might recognize
Engagements with overlapping problem shapes, industries, or stacks.
Have a similar challenge?
If any of this looks like the project on your desk, the conversation is the cheapest part. You can also browse other saas work or the full service list.