Case study 01 · Happy-Place · 2025 — present

A multi-tenant analytics platform, from whiteboard to production

Role
Architect & lead engineer
Status
In production, used by the FirstDrive team
Runs on
10-node bare-metal Kubernetes
Built with
Python, PySpark, Airflow, Pulsar, Delta Lake

The problem

Teams across the group needed analytics over a decade of operational data without writing pipeline code — and without their data ever mixing — on shared infrastructure, under data-protection rules.

Architecture

Three planes: tenancy, ingestion and query.

Control plane — four levels of data separation

  1. L1Conglomerate
  2. L2Organization
  3. L3Project
  4. L4Environment

Distributed Python services over pub/sub configure the entire cluster from code, with PostgreSQL as the control plane database.

Data plane — CDC to medallion lakehouse

  1. sourceMongoDB · any DB
  2. captureDebezium
  3. streamPulsar + ZooKeeper
  4. processPySpark
  5. storeDelta · MinIO
  6. serveTrino · ~30 ms

Any database can be a source; MongoDB is the main one in production. Bronze → silver → gold. Airflow orchestrates; a no-code builder generates the Spark jobs and DAGs.

Decisions that shaped it

  1. 01

    Concurrency first

    Race conditions and state synchronisation drove the ingestion design before throughput did.

  2. 02

    One bus, many tenants

    Every org, project and environment gets its own isolated space on Pulsar, with credentials that can reach only its own data — designed around data-protection requirements.

  3. 03

    Two freshness paths

    Medallion layers give historical analytics 30-minute freshness, and a real-time subsystem under a minute.

  4. 04

    GitOps on bare metal

    Argo CD and Helm deploy everything to an on-prem cluster, observed with Prometheus and Grafana.

Outcomes

ingested per day
~1.5 GB
of history, spanning a decade
3 TB
average historical query
~30 ms
freshness, real-time path
< 1 min
← Back to all workTalk about a project like this →

© 2026 George Thomas · georgev.dev