Writing

Notes from building data infrastructure.

What broke, what we changed, and what I'd tell someone starting out. RSS

  1. 8 min

    Debezium to Pulsar to Delta Lake: making silent failures loud

    How we stream MongoDB changes through Debezium Server and Apache Pulsar into a Delta Lake lakehouse, and the silent failures we had to turn into loud ones.

    Debezium · Apache Pulsar · CDC · MongoDB · Delta Lake · Apache Spark · Airflow