Built an event-driven ingestion lake
Replaced batch ETL with Kafka + S3 + Iceberg and contract tests, cutting data latency from 24h to 8m.
3
Challenges
2
Solved
1
Part of
Impact & Results
Contracts + DLQs let us ship fast without poisoning downstream consumers.
The Story
How this challenge was approached and solved
Implemented CDC with Debezium, enforced schema contracts per topic, and added SLAs per domain. Airflow orchestrated Iceberg compaction and partition evolution. Added DLQs plus replay tooling to keep data quality high during rapid iteration.
Focus Areas
Data PlatformsStreaming
Tools & Technologies
KafkaDebeziumAirflowIcebergS3
Screenshots
A closer look at the work
Videos
Demos and walkthroughs