📊 Data Engineering & Analytics · Data Flow
Change Data Capture Pipeline
How every insert, update and delete in an operational database is streamed to the data lake and search index in near real time using change data capture.
Drawing diagram…
What this diagram shows
- Changes are read from the database log, not by polling tables
- A schema registry keeps producers and consumers compatible
- The same change stream feeds several targets
Prompt used
CDC data flow: Debezium reads the PostgreSQL write-ahead log and publishes row changes to Kafka topics, with schemas in a schema registry. A stream job writes changes to an Iceberg table in the data lake, another updates an Elasticsearch index, and a third invalidates Redis cache keys.
Mermaid code
flowchart LR DB[(Orders PostgreSQL)] -->|Write-ahead log| DZ[Debezium Connector] DZ -->|Row changes| K[Kafka Topics] SR[(Schema Registry)] -->|Schemas| DZ K -->|Changes| LJ[Lake Writer - Flink] K -->|Changes| SJ[Search Indexer] K -->|Changes| CJ[Cache Invalidator] LJ -->|Upserts| ICE[(Iceberg Tables in S3)] SJ -->|Documents| ES[(Elasticsearch)] CJ -->|Delete keys| RC[(Redis Cache)] SR -->|Schemas| LJ
Related templates
Modern Data Platform Architecture
A typical modern data stack: data is loaded from apps and SaaS tools into a cloud warehouse, modelled with dbt, orchestrated with Airflow and served to BI dashboards.
Sales Star Schema
A classic star schema for sales analytics: one fact table of order lines surrounded by date, customer, product, store and promotion dimensions.
Nightly Batch ETL Workflow
The steps of a nightly batch pipeline with data quality gates: extract, validate, transform, load and publish, stopping safely when checks fail.
Real-time Streaming Analytics Deployment
Where a real-time analytics stack runs: Kafka for events, Flink for stream processing, a real-time OLAP database and live dashboards.
Data Pipeline Run States
The states of a single pipeline run in an orchestrator like Airflow, including retries, upstream failures and manual reruns.
Data Lake Zones (Bronze, Silver, Gold)
The medallion pattern for a data lake: raw data lands in bronze, is cleaned in silver, and turned into business-ready tables in gold.