
π Data Engineering & Analytics Β· Data Flow
How every insert, update and delete in an operational database is streamed to the data lake and search index in near real time using change data capture.
Opening a template is free and never counts toward your daily diagrams. Asking the AI to change it counts as one.
Drawing diagramβ¦
CDC data flow: Debezium reads the PostgreSQL write-ahead log and publishes row changes to Kafka topics, with schemas in a schema registry. A stream job writes changes to an Iceberg table in the data lake, another updates an Elasticsearch index, and a third invalidates Redis cache keys. Number the processes and label every flow.
flowchart LR
subgraph EXT ["External entities"]
pg_wal["PostgreSQL WAL"]
end
subgraph PROC ["Processes"]
p1("1.0 Capture log changes")
p2("2.0 Register and validate schemas")
p3("3.0 Ingest changes to data lake")
p4("4.0 Update search index")
p5("5.0 Invalidate application cache")
end
subgraph STORE ["Data stores"]
d1[("D1 Kafka Topics")]
d2[("D2 Schema Registry Store")]
d3[("D3 Iceberg Data Lake")]
d4[("D4 Elasticsearch Index")]
d5[("D5 Redis Cache")]
end
pg_wal -->|"Row change events"| p1
p1 -->|"Raw row change records"| p2
p2 -->|"Registered schema definitions"| d2
d2 -->|"Schema validation details"| p2
p2 -->|"Schema-formatted change events"| d1
d1 -->|"CDC event stream"| p3
p3 -->|"Formatted tabular change data"| d3
d1 -->|"CDC event stream"| p4
p4 -->|"Updated index documents"| d4
d1 -->|"CDC event stream"| p5
p5 -->|"Cache invalidation keys"| d5A data flow diagram with 6 processes and outside parties, 5 data stores and 11 data flows.
Describe your own system in plain English and CloudSketch AI draws it in seconds. See all AI diagram generators.
A typical modern data stack: data is loaded from apps and SaaS tools into a cloud warehouse, modelled with dbt, orchestrated with Airflow and served to BI dashboards.
A classic star schema for sales analytics: one fact table of order lines surrounded by date, customer, product, store and promotion dimensions.
The steps of a nightly batch pipeline with data quality gates: extract, validate, transform, load and publish, stopping safely when checks fail.
Where a real-time analytics stack runs: Kafka for events, Flink for stream processing, a real-time OLAP database and live dashboards.
The states of a single pipeline run in an orchestrator like Airflow, including retries, upstream failures and manual reruns.
The medallion pattern for a data lake: raw data lands in bronze, is cleaned in silver, and turned into business-ready tables in gold.