CloudSketch AI Logo
FeaturesTemplatesPricingEnterpriseAboutContactLog in
πŸŒ™β˜€οΈ
Log inStart free
Home/Templates/Data Engineering & Analytics

πŸ“Š Data Engineering & Analytics Β· Data Flow

Change Data Capture Pipeline

How every insert, update and delete in an operational database is streamed to the data lake and search index in near real time using change data capture.

More Data Engineering & Analytics templates

Opening a template is free and never counts toward your daily diagrams. Asking the AI to change it counts as one.

Drawing diagram…

What this diagram shows

  • Changes are read from the database log, not by polling tables
  • A schema registry keeps producers and consumers compatible
  • The same change stream feeds several targets

Prompt used

CDC data flow: Debezium reads the PostgreSQL write-ahead log and publishes row changes to Kafka topics, with schemas in a schema registry. A stream job writes changes to an Iceberg table in the data lake, another updates an Elasticsearch index, and a third invalidates Redis cache keys. Number the processes and label every flow.

Mermaid code
flowchart LR
subgraph EXT ["External entities"]
pg_wal["PostgreSQL WAL"]
end
subgraph PROC ["Processes"]
p1("1.0 Capture log changes")
p2("2.0 Register and validate schemas")
p3("3.0 Ingest changes to data lake")
p4("4.0 Update search index")
p5("5.0 Invalidate application cache")
end
subgraph STORE ["Data stores"]
d1[("D1 Kafka Topics")]
d2[("D2 Schema Registry Store")]
d3[("D3 Iceberg Data Lake")]
d4[("D4 Elasticsearch Index")]
d5[("D5 Redis Cache")]
end
pg_wal -->|"Row change events"| p1
p1 -->|"Raw row change records"| p2
p2 -->|"Registered schema definitions"| d2
d2 -->|"Schema validation details"| p2
p2 -->|"Schema-formatted change events"| d1
d1 -->|"CDC event stream"| p3
p3 -->|"Formatted tabular change data"| d3
d1 -->|"CDC event stream"| p4
p4 -->|"Updated index documents"| d4
d1 -->|"CDC event stream"| p5
p5 -->|"Cache invalidation keys"| d5

Inside this diagram

A data flow diagram with 6 processes and outside parties, 5 data stores and 11 data flows.

Processes and outside parties

  • PostgreSQL WAL
  • 1.0 Capture log changes
  • 2.0 Register and validate schemas
  • 3.0 Ingest changes to data lake
  • 4.0 Update search index
  • 5.0 Invalidate application cache

Data stores

  • D1 Kafka Topics
  • D2 Schema Registry Store
  • D3 Iceberg Data Lake
  • D4 Elasticsearch Index
  • D5 Redis Cache

Data flows

  • PostgreSQL WAL β†’ 1.0 Capture log changes: Row change events
  • 1.0 Capture log changes β†’ 2.0 Register and validate schemas: Raw row change records
  • 2.0 Register and validate schemas β†’ D2 Schema Registry Store: Registered schema definitions
  • D2 Schema Registry Store β†’ 2.0 Register and validate schemas: Schema validation details
  • 2.0 Register and validate schemas β†’ D1 Kafka Topics: Schema-formatted change events
  • D1 Kafka Topics β†’ 3.0 Ingest changes to data lake: CDC event stream
  • 3.0 Ingest changes to data lake β†’ D3 Iceberg Data Lake: Formatted tabular change data
  • D1 Kafka Topics β†’ 4.0 Update search index: CDC event stream
  • 4.0 Update search index β†’ D4 Elasticsearch Index: Updated index documents
  • D1 Kafka Topics β†’ 5.0 Invalidate application cache: CDC event stream
  • 5.0 Invalidate application cache β†’ D5 Redis Cache: Cache invalidation keys

Boundaries

  • External entities: PostgreSQL WAL
  • Processes: 1.0 Capture log changes, 2.0 Register and validate schemas, 3.0 Ingest changes to data lake, 4.0 Update search index and 5.0 Invalidate application cache
  • Data stores: D1 Kafka Topics, D2 Schema Registry Store, D3 Iceberg Data Lake, D4 Elasticsearch Index and D5 Redis Cache

Make your own diagram like this

Describe your own system in plain English and CloudSketch AI draws it in seconds. See all AI diagram generators.

Related templates

C4 ArchitectureProData Engineering & Analytics

Modern Data Platform Architecture

A typical modern data stack: data is loaded from apps and SaaS tools into a cloud warehouse, modelled with dbt, orchestrated with Airflow and served to BI dashboards.

Database ERDData Engineering & Analytics

Sales Star Schema

A classic star schema for sales analytics: one fact table of order lines surrounded by date, customer, product, store and promotion dimensions.

FlowchartData Engineering & Analytics

Nightly Batch ETL Workflow

The steps of a nightly batch pipeline with data quality gates: extract, validate, transform, load and publish, stopping safely when checks fail.

DeploymentProData Engineering & Analytics

Real-time Streaming Analytics Deployment

Where a real-time analytics stack runs: Kafka for events, Flink for stream processing, a real-time OLAP database and live dashboards.

State MachineData Engineering & Analytics

Data Pipeline Run States

The states of a single pipeline run in an orchestrator like Airflow, including retries, upstream failures and manual reruns.

Data FlowData Engineering & Analytics

Data Lake Zones (Bronze, Silver, Gold)

The medallion pattern for a data lake: raw data lands in bronze, is cleaned in silver, and turned into business-ready tables in gold.