
π Data Engineering & Analytics Β· Data Flow
The medallion pattern for a data lake: raw data lands in bronze, is cleaned in silver, and turned into business-ready tables in gold.
Opening a template is free and never counts toward your daily diagrams. Asking the AI to change it counts as one.
Drawing diagramβ¦
Medallion architecture data flow: batch files, API data and streaming events land unchanged in the bronze zone of a Delta Lake. A cleaning job de-duplicates, standardises types and masks personal data into silver. Aggregation jobs build gold tables such as daily revenue and customer 360. BI tools and ML features read from gold. Number the processes and label every flow.
flowchart LR
subgraph EXT ["External entities"]
batch_sys["Batch Source Systems"]
api_sys["API Data Sources"]
stream_sys["Streaming Event Sources"]
bi_tools["BI Tools"]
ml_sys["ML Feature Store / ML Systems"]
end
subgraph PROC ["Processes"]
p1("1.0 Ingest Raw Data")
p2("2.0 Clean and Standardize Data")
p3("3.0 Aggregate Business Metrics")
p4("4.0 Serve Analytics and Features")
end
subgraph STORE ["Data stores"]
d1[("D1 Bronze Zone - Raw Delta Tables")]
d2[("D2 Silver Zone - Cleaned Delta Tables")]
d3[("D3 Gold Zone - Aggregated Delta Tables")]
end
batch_sys -->|"Raw Batch Files"| p1
api_sys -->|"Raw API Data Payload"| p1
stream_sys -->|"Raw Streaming Events"| p1
p1 -->|"Unchanged Raw Data Records"| d1
d1 -->|"Unprocessed Bronze Records"| p2
p2 -->|"Deduplicated, Standardized & Masked Data"| d2
d2 -->|"Cleaned Silver Records"| p3
p3 -->|"Daily Revenue & Customer 360 Aggregations"| d3
d3 -->|"Gold Aggregated Tables Data"| p4
p4 -->|"Analytics Reports & Dashboard Data"| bi_tools
p4 -->|"Machine Learning Features"| ml_sysA data flow diagram with 9 processes and outside parties, 3 data stores and 11 data flows.
Describe your own system in plain English and CloudSketch AI draws it in seconds. See all AI diagram generators.
A typical modern data stack: data is loaded from apps and SaaS tools into a cloud warehouse, modelled with dbt, orchestrated with Airflow and served to BI dashboards.
How every insert, update and delete in an operational database is streamed to the data lake and search index in near real time using change data capture.
A classic star schema for sales analytics: one fact table of order lines surrounded by date, customer, product, store and promotion dimensions.
The steps of a nightly batch pipeline with data quality gates: extract, validate, transform, load and publish, stopping safely when checks fail.
Where a real-time analytics stack runs: Kafka for events, Flink for stream processing, a real-time OLAP database and live dashboards.
The states of a single pipeline run in an orchestrator like Airflow, including retries, upstream failures and manual reruns.