CloudSketch AI Logo
FeaturesTemplatesPricingEnterpriseAboutContactLog in
πŸŒ™β˜€οΈ
Log inStart free
Home/Templates/SaaS & Cloud

☁️ SaaS & Cloud · Data Flow

Data Lakehouse Pipeline

How data from apps, databases and events lands in a lakehouse and becomes dashboards and ML features: bronze, silver and gold layers.

More SaaS & Cloud templates

Opening a template is free and never counts toward your daily diagrams. Asking the AI to change it counts as one.

Drawing diagram…

What this diagram shows

  • Change data capture streams database changes without heavy queries
  • Bronze keeps raw data, silver cleans it, gold models it for reporting
  • BI and ML both read from the same governed tables

Prompt used

Data flow for a lakehouse: operational databases stream changes through CDC (Debezium) and app events through Kafka, SaaS tools are loaded by a batch connector (Fivetran). Everything lands raw in the bronze layer (Delta Lake on object storage), Spark jobs clean and de-duplicate into silver, then build business models (orders, customers, revenue) in gold. BI dashboards and ML feature pipelines read gold; a data catalog tracks lineage and access. Number the processes and label every flow.

Mermaid code
flowchart LR
subgraph EXT ["External entities"]
op_db["Operational Databases"]
app_evt["App Events"]
saas["SaaS Tools"]
end
subgraph PROC ["Processes"]
p1("1.0 Stream Database Changes")
p2("2.0 Ingest App Events")
p3("3.0 Batch Extract SaaS Data")
p4("4.0 Clean and Deduplicate Data")
p5("5.0 Build Business Models")
p6("6.0 Track Lineage and Access")
end
subgraph STORE ["Data stores"]
d1[("D1 Bronze Layer Raw Delta Lake")]
d2[("D2 Silver Layer Cleaned Delta Lake")]
d3[("D3 Gold Layer Business Models Delta Lake")]
d4[("D4 Data Catalog Storage")]
end
op_db-->|"Database CDC changes via Debezium"|p1
app_evt-->|"Application events via Kafka"|p2
saas-->|"SaaS data via Fivetran"|p3
p1-->|"Raw database CDC logs"|d1
p2-->|"Raw event payloads"|d1
p3-->|"Raw SaaS records"|d1
d1-->|"Raw data records"|p4
p4-->|"Cleaned and deduplicated data"|d2
d2-->|"Silver cleansed datasets"|p5
p5-->|"Business models data for orders, customers, and revenue"|d3
p1-->|"Streaming ingestion metadata"|p6
p2-->|"Event ingestion metadata"|p6
p3-->|"Batch ingestion metadata"|p6
p4-->|"Silver transformation lineage"|p6
p5-->|"Gold business model lineage and access logs"|p6
p6-->|"Catalog lineage and access metadata"|d4

Inside this diagram

A data flow diagram with 9 processes and outside parties, 4 data stores and 16 data flows.

Processes and outside parties

  • Operational Databases
  • App Events
  • SaaS Tools
  • 1.0 Stream Database Changes
  • 2.0 Ingest App Events
  • 3.0 Batch Extract SaaS Data
  • 4.0 Clean and Deduplicate Data
  • 5.0 Build Business Models
  • 6.0 Track Lineage and Access

Data stores

  • D1 Bronze Layer Raw Delta Lake
  • D2 Silver Layer Cleaned Delta Lake
  • D3 Gold Layer Business Models Delta Lake
  • D4 Data Catalog Storage

Data flows

  • Operational Databases β†’ 1.0 Stream Database Changes: Database CDC changes via Debezium
  • App Events β†’ 2.0 Ingest App Events: Application events via Kafka
  • SaaS Tools β†’ 3.0 Batch Extract SaaS Data: SaaS data via Fivetran
  • 1.0 Stream Database Changes β†’ D1 Bronze Layer Raw Delta Lake: Raw database CDC logs
  • 2.0 Ingest App Events β†’ D1 Bronze Layer Raw Delta Lake: Raw event payloads
  • 3.0 Batch Extract SaaS Data β†’ D1 Bronze Layer Raw Delta Lake: Raw SaaS records
  • D1 Bronze Layer Raw Delta Lake β†’ 4.0 Clean and Deduplicate Data: Raw data records
  • 4.0 Clean and Deduplicate Data β†’ D2 Silver Layer Cleaned Delta Lake: Cleaned and deduplicated data
  • D2 Silver Layer Cleaned Delta Lake β†’ 5.0 Build Business Models: Silver cleansed datasets
  • 5.0 Build Business Models β†’ D3 Gold Layer Business Models Delta Lake: Business models data for orders, customers, and revenue
  • 1.0 Stream Database Changes β†’ 6.0 Track Lineage and Access: Streaming ingestion metadata
  • 2.0 Ingest App Events β†’ 6.0 Track Lineage and Access: Event ingestion metadata
  • 3.0 Batch Extract SaaS Data β†’ 6.0 Track Lineage and Access: Batch ingestion metadata
  • 4.0 Clean and Deduplicate Data β†’ 6.0 Track Lineage and Access: Silver transformation lineage
  • 5.0 Build Business Models β†’ 6.0 Track Lineage and Access: Gold business model lineage and access logs
  • 6.0 Track Lineage and Access β†’ D4 Data Catalog Storage: Catalog lineage and access metadata

Boundaries

  • External entities: Operational Databases, App Events and SaaS Tools
  • Processes: 1.0 Stream Database Changes, 2.0 Ingest App Events, 3.0 Batch Extract SaaS Data, 4.0 Clean and Deduplicate Data, 5.0 Build Business Models and 6.0 Track Lineage and Access
  • Data stores: D1 Bronze Layer Raw Delta Lake, D2 Silver Layer Cleaned Delta Lake, D3 Gold Layer Business Models Delta Lake and D4 Data Catalog Storage

Make your own diagram like this

Describe your own system in plain English and CloudSketch AI draws it in seconds. See all AI diagram generators.

Related templates

DeploymentProSaaS & Cloud

Kubernetes Microservices Deployment

How a microservices app runs on Kubernetes in the cloud: ingress, namespaces, services with multiple replicas, managed databases and monitoring.

FlowchartSaaS & Cloud

CI/CD Pipeline Flowchart

A typical CI/CD pipeline from a pull request to production: build, tests, security scans, staging deploy, approval and a canary release.

SequenceSaaS & Cloud

OAuth 2.0 / OpenID Connect Login

The authorization code flow with PKCE used by most modern web and mobile apps to log users in through an identity provider.

C4 ArchitectureProSaaS & Cloud

Multi-Tenant SaaS Architecture

A B2B SaaS product serving many customer organisations from one platform: tenant-aware services, per-tenant data isolation, billing and admin.

DeploymentProSaaS & Cloud

AWS Three-Tier Web Application

The classic three-tier web app on AWS: CloudFront and a load balancer, an autoscaling app tier in private subnets, and a Multi-AZ database.

C4 ArchitectureProSaaS & Cloud

Event-Driven Architecture with Kafka

Services that talk through events instead of direct calls: an order service publishes events that inventory, payments, notifications and analytics react to.