How to Build a Data Platform Without a Warehouse (Lakehouse + Streaming + Semantic Layer)

If you've ever tried to stand up a "proper" data warehouse and ended up with spiraling costs, brittle pipelines, and a backlog of requests, you're not alone. The good news: you can build a modern data platform without buying into a traditional warehouse-first architecture.

This post walks through a practical approach: keep data in open formats on cheap storage, query it with flexible engines, and add the pieces that usually get conflated with "the warehouse" (governance, modeling, metrics, and sharing). You'll still deliver dashboards, self-serve analytics, ML features, and operational data products-just without putting everything behind one monolithic warehouse.

1) Start with the core idea: separate storage, compute, and meaning

A traditional warehouse often bundles three jobs into one system:

1) Storage (where data lives)
2) Compute (how you transform and query it)
3) Meaning (what a "customer," "revenue," or "active user" means)

A warehouse-less platform explicitly separates these.

Storage: Put raw and curated data in object storage (S3, GCS, Azure Data Lake). This is cheap, durable, and vendor-neutral.

Open table format: Use Apache Iceberg, Delta Lake, or Apache Hudi to manage tables on object storage. These formats give you the "database-like" features people assume require a warehouse: schema evolution, partitioning, ACID-ish behavior, snapshots/time travel, and efficient reads.

Compute: Query and transform using engines that can read those tables-Trino/Starburst, Athena/BigQuery external tables, Spark, Flink, DuckDB (locally), or Dask. You can swap engines based on workload and cost.

Meaning (semantic + metrics): Use a semantic layer (dbt Semantic Layer, Cube, Looker semantic model, MetricFlow, or even well-governed dbt models + shared metric definitions) so every team doesn't reinvent "net revenue."

Practical example:

You store `events` and `orders` as Iceberg tables in S3.
Analysts query with Trino for ad hoc analysis.
ETL/ELT transformations run in Spark on a schedule.
Your BI tool points to a curated "gold" layer plus a metric definition for "Weekly Active Users."

No warehouse required-just a well-structured platform.

2) Design the data layers: raw, refined, and product-ready

Without a warehouse, the "discipline" comes from how you organize data.

A simple, battle-tested layout is:

Bronze (raw): Immutable-ish landing zone. Keep source-aligned data with minimal changes.
Silver (refined): Cleaned, standardized, deduplicated, conformed keys, basic business logic.
Gold (product-ready): Data products: marts, metrics-ready tables, wide tables for BI, feature tables for ML.

You can implement this with folders and naming conventions (and ideally a catalog):

`s3://data/bronze/stripe/charges/...`
`s3://data/silver/payments/transactions_iceberg/...`
`s3://data/gold/finance/revenue_daily_iceberg/...`

Ingestion patterns that work well

1) Batch CDC for databases: Use Debezium, Airbyte, Fivetran, or DMS-like tools to land change data into Bronze. Save both the raw logs and a table representation.

2) Event streaming for product analytics: Push events to Kafka/Kinesis/PubSub, then write to Iceberg/Delta using Flink/Spark structured streaming.

3) SaaS sources: Ingest via API connectors (Airbyte, custom) into raw tables. Preserve the original payload for auditability.

A concrete mini-architecture

Sources: Postgres (orders), Salesforce (accounts), Segment events (product usage)
Landing: Object storage
Table format: Iceberg
Transform: dbt (running on Spark/Trino) for Silver/Gold
Query: Trino for interactive queries; Spark for heavy transformations
BI: Mode/Hex/Metabase/Looker pointing at Gold + semantic layer

Guardrails to avoid a messy "data swamp"

Enforce schemas early: Even in Bronze, define explicit columns where possible (and store raw JSON alongside if needed).
Partition intentionally: Example: partition events by `event_date`, and consider bucketing/clustering by `user_id` if supported.
Small file management: Streaming can create tons of small files. Plan for compaction jobs (Iceberg/Delta optimize).
Data contracts: Agree on key fields and meanings with producing teams (e.g., `user_id`, `account_id`, timestamps in UTC).

3) Replace "warehouse features" with the right platform services

When someone says "we need a warehouse," they often mean "we need reliability and governance." Here's how to get those without a monolith.

Metadata, discovery, and lineage

Add a data catalog so people can find and trust datasets:

DataHub, Amundsen, OpenMetadata, AWS Glue Catalog, Unity Catalog (if using Databricks)

Minimum viable catalog setup:

Register all Silver/Gold tables
Tag owners and domains (e.g., Finance, Growth)
Add freshness and quality indicators (last updated, row counts)
Track lineage from Bronze â†’ Silver â†’ Gold (dbt helps a lot here)

Data quality and observability

Without centralized constraints, you need automated checks:

Great Expectations, Soda, dbt tests, Deequ

Example checks that pay off immediately:

`orders.order_id` is unique
`orders.total_amount >= 0`
`events.event_time` not null and within expected range
Freshness: `revenue_daily` updated by 7am UTC

Add observability (Monte Carlo, Databand, or open-source + alerts) to catch:

Schema drift from sources
Sudden volume drops/spikes
Downstream breakage after a change

Identity, access, and governance

Object storage is permissive by default; governance is not.

Use IAM roles/groups and table-level permissions via your catalog or a policy engine (Lake Formation, Ranger, Unity Catalog).
Establish a clear pattern: Bronze is restricted; Silver/Gold has broader access.
Tokenize or hash PII where possible; maintain a controlled "PII zone."

The semantic layer: stop metric sprawl

This is the most overlooked piece.

If every dashboard defines "active user" differently, you'll lose trust fast. A semantic layer gives you a single definition for metrics and dimensions.

Practical example:

Define `active_user` as "a user with at least one `app_open` event in the last 7 days"
Define `net_revenue` as "gross revenue - refunds - chargebacks"

Then BI tools and notebooks pull from the same definitions. Even if your underlying compute engine changes, your metric meaning stays consistent.

4) Deliver outcomes: analytics, ML, and operational data products

A warehouse-less platform is only worth it if it ships real value. Here are three common outcomes and how to implement them.

Outcome A: Fast, self-serve analytics

Pattern:

Analysts query Gold tables with Trino/Athena
BI dashboards read pre-modeled tables
Ad hoc exploration uses notebooks (SQL + Python)

Tip: Create "BI-friendly" wide tables where it's helpful. Not everything needs to be perfectly normalized.

Example: a `customer_360_daily` Gold table that joins accounts, subscriptions, latest activity, and support stats.

Outcome B: ML features without copying everything into a warehouse

Pattern:

Build feature tables in Gold (e.g., `features_user_churn_daily`)
Register in a feature store (Feast, Tecton, SageMaker Feature Store)
Serve online features to your app, and use the same offline tables for training

Example features:

`sessions_7d`
`payments_failed_30d`
`days_since_last_login`

Because the data is in open tables, you can train with Spark, Ray, or even pandas depending on scale.

Outcome C: "Reverse ETL" / operational analytics

Getting data back into tools is often more valuable than another dashboard.

Pattern:

Curate a Gold audience table, e.g., `audience_high_intent_users`
Sync to Salesforce/HubSpot/Braze using Hightouch/Census or custom jobs

Example:

Create a daily list of users who viewed pricing twice and started checkout but didn't complete.
Push that list (with last activity timestamp and plan interest) to your CRM for targeted outreach.

A simple rollout plan (so this doesn't become a science project)

1) Week 1-2: Stand up storage + table format + one query engine (Trino or Athena) + minimal catalog
2) Week 3-4: Ingest 2-3 core sources, build Bronze/Silver/Gold for one domain (e.g., Revenue)
3) Month 2: Add quality tests + alerting + semantic definitions for top 5 metrics
4) Month 3: Add streaming for events (if needed) and one operational data product (reverse ETL)

The goal isn't perfection-it's a platform that can expand safely.

Building a data platform without a warehouse is less about avoiding warehouses and more about choosing open storage, pluggable compute, and shared meaning. If you get those three right-and add the unsexy but essential layers like governance, data quality, and a semantic model-you'll end up with a system that's cheaper, more flexible, and often easier to evolve than a warehouse-centered setup.

If you tell me your stack (cloud, data volume, key use cases), I can suggest a concrete reference architecture and tool choices that fit your constraints.

Search This Blog

tylers-blogger-blog