Data Lakehouse Architecture: How Apache Iceberg Won the Format War

For most of the last decade, every large company ran two parallel data stacks: a data lake for cheap, flexible storage of raw files, and a data warehouse for the fast, reliable, transactionally consistent queries that finance, product, and analytics teams actually depend on. Keeping both in sync meant duplicating data, running fragile ETL pipelines, and accepting that half your data was always slightly stale. The data lakehouse was supposed to fix that by putting warehouse-grade guarantees directly on top of lake storage. The idea took years to prove out, and for a long stretch it wasn't clear which underlying technology would make it work. That question has largely been settled, and the answer is Apache Iceberg.

If you run a data platform today, that settlement changes real decisions: which format new tables should use, whether to migrate existing Delta Lake or Hudi tables, which catalog to standardize on, and how much leverage you have when negotiating with warehouse and compute vendors. Getting those calls wrong means another round of duplicated storage and migration work in a few years.

This article explains what a data lakehouse actually is, why Iceberg pulled ahead of the alternatives, why the consolidation matters now, how teams can build on or migrate to it, and the limitations (streaming overhead, catalog fragmentation, uneven engine performance) that remain open.

What a data lakehouse actually is

A data lakehouse is an architecture, not a product. It combines three layers:

1. Cheap object storage — S3, Azure Data Lake Storage, Google Cloud Storage, or on-prem equivalents, holding data as files (usually Parquet) rather than in a proprietary database format.
2. A table format — a metadata layer that sits on top of those files and gives them the properties of a database table: schema, partitioning, versioning, and transactional guarantees like atomic commits and snapshot isolation.
3. Compute engines — Spark, Trino, Flink, DuckDB, Snowflake, Databricks, BigQuery, and others, which read and write through the table format rather than talking to raw files directly.

The table format is the layer that makes the whole thing work, and it's also the layer that was, until recently, contested. Before open table formats existed, a "lake" was just a folder of Parquet files with no shared understanding of which files belonged to which table version, no way to add or drop columns without rewriting everything, and no protection against two jobs writing to the same table at once. You could query it, but you couldn't trust it the way you trust a warehouse table.

Table formats solve this by writing metadata — manifests, snapshots, schema history — alongside the data files. A query engine reads the metadata first to figure out exactly which files constitute the current (or a historical) version of the table, then reads only those files. This is what gives lakehouses time travel, schema evolution, hidden partitioning, and ACID transactions without needing a centralized database engine to own the storage.

Why this mattered enough to fight over

Three open table formats emerged to do this job: Apache Iceberg (originated at Netflix), Delta Lake (originated at Databricks), and Apache Hudi (originated at Uber). All three solve roughly the same problem — versioned, transactional metadata over Parquet files — but with different internal designs, different governance models, and, for a few years, different levels of vendor backing. Databricks pushed Delta Lake as the format tied to its platform. Snowflake, AWS, Google, and much of the rest of the ecosystem coalesced around Iceberg. Hudi carved out a niche in streaming-heavy, incremental-upsert use cases, particularly at companies with Uber-like ingestion patterns.

For data teams, this wasn't an academic dispute. Picking a table format meant picking which engines could read your data natively, which vendors would prioritize your workload, and how much you'd need to convert or duplicate data to move between systems. A multi-engine shop — one query layer in Snowflake, another in Databricks, streaming in Flink — could easily end up maintaining the same dataset in two formats just to keep everyone fed.

Why Iceberg pulled ahead

Iceberg's advantage was never really about raw performance — all three formats perform comparably for most workloads once tuned. It was about neutrality and catalog interoperability.

• Vendor-neutral governance. Iceberg is an Apache Software Foundation project with contributors across Snowflake, AWS, Netflix, Apple, Tabular (acquired by Databricks), Google, and others. No single vendor controls its roadmap, which made it the safer default for companies that didn't want to bet their data layer on one cloud provider's incentives.
• Catalog standardization. The Iceberg REST catalog spec gave every engine — Spark, Trino, Snowflake, Flink, DuckDB — a common way to discover and commit table metadata, regardless of who wrote the data. This mattered more than the file format itself: it meant a table written by one engine could be read and updated by a completely different one without translation layers.
• Broad engine support arriving early. Snowflake, Databricks, AWS Athena and EMR, Google BigQuery, Trino, Flink, and Dremio all shipped native Iceberg support, in some cases years before matching that level of investment in the alternatives.
• Delta Lake's own convergence. Databricks open-sourced Delta Lake's core protocol and has increasingly supported Iceberg interoperability (via UniForm and similar bridging layers), acknowledging that customers wanted to read Iceberg tables without a format war getting in the way. That's a tell: even the company most invested in an alternative format decided compatibility mattered more than exclusivity.

None of this means Delta Lake or Hudi disappeared. Delta Lake remains deeply embedded in Databricks-centric shops, and Hudi still leads in certain high-throughput, upsert-heavy streaming pipelines. But when analysts, hiring managers, and platform teams talk about "the" open table format now, they usually mean Iceberg by default, and increasingly treat the others as either legacy or workload-specific choices rather than equally weighted options.

Why this consolidation matters right now

The clearest recent signal of where this settled is Iceberg's table format hitting version 3 and going generally available on major platforms — Snowflake announced GA support for Iceberg v3 tables at its Snowflake Summit 2026 event. That's notable for two reasons. First, Snowflake was historically a closed-format warehouse; its own proprietary tables were the whole product. Committing to first-class support for an open, community-governed table format inside its core engine is a concession that customers now expect data portability as a baseline requirement, not a nice-to-have. Second, v3 itself adds capabilities — including deletion vectors for more efficient row-level updates and deletes, and richer type support — that close much of the performance gap that used to be Delta Lake's main technical argument in upsert-heavy workloads.

Industry surveys of data engineering and platform teams over the past couple of years have consistently shown Iceberg pulling ahead as the default or preferred table format for new lakehouse builds, with adoption reported across finance, retail, ad tech, and other data-intensive verticals. The practical effect is that "which table format should we standardize on" is no longer the multi-quarter architectural debate it used to be for teams starting fresh. It's closer to a default choice, with format selection reserved for genuinely unusual workload shapes rather than general-purpose analytics.

That shift changes vendor behavior too. When a table format is contested, platform vendors compete partly by locking customers into their format. When a table format is settled, vendors have to compete on compute performance, cost, governance tooling, and developer experience instead — because the data itself is portable regardless of which engine you're running. That's a meaningfully different, and generally healthier, competitive dynamic for buyers.

Benefits of a data lakehouse architecture

With the table format question settled, the benefits of the lakehouse model are easier to realise than they were a few years ago. Most of them come from having one authoritative copy of the data that many engines can use.

One copy of the data for every engine

Analytics in a warehouse engine, feature pipelines in Spark, and ad hoc queries in Trino can all read and write the same Iceberg tables through a shared catalog. There is no need to maintain separate copies per engine or build sync jobs between them. That removes a whole category of "which copy is correct" bugs and the reconciliation work that comes with them.

Lower storage costs

Data sits once in object storage, which is far cheaper per terabyte than proprietary warehouse storage, and it isn't duplicated across systems. Storage and compute scale independently, so a team can keep years of history without paying for compute capacity it only needs occasionally. Cold data can sit in cheaper storage tiers while staying queryable.

Fresher data across teams

When the lake and the warehouse were separate, one of them was always behind, waiting for the next ETL run. With a single set of tables, every engine sees the same committed snapshot. Dashboards, models, and reports stop disagreeing because they were built on copies refreshed at different times, and the meetings spent reconciling numbers get shorter.

Warehouse-grade reliability on open files

Atomic commits, snapshot isolation, schema evolution, hidden partitioning, and time travel give lake storage the guarantees teams used to get only from a warehouse. Analysts can query a table as it looked at a past point, engineers can add columns without rewriting data, and concurrent jobs don't corrupt each other's writes.

Negotiating leverage with vendors

Because the format is open and portable, switching or adding a compute engine no longer means migrating the data. Vendors have to compete on performance, cost, and tooling. Buyers can run a new engine against existing tables to test it, which is a much stronger position than being locked into whichever platform owns the storage format.

Data lakehouse use cases

The lakehouse pattern is being adopted across finance, retail, ad tech, and other data-intensive sectors. These are the workloads where it most clearly pays off, and where teams usually start.

Multi-engine analytics on shared tables

A company runs business intelligence in Snowflake, data science in Spark, and exploratory queries in Trino. Before, each needed its own copy of key datasets. With Iceberg tables and a shared REST catalog, all three work against the same data directly. Storage costs drop, the staleness window between systems disappears, and teams can pick the engine that suits each job without negotiating a new data pipeline.

Consolidating a lake and a warehouse

Many organisations still run a raw-file lake alongside a warehouse, joined by fragile ETL. Moving curated tables onto Iceberg in object storage lets the warehouse engine query them in place, while the lake keeps its flexibility. The result is one platform to govern, fewer pipelines to maintain, and less duplicated storage, with the warehouse engine still available for the workloads it handles best.

Machine learning features built on the same data as reporting

Feature pipelines often drift from the numbers used in dashboards because they read different copies. Building both from the same Iceberg tables means models and reports agree on definitions. Time travel also lets data scientists reproduce the exact training snapshot used for a model, which helps with debugging and audits.

Point-in-time reporting and audit

Regulated teams need to show what the data looked like when a report was produced. Snapshot history lets them query a table as of a past commit without maintaining separate archived copies. Audits and restatements become queries rather than restoration projects, as long as snapshot retention is set long enough to cover the reporting period.

Mixed batch and incremental workloads

Teams with both nightly batch loads and frequent incremental updates can run them against the same tables, with Iceberg v3 deletion vectors improving row-level updates. Very high-frequency streaming still needs careful compaction, and some teams keep Hudi or a dedicated streaming layer for those pipelines while using Iceberg for everything downstream.

Practical implications for teams building on this

For platform and data engineering teams

If you're standing up new data infrastructure, or evaluating whether to consolidate a lake-plus-warehouse setup into a single lakehouse, the calculus has changed:

The practical upshot: a company running analytics in Snowflake, ML feature pipelines in Spark, and ad hoc queries in Trino no longer needs three copies of the same data or a bespoke sync pipeline between them. One set of Iceberg tables in object storage, one shared catalog, and each engine reads and writes against it directly. That reduces storage cost, reduces the staleness window between systems, and removes an entire category of "which copy is correct" bugs.

For teams choosing a build vs. buy path

Adopting a lakehouse architecture doesn't remove complexity — it relocates it. Where a managed warehouse used to hide storage layout, compaction, and file management behind a proprietary engine, a lakehouse exposes more of that machinery, even with a mature table format doing the heavy lifting. Teams taking this on should expect to own, or pay a vendor to own:

• Catalog operations — running or subscribing to a REST catalog service (Snowflake's Horizon, AWS Glue, Databricks Unity Catalog, Tabular/Polaris, or a self-hosted option) that all engines point to — an approach that echoes the federated governance model of a data mesh.
• Compaction and maintenance jobs — small-file compaction, snapshot expiration, and orphan file cleanup, which Iceberg needs periodically to keep query planning fast and storage costs sane.
• Access control mapped across engines — row- and column-level permissions that need to mean the same thing whether the query comes from Spark or Snowflake.
• Schema governance — Iceberg makes schema evolution safe at the format level, but someone still has to decide, often through a semantic layer or governance policy, what "safe" changes are allowed organizationally.

None of this is exotic engineering, but it is real operational surface area that a fully closed warehouse would have handled for you. The tradeoff is flexibility and cost control in exchange for owning more of the plumbing — a tradeoff that's become easier to justify now that the format underneath it is stable and widely supported rather than a moving target. Teams without in-house platform expertise often bring in outside tech consulting support to scope the catalog and compaction work realistically before committing to a build-it-yourself path.

A practical migration path

Teams moving an existing Hive-style lake, or Delta/Hudi tables, onto Iceberg generally follow a similar sequence rather than a big-bang cutover:

1. Inventory and prioritize tables by query volume and downstream dependency count — migrate the highest-value, lowest-risk tables first to validate the pipeline before touching anything business-critical.
2. Stand up a catalog (Glue, Unity Catalog, Polaris, or a vendor-managed REST catalog) that every consuming engine can authenticate against.
3. Convert or register tables using the source format's built-in conversion tooling where available — both Delta Lake and Hudi ship utilities that generate Iceberg metadata without rewriting the underlying Parquet files, which keeps the initial conversion fast.
4. Validate query results in parallel by running critical dashboards and jobs against both the old and new tables for a defined period before cutting reads over.
5. Cut over writers last, once readers are confirmed stable, so you're never in a state where two systems believe they own the authoritative copy of a table.

The files themselves rarely need to move in this process — only the metadata layer changes — which is a large part of why conversions are cheaper than they sound at first.

Common data lakehouse mistakes

Teams adopting a lakehouse tend to stumble in the same places. Most of the problems are operational rather than architectural, which is good news: they can be fixed with process rather than a redesign.

Attempting a big-bang migration

Converting every table and cutting over every reader and writer at once concentrates risk into a single weekend. When something breaks, it is hard to tell which table or job caused it. Migrating in waves, starting with high-value, low-risk tables and validating in parallel, keeps problems small and reversible, and builds confidence before business-critical tables move.

Skipping compaction and maintenance

Iceberg needs periodic small-file compaction, snapshot expiration, and orphan file cleanup. Without them, query planning slows down, storage costs creep up, and metadata grows unwieldy. Teams coming from a managed warehouse often don't realise this work used to happen behind the scenes until performance degrades.

Choosing a catalog casually

The table format is standardised, but the catalog is where lock-in can return. Picking whichever catalog came bundled with the first engine, without checking how other engines authenticate against it and how governance works across them, can recreate the vendor dependence the lakehouse was meant to remove.

Assuming the format equalises performance

Iceberg makes data portable, but each engine's planner, caching, and optimisation behave differently on the same tables. Teams that expect identical performance everywhere are surprised when one engine is much faster for a given workload. Benchmarking engines against your own tables and queries avoids building on that assumption.

Expecting the lakehouse to fix data quality

A new storage architecture doesn't improve lineage, documentation, or the accuracy of the data going into it. Organisations that migrate expecting cleaner data end up with the same problems on a more flexible platform. Quality and governance remain organisational work, and they deserve their own owners and budget alongside the migration.

Data lakehouse best practices

These practices reflect what tends to keep lakehouse platforms fast, governed, and portable after the initial build. Most of them are about owning the operational work that a closed warehouse used to hide.

• Default new tables to Iceberg. Use it for general-purpose analytics unless a specific engine or workload, such as very high-frequency upserts, gives you a concrete reason to choose otherwise.
• Pick one catalog every engine can use. Standardise on a REST-compatible catalog that all consuming engines can authenticate against, and document why you chose it so the decision can be revisited if catalog options change.
• Automate table maintenance. Schedule compaction, snapshot expiration, and orphan file cleanup per table based on write patterns, and monitor file counts and planning times so problems show up in monitoring before users notice slow queries.
• Define access control once and map it across engines. Make sure row- and column-level permissions mean the same thing whether a query comes from Spark, Trino, or a warehouse engine, and test it.
• Set a schema change policy. Iceberg makes evolution technically safe; decide organisationally which changes are allowed, who approves them, and how downstream consumers are notified before a change lands.
• Migrate incrementally with parallel validation. Convert metadata with built-in tooling, run critical jobs against old and new tables side by side, cut over readers first and writers last, so two systems never both believe they own a table.
• Benchmark engines on your own tables. Compare cost and performance using real queries before routing workloads, and re-test when engines add support for newer Iceberg features.
• Track feature support per engine. Check that every engine you rely on fully supports features such as v3 deletion vectors before depending on them in production. A feature that one engine writes but another can't read will break multi-engine access.

Limitations and open questions

The table format question being settled doesn't mean the lakehouse model is a solved problem. A few real limitations remain:

• Small-file and streaming ingestion overhead. Iceberg's metadata model works best with batched writes. High-frequency, low-latency streaming writes (think per-second event ingestion) can generate large numbers of small files and metadata commits, which still requires careful compaction strategy and, for some workloads, keeps Hudi or a purpose-built streaming layer in the running.
• Catalog fragmentation, one level up. The table format is standardized, but catalogs — the service that tracks which tables exist and their current metadata pointer — are not fully unified yet. REST catalog support is converging, but organizations still end up choosing between Glue, Unity Catalog, Polaris, Snowflake Horizon, and others, each with different governance and interoperability characteristics. That's a smaller fight than the table format war, but it's not over.
• Query performance still varies by engine, not just format. Adopting Iceberg doesn't equalize performance across Snowflake, Databricks, and Trino — each engine's query planner, caching, and optimization work differently on the same underlying tables. Format neutrality removes one variable, not all of them.
• Migration cost is nontrivial. Converting existing Delta Lake or Hudi tables, or legacy Hive-style tables, to Iceberg is generally supported via conversion tooling, but at scale it still means rewriting or re-registering large volumes of metadata, validating downstream jobs, and coordinating cutover windows. "The format won" doesn't mean migration is free.
• Governance maturity varies by organization, not by format. A lakehouse doesn't automatically improve data quality, lineage, or documentation. Those remain organizational disciplines that the underlying storage format doesn't solve on its own.

What to watch next

A few developments will shape how this settles further over the next year or two:

1. Catalog interoperability maturing. Watch whether REST catalog implementations converge on shared behavior the way the table format itself did, or whether catalog choice becomes the next axis of lock-in.
2. Iceberg v3 feature adoption in practice. Deletion vectors and expanded type support are GA on paper; the real test is how quickly engines other than the ones that shipped them first pick up full support, and how upsert-heavy workloads perform against Hudi and Delta in production once that's widespread.
3. Consolidation of the "second tier" formats. Whether Hudi and Delta Lake continue serving distinct niches (streaming-heavy upserts, Databricks-native shops) or gradually get absorbed into Iceberg-compatible tooling via bridging layers.
4. Lakehouse-native governance and security tooling. As more sensitive workloads move onto lakehouse architectures, expect more investment in fine-grained access control, data masking, and audit tooling that works consistently across engines reading the same Iceberg tables.
5. Cost transparency. As compute becomes the main differentiator between vendors (rather than data format lock-in), expect more public benchmarking and price competition among engines reading the same underlying Iceberg tables — a dynamic that's better for buyers than the previous format-locked landscape.

Teams evaluating or migrating to a lakehouse architecture can get hands-on help with catalog setup, format migration, and multi-engine access design from Woyce Technologies.

FAQ

What's the difference between a data lake, a data warehouse, and a data lakehouse?

A data lake stores raw files cheaply but historically lacked transactional guarantees; a data warehouse offers fast, reliable queries but at higher cost and with less flexible storage. A lakehouse combines cheap object storage with a table format layer (like Iceberg) that adds warehouse-grade transactions, schema management, and time travel directly on top of the lake, removing the need to maintain both separately.

Why did Apache Iceberg become the dominant table format?

Iceberg's vendor-neutral governance under the Apache Software Foundation, combined with a standardized REST catalog spec that let many different query engines read and write the same tables natively, made it the safer, more portable default. Broad early adoption by Snowflake, AWS, Google, and Trino reinforced that momentum, and even Databricks — the company behind the main alternative, Delta Lake — has built interoperability bridges toward Iceberg.

Is Delta Lake or Apache Hudi still worth using?

Yes, in specific contexts. Delta Lake remains well-suited to Databricks-native environments, and Hudi still has real strengths in high-throughput, upsert-heavy streaming pipelines. Iceberg has become the default for general-purpose, multi-engine lakehouse builds, but format choice is still workload-dependent rather than a single universal answer. If you already run one of them successfully, interoperability bridges often make a full migration unnecessary in the short term.

Do I need a data lakehouse if I already use a cloud data warehouse?

Not necessarily. If your data volumes and multi-engine requirements are modest, a managed warehouse alone may still be simpler to operate. Lakehouse architecture pays off most when you need multiple compute engines reading the same data, want to avoid storage duplication, or need to control storage costs independently of compute.

What is a table format, exactly?

A table format is a metadata layer — manifests, snapshots, and schema history — stored alongside data files in object storage. It tells query engines exactly which files make up a table's current (or historical) version, enabling transactions, schema evolution, and time travel without requiring a single database engine to own the storage.

Does adopting Iceberg mean I no longer need a data engineering team?

No. Iceberg standardizes the table format, but someone still needs to run catalog services, manage compaction and file maintenance, set access controls consistently across engines, and govern schema changes. The format being standardized reduces integration work; it doesn't eliminate operational ownership. If anything, multi-engine access raises the bar for consistent governance, because more tools can now read and write the same tables.

What does Iceberg v3 add over earlier versions?

Iceberg v3 introduces deletion vectors for more efficient row-level updates and deletes, along with expanded type support, narrowing the performance gap that previously favored formats like Delta Lake or Hudi for upsert-heavy workloads. Major platforms, including Snowflake, have moved to GA support for v3 tables. The real test is how quickly other engines pick up full support, so check your engines before relying on v3 features.

Conclusion

The data lakehouse set out to end the two-stack problem: cheap but loosely governed lakes on one side, reliable but expensive warehouses on the other, with fragile pipelines keeping them roughly in sync. Open table formats made that possible by adding transactions, schema evolution, and time travel on top of files in object storage, and Apache Iceberg has become the default because of its vendor-neutral governance, REST catalog spec, and broad engine support.

That settlement is good news for data teams. Storage can be shared across engines, compute vendors compete on price and performance rather than format lock-in, and moving from Delta Lake or Hudi is mostly a metadata exercise rather than a full data rewrite. The caveats still matter: streaming-heavy ingestion needs careful compaction, the catalog layer is the next likely point of lock-in, query performance varies by engine, and no table format fixes weak data quality or governance.

A sensible next step is to inventory your existing tables by format and catalog, pick a catalog every engine can authenticate against, and pilot Iceberg on one or two high-value tables with parallel validation before converting more. If you'd like help designing the catalog, migration, or multi-engine access layer, our database engineering team can work through it with you.