For most of the last decade, every large company ran two parallel data stacks: a data lake for cheap, flexible storage of raw files, and a data warehouse for the fast, reliable, transactionally consistent queries that finance, product, and analytics teams actually depend on. Keeping both in sync meant duplicating data, running fragile ETL pipelines, and accepting that half your data was always slightly stale. The data lakehouse was supposed to fix that by putting warehouse-grade guarantees directly on top of lake storage. The idea took years to prove out, and for a long stretch it wasn't clear which underlying technology would make it work. That question has largely been settled, and the answer is Apache Iceberg.
What a data lakehouse actually is
A data lakehouse is an architecture, not a product. It combines three layers:
- Cheap object storage — S3, Azure Data Lake Storage, Google Cloud Storage, or on-prem equivalents, holding data as files (usually Parquet) rather than in a proprietary database format.
- A table format — a metadata layer that sits on top of those files and gives them the properties of a database table: schema, partitioning, versioning, and transactional guarantees like atomic commits and snapshot isolation.
- Compute engines — Spark, Trino, Flink, DuckDB, Snowflake, Databricks, BigQuery, and others, which read and write through the table format rather than talking to raw files directly.
The table format is the layer that makes the whole thing work, and it's also the layer that was, until recently, contested. Before open table formats existed, a "lake" was just a folder of Parquet files with no shared understanding of which files belonged to which table version, no way to add or drop columns without rewriting everything, and no protection against two jobs writing to the same table at once. You could query it, but you couldn't trust it the way you trust a warehouse table.
Table formats solve this by writing metadata — manifests, snapshots, schema history — alongside the data files. A query engine reads the metadata first to figure out exactly which files constitute the current (or a historical) version of the table, then reads only those files. This is what gives lakehouses time travel, schema evolution, hidden partitioning, and ACID transactions without needing a centralized database engine to own the storage.
Why this mattered enough to fight over
Three open table formats emerged to do this job: Apache Iceberg (originated at Netflix), Delta Lake (originated at Databricks), and Apache Hudi (originated at Uber). All three solve roughly the same problem — versioned, transactional metadata over Parquet files — but with different internal designs, different governance models, and, for a few years, different levels of vendor backing. Databricks pushed Delta Lake as the format tied to its platform. Snowflake, AWS, Google, and much of the rest of the ecosystem coalesced around Iceberg. Hudi carved out a niche in streaming-heavy, incremental-upsert use cases, particularly at companies with Uber-like ingestion patterns.
For data teams, this wasn't an academic dispute. Picking a table format meant picking which engines could read your data natively, which vendors would prioritize your workload, and how much you'd need to convert or duplicate data to move between systems. A multi-engine shop — one query layer in Snowflake, another in Databricks, streaming in Flink — could easily end up maintaining the same dataset in two formats just to keep everyone fed.
Why Iceberg pulled ahead
Iceberg's advantage was never really about raw performance — all three formats perform comparably for most workloads once tuned. It was about neutrality and catalog interoperability.
- Vendor-neutral governance. Iceberg is an Apache Software Foundation project with contributors across Snowflake, AWS, Netflix, Apple, Tabular (acquired by Databricks), Google, and others. No single vendor controls its roadmap, which made it the safer default for companies that didn't want to bet their data layer on one cloud provider's incentives.
- Catalog standardization. The Iceberg REST catalog spec gave every engine — Spark, Trino, Snowflake, Flink, DuckDB — a common way to discover and commit table metadata, regardless of who wrote the data. This mattered more than the file format itself: it meant a table written by one engine could be read and updated by a completely different one without translation layers.
- Broad engine support arriving early. Snowflake, Databricks, AWS Athena and EMR, Google BigQuery, Trino, Flink, and Dremio all shipped native Iceberg support, in some cases years before matching that level of investment in the alternatives.
- Delta Lake's own convergence. Databricks open-sourced Delta Lake's core protocol and has increasingly supported Iceberg interoperability (via UniForm and similar bridging layers), acknowledging that customers wanted to read Iceberg tables without a format war getting in the way. That's a tell: even the company most invested in an alternative format decided compatibility mattered more than exclusivity.
None of this means Delta Lake or Hudi disappeared. Delta Lake remains deeply embedded in Databricks-centric shops, and Hudi still leads in certain high-throughput, upsert-heavy streaming pipelines. But when analysts, hiring managers, and platform teams talk about "the" open table format now, they usually mean Iceberg by default, and increasingly treat the others as either legacy or workload-specific choices rather than equally weighted options.
Why this consolidation matters right now
The clearest recent signal of where this settled is Iceberg's table format hitting version 3 and going generally available on major platforms — Snowflake announced GA support for Iceberg v3 tables at its Snowflake Summit 2026 event. That's notable for two reasons. First, Snowflake was historically a closed-format warehouse; its own proprietary tables were the whole product. Committing to first-class support for an open, community-governed table format inside its core engine is a concession that customers now expect data portability as a baseline requirement, not a nice-to-have. Second, v3 itself adds capabilities — including deletion vectors for more efficient row-level updates and deletes, and richer type support — that close much of the performance gap that used to be Delta Lake's main technical argument in upsert-heavy workloads.
Industry surveys of data engineering and platform teams over the past couple of years have consistently shown Iceberg pulling ahead as the default or preferred table format for new lakehouse builds, with adoption reported across finance, retail, ad tech, and other data-intensive verticals. The practical effect is that "which table format should we standardize on" is no longer the multi-quarter architectural debate it used to be for teams starting fresh. It's closer to a default choice, with format selection reserved for genuinely unusual workload shapes rather than general-purpose analytics.
That shift changes vendor behavior too. When a table format is contested, platform vendors compete partly by locking customers into their format. When a table format is settled, vendors have to compete on compute performance, cost, governance tooling, and developer experience instead — because the data itself is portable regardless of which engine you're running. That's a meaningfully different, and generally healthier, competitive dynamic for buyers.
Practical implications for teams building on this
For platform and data engineering teams
If you're standing up new data infrastructure, or evaluating whether to consolidate a lake-plus-warehouse setup into a single lakehouse, the calculus has changed:
| Decision point | Old default (pre-consolidation) | Current default |
|---|---|---|
| Table format for new builds | Pick based on primary compute vendor | Iceberg unless a specific engine mandates otherwise |
| Multi-engine access | Expect conversion jobs or duplicate copies | Expect native reads via REST catalog |
| Vendor lock-in risk | High — format tied to platform | Lower — format is portable, compute is the lock-in surface |
| Streaming/high-upsert workloads | Hudi or Delta, evaluated case by case | Still worth evaluating Hudi/Delta; Iceberg v3 narrows the gap |
| Governance/catalog | Proprietary catalogs per vendor | Iceberg REST catalog spec as common interface |
The practical upshot: a company running analytics in Snowflake, ML feature pipelines in Spark, and ad hoc queries in Trino no longer needs three copies of the same data or a bespoke sync pipeline between them. One set of Iceberg tables in object storage, one shared catalog, and each engine reads and writes against it directly. That reduces storage cost, reduces the staleness window between systems, and removes an entire category of "which copy is correct" bugs.
For teams choosing a build vs. buy path
Adopting a lakehouse architecture doesn't remove complexity — it relocates it. Where a managed warehouse used to hide storage layout, compaction, and file management behind a proprietary engine, a lakehouse exposes more of that machinery, even with a mature table format doing the heavy lifting. Teams taking this on should expect to own, or pay a vendor to own:
- Catalog operations — running or subscribing to a REST catalog service (Snowflake's Horizon, AWS Glue, Databricks Unity Catalog, Tabular/Polaris, or a self-hosted option) that all engines point to.
- Compaction and maintenance jobs — small-file compaction, snapshot expiration, and orphan file cleanup, which Iceberg needs periodically to keep query planning fast and storage costs sane.
- Access control mapped across engines — row- and column-level permissions that need to mean the same thing whether the query comes from Spark or Snowflake.
- Schema governance — Iceberg makes schema evolution safe at the format level, but someone still has to decide what "safe" changes are allowed organizationally.
None of this is exotic engineering, but it is real operational surface area that a fully closed warehouse would have handled for you. The tradeoff is flexibility and cost control in exchange for owning more of the plumbing — a tradeoff that's become easier to justify now that the format underneath it is stable and widely supported rather than a moving target.
A practical migration path
Teams moving an existing Hive-style lake, or Delta/Hudi tables, onto Iceberg generally follow a similar sequence rather than a big-bang cutover:
- Inventory and prioritize tables by query volume and downstream dependency count — migrate the highest-value, lowest-risk tables first to validate the pipeline before touching anything business-critical.
- Stand up a catalog (Glue, Unity Catalog, Polaris, or a vendor-managed REST catalog) that every consuming engine can authenticate against.
- Convert or register tables using the source format's built-in conversion tooling where available — both Delta Lake and Hudi ship utilities that generate Iceberg metadata without rewriting the underlying Parquet files, which keeps the initial conversion fast.
- Validate query results in parallel by running critical dashboards and jobs against both the old and new tables for a defined period before cutting reads over.
- Cut over writers last, once readers are confirmed stable, so you're never in a state where two systems believe they own the authoritative copy of a table.
The files themselves rarely need to move in this process — only the metadata layer changes — which is a large part of why conversions are cheaper than they sound at first.
Limitations and open questions
The table format question being settled doesn't mean the lakehouse model is a solved problem. A few real limitations remain:
- Small-file and streaming ingestion overhead. Iceberg's metadata model works best with batched writes. High-frequency, low-latency streaming writes (think per-second event ingestion) can generate large numbers of small files and metadata commits, which still requires careful compaction strategy and, for some workloads, keeps Hudi or a purpose-built streaming layer in the running.
- Catalog fragmentation, one level up. The table format is standardized, but catalogs — the service that tracks which tables exist and their current metadata pointer — are not fully unified yet. REST catalog support is converging, but organizations still end up choosing between Glue, Unity Catalog, Polaris, Snowflake Horizon, and others, each with different governance and interoperability characteristics. That's a smaller fight than the table format war, but it's not over.
- Query performance still varies by engine, not just format. Adopting Iceberg doesn't equalize performance across Snowflake, Databricks, and Trino — each engine's query planner, caching, and optimization work differently on the same underlying tables. Format neutrality removes one variable, not all of them.
- Migration cost is nontrivial. Converting existing Delta Lake or Hudi tables, or legacy Hive-style tables, to Iceberg is generally supported via conversion tooling, but at scale it still means rewriting or re-registering large volumes of metadata, validating downstream jobs, and coordinating cutover windows. "The format won" doesn't mean migration is free.
- Governance maturity varies by organization, not by format. A lakehouse doesn't automatically improve data quality, lineage, or documentation. Those remain organizational disciplines that the underlying storage format doesn't solve on its own.
What to watch next
A few developments will shape how this settles further over the next year or two:
- Catalog interoperability maturing. Watch whether REST catalog implementations converge on shared behavior the way the table format itself did, or whether catalog choice becomes the next axis of lock-in.
- Iceberg v3 feature adoption in practice. Deletion vectors and expanded type support are GA on paper; the real test is how quickly engines other than the ones that shipped them first pick up full support, and how upsert-heavy workloads perform against Hudi and Delta in production once that's widespread.
- Consolidation of the "second tier" formats. Whether Hudi and Delta Lake continue serving distinct niches (streaming-heavy upserts, Databricks-native shops) or gradually get absorbed into Iceberg-compatible tooling via bridging layers.
- Lakehouse-native governance and security tooling. As more sensitive workloads move onto lakehouse architectures, expect more investment in fine-grained access control, data masking, and audit tooling that works consistently across engines reading the same Iceberg tables.
- Cost transparency. As compute becomes the main differentiator between vendors (rather than data format lock-in), expect more public benchmarking and price competition among engines reading the same underlying Iceberg tables — a dynamic that's better for buyers than the previous format-locked landscape.
FAQ
What's the difference between a data lake, a data warehouse, and a data lakehouse?
A data lake stores raw files cheaply but historically lacked transactional guarantees; a data warehouse offers fast, reliable queries but at higher cost and with less flexible storage. A lakehouse combines cheap object storage with a table format layer (like Iceberg) that adds warehouse-grade transactions, schema management, and time travel directly on top of the lake, removing the need to maintain both separately.
Why did Apache Iceberg become the dominant table format?
Iceberg's vendor-neutral governance under the Apache Software Foundation, combined with a standardized REST catalog spec that let many different query engines read and write the same tables natively, made it the safer, more portable default. Broad early adoption by Snowflake, AWS, Google, and Trino reinforced that momentum, and even Databricks — the company behind the main alternative, Delta Lake — has built interoperability bridges toward Iceberg.
Is Delta Lake or Apache Hudi still worth using?
Yes, in specific contexts. Delta Lake remains well-suited to Databricks-native environments, and Hudi still has real strengths in high-throughput, upsert-heavy streaming pipelines. Iceberg has become the default for general-purpose, multi-engine lakehouse builds, but format choice is still workload-dependent rather than a single universal answer.
Do I need a data lakehouse if I already use a cloud data warehouse?
Not necessarily. If your data volumes and multi-engine requirements are modest, a managed warehouse alone may still be simpler to operate. Lakehouse architecture pays off most when you need multiple compute engines reading the same data, want to avoid storage duplication, or need to control storage costs independently of compute.
What is a table format, exactly?
A table format is a metadata layer — manifests, snapshots, and schema history — stored alongside data files in object storage. It tells query engines exactly which files make up a table's current (or historical) version, enabling transactions, schema evolution, and time travel without requiring a single database engine to own the storage.
Does adopting Iceberg mean I no longer need a data engineering team?
No. Iceberg standardizes the table format, but someone still needs to run catalog services, manage compaction and file maintenance, set access controls consistently across engines, and govern schema changes. The format being standardized reduces integration work; it doesn't eliminate operational ownership.
What does Iceberg v3 add over earlier versions?
Iceberg v3 introduces deletion vectors for more efficient row-level updates and deletes, along with expanded type support, narrowing the performance gap that previously favored formats like Delta Lake or Hudi for upsert-heavy workloads. Major platforms, including Snowflake, have moved to GA support for v3 tables.
Teams evaluating or migrating to a lakehouse architecture can get hands-on help with catalog setup, format migration, and multi-engine access design from Woyce Technologies.