How Iceberg Catalogs Work — and Where Lakehouse Lock-In Now Lives

Ask a data engineer what Apache Iceberg gives them and you'll hear about open file formats, schema evolution, and time travel. Ask them which vendor they're locked into, and the honest answer is usually a catalog they never chose deliberately — it just came bundled with whichever query engine they adopted first. That's the quiet shift in the lakehouse world: the table format is open, but the thing that decides who can read your tables, how fast, and under whose governance rules is the catalog. And catalogs have not been nearly as open as the format they sit on top of.

This matters more than it used to. For years, "Iceberg" and "lakehouse openness" were treated as synonyms. That equivalence is breaking down, and understanding why requires understanding what a catalog actually does — a piece of the stack most people outside data platform teams have never had to think about.

What an Iceberg catalog actually is

Apache Iceberg's core trick is that a "table" is not a directory of files — it's a chain of immutable metadata pointing at data files. Every time you write to an Iceberg table, you're not overwriting anything; you're creating a new metadata file that describes a new snapshot of the table, and appending it to a metadata log. Readers ask a simple question: "what is the current metadata file for this table?" Whoever can answer that question authoritatively controls the table.

That's the catalog's job. Specifically, a catalog:

• Tracks the mapping from table names (db.schema.table) to the location of their current metadata file
• Provides atomic commit semantics — when two writers try to update a table at the same time, the catalog ensures only one wins and the table never ends up in a torn, half-written state
• Enforces table-level and often column-level access control
• Often stores or proxies additional metadata: table properties, schema history, statistics used for query planning

Without a catalog, Iceberg is just a very clever way to organize Parquet files and JSON pointers in object storage. With one, it becomes something query engines can trust to give consistent, concurrent, transactional access. The catalog is what turns a pile of files into a database.

The pointer chain, concretely

It helps to walk through the actual chain of indirection, because the catalog's narrow job becomes obvious once you see it:

1. A catalog entry for sales.orders stores one thing: the path to the table's current metadata.json file.
2. That metadata.json lists the table's schema history, partition spec, and a pointer to the current manifest list.
3. The manifest list enumerates manifest files, each of which lists actual data files (Parquet, ORC, or Avro) along with column-level statistics used to skip irrelevant files at query time.
4. Data files themselves never move or get rewritten for a simple update — new files are added and old ones are marked as no longer part of the current snapshot.

The catalog only owns step 1. Everything downstream is just files in object storage that any engine with read access can open. This is precisely why the catalog became the leverage point: it's the smallest, cheapest piece of the system to control, and controlling it controls everything built on top.

Why catalog implementations differ so much

If the catalog's job is "store a pointer and make commits atomic," you'd expect implementations to be interchangeable. They are not, because the atomicity guarantee has to be built on top of whatever storage or database substrate is available, and that substrate shapes everything about performance, concurrency limits, and operational cost.

The REST catalog spec, formalized within the Iceberg project itself, matters because it's the first catalog interface designed explicitly to be an interoperability layer rather than an implementation. Instead of every query engine needing native code for Hive Metastore, Glue, and half a dozen proprietary systems, engines just need to speak one HTTP protocol. Any catalog vendor can implement the REST spec as a facade in front of their own storage.

That sounds like it should have ended lock-in. It didn't, for reasons worth unpacking.

Why atomic commits are harder than they sound

The "atomic pointer swap" description undersells how much engineering goes into making catalog commits safe under real-world concurrency. Consider two writers racing to update the same table at the same time — one appending a batch of new rows, another running a compaction job that rewrites small files into larger ones. Both start from the same base metadata snapshot. Both compute a new metadata file. Only one can win.

A correct catalog implementation has to:

• Detect that both writers started from the same base snapshot
• Accept the first commit and atomically advance the "current" pointer
• Reject the second commit with a retriable conflict error rather than silently corrupting the table
• Give the losing writer enough information to re-read the new current state, reapply its change on top, and retry

This is the same optimistic-concurrency pattern used in distributed version control, and it's precisely why catalogs built on systems with native compare-and-swap or transactional guarantees (a relational database, a strongly consistent key-value store) tend to be more reliable under heavy concurrent write load than catalogs bolted onto storage systems that were never designed for this kind of coordination. It's also why "just point Iceberg at a folder in object storage with no catalog" is a valid pattern for single-writer, low-concurrency use cases, but breaks down fast the moment multiple pipelines write to the same table.

Catalogs vs. metastores: a naming note

Worth clarifying, because the terminology gets muddled in practice: "metastore" and "catalog" are often used interchangeably, but the Iceberg project reserves "catalog" for the specific interface that resolves table identifiers to metadata locations and manages commits. A metastore in the older Hive sense did that job plus a lot more — column statistics, partition listings, view definitions — much of which Iceberg now handles itself through its own metadata layer rather than delegating to the catalog. That's part of why Iceberg catalogs can be comparatively lightweight services: they don't need to re-implement everything Hive Metastore did, just the identity-resolution and commit-coordination slice of it.

Why it matters right now

The catalog layer has been the site of the most consequential open-source governance move in the lakehouse space this year: Apache Polaris, the REST catalog project originally open-sourced by Snowflake, graduated to a top-level Apache Software Foundation project in February 2026. That graduation is not a technical milestone so much as a trust milestone — top-level ASF status signals that a project has an independent, vendor-neutral governance structure rather than being steered by whichever company donated the code.

That distinction is the entire story of catalog lock-in. A REST-compliant catalog API doesn't guarantee vendor neutrality any more than a SQL-compliant database guarantees your queries will run unmodified elsewhere. What matters is who controls the roadmap of the catalog implementation you actually run: who decides which access-control model it supports, which engines get first-class integration, which enterprise features (row-level security, credential vending, cross-catalog federation) ship and on what schedule. A catalog can expose an open REST interface for reads while keeping all of the governance, security, and multi-engine orchestration logic proprietary. That's the exact shape lock-in has taken in this generation of the lakehouse: not "can you read the files" but "who administers the control plane that decides who can read the files, and on what terms."

Polaris reaching top-level status is significant precisely because it removes single-vendor control from one of the catalogs credible enough to be a real default choice — the same trajectory Iceberg itself followed years earlier when it moved from a single company's project to an ASF top-level project with multi-vendor committers.

This also reframes how the rest of the ecosystem gets evaluated. Once one credible REST catalog has independent governance, every competing catalog — whether it's a cloud provider's managed offering, a data platform vendor's control plane, or another open-source project still incubating — gets implicitly compared against that baseline. "Does this catalog have a governance structure comparable to Polaris's, or is it still effectively controlled by one company's product roadmap" becomes a fair diligence question in a way it wasn't when every REST catalog option was vendor-controlled by default. That's a structural change in negotiating leverage for teams choosing infrastructure, even for teams that don't end up choosing Polaris itself: the existence of a neutral, credible alternative pressures every other catalog vendor to be more explicit about what "open" actually means in their offering, and in some cases to open-source components they might otherwise have kept proprietary.

The practical implications for teams building on Iceberg

If you're choosing or evaluating a catalog today, the decision carries weight well beyond initial setup, because catalogs are unusually expensive to migrate away from once table metadata, access policies, and engine integrations accumulate around them.

Questions worth asking before committing to a catalog

• Governance: Is the catalog's roadmap controlled by a foundation with multiple independent committers, or by one vendor's product team?
• Protocol compliance vs. feature parity: Does it implement the full Iceberg REST catalog spec, including newer additions like views and credential vending — or a subset that works today but may diverge later?
• Engine breadth: Which query engines (Spark, Trino, Flink, DuckDB, Snowflake, Databricks) have first-class, actively maintained connectors — not just a community adapter that lags behind spec changes?
• Multi-cloud story: Can the catalog run, or be self-hosted, outside the vendor's own cloud, or does the "open" catalog only work smoothly inside one cloud's ecosystem?
• Migration cost: What does moving to a different catalog actually require — a metadata export tool, a re-registration of every table, a rebuild of access policies? Some catalogs make this a scripted afternoon; others make it a multi-quarter project.
• Access-control model: Does it support fine-grained, table- and column-level permissions natively, or is that bolted on by the surrounding platform?

None of these questions have a universally correct answer — a single-cloud shop fully committed to one warehouse vendor may reasonably accept a proprietary catalog in exchange for tighter integration and less operational overhead. The mistake is not making the trade-off; it's not realizing a trade-off is being made, because "Iceberg" gets marketed as inherently open regardless of which catalog sits underneath it.

A short list of what changes operationally

For platform teams, catalog choice shows up in day-to-day work in a few concrete ways:

1. Cross-engine consistency — a REST catalog with broad engine support means a Spark job and a Trino query see the same committed snapshot without custom sync jobs.
2. Concurrency limits — catalogs backed by a relational database with strict locking can become a bottleneck under high-frequency streaming writes; some managed catalogs handle this better than a self-hosted Hive Metastore.
3. Credential vending — modern catalogs can hand out short-lived, scoped storage credentials per query rather than requiring every engine to hold broad bucket access, which is a meaningful security improvement over older Hive Metastore setups.
4. Disaster recovery — because the catalog is the single source of truth for "what is current," its backup and recovery story matters as much as the data files' durability. Losing catalog state without a recovery path can make technically intact data files unreadable as a coherent table.

How catalog choice shapes team topology

There's a less obvious effect worth naming: catalog choice quietly determines which teams get to own which parts of the platform. A catalog with strong, self-service credential vending and fine-grained access control lets a central platform team hand table ownership to individual domain teams without becoming a bottleneck for every new dataset or permission change. A catalog that only supports coarse, bucket-level access control tends to concentrate control back in the platform team, because anything finer-grained than "can read this bucket" has to be enforced somewhere else — often in application code or a separate authorization layer that has to stay in sync with the catalog by convention rather than by design.

This matters more as organizations scale the number of teams producing and consuming Iceberg tables. A five-person data team can tolerate a catalog with weak access control because everyone with access is trusted by default. A hundred-person organization with dozens of domain teams, external data-sharing requirements, and regulatory obligations around specific data categories cannot. Catalog selection made early, before those constraints are visible, has a way of becoming the reason a platform team spends a quarter re-architecting permissions two years later.

Cost signals that are easy to miss

Catalog pricing is rarely presented as a line item on its own — it's usually bundled into a managed data platform's overall bill, which makes it easy to underweight when comparing options. A few cost dimensions worth pricing out explicitly rather than assuming they're negligible:

None of these are reasons to avoid a given catalog — they're reasons to run a realistic load test against your actual write patterns before signing a contract or committing engineering time to an integration.

Limitations and open questions

The REST catalog spec is still maturing, and treating it as a finished, universally-implemented standard overstates where things are. A few real gaps:

• Spec coverage lags implementation needs. Features like multi-table transactions, views, and fine-grained access control have been added to the spec incrementally, and not every catalog vendor implements the newest additions at the same pace, so "REST-compatible" doesn't always mean "feature-equivalent."
• Access control is not standardized. The REST protocol governs how you read and commit metadata; it says very little about how permissions, roles, or row/column-level policies are expressed, which means switching catalogs can mean re-implementing your entire security model from scratch.
• Interoperability claims need testing, not trusting. Two catalogs can both claim REST compliance and still behave differently enough under concurrent writes, large partition counts, or cross-region latency that a proof-of-concept against your actual workload is the only reliable way to know.
• Governance neutrality is a spectrum, not a binary. A project reaching top-level Apache status is a strong signal, but the practical vendor-neutrality of a catalog also depends on who's actually running the managed service you use day to day, which committers work for which company, and how quickly community-contributed features actually ship versus vendor-only features.
• Format-level openness doesn't transfer automatically. Because Iceberg the table format is genuinely open and well-specified, it's easy to assume every layer built on top inherits that openness. The catalog layer has to be evaluated on its own terms.

What to watch next

The next eighteen months will likely settle a few open questions that currently make catalog selection harder than it should be:

• Whether REST catalog federation matures — the ability for one catalog to transparently proxy or federate tables registered in a different catalog, which would reduce the cost of catalog choice by making migration less of an all-or-nothing decision.
• How access-control standardization evolves — whether a common permissions model emerges across catalogs, or whether this remains the primary differentiator (and lock-in mechanism) vendors compete on.
• Whether managed-service pricing diverges from self-hosted cost — as more vendors offer hosted REST catalogs, the gap between "open protocol" and "open economics" will become clearer.
• Consolidation versus fragmentation among catalog projects — Polaris's graduation raises the bar for what counts as a credible neutral option; whether other vendor-originated catalogs follow the same path toward independent governance, or stay tightly coupled to their originating company, will shape how much real choice teams have.

FAQ

What is an Apache Iceberg catalog, in one sentence?

It's the service that tracks which metadata file currently represents each Iceberg table and guarantees that updates to that pointer happen atomically, so concurrent readers and writers always see a consistent version of the table.

Is Apache Iceberg itself open source, and does that mean I avoid lock-in?

Yes, the Iceberg table format and specification are open source under the Apache Software Foundation, but the format alone doesn't determine lock-in — the catalog implementation you choose to manage those tables can still be proprietary, single-vendor, or tightly coupled to one cloud, which is where practical lock-in now tends to occur.

What's the difference between Hive Metastore and a REST catalog?

Hive Metastore is an older, database-backed catalog interface tied closely to Hadoop-era tooling and requires engine-specific client libraries; a REST catalog exposes a standardized HTTP API that any compliant engine can speak, which reduces the amount of custom integration code needed per engine.

Why did Apache Polaris graduating to a top-level project matter?

Top-level Apache Software Foundation status means a project is governed independently, with decisions made by a diverse set of committers rather than a single company — a meaningful signal for a catalog specifically because catalog governance determines who controls access rules, roadmap priorities, and cross-engine support over time.

Can I migrate between Iceberg catalogs later if I choose wrong?

Migration is usually possible but rarely trivial — it typically involves re-registering every table's metadata pointer with the new catalog and, more painfully, rebuilding whatever access-control policies were defined at the old catalog, so it's worth testing migration tooling before committing at scale rather than after.

Does using Iceberg mean I no longer need a data warehouse?

Not necessarily — many teams run Iceberg tables alongside traditional warehouses, using Iceberg for open, engine-agnostic storage of large or shared datasets while keeping a warehouse for workloads that benefit from tighter, vendor-optimized query performance; the two are increasingly complementary rather than strictly competing choices.

Which catalog should a new project default to?

There's no universal answer, but a reasonable starting heuristic is to prioritize catalogs with independent governance and broad, actively maintained engine support if cross-engine flexibility matters to you, and to explicitly evaluate migration cost as part of the decision rather than assuming it will be cheap later.

Teams weighing catalog choice against real workload and governance requirements can get hands-on help scoping that decision from Woyce Technologies.