How Iceberg Catalogs Work — and Where Lakehouse Lock-In Now Lives

Ask a data engineer what Apache Iceberg gives them and you'll hear about open file formats, schema evolution, and time travel. Ask them which vendor they're locked into, and the honest answer is usually a catalog they never chose deliberately — it just came bundled with whichever query engine they adopted first. That's the quiet shift in the lakehouse world: the table format is open, but the thing that decides who can read your tables, how fast, and under whose governance rules is the catalog. And catalogs have not been nearly as open as the format they sit on top of.

This matters more than it used to. For years, "Iceberg" and "lakehouse openness" were treated as synonyms. That equivalence is breaking down, and understanding why requires understanding what a catalog actually does — a piece of the stack most people outside data platform teams have never had to think about.

What an Iceberg catalog actually is

Apache Iceberg's core trick is that a "table" is not a directory of files — it's a chain of immutable metadata pointing at data files. Every time you write to an Iceberg table, you're not overwriting anything; you're creating a new metadata file that describes a new snapshot of the table, and appending it to a metadata log. Readers ask a simple question: "what is the current metadata file for this table?" Whoever can answer that question authoritatively controls the table.

That's the catalog's job. Specifically, a catalog:

• Tracks the mapping from table names (db.schema.table) to the location of their current metadata file
• Provides atomic commit semantics — when two writers try to update a table at the same time, the catalog ensures only one wins and the table never ends up in a torn, half-written state
• Enforces table-level and often column-level access control
• Often stores or proxies additional metadata: table properties, schema history, statistics used for query planning — adjacent to what a semantic layer does one level up the stack

Without a catalog, Iceberg is just a very clever way to organize Parquet files and JSON pointers in object storage. With one, it becomes something query engines can trust to give consistent, concurrent, transactional access. The catalog is what turns a pile of files into a database.

The pointer chain, concretely

It helps to walk through the actual chain of indirection, because the catalog's narrow job becomes obvious once you see it:

1. A catalog entry for sales.orders stores one thing: the path to the table's current metadata.json file.
2. That metadata.json lists the table's schema history, partition spec, and a pointer to the current manifest list.
3. The manifest list enumerates manifest files, each of which lists actual data files (Parquet, ORC, or Avro) along with column-level statistics used to skip irrelevant files at query time.
4. Data files themselves never move or get rewritten for a simple update — new files are added and old ones are marked as no longer part of the current snapshot.

The catalog only owns step 1. Everything downstream is just files in object storage that any engine with read access can open. This is precisely why the catalog became the chokepoint: it's the smallest, cheapest piece of the system to control, and controlling it controls everything built on top.

Why catalog implementations differ so much

If the catalog's job is "store a pointer and make commits atomic," you'd expect implementations to be interchangeable. They are not, because the atomicity guarantee has to be built on top of whatever storage or database substrate is available, and that substrate shapes everything about performance, concurrency limits, and operational cost.

The REST catalog spec, formalized within the Iceberg project itself, matters because it's the first catalog interface designed explicitly to be an interoperability layer rather than an implementation. Instead of every query engine needing native code for Hive Metastore, Glue, and half a dozen proprietary systems, engines just need to speak one HTTP protocol. Any catalog vendor can implement the REST spec as a facade in front of their own storage.

That sounds like it should have ended lock-in. It didn't, for reasons worth unpacking.

Why atomic commits are harder than they sound

The "atomic pointer swap" description undersells how much engineering goes into making catalog commits safe under real-world concurrency. Consider two writers racing to update the same table at the same time — one appending a batch of new rows, another running a compaction job that rewrites small files into larger ones. Both start from the same base metadata snapshot. Both compute a new metadata file. Only one can win.

A correct catalog implementation has to:

• Detect that both writers started from the same base snapshot
• Accept the first commit and atomically advance the "current" pointer
• Reject the second commit with a retriable conflict error rather than silently corrupting the table
• Give the losing writer enough information to re-read the new current state, reapply its change on top, and retry

This is the same optimistic-concurrency pattern used in distributed version control, and it's precisely why catalogs built on systems with native compare-and-swap or transactional guarantees (a relational database, a strongly consistent key-value store) tend to be more reliable under heavy concurrent write load than catalogs bolted onto storage systems that were never designed for this kind of coordination. It's also why "just point Iceberg at a folder in object storage with no catalog" is a valid pattern for single-writer, low-concurrency use cases, but breaks down fast the moment multiple pipelines write to the same table.

Catalogs vs. metastores: a naming note

Worth clarifying, because the terminology gets muddled in practice: "metastore" and "catalog" are often used interchangeably, but the Iceberg project reserves "catalog" for the specific interface that resolves table identifiers to metadata locations and manages commits. A metastore in the older Hive sense did that job plus a lot more — column statistics, partition listings, view definitions — much of which Iceberg now handles itself through its own metadata layer rather than delegating to the catalog. That's part of why Iceberg catalogs can be comparatively lightweight services: they don't need to re-implement everything Hive Metastore did, just the identity-resolution and commit-coordination slice of it.

Benefits of an Iceberg Catalog

A catalog's job is narrow, but what it makes possible is not. These are the capabilities a well-chosen catalog gives a lakehouse that a folder of files never could.

Safe Concurrent Writes

The catalog's atomic pointer swap is what lets several pipelines write to the same table without corrupting it. An ingestion job, a compaction job, and a backfill can all run, and the catalog ensures each commit either lands cleanly or fails with a retriable conflict. Without that coordination, teams are forced into single-writer designs or fragile locking conventions, both of which limit how quickly data can move through the platform.

One Table, Many Engines

When Spark, Trino, Flink, and other engines all resolve a table through the same catalog, they see the same committed snapshot. Analysts query with one engine while pipelines write with another, and nobody runs sync jobs to copy data between systems. A REST catalog makes this easier still, since each engine only needs to speak one HTTP protocol instead of carrying a native integration for every catalog type.

Centralised Access Control

Because every read and commit passes through the catalog, it is a natural place to enforce who can see which tables and columns. Fine-grained permissions defined once apply regardless of which engine is used, instead of being re-implemented in each tool. That consistency simplifies audits and reduces the chance that a new engine quietly bypasses the rules applied everywhere else.

Scoped, Short-Lived Storage Credentials

Modern catalogs can vend temporary credentials limited to the files a specific query needs. Engines no longer hold broad, long-lived access to whole storage buckets. This is a real security improvement over older setups, and it makes it practical to give more teams and tools access to shared data without widening the blast radius of a leaked key.

Time Travel and Reproducibility

Since the catalog tracks the current metadata file and each snapshot remains addressable, teams can query a table as it was at an earlier point, audit changes, and roll back a bad write. Reproducing a report or a model's training data from a specific moment becomes a query rather than a forensic exercise, which matters for debugging, compliance, and machine learning workflows alike.

Iceberg Catalog Use Cases

The right catalog depends on the job. These are the common patterns and the catalog characteristics each one leans on.

Multi-Engine Analytics Platforms

Organisations that run Spark for transformation, Trino for interactive queries, and a warehouse for business reporting need every engine to agree on what a table contains. A REST catalog with broad, actively maintained engine connectors lets them share tables without copies or sync jobs. The outcome is one governed copy of the data, queried by whichever engine suits each task, and fewer arguments about which number is correct.

Streaming and Micro-Batch Ingestion

Pipelines that commit small batches every few seconds or minutes put heavy load on the catalog's commit path. Here, the backing store's concurrency behaviour matters more than almost anything else, since a catalog that serialises commits poorly becomes the bottleneck. Teams in this situation load-test commit throughput against their real write pattern and schedule compaction so maintenance commits don't compete with ingestion at peak times.

Domain-Owned Data Products

In a data mesh or similar model, many domain teams publish tables for others to consume. A catalog with fine-grained, self-service access control and credential vending lets each team manage permissions on its own tables without routing every change through a central platform team. Ownership scales with the organisation instead of concentrating in one group that becomes a queue for every new dataset.

Multi-Cloud and Sovereignty Requirements

Some organisations must keep data in particular regions or across more than one cloud provider. A catalog that can be self-hosted, or that runs consistently across providers, avoids tying the whole lakehouse to one cloud's managed service. Teams keep the option to move workloads without re-registering every table or rebuilding access policies.

Branching for Data Development

Teams that want to test pipeline changes against production-like data without affecting production use catalogs with Git-like branching, such as Nessie. A change runs on a branch, gets validated, and merges when ready. Data changes follow a review process similar to code, which reduces the risk of a bad transformation reaching downstream consumers. It also gives reviewers a concrete diff to approve.

Why it matters right now

The catalog layer has been the site of the most consequential open-source governance move in the lakehouse space this year: Apache Polaris, the REST catalog project originally open-sourced by Snowflake, graduated to a top-level Apache Software Foundation project in February 2026 — the kind of governance milestone that matters increasingly across the open-source AI era. That graduation is not a technical milestone so much as a trust milestone — top-level ASF status signals that a project has an independent, vendor-neutral governance structure rather than being steered by whichever company donated the code.

That distinction is the entire story of catalog lock-in. A REST-compliant catalog API doesn't guarantee vendor neutrality any more than a SQL-compliant database guarantees your queries will run unmodified elsewhere. What matters is who controls the roadmap of the catalog implementation you actually run: who decides which access-control model it supports, which engines get first-class integration, which enterprise features (row-level security, credential vending, cross-catalog federation) ship and on what schedule. A catalog can expose an open REST interface for reads while keeping all of the governance, security, and multi-engine orchestration logic proprietary. That's the exact shape lock-in has taken in this generation of the lakehouse: not "can you read the files" but "who administers the control plane that decides who can read the files, and on what terms."

Polaris reaching top-level status is significant precisely because it removes single-vendor control from one of the catalogs credible enough to be a real default choice — the same trajectory Iceberg itself followed years earlier when it moved from a single company's project to an ASF top-level project with multi-vendor committers.

This also reframes how the rest of the ecosystem gets evaluated. Once one credible REST catalog has independent governance, every competing catalog — whether it's a cloud provider's managed offering, a data platform vendor's control plane, or another open-source project still incubating — gets implicitly compared against that baseline. "Does this catalog have a governance structure comparable to Polaris's, or is it still effectively controlled by one company's product roadmap" becomes a fair diligence question in a way it wasn't when every REST catalog option was vendor-controlled by default. That's a structural change in bargaining power for teams choosing infrastructure, even for teams that don't end up choosing Polaris itself: the existence of a neutral, credible alternative pressures every other catalog vendor to be more explicit about what "open" actually means in their offering, and in some cases to open-source components they might otherwise have kept proprietary.

The practical implications for teams building on Iceberg

If you're choosing or evaluating a catalog today, the decision carries weight well beyond initial setup, because catalogs are unusually expensive to migrate away from once table metadata, access policies, and engine integrations accumulate around them.

Questions worth asking before committing to a catalog

• Governance: Is the catalog's roadmap controlled by a foundation with multiple independent committers, or by one vendor's product team?
• Protocol compliance vs. feature parity: Does it implement the full Iceberg REST catalog spec, including newer additions like views and credential vending — or a subset that works today but may diverge later?
• Engine breadth: Which query engines (Spark, Trino, Flink, DuckDB, Snowflake, Databricks) have first-class, actively maintained connectors — not just a community adapter that lags behind spec changes?
• Multi-cloud story: Can the catalog run, or be self-hosted, outside the vendor's own cloud — a question central to sovereign cloud strategies — or does the "open" catalog only work smoothly inside one cloud's ecosystem?
• Migration cost: What does moving to a different catalog actually require — a metadata export tool, a re-registration of every table, a rebuild of access policies? Some catalogs make this a scripted afternoon; others make it a multi-quarter project.
• Access-control model: Does it support fine-grained, table- and column-level permissions natively, or is that bolted on by the surrounding platform?

None of these questions have a universally correct answer — a single-cloud shop fully committed to one warehouse vendor may reasonably accept a proprietary catalog in exchange for tighter integration and less operational overhead. The mistake is not making the trade-off; it's not realizing a trade-off is being made, because "Iceberg" gets marketed as inherently open regardless of which catalog sits underneath it.

A short list of what changes operationally

For platform teams, catalog choice shows up in day-to-day work in a few concrete ways:

1. Cross-engine consistency — a REST catalog with broad engine support means a Spark job and a Trino query see the same committed snapshot without custom sync jobs.
2. Concurrency limits — catalogs backed by a relational database with strict locking can become a bottleneck under high-frequency streaming writes; some managed catalogs handle this better than a self-hosted Hive Metastore.
3. Credential vending — modern catalogs can hand out short-lived, scoped storage credentials per query rather than requiring every engine to hold broad bucket access, which is a meaningful security improvement over older Hive Metastore setups.
4. Disaster recovery — because the catalog is the single source of truth for "what is current," its backup and recovery story matters as much as the data files' durability. Losing catalog state without a recovery path can make technically intact data files unreadable as a coherent table.

How catalog choice shapes team topology

There's a less obvious effect worth naming: catalog choice quietly determines which teams get to own which parts of the platform. A catalog with strong, self-service credential vending and fine-grained access control lets a central platform team hand table ownership to individual domain teams without becoming a bottleneck for every new dataset or permission change. A catalog that only supports coarse, bucket-level access control tends to concentrate control back in the platform team, because anything finer-grained than "can read this bucket" has to be enforced somewhere else — often in application code or a separate authorization layer that has to stay in sync with the catalog by convention rather than by design.

This matters more as organizations scale the number of teams producing and consuming Iceberg tables. A five-person data team can tolerate a catalog with weak access control because everyone with access is trusted by default. A hundred-person organization with dozens of domain teams, external data-sharing requirements, and regulatory obligations around specific data categories cannot. Catalog selection made early, before those constraints are visible, has a way of becoming the reason a platform team spends a quarter re-architecting permissions two years later.

Cost signals that are easy to miss

Catalog pricing is rarely presented as a line item on its own — it's usually bundled into a managed data platform's overall bill, which makes it easy to underweight when comparing options. A few cost dimensions worth pricing out explicitly rather than assuming they're negligible:

None of these are reasons to avoid a given catalog — they're reasons to run a realistic load test against your actual write patterns before signing a contract or committing engineering time to an integration.

Common Iceberg Catalog Mistakes

Most catalog regret comes from decisions that seemed minor at the time. These are the ones that tend to cost the most later.

Accepting Whatever Catalog Came With the First Engine

Many teams never choose a catalog deliberately; it arrives bundled with the first query engine or platform they adopt. Years later, they discover that their tables, permissions, and integrations all depend on a control plane they never evaluated. Treat the catalog as a separate decision from the engine, even if you end up with the bundled option.

Equating REST Compatibility With Portability

A catalog that implements the REST spec can still keep access-control models, credential vending, and federation features proprietary. Teams that assume "REST-compatible" means "easy to leave" are surprised when a migration requires rebuilding the entire permissions model. Check which spec features are implemented and how policies are expressed before relying on compatibility claims.

Running Multiple Writers Without a Proper Catalog

Pointing Iceberg at object storage with no catalog, or with one that lacks strong commit guarantees, works for a single writer. Once several pipelines write to the same table, conflicts and lost updates appear. Any table with concurrent writers needs a catalog built on a substrate with real transactional or compare-and-swap semantics.

Neglecting Catalog Backup and Recovery

Teams protect data files with replication and versioning but treat the catalog as stateless infrastructure. If catalog state is lost, intact data files may no longer be readable as coherent tables. The catalog needs its own backup, recovery testing, and disaster-recovery plan.

Choosing Coarse Access Control for a Growing Organisation

Bucket-level permissions are fine for a small, trusted team. As domains, external sharing, and regulated data categories multiply, coarse controls force fine-grained rules into application code or separate systems that drift out of sync. Choose for the access model you will need in two years, not the one that suffices today.

Iceberg Catalog Best Practices

• Decide the catalog explicitly and write down why. Record the governance model, spec coverage, engine support, and migration path of the catalog you choose, so the trade-off is visible to whoever inherits the platform. Revisit the record when engines, clouds, or regulatory needs change.
• Prefer independent governance where cross-engine flexibility matters. If several engines or clouds are in your future, favour catalogs whose roadmap is controlled by a foundation with diverse committers rather than a single vendor.
• Load-test commits with your real write pattern. Simulate your busiest streaming or micro-batch ingestion together with compaction jobs, and measure conflict rates and commit latency before production. Repeat the test after major version upgrades, since commit behaviour can change between releases.
• Rehearse a migration early. Export metadata, re-register a sample of tables in a second catalog, and rebuild their access policies. The exercise reveals switching costs while they are still cheap to address. Time the exercise and note every manual step, because those steps become the real cost estimate for any future move.
• Use credential vending instead of broad bucket access. Give engines short-lived, scoped credentials from the catalog so no tool holds standing access to entire storage locations. Audit any remaining static keys and set an expiry plan for each.
• Define permissions at the catalog, not in each engine. Keep table- and column-level policies in one place so new engines inherit them automatically and audits have a single source of truth. Review those policies on a schedule as teams and datasets change.
• Back up catalog state and test restores. Include the catalog in disaster-recovery drills and confirm that restored state resolves tables correctly. Measure how long a restore takes, so recovery time objectives are based on evidence.
• Schedule table maintenance deliberately. Plan snapshot expiry and manifest rewriting so maintenance commits don't collide with ingestion peaks or let metadata chains grow unchecked.

Limitations and open questions

The REST catalog spec is still maturing, and treating it as a finished, universally-implemented standard overstates where things are. A few real gaps:

• Spec coverage lags implementation needs. Features like multi-table transactions, views, and fine-grained access control have been added to the spec incrementally, and not every catalog vendor implements the newest additions at the same pace, so "REST-compatible" doesn't always mean "feature-equivalent."
• Access control is not standardized. The REST protocol governs how you read and commit metadata; it says very little about how permissions, roles, or row/column-level policies are expressed, which means switching catalogs can mean re-implementing your entire security model from scratch.
• Interoperability claims need testing, not trusting. Two catalogs can both claim REST compliance and still behave differently enough under concurrent writes, large partition counts, or cross-region latency that a proof-of-concept against your actual workload is the only reliable way to know.
• Governance neutrality is a spectrum, not a binary. A project reaching top-level Apache status is a strong signal, but the practical vendor-neutrality of a catalog also depends on who's actually running the managed service you use day to day, which committers work for which company, and how quickly community-contributed features actually ship versus vendor-only features.
• Format-level openness doesn't transfer automatically. Because Iceberg the table format is genuinely open and well-specified, it's easy to assume every layer built on top inherits that openness. The catalog layer has to be evaluated on its own terms.

What to watch next

The next eighteen months will likely settle a few open questions that currently make catalog selection harder than it should be:

• Whether REST catalog federation matures — the ability for one catalog to transparently proxy or federate tables registered in a different catalog, which would reduce the cost of catalog choice by making migration less of an all-or-nothing decision.
• How access-control standardization evolves — whether a common permissions model emerges across catalogs, or whether this remains the primary differentiator (and lock-in mechanism) vendors compete on.
• Whether managed-service pricing diverges from self-hosted cost — as more vendors offer hosted REST catalogs, the gap between "open protocol" and "open economics" will become clearer.
• Consolidation versus fragmentation among catalog projects — Polaris's graduation raises the bar for what counts as a credible neutral option; whether other vendor-originated catalogs follow the same path toward independent governance, or stay tightly coupled to their originating company, will shape how much real choice teams have.

Teams weighing catalog choice against real workload and governance requirements can get hands-on help scoping that decision from Woyce Technologies.

FAQ

What is an Apache Iceberg catalog, in one sentence?

It's the service that tracks which metadata file currently represents each Iceberg table and guarantees that updates to that pointer happen atomically, so concurrent readers and writers always see a consistent version of the table. Everything else, from manifests to Parquet data files, lives in object storage. The catalog is small, but because it decides what "current" means, it also tends to be where access control and governance get enforced.

Is Apache Iceberg itself open source, and does that mean I avoid lock-in?

Yes, the Iceberg table format and specification are open source under the Apache Software Foundation, but the format alone doesn't determine lock-in — the catalog implementation you choose to manage those tables can still be proprietary, single-vendor, or tightly coupled to one cloud, which is where practical lock-in now tends to occur.

What's the difference between Hive Metastore and a REST catalog?

Hive Metastore is an older, database-backed catalog interface tied closely to Hadoop-era tooling and requires engine-specific client libraries; a REST catalog exposes a standardized HTTP API that any compliant engine can speak, which reduces the amount of custom integration code needed per engine. Hive Metastore also carries extra responsibilities, such as partition listings and statistics, that Iceberg now handles in its own metadata. For new projects that need several engines to share tables, a REST catalog is usually the cleaner choice; Hive Metastore mostly persists where existing Hadoop-era clusters still depend on it.

Why did Apache Polaris graduating to a top-level project matter?

Top-level Apache Software Foundation status means a project is governed independently, with decisions made by a diverse set of committers rather than a single company — a meaningful signal for a catalog specifically because catalog governance determines who controls access rules, roadmap priorities, and cross-engine support over time. It also changes the bargaining position for every team choosing a catalog, even those that pick something else.

Can I migrate between Iceberg catalogs later if I choose wrong?

Migration is usually possible but rarely trivial — it typically involves re-registering every table's metadata pointer with the new catalog and, more painfully, rebuilding whatever access-control policies were defined at the old catalog, so it's worth testing migration tooling before committing at scale rather than after. Include a trial migration and a permissions rebuild in your proof of concept.

Does using Iceberg mean I no longer need a data warehouse?

Not necessarily — many teams run Iceberg tables alongside traditional warehouses as part of a broader data lakehouse strategy, using Iceberg for open, engine-agnostic storage of large or shared datasets while keeping a warehouse for workloads that benefit from tighter, vendor-optimized query performance; the two are increasingly complementary rather than strictly competing choices.

Which catalog should a new project default to?

There's no universal answer, but a reasonable starting heuristic is to prioritize catalogs with independent governance and broad, actively maintained engine support if cross-engine flexibility matters to you, and to explicitly evaluate migration cost as part of the decision rather than assuming it will be cheap later. Test any shortlisted catalog against your busiest streaming or compaction workload before deciding.

Conclusion

The core problem this article set out to explain is that an open table format does not make the whole lakehouse open. Iceberg's data files and metadata are readable by any engine, but the catalog decides which metadata file is current, who may commit, and who may read — and that control plane is where switching costs build up.

The key insights: the catalog's technical job is narrow (resolve a name, swap a pointer atomically), but implementations differ widely in concurrency behavior, credential vending, and access-control models. The REST catalog spec makes engine interoperability far easier, yet it says little about permissions, so moving between two "REST-compatible" catalogs can still mean rebuilding your security model. Independent governance, as with Apache Polaris, changes the bargaining position for every team choosing a catalog, even those that pick something else.

The caveats are practical. Spec coverage varies between vendors, catalog costs are often hidden inside platform bills, and concurrency limits only show up under your real write patterns. Treat compatibility claims as hypotheses to test.

The next step is to run a proof of concept against your busiest streaming or compaction workload, including a trial migration and a permissions rebuild, before committing. If you want help designing that evaluation, see our database engineering services.