XenonStack BlogModernise in Place: HDFS to Apache Iceberg on Private Cloud

Can you adopt Apache Iceberg without leaving Hadoop? Yes. Apache Iceberg is an open table format, not a storage system. It can sit on the HDFS storage you already run, use the Hive Metastore you already have and be read by the Spark you already operate. That makes it possible to modernise a data platform in place, table by table, without a single high-risk migration to a new platform or a public cloud.

The 60-second brief

• The pressure: Hadoop estates work, but classic Hive tables make deletes, schema changes and concurrent writes hard.
• The option: Iceberg adds transactional tables, schema evolution, time travel and row-level deletes on top of existing storage.
• The approach: three stages. Adopt the table format, open up compute, then decide on storage.
• The governance gain: row-level deletes and snapshot history help with erasure, audit and reproducibility.
• The caution: Iceberg adds its own operations: compaction, snapshot expiry and catalogue management. Someone must run them.

Part of a series. Start with the overview: Open-Source Data Platforms on Private Cloud: Why Managed Services and Governance Must Evolve Together.

Why "Rip and Replace" Is the Wrong Default

Storage and analytics vendors are publishing a steady stream of Hadoop-to-lakehouse migration guides. Most assume the destination is their platform. For an enterprise on a private cloud, with regulated data, tuned pipelines and years of operating knowledge, a full replacement carries three costs that rarely appear in the comparison:

• Parallel running. Two platforms must be operated and secured for the length of the migration.
• Control re-validation. Every access policy, encryption setting and audit feed has to be rebuilt and proven again.
• Knowledge loss. Runbooks, known errors and tuning experience do not transfer automatically.

Modernisation is a change to how tables behave. It does not have to be a change of address.

What Iceberg Changes

A Three-Stage Path

Stage 1: Adopt the table format on existing storage

Keep HDFS, YARN and the Hive Metastore. Add Iceberg support to Spark and create new tables as Iceberg. For existing Hive tables, Iceberg provides procedures to snapshot a table for testing or migrate it in place, reusing existing data files rather than copying them. Start with tables that hurt most: those needing deletes, late-arriving corrections or frequent schema change.

Stage 2: Open up compute

Once tables are in an open format, more than one engine can use them. Add an interactive SQL engine such as Trino for analysts while Spark continues batch processing. Run new compute on Kubernetes if that is where the enterprise is heading, while YARN continues to serve existing jobs.

Stage 3: Decide on storage, deliberately

With data in Iceberg, the storage layer becomes a separate decision. Options on private cloud include staying on HDFS, or moving progressively to S3-compatible object storage. Because the table format is the same, this can be done dataset by dataset, on evidence of cost and performance rather than as a prerequisite.

Each stage delivers value on its own. An enterprise can stop after stage 1 and still be better off.

Governance Benefits, and New Duties

Iceberg helps directly with expectations now reaching regulated enterprises:

• Erasure. Row-level deletes make it practical to remove one customer's records. Note that old snapshots still hold the data until they are expired, so snapshot retention must be part of the erasure procedure.
• Audit and reproducibility. Snapshot history shows what a table contained at a point in time.
• Controlled change. Schema and partition changes are recorded in table metadata.

These connect to the tagging, lineage and erasure themes in the RBI draft data governance guidance and the safeguards in our DPDP 2027 checklist.

Iceberg also adds operating work that must be owned:

What to Check Before You Start

1. Versions. Confirm that your Spark, Hive and Hadoop versions support the Iceberg release you plan to use. Unsupported base versions should be dealt with first. See the lifecycle risk register.
2. Consumers. List every reader of a table before migrating it, including serving layers and exports.
3. Security. Test authorisation policies and audit logging against Iceberg tables in nonproduction.
4. Rollback. Use snapshot-based testing first so the original table remains untouched until validation passes.
5. Operations. Schedule and monitor maintenance jobs from day one.

The Bottom Line

Enterprises do not need to replace a working platform to gain modern table behaviour. Change the table format first, keep what works and let evidence decide the rest. Investment then follows validated need rather than technology replacement for its own sake.

Modernise Your Data Platform with XenonStack

XenonStack helps enterprises modernise open-source data platforms on private cloud in stages, with co-managed L2/L3 support for the platform they run today and the one they are moving towards. Explore Data Modernization, Data Managed Services and the Open-Source Data Platform solution.

Talk to a Specialist

Frequently Asked Questions

Does Apache Iceberg work on HDFS?

Yes. Iceberg is a table format that works on HDFS as well as on object storage.

Do we have to copy all our data to migrate Hive tables?

Not necessarily. Iceberg provides procedures that can create Iceberg metadata over existing data files, so many tables can be migrated in place.

Can Hive and Iceberg tables coexist?

Yes. Both can be registered in the same metastore, which allows table-by-table migration.

Does Iceberg solve data erasure?

It makes row-level deletion practical. Complete erasure also requires expiring older snapshots and handling copies elsewhere in the platform.

What new operations does Iceberg need?

Regular compaction, snapshot expiry, orphan file clean-up and a highly available catalogue.

Confirm feature and version compatibility in the Apache Iceberg documentation for the releases you run.

Explore the XenonStack Platform and Agent Network

Platforms: ElixirData · ElixirClaw · ElixirHub · NexaStack · MetaSecure · Neural AI · AutonomousOps · CohortIQ · AI Aviator · SAIF Aviator

AI agents: AgentAnalyst · AgentEvaluation · AgentForce · AgentGRC · AgentHR · AgentInstruct · AgentLabel · AgentPixel · AgentQA · AgentRAI · AgentScrum · AgentSearch · AgentSketch · AgentSRE

XenonStack: xenonstack.com · xenonstack.ai · xenonstack.ae · xenonstack.jobs