Open data lakehouse: A practical guide to what, why, and how

Most enterprise data teams are running a lake and a warehouse side by side. The challenge is familiar: reconciling numbers between the two, waiting on ETL to catch up, paying for storage nobody queries. An open data lakehouse promises to close that gap, but "open" gets used loosely. Adopting one open table format doesn't make the rest of the stack open. The catalog, compute engine, and table maintenance layer can all still be closed, and that's usually where lock-in shows up.

N-iX has built these architectures for over 90 global enterprises, migrating them off legacy warehouses onto open lakehouse platforms as part of a broader data analytics modernization practice. We've spent 23 years turning legacy data infrastructure into platforms that hold up under real production load. This guide covers what "open" means at every layer, how the architecture fits together, and why AI workloads specifically need it, closing with a quick way to check if your organization is ready to implement one.

Key takeaways

• Adopting an open table format, such as Iceberg or Delta Lake, doesn't make a lakehouse open. The catalog, compute engine, and table maintenance layer can still lock you in.
• An open lakehouse works across five separate layers: storage, file format, table format, catalog, and compute engine, and each one has to stay swappable on its own.
• Iceberg, Delta Lake, and Hudi each optimize for a different workload, so the right format depends on the read and write patterns, not market popularity.
• Enterprises take one of three paths to build a lakehouse: DIY open-source, vendor-managed, or CSP-native. Team size and timeline decide the right fit more than any platform comparison.
• Openness matters most for AI workloads, since it lets training pipelines and agents read the same governed data instead of running on a separate, ungoverned pipeline.
• Governance has to get built in from day one. Deferring it until after launch is the single most common reason an open lakehouse architecture turns into a data swamp.

What an open data lakehouse solves

Most enterprises run two systems that were never built to talk to each other. Raw, unstructured data lands in a lake, and a subset gets cleaned, transformed, and copied into a warehouse for BI. Each copy is a new ETL job, a new place for the numbers to drift apart, and a new delay between when data arrives and when anyone can act on it. Data teams end up spending the week reconciling dashboards instead of building the analytics the business asked for.

An open data lakehouse folds that two-tier setup into one system, putting warehouse-grade reliability directly on top of the cheap object storage a lake runs on:

• ACID transactions and schema enforcement, at lake storage prices;
• Structured BI workloads and unstructured AI/ML workloads pulling from the same copy of data;
• Reporting data and machine learning data collapsing into the same dataset, with no export step splitting them apart;
• Schema evolution and time travel, so historical queries stay valid even after the data model changes;
• One governance layer covering both workload types, instead of separate access rules for the lake and the warehouse;
• Compute scaling independently from storage, so a demanding ML workload doesn't mean paying for more warehouse capacity nobody else needs.

Here's what that scale looks like in practice. One manufacturing enterprise N-iX worked with pushed its data processing to 5x the previous volume in a single year, on a lakehouse architecture built to absorb that kind of growth without infrastructure spending climbing at the same pace. That's the real argument for a lakehouse. 

What makes a data lakehouse open

Open-source and open are different claims, and conflating them is where most of the confusion starts. Apache Iceberg is open-source. Delta Lake is open-source. A platform can run either one and still lock your data in, because the table format was never the layer that did the locking. Data engineers who've lived through a lakehouse migration know this from the inside: the format sat there being open the entire time. The catalog underneath it was where the lock-in actually lived.

A catalog tracks which files belong to which tables, the current schema, and which snapshot represents the latest state. Spark writes to that catalog one way. Trino needs to read it the same way, or the two engines disagree about what the table currently looks like.

If a team registers the same Iceberg table in two independent catalogs using a register_table call, the catalogs don't share transaction state, so a write committed in one catalog is invisible to the other. Apache Iceberg's own documentation calls this out directly: doing so can lead to missing updates, data loss, and table corruption.

That risk has everything to do with the catalog layer refusing to speak the same protocol across engines, and nothing to do with Iceberg being closed. An open catalog, Unity Catalog or Apache Polaris, closes that gap by exposing a standard REST interface any Iceberg-compatible engine can call. Open security lakehouse architecture rules live at the catalog layer itself and are enforced once, regardless of whether Spark, Trino, or Flink is running the query that day.

Four questions test real openness in practice, and every one exposes a different layer where vendors quietly reintroduce lock-in after selling the open table format as the whole story.

The schema and partitioning question is where Iceberg's hidden partitioning matters most, and it's what earns Iceberg its reputation beyond ACID transactions. Hidden partitioning maps a query to the correct physical partition. That lets a data engineering team move from daily to hourly partitions without rewriting historical data or breaking dashboards built against the old layout. It's a small mechanical detail with a large practical consequence: architecture decisions stop being permanent.

N-iX has conducted more than one migration where Iceberg was already standardized, and the real constraint was still sitting one layer up, in a catalog nobody had scrutinized. The table format was never the question that mattered. The catalog was.

5 key layers of an open data lakehouse architecture

An open data lakehouse is five layers stacked on top of each other, and each one has to stand on its own for the word "open" to mean anything real. A closed platform blurs those layers together, so swapping one out means rebuilding the whole stack. An open one keeps them separate enough that a decision at the storage layer stays independent of a decision at the compute layer.

Storage layer

Everything sits on commodity cloud object storage: Amazon S3, Azure Blob Storage, Google Cloud Storage, or self-hosted options like MinIO and Ceph are all in active use. This layer earns its "commodity" label. None of these options demand proprietary access patterns to read the files sitting inside them, so storage ends up being the layer where lock-in shows up least. A German industrial manufacturer N-iX worked with rebuilt its platform on Azure Data Lake Storage for that reason alone: the storage layer was the one place it wanted zero vendor-specific dependencies. That choice kept everything built on top free to change later.

Open file formats

Raw data physically lives in a column-oriented file format below the table format layer. Apache Parquet and Apache ORC are two of the most popular open-source data formats for building an enterprise data lakehouse architecture that support most production workloads. Both compress heavily and let a query engine skip entire blocks of irrelevant data instead of scanning row by row. The whole reason analytical queries over these formats consistently beat the same query over a row-based one. Apache Avro is an example of a different category entirely: row-based, commonly seen in write-heavy streaming pipelines, Kafka ingestion, and frequently shifting schemas.

Open table formats

This layer turns a folder of Parquet or ORC files into something that behaves like a database table, adding transactional guarantees a raw file system can never provide on its own. Apache Iceberg, Delta Lake, and Apache Hudi are the three most prominent examples in this space. All three solve the same core problem: tracking schema, partitions, and snapshot history as metadata alongside the data, but each was built around a different original use case.

The differences here map to workload shape more than to any format being objectively stronger.

• A team running heavy CDC pipelines against a transactional source system tends to lean on the kind of upsert handling Hudi was built around. 
• A team standardized on Spark for most of its processing often gets immediate value from the Spark-native optimizations that Delta Lake provides. 
• A team that needs several engines- Spark, Trino, Flink- all reading the same tables without one engine's assumptions breaking another's tends to gravitate toward the engine-neutral design Iceberg was built on from the outset.

Whichever format a team lands on, the decision should follow the read-and-write pattern of the actual workload. Get it wrong, and the cost shows up later as a full migration, never as a line item today.

Unified catalog and governance

This layer decides whether the table formats above deliver the interoperability they promise, making it the most consequential layer in the stack and the one teams underestimate most often. Unity Catalog, Apache Polaris, and Project Nessie are examples of catalogs that expose a standard interface, so Spark, Trino, and Flink can read the same table without disagreeing about its current state. Skip this layer, or hand a single vendor exclusive control over it, and every guarantee the table format layer made becomes conditional on staying with that vendor indefinitely.

Evaluate the catalog before evaluating the table format. An open-table format sitting behind a closed catalog delivers none of the portability that an open lakehouse architecture data platform review would assume.

Compute engines

The processing layer is where workload-matching carries direct, measurable cost, and getting it wrong is an expensive decision to unwind once pipelines depend on it. Apache Spark is a common example for heavy ETL and Machine Learning training. Still, its batch-oriented execution model becomes a bottleneck the moment a team needs sub-second, interactive query response. Trino and Presto are examples used to fill that gap for high-performance federated SQL. Apache Flink is an example built for true real-time stream processing, distinct from a micro-batched approximation.

High-performance analytical engines increasingly build on Apache Arrow, an in-memory columnar format that eliminates a cost most engineering teams only notice once they hit it directly. Dremio is one example built this way from the ground up. Reading a Parquet file off disk still means converting it into row-based structures before a query engine can process it. That conversion, often called the serialization tax, consumes CPU cycles the engine could spend answering the query itself. Arrow processes columnar data directly in memory, skipping that conversion step entirely, and that's a big share of why Arrow-native engines outperform non-Arrow engines on identical hardware.

Matching the engine to the workload's latency requirement matters more than defaulting to whichever engine the team already knows best. A Spark-only stack forced into sub-second interactive queries will underperform an architecture pairing Spark for batch with Trino or Flink for the workloads that need speed.

Five layers, five decisions, each carrying real operational and cost consequences. The architecture stays open only when each can be swapped independently, with a change in one layer never forcing a rebuild in another.

How enterprises can build an open lakehouse: 3 paths compared

Enterprises can build an lakehouse open in three different ways, and the choice among them shapes what a team experiences over the next year more than any single technology decision buried within the architecture.

The DIY open-source stack

A team assembles every layer directly: object storage, an open table format such as Apache Iceberg or Apache Hudi, a catalog such as Apache Polaris or Project Nessie, and compute engines such as Spark and Trino. Every piece gets wired together without an intermediating platform. This path delivers full vendor neutrality and complete control over every layer, since the entire stack belongs to the team running it.

The cost sits in engineering time. Someone owns the glue code that connects each layer, the infrastructure itself, and table maintenance, compaction, snapshot cleanup, and indexing, either manually or through custom in-house automation. This path fits a team that already has several dedicated data platform engineers with hands-on experience in Spark and Iceberg. It also fits a team with the runway to invest in infrastructure before shipping the first production workload.

Vendor-managed platforms

Best open source data lakehouse platforms like Databricks and Snowflake package these same open-source components behind a managed interface, handling infrastructure complexity, built-in governance, and automated performance tuning. A team can move from zero to a working pipeline in weeks instead of months, without standing up a platform engineering function first.

The real consideration here lies in the decision-making a team does up front: which features are core to the open standard and which are specific to that platform. That split determines what portability looks like down the line. That question belongs in the evaluation phase, before signing, and answering it well matters more than the choice of platform itself. 

Cloud service provider native architectures

Major cloud providers offer their own managed services for building a lakehouse using native tools: AWS with S3, Redshift, and Athena; Google Cloud with BigQuery and Dataproc; Azure with Synapse. This path integrates tightly with an ecosystem that a team likely already operates in, with auto-scaling and much of the operational overhead handled directly by the provider.

The consideration here is architectural. Building deeply on one provider's native services carries a cost later. A future multi-cloud strategy or a migration to a different provider requires more planning than with a more portable stack underneath. One industrial supply enterprise N-iX worked with optimized computational costs by 15% through this kind of cloud migration, without adding a second platform layer on top.

Team size, timeline, existing cloud commitment, and in-house Iceberg or Spark expertise determine the right path far more reliably than any comparison between the platforms themselves. N-iX works across all three, helping teams choose the path that fits before writing a line of infrastructure code, and building out whichever one they land on.

Why an open data lakehouse works better for AI workloads

AI models need something warehouses were never built to hand over cleanly: massive volumes of diverse data. That means structured transactional records sit alongside unstructured text, images, and logs, all accessible to a training pipeline without a separate export process. A warehouse's rigid schema makes storing that raw variety expensive before a single model gets trained. An unmanaged data lake solves storage cheaply but introduces a different risk. Ungoverned files accumulate into what teams call a data swamp, degrading model accuracy the moment the quality of the training data becomes questionable.

An open lakehouse closes that gap because openness is what lets AI workloads sit on the same governed data as everything else. Otherwise, they get siphoned into a separate, ungoverned pipeline built just for Machine Learning. That's the part a closed platform can't offer. Locking the catalog and compute layers into one vendor means every new AI framework, training engine, or agent framework either has to work around that lock-in, or the data gets duplicated into a separate system built just for it. 

Two specific technical bottlenecks make this concrete.

The first appears in the storage layer. TensorFlow, PyTorch, and XGBoost don't query data the way a BI dashboard does. They process massive datasets through custom, non-SQL code.  Pulling that volume from a data warehouse via a standard ODBC or JDBC connection becomes a serious performance bottleneck, since those connections were designed for query results. An data lakehouse open removes that bottleneck by storing data in open formats like Apache Parquet, which ML libraries read directly off storage, no warehouse query layer in the way. Paired with an in-memory format like Apache Arrow, a training pipeline skips the serialization tax entirely and processes columnar data straight from memory instead of converting formats on every read.

The second shows up once AI agents enter the picture. The common scenario is building AI infrastructure off to the side of the main data platform. That creates an ungoverned stack that answers to none of the access rules the rest of the organization follows. An open data lakehouse closes that gap by treating an agent as a member of the same class of data consumers as a human analyst, governed by the same catalog and the same rules. 

Two different systems do two different jobs here, and it's worth being precise about which is which. The open catalog, Unity Catalog, Apache Polaris, or similar, evaluates the request and denies reads to data the agent isn't authorized to see. A tool like MLflow, meanwhile, records the agent's identity, its tool calls, and the full lineage from raw source data to its final output. Enforcement and tracing are separate jobs. An open lakehouse architecture data platform is what lets both attach cleanly to the same governed data.

Medallion architecture makes this real

Data moves through three progressively refined zones:

• Bronze. Data lands as it arrived from the source system, unmodified. Nothing gets cleaned or dropped at this stage, which matters because AI training sometimes needs a signal that a cleansing process further downstream would have discarded.
• Silver. Data gets cleansed, standardized, and reconciled into a single, trustworthy version, the layer where duplicate records, schema mismatches, and obvious errors get resolved before anything downstream depends on them.
• Gold. Data is aggregated and shaped for a specific consumer, a BI dashboard, a reporting workflow, or a model training pipeline, tailored to the workload's actual needs rather than a generic export.

The traceability this creates matters more for AI than it does for BI. A dashboard showing a wrong number gets noticed and questioned. A model trained on bad data produces confident, plausible-looking predictions that are simply wrong, with no obvious signal that anything went sideways. Medallion zoning gives a team a way to trace a prediction back through gold, into silver, and into the original bronze record when a model's behavior needs investigation. 

Governed data quality and AI accuracy aren't two separate goals here. They're the same requirement, stated for two different audiences. One manufacturing enterprise N-iX worked with saw a 5% improvement in prediction accuracy after moving its ML models onto this kind of governed lakehouse foundation.

None of this needs a closed platform to work. Zoning is a data design pattern applicable to Delta Lake, Iceberg, or Hudi alike. The ODBC bottleneck and the agent governance split are architectural facts independent of vendor. That's why open data lakehouse architecture and AI readiness deserve the same budget line on the roadmap, treated as one investment.

Best practices for building and maintaining an open data lakehouse

Getting the architecture right is the visible half of the work. The other half is the set of decisions that only matter months after launch, when the first small files pile up, the first schema needs to evolve, or the first audit asks a question for which nobody has prepared an answer. Here's what N-iX has learned from running open source data lakehouse builds end-to-end. These are the practices that separate a lakehouse that stays fast and governed from one that quietly turns into the data swamp it was supposed to prevent.

We treat governance as a day-one requirement, never a phase-two cleanup

N-iX's data engineers have seen the same pattern often enough to state it plainly. The single most common reason an open architecture degrades into fragmentation is deferring metadata standards and lineage tracking until after the platform is already in production. By then, every table already carries an inconsistent history to untangle. 

We centralize the catalog from the start, using Apache Polaris or Unity Catalog as the single source of truth, rather than letting each compute engine manage its own permissions. Role-based access control, row-level filtering, and column-level masking get enforced at the catalog layer itself. That way, the same security rules apply no matter which engine a user queries with. That decision, made before a single production table exists, costs a fraction of what retrofitting it costs later.

We automate table maintenance from the start

Open table formats accumulate small files as data streams continuously, and query performance degrades steadily as that accumulation grows. We schedule compaction as a recurring job, OPTIMIZE in Delta Lake, clustering in Apache Hudi, rewriting scattered small files into larger, query-efficient ones on a fixed cadence instead of waiting for a slowdown to trigger it manually. 

Snapshot retention gets the same treatment. Table open lakehouse format solutions retain historical versions for time travel, but those snapshots consume real storage space. We set automated expiration policies, Delta's VACUUM, and Iceberg's snapshot expiration settings to clear unreferenced files before they accumulate into a cost problem nobody budgeted for. Indexing rounds this out. ZORDER BY in Delta Lake physically co-locates related data, and Bloom filters in Iceberg let engines skip entire files that can't contain a match, both shaving real time off queries that would otherwise scan data they never needed to touch.

We choose table format and partitioning by workload

Covered in depth earlier in this guide, but worth restating as a standalone rule we hold ourselves to: Iceberg, Delta Lake, and Hudi each optimize for a different read and write pattern. We match the format to the client's actual workload instead of whichever option happens to be loudest in the market that year. 

Partitioning gets the same discipline. Splitting data by date or region speeds up queries by avoiding full-table scans, but over-partitioning creates its own drag; too many tiny partitions can slow a system down as badly as too few large ones. On Iceberg builds, we lean on hidden partitioning specifically to sidestep this trade-off. The mapping runs automatically in the background, allowing a partition scheme to evolve later without rewriting historical data or breaking existing queries.

We version both infrastructure and data as code

Every environment is deployed via Terraform from day one, including the very first one, alongside GitHub Actions and ArgoCD for the deployment pipeline. We don't retrofit Infrastructure as Code after manual deployments have already accumulated undocumented drift. We extend that same discipline to the data itself, using Git-like branching tools such as Project Nessie or LakeFS to isolate an ETL job or a data quality check on its own table branch. 

Nothing merges into production until it's validated on that branch, so downstream business users and models always see complete, working data. One industrial supply enterprise migrated off on-premises infrastructure using this same branching approach. Its heaviest transformation jobs ran 6x faster afterward, a 12-hour job cut down to about 2.

We validate data quality at ingestion, never at the BI layer

Catching a bad record when it lands is cheap. Catching it three transformations later is expensive. By then, a dashboard or a model has already consumed it, and tracing the error back means retracing every step it passed through. We apply tools like Great Expectations specifically at the ingestion boundary, catching schema drift, null spikes, and type mismatches before they ever reach the silver and gold zones.

N-iX learned the governance lesson directly on a project with a German industrial manufacturer. An early phase treated cataloging as a follow-up task. The team later spent real engineering time reconciling lineage gaps that day-one standards would have prevented. None of these practices are unusual. However, most are simple to state and easy to skip under deadline pressure. That's precisely why N-iX builds to this standard on every data lakehouse open source engagement. 

Is your organization ready for an open data lakehouse? Take the readiness scoreboard

Everything so far covers what an open source lakehouse is and how to build one. It doesn't cover whether now is the right moment for a specific organization to start, and that's a separate question, one that depends less on the technology and more on where a team already stands. N-iX built this scoreboard to answer that, honestly, before a single conversation about architecture or vendors even starts.

Six questions below, scored the same way N-iX scopes an assessment for any client. Score each based on where things stand today, then find your range below.

• If your score is 0–4: the fundamentals need attention before a migration makes sense. Governance gaps and team capacity constraints aren't a verdict against moving to an lakehouse. They're the actual starting project. N-iX works with organizations at this stage through a scoped assessment, closing those gaps first. That way, the migration itself carries far less risk.
• If your score is 5–8: the organization is pilot-ready. Skip the full-scale rebuild and instead pick one workload to run as a real Proof of Concept. N-iX scopes these pilots specifically to let results make the case for scaling further.
• If your score is 9–12: the fundamentals are already solid, and the organization is lakehouse-ready. The only open question now is sequencing: which workload moves first, and which architecture path- DIY, managed, or CSP-native- fits the team building it. N-iX helps map that sequencing before any infrastructure code gets written.

Every architecture decision in this guide, table format, catalog, compute engine, comes down to one thing on implementation day: whether it fits the workload running, or the workload a whiteboard assumed. N-iX has made those decisions on production systems, backed by 200 data engineers, a Databricks partnership, and 23 years of turning legacy data infrastructure into platforms.

FAQ

What is an open data lakehouse?

An open data lakehouse is a data architecture that combines a data warehouse's reliability, ACID transactions, schema enforcement, and fast queries, with a data lake's low-cost, scalable storage, built entirely on open standards. Hence, no layer locks you into one vendor. Storage, table format, catalog, and compute engine all stay interoperable, meaning any compatible tool can read the same data without a rewrite. N-iX designs open data lakehouse architecture specifically so a client owns that interoperability.

How is an open data lakehouse different from a data warehouse?

A data warehouse relies on schema-on-write, structure decided before data lands, which makes it reliable for BI but expensive and rigid for the unstructured data AI and ML workloads need. An open data lakehouse applies warehouse-grade transactions and governance directly on top of low-cost object storage, so structured and unstructured data live in the same system instead of two disconnected ones. That removes the constant ETL copying between a lake and a warehouse, which most two-tier architectures rely on.

Does adopting Apache Iceberg or Delta Lake automatically make a platform open?

No, adopting an open table format is a necessary step but not a sufficient one. A platform can run entirely on open-source table formats and still lock data behind a proprietary catalog, meaning a different compute engine can't read the same tables without an export and rewrite. Real openness requires that the catalog, compute engine, and table maintenance layer remain interoperable, too.

What are the best open source table formats for a data lakehouse?

Apache Iceberg, Delta Lake, and Apache Hudi are the three formats of open source lakehouse that dominate this space, and none is universally better, as each optimizes for a different workload. Iceberg favors multi-engine interoperability at large scale, Delta Lake integrates most deeply with Apache Spark, and Hudi handles frequent upserts and change data capture better than either. N-iX matches the format to the client's actual read and write pattern.

Can an open data lakehouse fully replace a data warehouse?

Yes, for most organizations, the lakehouse's transactional guarantees and query performance now match what a dedicated warehouse offers, while also handling workloads a warehouse can't touch affordably. The migration itself needs to be incremental since undocumented pipelines and legacy transformations rarely surface until the move is already underway. Within open source data lakehouse consulting, N-iX federates the legacy warehouse through a unified catalog first, so governance reaches it before any table physically moves.

How does an open data lakehouse architecture support AI?

An open source lakehouse gives AI workloads direct access to both structured and unstructured data, without the schema restrictions or storage costs that a traditional warehouse would impose on training data. Because every layer stays interoperable, a new training framework or agent tool can attach to the same governed data. Medallion architecture then feeds models a continuous stream of quality-checked data, which is what drives prediction accuracy.