ゼロコピー：ETLの終焉とAIの未来

It’s the dirty secret of the AI boom: the models are brilliant, but the plumbing is ancient.

You build a state-of-the-art transformer model. You secure H100 GPU clusters. But then you wait 24 hours for the ETL (Extract, Transform, Load) pipeline to run so your model can actually see the new sales data. By the time the inference runs, the customer has already churned.

This is the “Data Friction Tax,” and it is killing AI ROI.

In 2025, the industry is finally fixing the plumbing. Engineering teams are moving from the era of Data Warehousing to the era of Unified Data Platforms, specifically architectures built around a “Zero-Copy” philosophy. If you’re building AI systems, you need to understand why copying data is becoming an architectural anti-pattern.

The Physics of the Problem: Why ETL Scales Poorly

To understand why Zero-Copy matters, one must look at the inefficiency of traditional stacks.

In a standard enterprise, data lives effectively in “gravity wells”—Salesforce for CRM, SAP for ERP, AWS S3 for logs. To analyze this data, engineers historically built pipelines to physically copy it from Source A to Destination B (usually a Data Warehouse like Redshift or Snowflake).

This introduces a fundamental latency equation:

$$\text{Latency} = T_{extract} + T_{transfer} + T_{load} + T_{indexing}$$

Every time you copy data, you introduce:

1. Latency: The “stale data” problem. If the pipeline runs nightly, the AI is always 24 hours behind reality.
2. Cost: Enterprises pay specific storage costs for the duplicate. Storing 1PB of data is manageable; storing 5 copies of it (Raw, Bronze, Silver, Gold, Warehouse) is exorbitant.
3. Serialization Overhead: The CPU cost of standardizing JSON/CSV into Parquet/Avro (SerDes) consumes massive amounts of compute that could be used for inference.
4. Drift: The schema in Source A changes (e.g., a developer renames user_id to uuid), breaking the pipeline to Destination B.

Dataversity warns that poor data quality, often stemming from these fragile pipelines, could cause 60% of AI projects to be abandoned in the coming year. The solution isn’t faster pipelines; it’s no pipelines.

The Evolution of Data Gravity

To appreciate the Zero-Copy revolution, it helps to map the trajectory of data architecture over the last two decades. The industry has oscillated between coupling and decoupling.

Phase 1: The Monolith (1990-2010)

In the beginning, there was the Oracle Database. Storage and compute were tightly coupled. If you wanted to run a faster query, you bought a bigger box. It was consistent and fast (ACID compliant), but it couldn’t scale horizontally. It choked on the volume of web-scale data.

Phase 2: The Data Lake / The “Swamp” (2010-2018)

The Hadoop era introduced HDFS. The philosophy was “Schema on Read.” Just dump all the JSON logs into cheap storage and figure it out later. This solved the storage cost problem but created a “swamp.” Query performance was terrible, and without transactions, data integrity vanished. AI models trained on this data frequently hallucinated because the input data was garbage or incomplete.

Phase 3: The Cloud Warehouse (2018-2023)

Snowflake and BigQuery separated storage (S3/GCS) from compute. This was a breakthrough. You could scale storage infinitely and spin up compute clusters on demand. However, the format was still proprietary. To use Snowflake, you had to COPY INTO Snowflake. The data was locked in their micro-partitions. You still had to move the bytes.

Phase 4: The Zero-Copy Lakehouse (2024+)

This is the current inflection point. The data stays in S3, but in open formats (like Apache Iceberg). The compute engine (Snowflake, Spark, Trino, Dremio) visits the data where it lives. There is no COPY INTO. There is only SELECT * FROM.

Technical Deep Dive: How Zero-Copy Actually Works

“Zero-Copy” is a misnomer. The bits exist on a disk somewhere. Architecturally, however, it means the system does not move the bytes to the compute; it brings the compute permissions to the bytes (or specifically, shares the pointers).

Modern “Open Table Formats” like Apache Iceberg, Delta Lake, and Apache Hudi are the enablers here. They allow different compute engines to look at the same files in object storage without needing to “own” them.

The Metadata Layer

The magic happens in the metadata. Instead of copying a 10TB table, a Zero-Copy system shares a manifest file: a list of pointers to the Parquet files sitting in S3.

When an engineer “clones” a database in Snowflake or creates a branch in Dremio Arctic, the system doesn’t duplicate the 10TB. It duplicates the metadata (kilobytes) and points it to the same underlying storage blocks.

The Mechanics of Isolation: Zero-Copy relies heavily on Snapshot Isolation.

1. The Manifest List points to a specific “Snapshot” (e.g., Snapshot S1).
2. Snapshot S1 points to a set of Manifest Files.
3. Manifest Files point to the actual Data Files (Parquet).

When an AI model starts training at 10:00 AM, it locks onto Snapshot S1. If an ETL job updates the table at 10:05 AM, it creates Snapshot S2 (writing new Parquet files and a new Manifest). The AI model continues to read S1 undisturbed. There are no locks, and no readers blocking writers.

Query Pushdown Optimization: The analytic engine pushes filters to the source database. If the query asks for SELECT * WHERE region = 'Western-Region', the source system uses the metadata to identify which specific micro-partitions contain ‘Western-Region’ data. It only returns those specific blocks. This effectively reduces network transfer by orders of magnitude compared to a full table scan.

The “Unistore” Concept

The ultimate goal of Zero-Copy is the unification of Transactional (OLTP) and Analytical (OLAP) workloads, a concept often called HTAP (Hybrid Transactional/Analytical Processing).

Snowflake’s Unistore and Databricks’ Lakehouse paradigms aim to bridge this gap. Imagine a retail application where a purchase is written to a transactional row-store. In a traditional world, that row wouldn’t appear in the analytics warehouse until the nightly batch job. In a Zero-Copy Unistore world, that row is immediately visible to the analytical engine via a unified table abstraction.

For AI, this is game-changing. Recommend systems can react to a user’s click within the same session, not just the next day.

The 2026 Reality: “Living Ecosystems”

This architectural shift is not just about saving disk space. It is about organizational agility.

Slalom’s 2026 Financial Services Outlook predicts a shift from “static, siloed solutions” to “living, adaptive ecosystems” where data flows freely. They identify Unified Data Foundations as the prerequisite for this. If a bank’s fraud detection AI has to wait for a nightly batch job, it is useless against real-time 2026 threats. The report emphasizes that “governance enables both speed and confidence,” suggesting that the metadata layer will become the new control plane.

McKinsey’s Tech Trends reinforce this, highlighting that the “execution gap” in AI is largely a data availability problem. Their analysis suggests that organizations capable of acting on real-time insights are 1.6x more likely to see double-digit growth. The ability to query data “in-place” allows for the creation of Data Products that are instantly consumable by other teams without the friction of setting up new pipelines.

The Rise of the “Headless” Data Architecture

Throughout 2026, analysts expect to see the dominance of “Headless” Data Architectures. In this model, the storage and the semantic layer are completely decoupled from the consumption tool.

• Storage: S3/Azure Blob (Cheap, Infinite).
• Format: Iceberg/Delta (Open, Transactional).
• Catalog: Nessie/Unity Catalog (The “Brain” keeping track of pointers).
• Compute: Whatever the user wants. The Data Scientist uses PySpark; the Analyst uses SQL; the CEO uses a Dashboard. They all hit the same Single Source of Truth.

Conclusion: The Platform is the Pipeline

For the data engineer, the skill set is shifting. Writing efficient PySpark ETL scripts to move data from A to B is becoming less valuable than designing robust metadata governance strategies. The value isn’t in moving data; it’s in making data accessible without moving it.

The pipeline, as traditionally understood, is dead. Long live the platform.