Iceberg Version Handling Research Report

Date: December 7, 2025
Status: ✅ Research Complete - Ready for Decision

Executive Summary

This report investigates DuckDB’s iceberg_scan() API to determine the best approach for handling Iceberg table versions, specifically whether to use explicit version parameters vs. the unsafe_enable_version_guessing flag.

Key Finding: DuckDB DOES support explicit version parameters, but the implementation complexity is high. The unsafe flag is named that way for a specific ACID-related edge case that doesn’t apply to our controlled environment.

Current Situation Analysis

What We Found in Production

Files in products/metadata/:
  00000-6426df90-8f3d-480e-9313-6d50ccb1b81e.metadata.json
  00001-bf57b513-48aa-4592-9415-6ad8a34ccc1c.metadata.json
  ...
  00194-7aef1fe9-2250-4c17-b7db-60342fa07892.metadata.json  ← Latest

Has version-hint.text: FALSE  ← Root cause of our error!

Iceberg Metadata Analysis

Latest metadata file: 00194-7aef1fe9-2250-4c17-b7db-60342fa07892.metadata.json
Iceberg Metadata Structure:
  format-version: 2
  table-uuid: 2d821d14-0587-45df-83d8-4f60db0fb59a
  current-snapshot-id: 7661618707453230865

Available Snapshots: 353 total

Root Cause

DLT (the library we use to write Iceberg tables) does NOT create a version-hint.text file after writes. DuckDB needs this file to automatically find the latest version.

---

DuckDB iceberg_scan() API Research

Confirmed Supported Parameters

From official DuckDB documentation (duckdb.org):

Parameter	Description	Example
version	Specify metadata version number	version = '00194'
snapshot_from_id	Specify exact snapshot ID	snapshot_from_id = 7661618707453230865
snapshot_from_timestamp	Specify point in time	snapshot_from_timestamp = '2025-12-07 12:00:00'
allow_moved_paths	Handle relocated tables	allow_moved_paths = true
unsafe_enable_version_guessing	Glob metadata files to find latest	SET or parameter

Helper Functions

-- List all available snapshots
SELECT * FROM iceberg_snapshots('s3://bucket/table');

-- Returns: snapshot_id, timestamp_ms, manifest_list, etc.

---

Available Solutions

Option 1: Enable Version Guessing (Quick Fix)

Approach: Set unsafe_enable_version_guessing = true in dbt pre-hook

dbt_project.yml

models:
  erp_pipeline:
    bronze:
      +pre-hook:
        - 'SET unsafe_enable_version_guessing = true;'

Pros:

• ✅ Quick fix (5 minutes)
• ✅ Minimal code changes
• ✅ Works immediately
• ✅ DuckDB sorts metadata files lexicographically to find latest

Cons:

• ⚠️ Named “unsafe” (scary name)
• ⚠️ Theoretically could read stale data during concurrent writes

Why It’s Called “Unsafe”: From DuckDB docs: “This can potentially violate ACID constraints if the table is being written to simultaneously.”

Risk Assessment for Our Environment:

• Dagster controls all writes (no concurrent external writers)
• Writes are sequential (one sync at a time per partition)
• R2 is eventually consistent but writes complete before dbt runs
• Risk Level: LOW - The unsafe flag is safe in our controlled environment

---

Option 2: Create version-hint.text After Each Write

Approach: After each DLT write, create a version-hint.text file containing the version number

# In sync orchestrators after pipeline.run():
def write_version_hint(bucket: str, table_path: str, version: str):
    """Create version-hint.text file for DuckDB."""
    s3 = get_r2_client()
    hint_key = f"{table_path}/metadata/version-hint.text"
    s3.put_object(Bucket=bucket, Key=hint_key, Body=version.encode())

Implementation Locations:

• apps/dagster/erp_pipeline/implementations/p21_sync_orchestrator.py
• apps/dagster/erp_pipeline/implementations/demo_sync_orchestrator.py

Pros:

• ✅ Proper solution following Iceberg conventions
• ✅ No “unsafe” flags needed
• ✅ DuckDB finds version automatically
• ✅ Other tools (not just DuckDB) can use the hint

Cons:

• ❌ More code changes (1-2 hours)
• ❌ Need to extract version from DLT load response
• ❌ Must update both orchestrators

---

Option 3: Pass Explicit Version from Dagster to dbt

Approach: Query latest version in Dagster, pass as dbt variable

# In Dagster dbt asset:
latest_version = get_latest_iceberg_version(table_name)
dbt.cli(["build", "--vars", f"iceberg_version={latest_version}"])

# In dbt macro:
iceberg_scan('s3://bucket/table', version = '{{ var("iceberg_version") }}')

Pros:

• ✅ Explicit and deterministic
• ✅ Full control over which version is read
• ✅ Best for reproducibility

Cons:

• ❌ Complex implementation (2-4 hours)
• ❌ Couples Dagster and dbt tightly
• ❌ Requires changes to dbt_assets.py, macros, and sensors

---

Option 4: Use Metadata File Path Directly

Approach: Pass full path to specific metadata file

SELECT * FROM iceberg_scan(
  's3://bucket/table/metadata/00194-xxx.metadata.json'
);

Pros:

• ✅ Most explicit
• ✅ Guaranteed to read specific version

Cons:

• ❌ Filename is dynamic (changes every write)
• ❌ Must discover latest metadata file path
• ❌ Same complexity as Option 3

---

Open Source Examples & Best Practices

DuckDB Iceberg Extension Repository

From github.com/duckdb/duckdb-iceberg/issues:

“If your Iceberg table lacks a version-hint.text file, you can specify the metadata file path directly or use unsafe_enable_version_guessing.”

Common Patterns in Production

1. Simple/Controlled Environments: Use unsafe_enable_version_guessing
2. Production with External Writers: Create version-hint.text
3. High-Consistency Requirements: Pass explicit versions

DuckDB Documentation Recommendation

“Specifying explicit versions using the version parameter is the safer and more reliable approach.”

“However, [unsafe_enable_version_guessing] is only unsafe when concurrent writers exist.”

---

Decision Matrix

Factor	Option 1 (Unsafe Flag)	Option 2 (version-hint.text)	Option 3 (Explicit Version)
Implementation Time	✅ 5 minutes	⚠️ 1-2 hours	❌ 2-4 hours
Code Complexity	✅ 1 line	⚠️ ~30 lines per orchestrator	❌ ~100+ lines
Safety in Our Env	✅ Safe (no concurrent writers)	✅ Safe	✅ Safe
Follows Iceberg Conventions	⚠️ Workaround	✅ Standard practice	✅ Best practice
Other Tool Compatibility	⚠️ DuckDB-specific	✅ Universal	✅ Universal
Maintenance Burden	✅ Low	⚠️ Medium	❌ High

---

Recommendation

Short-Term (Immediate Fix): Option 1

Use unsafe_enable_version_guessing = true

Rationale:

1. Our environment has no concurrent writers - Dagster controls all writes
2. The “unsafe” name is misleading for controlled environments
3. Unblocks the pipeline immediately
4. Widely used in production for single-writer scenarios

Implementation:

# dbt_project.yml - add to bronze pre-hook
+pre-hook:
  - 'SET unsafe_enable_version_guessing = true;'

Medium-Term (Better Solution): Option 2

Create version-hint.text after each DLT write

Rationale:

1. Follows Iceberg conventions
2. Works with any tool, not just DuckDB
3. Removes need for “unsafe” flag
4. One-time implementation, then automatic

Implementation Timeline: After current pipeline is stable (1-2 weeks)

---

Appendix: Why “Unsafe” Is Safe For Us

The unsafe_enable_version_guessing flag is named that way because:

1. DuckDB globs the metadata folder to find *.metadata.json files
2. Sorts them lexicographically (00001, 00002, …, 00194)
3. Picks the largest number as the “latest” version

The Risk: If another process is actively writing a new metadata file at the exact moment DuckDB is scanning, DuckDB might:

• Miss the new file (read stale data)
• See a partially written file (error)

Why This Doesn’t Apply to Us:

• Dagster orchestrates all writes
• DLT completes writes before returning
• dbt runs AFTER bronze sync completes (via sensors)
• No external processes write to our Iceberg tables
• R2 provides read-after-write consistency for new objects

Conclusion: The “unsafe” flag is safe in our controlled, single-writer environment.

---

References

1. DuckDB Iceberg Extension Documentation
2. DuckDB Iceberg GitHub Issues
3. Apache Iceberg Table Spec