profiles.yml Setting Overrides on-run-start Hooks - Investigation Report

Date: December 7, 2025
Status: 🔴 Issue Identified - Awaiting Fix Approval

Executive Summary

The on-run-start hooks ARE executing (logs show “Finished running 3 project hooks”), but the SET unsafe_enable_version_guessing = true; command is being overridden by the profiles.yml setting which explicitly sets it to false.

Evidence

Logs Show Hooks Running

Finished running 3 project hooks, 25 data tests, 14 view models in 0 hours 0 minutes and 1.34 seconds (1.34s).

profiles.yml Overrides the Setting

# profiles.yml (current)
erp_pipeline:
  target: dev
  outputs:
    dev:
      type: duckdb
      settings:
        # This takes precedence over SET commands!
        unsafe_enable_version_guessing: false # ← PROBLEM

Root Cause

DuckDB’s settings in profiles.yml are applied at connection creation time and persist for the entire session. The SET command in on-run-start cannot override a setting that was already configured via the profile.

Order of Operations:

1. dbt creates DuckDB connection using profiles.yml settings
2. unsafe_enable_version_guessing: false is set at connection level
3. on-run-start hooks execute SET unsafe_enable_version_guessing = true;
4. But the connection-level setting already determined the behavior
5. Model SQL executes and fails because the setting remains false

Solution

Change profiles.yml to set unsafe_enable_version_guessing: true

This is the authoritative location for DuckDB settings. The on-run-start hooks can be kept for defense-in-depth but the profile setting is what actually controls the behavior.

File: apps/dagster/dbt/profiles.yml

# Change from:
unsafe_enable_version_guessing: false

# To:
unsafe_enable_version_guessing: true

Both the dev and prod profiles need this change.

Recommended Fix

erp_pipeline:
  target: dev
  outputs:
    dev:
      type: duckdb
      path: 'erp_pipeline.duckdb'
      extensions:
        - httpfs
        - iceberg
      config_options:
        allow_unsigned_extensions: true
      settings:
        s3_region: auto
        s3_endpoint: '{{ ... }}'
        s3_access_key_id: '{{ ... }}'
        s3_secret_access_key: '{{ ... }}'
        # Version guessing is safe in our single-writer environment
        # (Dagster controls all writes, no concurrent external writers)
        # See: debt/bug-report-analysis/iceberg-version-handling-research-2025-12-07.md
        unsafe_enable_version_guessing: true # ← CHANGE THIS

    prod:
      type: duckdb
      path: '/tmp/erp_pipeline.duckdb'
      extensions:
        - httpfs
        - iceberg
      config_options:
        allow_unsigned_extensions: true
      settings:
        s3_region: auto
        s3_endpoint: '{{ ... }}'
        s3_access_key_id: '{{ ... }}'
        s3_secret_access_key: '{{ ... }}'
        # See: debt/bug-report-analysis/iceberg-version-handling-research-2025-12-07.md
        unsafe_enable_version_guessing: true # ← CHANGE THIS

Optional Cleanup

After fixing profiles.yml, the on-run-start hooks in dbt_project.yml can optionally be removed since the profile setting handles it. However, keeping them doesn’t hurt and provides defense-in-depth.

Files to Modify

1. apps/dagster/dbt/profiles.yml - Change unsafe_enable_version_guessing: false to true (both dev and prod profiles)