INTERNAL Error "Vector::Reference used on vector of different type (source VARCHAR[] referenced STRUCT(list VARCHAR))" in IcebergMultiFileReader::FinalizeChunk — list-encoding drift across files

[siemonster]

Versions / environment

• DuckDB 1.5.5 (Python client), macOS arm64
• iceberg extension 45163a28 · httpfs 827222f · aws efa54a9
• Table: AWS Security Lake native source CLOUD_TRAIL_MGMT v2.0 — Glue-managed Iceberg on S3,
  data files written continuously by Security Lake over months (writer-version drift across files),
  read via iceberg_scan('s3://…/metadata/<uuid>.metadata.json') (catalog-free mode)

Symptom

Projecting the whole table (SELECT *, to_json(t)) or the metadata struct column crashes:

INTERNAL Error: Vector::Reference used on vector of different type
(source VARCHAR[] referenced STRUCT(list VARCHAR))

Stack (abridged):

Vector::Reference
(anon)::RemapChildVectors
(anon)::RemapNested
(anon)::RemapStructFunction
ExpressionExecutor::Execute
MultiFileReader::FinalizeChunk
IcebergMultiFileReader::FinalizeChunk
MultiFileFunction<ParquetMultiFileInfo>::MultiFileScan

The struct's current logical type (via DESCRIBE):

STRUCT(product STRUCT("version" VARCHAR, "name" VARCHAR, vendor_name VARCHAR,
       feature STRUCT("name" VARCHAR)),
       event_code VARCHAR, uid VARCHAR, profiles VARCHAR[], "version" VARCHAR)

The error's type pair (VARCHAR[] vs STRUCT("list" VARCHAR)) points at metadata.profiles:
older Security-Lake-written files evidently encode the list such that the reader surfaces it as a
single-field struct named list (parquet 3-level list wrapper surfaced literally), while newer
files/the table schema read as a proper VARCHAR[]. The multi-file remap path References the
child vector rather than converting between the two shapes.

Aggravator

The InternalException invalidates the whole DuckDB instance — every subsequent query on any
connection of that instance fails with FATAL Error: … database has been invalidated because of a previous fatal error. For long-lived embedded readers this turns one bad file into a full restart.

What still works (workarounds we use)

• Explicit column lists excluding the affected struct — every other column reads fine, including
  other nested structs and nested-field filters (for what it's worth, we could not reproduce
  [BUG] Scan filters broken for nested types #760's nested-filter-pushdown crash on this build).
• Child projection works: SELECT metadata.uid, metadata.event_code … is fine — only
  materializing the whole parent struct crashes.

Repro

We cannot share the table (security telemetry), but any Security Lake cloud_trail_mgmt 2.0 table
whose files span several months of SL writer versions should reproduce with SELECT * … LIMIT 1.
Synthetic shape: an Iceberg table with a list<string> field inside a struct, where data files
disagree on the parquet list element encoding/naming across commits; project the parent struct.
Happy to run diagnostics against our table on request.

[Tishj]

I think we'd need synthetic data that reproduces this.
It sounds like the iceberg schema has VARCHAR[] type for the column, and the mapper (which is part of duckdb core) has a bug where at bind-time the code is too lenient and constructs an expression that errors at execution. That should either throw a regular binder error, or somehow map "correctly", I don't see how it could map STRUCT("list" VARCHAR) to VARCHAR[] however.

I was able to write a test on a uncommitted branch of duckdb/duckdb that has extended read_parquet schema support to define child columns in the schema: https://github.com/Tishj/duckdb/tree/map_struct_field_to_varchar_list

Test is test/parquet/test_parquet_schema_struct_to_list.test:

# name: test/parquet/test_parquet_schema_struct_to_list.test
# description: Test mapping a Parquet STRUCT field to a LIST
# group: [parquet]

require parquet

statement ok
COPY (
	SELECT struct_pack(
		profiles := struct_pack(list := 'profile')
	) AS event
) TO '{TEMP_DIR}/struct_to_list.parquet'

statement ok
SELECT event.profiles
FROM read_parquet(
	'{TEMP_DIR}/struct_to_list.parquet',
	schema=map {
		'event': ({
			name: 'event',
			type: 'STRUCT(profiles VARCHAR[])',
			default_value: NULL,
			children: map {
				'profiles': ({
					name: 'profiles',
					type: 'VARCHAR[]',
					default_value: NULL,
					children: map {
						'list': ({
							name: 'list',
							type: 'VARCHAR',
							default_value: NULL
						})::VARIANT
					}
				})::VARIANT
			}
		})::VARIANT
	}
)

Which produces:

INTERNAL Error: Invalid PhysicalType for GetTypeIdSize

in reldebug

[siemonster]

Thanks — here's a self-contained synthetic reproducer, plus a data point that I think localises it.

Reproducer (no AWS, no proprietary data; needs duckdb, pyarrow, pyiceberg[sql], sqlalchemy):
inlined below as repro_1270.py. It builds a local SQL-catalog Iceberg table whose schema is

metadata STRUCT(uid VARCHAR, profiles VARCHAR[])

with two data files that disagree on the physical encoding of metadata.profiles:

• file A — annotated 3-level LIST (profiles reads back as VARCHAR[])
• file B — same leaf paths (metadata.profiles.list.element) but written as plain nested groups
  with no LIST logical type, so it reads back as STRUCT("list" STRUCT("element" VARCHAR))

Then:

SELECT id            FROM iceberg_scan(<meta>)  -- OK
SELECT metadata.uid  FROM iceberg_scan(<meta>)  -- OK  (child projection works)
SELECT metadata      FROM iceberg_scan(<meta>)  -- INTERNAL Error (below)
SELECT *             FROM iceberg_scan(<meta>)  -- same, then instance is invalidated

Output on duckdb 1.5.5 / iceberg 45163a28:

[SELECT metadata] InternalException: INTERNAL Error: Vector::Reference used on vector of different
type (source VARCHAR[] referenced STRUCT(list VARCHAR))
[SELECT *]        FatalException: FATAL Error: Failed: database has been invalidated because of a
previous fatal error. The database must be restarted prior to being used again.

which is the original error verbatim.

Note on how the divergence is created: the script registers file B via add_files and then rewrites
its bytes in place with the unannotated encoding. That's a shortcut to manufacture the condition —
pyiceberg deliberately refuses to register a file whose arrow schema doesn't match the table
(_check_pyarrow_schema_compatible), and its add_files path→field-id mapping also rejects legacy
list.item naming outright (KeyError: '<col>.list.item'). In the real table (AWS Security Lake
cloud_trail_mgmt 2.0) nobody overwrote anything — files written months apart by different Security
Lake writer versions simply disagree. The essential condition either way is: the table schema says
list<string> and some data file's physical encoding reads back as a struct.

The data point that may localise it: the same divergence through plain read_parquet does NOT
crash — it produces exactly the clean binder error you described:

BinderException: Binder Error: Can't change source type (STRUCT(list VARCHAR)) to target type
(VARCHAR[]), type conversion not allowed

(reproduced with two parquet files where one has profiles VARCHAR[] and the other
profiles STRUCT(list VARCHAR), read via read_parquet([...], union_by_name=true) and also with
default binding — script probe_core.py.)

So core's generic multi-file path already rejects this mismatch at bind time; it's the iceberg
reader's schema mapping that accepts it and then fails at execution in
IcebergMultiFileReader::FinalizeChunk → RemapStructFunction. Whatever check read_parquet gets
for free seems not to be applied on the iceberg path.

Happy to run anything else against the real table.

repro_1270.py — the Iceberg reproducer (prints REPRODUCED, exits 0)

#!/usr/bin/env python3
"""Self-contained reproducer for duckdb/duckdb-iceberg#1270.

Builds a local (SQL-catalog) Iceberg table whose schema is

    metadata STRUCT(uid VARCHAR, profiles VARCHAR[])

with TWO data files that disagree on the physical encoding of `metadata.profiles`:

  file A: annotated 3-level LIST            -> reads back as VARCHAR[]
  file B: same leaf paths, but written as plain nested groups with no LIST logical type
                                            -> reads back as STRUCT("list" STRUCT("element" VARCHAR))

Then duckdb's iceberg_scan projects the parent struct.

Expected (the bug): INTERNAL Error: Vector::Reference used on vector of different type
(source VARCHAR[] referenced STRUCT(list VARCHAR)) raised from
IcebergMultiFileReader::FinalizeChunk -> RemapStructFunction; a following query then hits
"database has been invalidated".

NOTE on how the divergence is manufactured: file B is written schema-conformant, registered
via add_files, and then its bytes are rewritten in place with the unannotated encoding.
That is a shortcut — pyiceberg deliberately refuses to register a non-conforming file
(_check_pyarrow_schema_compatible), and its path->field-id mapping also rejects legacy
`list.item` naming (KeyError: '<col>.list.item'). In the real table (AWS Security Lake
cloud_trail_mgmt 2.0) nothing was overwritten: files written months apart by different
writer versions simply disagree. The essential condition is the same either way — the table
schema says list<string> while some data file's physical encoding reads back as a struct.

No AWS, no network, no proprietary data.
Requires: duckdb, pyarrow, pyiceberg[sql], sqlalchemy.
"""
import os, shutil, sys, tempfile

import duckdb
import pyarrow as pa
import pyarrow.parquet as pq
from pyiceberg.catalog.sql import SqlCatalog

WAREHOUSE = os.path.join(tempfile.gettempdir(), "iceberg1270")
shutil.rmtree(WAREHOUSE, ignore_errors=True)
os.makedirs(f"{WAREHOUSE}/data", exist_ok=True)

# ---- 1. table schema: a LIST nested inside a STRUCT (the real-world shape) --------------------
# metadata STRUCT(uid VARCHAR, profiles VARCHAR[])
SCHEMA = pa.schema([("id", pa.int64()),
                    ("metadata", pa.struct([("uid", pa.string()),
                                            ("profiles", pa.list_(pa.string()))]))])

# File B's physical shape: the SAME leaf paths (metadata.profiles.list.element) but written as
# plain nested GROUPS instead of an annotated LIST — i.e. a writer that emitted the 3-level
# structure without the LIST logical type. This is what makes duckdb read the column as
# STRUCT(...) while the Iceberg schema says VARCHAR[]. Matching leaf paths is also what lets
# pyiceberg's add_files() column-path->field-id mapping accept the file.
SCHEMA_UNANNOTATED = pa.schema([("id", pa.int64()),
                                ("metadata", pa.struct([("uid", pa.string()),
                                                        ("profiles", pa.struct([("list", pa.struct([("element", pa.string())]))]))]))])

# ---- 2. two data files: one annotated LIST, one unannotated group ----------------------------
p_list = f"{WAREHOUSE}/data/a_annotated_list.parquet"
pq.write_table(pa.table({"id": [1, 2],
                         "metadata": [{"uid": "row-1", "profiles": ["cloud_profile", "datetime_profile"]},
                                      {"uid": "row-2", "profiles": ["cloud_profile"]}]},
                        schema=SCHEMA), p_list)
p_group = f"{WAREHOUSE}/data/b_unannotated_group.parquet"
# written schema-conformant FIRST so pyiceberg will register it (it validates on add_files);
# its bytes are swapped for the unannotated-group encoding below, simulating a writer that
# changed physical encoding over time while the Iceberg schema stayed put.
pq.write_table(pa.table({"id": [3, 4],
                         "metadata": [{"uid": "row-3", "profiles": ["cloud_profile"]},
                                      {"uid": "row-4", "profiles": ["datetime_profile"]}]},
                        schema=SCHEMA), p_group)
files = [p_list, p_group]
for p in files:
    leaves = [c.path for c in
              [pq.ParquetFile(p).metadata.schema.column(i) for i in range(pq.ParquetFile(p).metadata.num_columns)]]
    print(f"[repro] wrote {os.path.basename(p):28} leaves={leaves}")

# ---- 3. register BOTH files into one Iceberg table (add_files = no rewrite) --------------------
catalog = SqlCatalog(
    "repro",
    **{"uri": f"sqlite:///{WAREHOUSE}/catalog.db", "warehouse": f"file://{WAREHOUSE}"},
)
catalog.create_namespace_if_not_exists("db")
tbl = catalog.create_table("db.t", schema=SCHEMA)
tbl.add_files([f"file://{p}" for p in files])
meta = tbl.metadata_location
print(f"[repro] iceberg table built, 2 data files, metadata: {meta}")

# ---- 3b. swap file B's bytes for the UNANNOTATED-group encoding (same path, same row count) ---
# Simulates a writer emitting a different physical encoding for the same Iceberg field over time.
pq.write_table(pa.table({"id": [3, 4],
                         "metadata": [{"uid": "row-3", "profiles": {"list": {"element": "cloud_profile"}}},
                                      {"uid": "row-4", "profiles": {"list": {"element": "datetime_profile"}}}]},
                        schema=SCHEMA_UNANNOTATED), p_group)
print("[repro] file B re-encoded in place: metadata.profiles is now a plain group "
      "(reads as STRUCT), while the Iceberg schema still says list<string>")

# ---- 4. read it back through duckdb's iceberg_scan --------------------------------------------
con = duckdb.connect()
con.execute("INSTALL iceberg; LOAD iceberg;")
EXT_V = con.execute(
    "SELECT extension_version FROM duckdb_extensions() WHERE extension_name='iceberg'"
).fetchone()[0]
print(f"[repro] duckdb {duckdb.__version__} | iceberg {EXT_V}")
scan = f"iceberg_scan('{meta}')"

def attempt(label, sql):
    try:
        out = con.execute(sql).fetchall()
        print(f"  [{label}] OK -> {out}")
        return True
    except Exception as e:
        print(f"  [{label}] {type(e).__name__}: {str(e).splitlines()[0][:160]}")
        return False

print("[repro] scalar column (control, expected OK):")
attempt("SELECT id", f"SELECT id FROM {scan} ORDER BY id")

print("[repro] child projection (expected OK — the documented workaround):")
attempt("SELECT metadata.uid", f"SELECT metadata.uid FROM {scan} ORDER BY 1")

print("[repro] PARENT STRUCT projection (expected: the #1270 crash):")
crashed = not attempt("SELECT metadata", f"SELECT metadata FROM {scan}")

print("[repro] SELECT * (expected: same crash):")
attempt("SELECT *", f"SELECT * FROM {scan}")

print("\n[repro] RESULT:", "REPRODUCED" if crashed else "NOT reproduced on this build")
sys.exit(0 if crashed else 2)

probe_core.py — the read_parquet contrast (clean BinderException)

#!/usr/bin/env python3
"""Probe: does the #1270 remap crash reproduce WITHOUT iceberg, via plain multi-file
read_parquet? The crash frame (MultiFileReader::FinalizeChunk -> RemapStructFunction) is
duckdb core, shared by read_parquet and iceberg_scan.

File A: metadata STRUCT(uid VARCHAR, profiles VARCHAR[])            <- annotated LIST
File B: metadata STRUCT(uid VARCHAR, profiles STRUCT(list VARCHAR)) <- unannotated group,
        which is exactly the type named in the crash message.
"""
import os, tempfile
import duckdb, pyarrow as pa, pyarrow.parquet as pq

D = os.path.join(tempfile.gettempdir(), "core1270")
os.makedirs(D, exist_ok=True)

A = pa.schema([("id", pa.int64()),
               ("metadata", pa.struct([("uid", pa.string()), ("profiles", pa.list_(pa.string()))]))])
B = pa.schema([("id", pa.int64()),
               ("metadata", pa.struct([("uid", pa.string()),
                                       ("profiles", pa.struct([("list", pa.string())]))]))])

pa_path = f"{D}/a_list.parquet"
pb_path = f"{D}/b_structlist.parquet"
pq.write_table(pa.table({"id": [1], "metadata": [{"uid": "a", "profiles": ["cloud", "datetime"]}]},
                        schema=A), pa_path)
pq.write_table(pa.table({"id": [2], "metadata": [{"uid": "b", "profiles": {"list": "cloud"}}]},
                        schema=B), pb_path)
print("[probe] file A profiles = VARCHAR[]   | file B profiles = STRUCT(list VARCHAR)")

con = duckdb.connect()
V = con.execute("SELECT version()").fetchone()[0]
print(f"[probe] duckdb {V}")

def go(label, sql):
    try:
        print(f"  [{label}] OK -> {con.execute(sql).fetchall()}")
        return False
    except Exception as e:
        print(f"  [{label}] {type(e).__name__}: {str(e).splitlines()[0][:170]}")
        return True

files = f"['{pa_path}', '{pb_path}']"
crashed = False
crashed |= go("union_by_name, SELECT metadata", f"SELECT metadata FROM read_parquet({files}, union_by_name=true)")
con2 = duckdb.connect()
crashed |= go("union_by_name, SELECT *", f"SELECT * FROM read_parquet({files}, union_by_name=true)")
con3 = duckdb.connect()
try:
    print("  [default (no union_by_name)]", con3.execute(f"SELECT metadata FROM read_parquet({files})").fetchall())
except Exception as e:
    print(f"  [default (no union_by_name)] {type(e).__name__}: {str(e).splitlines()[0][:170]}")
    crashed = True

print("\n[probe] core-only reproduction:", "YES" if crashed else "NO — needs the iceberg path")

[Tishj]

I managed to reproduce it
It looks like it's caused by a mismatch between the type of the default for the column, and the result type:

(lldb) p remap.default_index
(const duckdb::optional_idx) $13 = (index = 0)

(lldb) p defaults[0].Print()
CONSTANT STRUCT(list VARCHAR): 2 = [ {'list': NULL}]

(lldb) p result_vectors[i].get().type.ToString()
(std::string) $7 = "VARCHAR[]"

frame #2: 0x000000012efb9d68 libduckdb.dylib`duckdb::(anonymous namespace)::RemapChildVectors(result=0x00006070000a6ad8, input_vectors=0x00000001700b1960, result_vectors=0x00000001700b19e0, remap_info=0x000060300036d5a8, default_vector=0x000060e0000b8658, has_top_level_null=false, count=2) at remap_struct.cpp:73:28
   70                   // primitive type remap
   71                   if (remap.default_index.IsValid()) {
   72                           auto &defaults = StructVector::GetEntries(default_vector);
-> 73                           result_vectors[i].get().Reference(defaults[remap.default_index.GetIndex()]);
   74                           if (result_vectors[i].get().GetVectorType() != VectorType::CONSTANT_VECTOR) {
   75                                   throw InternalException("Default value in remap struct must be a constant");
   76                           }

So there's some blind trust in the mapper that's likely the problem

[siemonster]

That lines up exactly with what the reproducer produces, and I think it localises the bug a little further.

STRUCT(list VARCHAR) in your defaults[0] is file B's physical shape — the data file written as plain nested groups with no LIST logical annotation. VARCHAR[] in result_vectors[i] is the Iceberg schema's declared type for metadata.profiles. So the two values you printed are the two halves of the divergence the reproducer manufactures, seen from inside the remap.

The part I'd flag: this is reached via remap.default_index.IsValid(), i.e. the default-substitution path — but there is no Iceberg initial-default on this field. The column exists in both files; they just disagree on physical encoding. So the mapper appears to conclude "this field is absent from the file" (reasonably — the annotated-list shape it expects isn't there) and falls back to substituting a default.

That suggests the defect is where the default vector's type comes from, rather than the substitution itself. defaults[0] is a NULL constant typed STRUCT(list VARCHAR) — the source file's type. A default standing in for a missing VARCHAR[] column should surely be typed from the result/Iceberg schema type (VARCHAR[] NULL), in which case the Reference() at remap_struct.cpp:73 would be type-correct and the whole thing would degrade to nulls instead of failing. If that's right, the "blind trust" is upstream of line 73 — the default is constructed against the wrong schema.

Two smaller notes:

• count=2 matches the reproducer's two rows (one per file), so the vector you're looking at is the file-B row plus the file-A row remapped together — consistent with the failure only appearing once both files are scanned. Scanning file A alone is fine.
• Whatever the fix, a type assertion before the Reference() would turn this class of mismatch into a diagnosable error rather than the current failure — useful independently, since any future mapper bug that produces a mistyped default would otherwise land in the same place with no indication of which column or which file.

For context on prioritisation: this isn't synthetic in origin. It surfaced on AWS Security Lake OCSF data, where some files in a table carry that unannotated encoding — the synthetic reproducer was only needed because the original data isn't shareable. So it's an interop case against a fairly widely used producer, not a contrived edge case.

[Tishj]

Yea, I'm gonna have to look further into it than duckdb/duckdb#24433 it seems. My hopes were that if the file-column and result-column have to be of the same type in the lookup, we can be sure that the default is correctly typed as a result.

But I agree that there's likely a bug there where somehow the file-column's default is used instead of the result-column