S3 Tables: INSERT fails with "Table with name X does not exist!" after successful ATTACH and SELECT

[bella-wilson]

What happens?

1. On an S3 Tables Catalog
2. Create a table using DuckDB-Iceberg (CREATE TABLE or CREATE TABLE ... AS)
3. INSERT with DuckDB-Iceberg and observe the error

After attaching an S3 Tables bucket, SHOW ALL TABLES and SELECT both work and correctly list/query existing tables. However, INSERT INTO against the same table fails with a catalog error claiming the table doesn't exist even though it clearly does, and DuckDB's own suggestion in the error message points to the exact same qualified name.

This happens both for a preexisting table (created outside the current session, via CloudFormation/other tooling) and for a table created fresh within the same session (CREATE TABLE + INSERT immediately after, in the same transaction).

A separate CREATE TABLE AS SELECT attempt in the same environment throws a more severe internal error (stack trace inside the iceberg extension), suggesting this may be related to how the extension resolves/commits against the catalog during write operations specifically, rather than a simple naming/qualification issue.

INSERT INTO s3_tables_db.default.my_table VALUES (...);
-- Catalog Error: Table with name my_table does not exist!
-- Did you mean "s3_tables_db.default.my_table"?

Update: narrowed down to credential chain, not DuckDB/extension version

I was able to test the exact same setup (DuckDB v1.4.3, iceberg extension 1c0c4c60)
running locally against the same S3 Tables bucket, and it works correctly there
all steps (SELECT, INSERT, CREATE TABLE AS) succeed.

Since the DuckDB version and iceberg extension build are identical in both
environments, this rules out a version specific bug. The key difference between
the two environments is the AWS credential chain:

Failing environment (containerized/EKS pod):

• Credentials obtained via IAM Roles for Service Accounts (IRSA) —
  AWS_ROLE_ARN set, AWS_WEB_IDENTITY_TOKEN_FILE set, no static access key
• Involves a cross-account role assumption (the pod's service account role
  assumes a separate cross-account role via STS)

Working environment (local machine):

• Credentials from AWS SSO static AWS_ACCESS_KEY_ID + session token present,
  AWS_ROLE_ARN unset
• No cross-account role assumption involved

Both environments use CREATE SECRET (TYPE s3, PROVIDER credential_chain), so the
extension picks up whatever's in the environment either way but only the
IRSA + cross-account-assumed credentials path fails on write operations (INSERT,
CREATE TABLE AS), while reads (SELECT, SHOW ALL TABLES) succeed in both.

This suggests the issue may be related to how the extension handles STS-derived,
web-identity-federated, or multi-hop-assumed credentials specifically for write/commit
operations, rather than a general S3 Tables write bug.

Related issues checked

#1146 describes a related sounding lazy catalog-metadata issue (information_schema.columns requiring a prior table scan to populate correctly, per @Tishj's explanation there about lazy metadata population and the catalog not being notified of certain query types). In my case, however, a SELECT against the table already succeeds before the INSERT is attempted so this doesn't appear to be explained by the same "not yet scanned" mechanism, and the failure mode is a hard error rather than incomplete metadata. Flagging in case the underlying catalog notification gap is shared, but wanted to report separately since the trigger and symptom differ.

I also checked #453 and #488, which describe similar sounding S3 Tables write errors, but both were fixed by extension updates that predate my build (iceberg extension build from Dec 2025) so this appears to be a different or still open issue.

To Reproduce

Setup:

INSTALL aws;
INSTALL httpfs;
INSTALL iceberg;

CREATE SECRET (
    TYPE s3,
    PROVIDER credential_chain
);

ATTACH 'arn:aws:s3tables:REGION:ACCOUNT_ID:bucket/BUCKET_NAME' AS s3_tables_db (
    TYPE iceberg,
    ENDPOINT_TYPE s3_tables
);

Step 1 — confirm the table is visible and readable:

SHOW ALL TABLES;

Returns the expected table(s), e.g. my_table, under schema default.

SELECT count(*) FROM s3_tables_db.default.my_table;

Succeeds and returns the correct row count.

Step 2 — attempt an INSERT against the same table:

INSERT INTO s3_tables_db.default.my_table VALUES (1, 'test');

Fails with: Catalog Error: Table with name my_table does not exist!
Did you mean "s3_tables_db.default.my_table"?

Step 3 — retry with unqualified name after USE, same result:

USE s3_tables_db.default;
INSERT INTO my_table VALUES (1, 'test');

Catalog Error: Table with name my_table does not exist!
Did you mean "s3_tables_db.default.my_table"?

Step 4 — same catalog, CREATE TABLE AS instead of INSERT:

CREATE TABLE s3_tables_db.default.new_table AS SELECT 1 AS id, 'test' AS name;

Fails differently, with an internal error rather than a catalog error:

INTERNAL Error: Table could not be created
The stack trace for this internal error originates inside iceberg.duckdb_extension, suggesting the write path (rather than read/scan) is where catalog resolution breaks down.

OS:

Linux, x86_64 (containerized Docker/Kubernetes)

DuckDB Version:

v1.4.3

DuckDB Client:

Node.js (@duckdb/node-bindings-linux-x64)

Hardware:

N/A

Full Name:

Bella Wilson

Affiliation:

Eli Lilly

Did you include all relevant data sets for reproducing the issue?

Not applicable - the reproduction does not require a data set

Did you include all code required to reproduce the issue?

• Yes, I have

Did you include all relevant configuration (e.g., CPU architecture, Python version, Linux distribution) to reproduce the issue?

• Yes, I have

[Tmonster]

Hi @bella-wilson ,

Is this also reproducible with v1.4.5 or v1.5.4?