SSL Issue Take 3

[mattmartin14]

What happens?

Hi,
I had previously worked with you all on a potential SSL issue that I found on duckdb version 1.5 and above, but I was able to resolve the problem as I was not directing my s3 calls through our corp aws private link vpc. But now, I'm seeing an issue where when the duckdb iceberg extension attempts to write a data file to s3, it does not appear its utilizing the S3 vpc endpoint specified in the aws s3 secrets I setup.

To Reproduce

For context, this script works on my local workstation at my corp office to read and write data to s3:

cn = duckdb.connect()
        cn.execute("install aws; load aws;")

        vpc = "#######"
        bucket = "#####"
        cn.execute(f"""create secret aws_s3 (
                    type s3, 
                provider credential_chain,
                region 'us-east-1',
                use_ssl true,
                endpoint 'bucket.{vpc}.s3.us-east-1.vpce.amazonaws.com',
                url_style 'path'
                )
                
        """)
        cn.execute(f"copy (select 1 as id) to 's3://{bucket}/test_table/data/test.csv' (format csv, header true);")

        cn.sql(f"select * from read_csv_auto('s3://{bucket}/test_table/data/test.csv');").show()

However, the following Iceberg code with ducdb only works up to the create of the table; the insert statement fails

#works
cn.execute(f"""
            create or replace secret aws_s3 (
                type s3,
                provider credential_chain,
                region '{region}',
                use_ssl true,
                endpoint 'bucket.{s3_vpc_endpoint_id}.s3.{region}.vpce.amazonaws.com',
                url_style 'path'
            )
        """)

#works
cn.execute(f"""
            attach '{aws_account_id}' as glue_catalog (
                type iceberg,
                endpoint '{glue_endpoint_override}',
                authorization_type 'sigv4',
                support_stage_create false,
                secret aws_s3
            )
        """)

# works
cn.execute(f"""
        create table glue_catalog.{glue_db_name}.{table_name} (id integer, val varchar)
        with ('location' = '{s3_path}')
    """)

## does not work
cn.execute(f"""
        insert into glue_catalog.{glue_db_name}.{table_name}
        values (1, 'a'), (2, 'b')
    """)

Error message on the insert attempt:

IO Error: SSL peer certificate or SSH remote key was not OK error for HTTP PUT to 'https://#######.s3.us-east-1.amazonaws.com/duckdb/icehouse/test_iceberg_stage_3bc7834b/data/019de4c6-0cb6-7cca-af1e-80ebdf6175db.parquet'

OS:

Mac OS Tahoe 26.4.1

DuckDB Version:

1.5.2

DuckDB Client:

python

Hardware:

Mac M2 Pro

Full Name:

Matt Martin

Affiliation:

State Farm

Did you include all relevant data sets for reproducing the issue?

Yes

Did you include all code required to reproduce the issue?

• Yes, I have

Did you include all relevant configuration (e.g., CPU architecture, Python version, Linux distribution) to reproduce the issue?

• Yes, I have

[mattmartin14]

hi @Tmonster - wanted to check with you to see if you had any insights into this issue?

[Tmonster]

No insights currently. I believe you also filed an issue on duckdb/duckdb? Has that gotten any more movement?

[mattmartin14]

No insights currently. I believe you also filed an issue on duckdb/duckdb? Has that gotten any more movement?

Hi,
I did have an issue opened on duckdb core, but what we found out is that between duckdb version 1.4 and 1.5, a change was made on how SSL was routed through S3 and vpc endpoints. Thus, when I made this adjustment to my aws secrets, I could read/write standard parquet files in s3 on version 1.5 and above:

cn.execute(f"""
            create or replace secret aws_s3 (
                type s3,
                provider credential_chain,
                region '{region}',
                use_ssl true,
                endpoint 'bucket.{s3_vpc_endpoint_id}.s3.{region}.vpce.amazonaws.com',
                url_style 'path'
            )
        """)

so then, a command such as this works:

copy (select 42 as val) to 's3://bucket/folder/data.parquet'

However, for the DuckDB iceberg extension, I don't think its using the vpc endpoint that we specify in our aws s3 secrets when we attempt to write a data file.

Does that make sense?

Thanks,
Matt