OSError: 500 Internal Server Error when writing Delta table to OneLake using deltalake (delta-rs) Python library

[nitheesh1398]

When using write_deltalake() from the deltalake Python package to overwrite a Delta table stored in OneLake (Microsoft Fabric Lakehouse), we get a persistent 500 Internal Server Error after 10 retries.

Error:
OSError: Generic MicrosoftAzure error: Error after 10 retries in 4.429440819s, max_retries:10, retry_timeout:180s, source:HTTP status server error (500 Internal Server Error) for url (https://onelake.blob.fabric.microsoft.com//.Lakehouse/Tables/<table_name>/_delta_log/00000000000000000015.json)

Stack trace:
File ~/jupyter-env/python3.11/lib/python3.11/site-packages/deltalake/writer.py:556, in write_deltalake(...)
555 else:
--> 556 table._table.create_write_transaction(
557 add_actions,
558 mode,
559 partition_by or [],
560 schema,
561 partition_filters,
562 custom_metadata,
563 )

OSError: Generic MicrosoftAzure error: Error after 10 retries in 4.429440819s, max_retries:10, retry_timeout:180s, source:HTTP status server error (500 Internal Server Error) for url (https://onelake.blob.fabric.microsoft.com/.../_delta_log/00000000000000000015.json)

Environment:

Platform: Microsoft Fabric Notebook (Jupyter kernel)
Python: 3.11
deltalake (delta-rs): installed via pip in Fabric notebook environment
Storage: OneLake (abfss:// path)
Table type: Managed Delta table in a Fabric Lakehouse

Sample Code:

from deltalake import DeltaTable, write_deltalake
import pyarrow as pa
import pandas as pd

delta_path = "abfss://@onelake.dfs.fabric.microsoft.com/.Lakehouse/Tables/<table_name>"

Read existing table

dt = DeltaTable(delta_path, storage_options={"azure_storage_account_name": "onelake"})
df = dt.to_pandas()

Modify dataframe (e.g., append a row)

new_row = pd.DataFrame({'column_name': ['new_value']})
updated_df = pd.concat([df, new_row], ignore_index=True)

Convert and write back

arrow_table = pa.Table.from_pandas(updated_df)
write_deltalake(
delta_path,
data=arrow_table,
mode='overwrite'
)

^^^ This fails with 500 Internal Server Error

What we tried:

Retrying multiple times — same error consistently
Confirmed no concurrent writes/reads on the table
Table is accessible for reads via DeltaTable().to_pandas() — works fine
The error occurs specifically on the _delta_log commit (JSON write), not on data file writes

Can one please support here