Skip to content

Lite: fix compaction OOM by setting DuckDB temp_directory (#933) #935

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
erikdarlingdata merged 1 commit into from
May 5, 2026
Merged

Lite: fix compaction OOM by setting DuckDB temp_directory (#933) #935

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
12 changes: 10 additions & 2 deletions Lite/Services/ArchiveService.cs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -322,6 +322,14 @@ Each group gets its own DuckDB connection so memory is fully released between gr
var totalMerged = 0;
var totalRemoved = 0;

/* Spill directory for the in-memory compaction connections. Without this,
the memory_limit pragma is a hard wall — DuckDB has nowhere to spill and
OOMs the moment the cap is hit. Co-locating with the archive keeps the
write on the same volume the parquet files already live on. */
var spillDir = Path.Combine(_archivePath, "duckdb_tmp");
Directory.CreateDirectory(spillDir);
var spillDirSql = spillDir.Replace("\\", "/");

foreach (var ((month, table), files) in groups)
{
/* If there's exactly one file and it's already in monthly format, skip */
Expand Down Expand Up @@ -376,7 +384,7 @@ Each group gets its own DuckDB connection so memory is fully released between gr
con.Open();
using (var pragma = con.CreateCommand())
{
pragma.CommandText = "SET memory_limit = '4GB'; SET preserve_insertion_order = false;";
pragma.CommandText = $"SET memory_limit = '4GB'; SET preserve_insertion_order = false; SET temp_directory = '{EscapeSqlPath(spillDirSql)}';";
pragma.ExecuteNonQuery();
}

Expand Down Expand Up @@ -407,7 +415,7 @@ Sort smallest-first so early merges are cheap. */
con.Open();
using (var pragma = con.CreateCommand())
{
pragma.CommandText = "SET memory_limit = '4GB'; SET preserve_insertion_order = false;";
pragma.CommandText = $"SET memory_limit = '4GB'; SET preserve_insertion_order = false; SET temp_directory = '{EscapeSqlPath(spillDirSql)}';";
pragma.ExecuteNonQuery();
}

Expand Down
Loading