Skip to content

Use a concurrent buffer deque in FragmentConsolidation. #5700

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
bekadavis9 wants to merge 18 commits into
base: main
Choose a base branch
from
Open

Use a concurrent buffer deque in FragmentConsolidation. #5700

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
18 commits
Select commit Hold shift + click to select a range
ce277b4
Use a concurrent buffer deque in FragmentConsolidation.
bekadavis9 Nov 24, 2025
3ab6232
Minor changes.
bekadavis9 Nov 25, 2025
3efa3be
Wait for read task to finish when writer fails, finalize in copy_array.
bekadavis9 Dec 5, 2025
7d41804
Add missing header.
bekadavis9 Dec 5, 2025
d0968de
Add back test which ensures consolidation can progress.
bekadavis9 Dec 5, 2025
6e3ff56
Address most comments.
bekadavis9 Dec 8, 2025
04fecba
Add additional test coverage.
bekadavis9 Dec 11, 2025
191a11d
Minor test updates.
bekadavis9 Dec 22, 2025
963d2d5
Some test changes and debug WIP.
bekadavis9 Jan 12, 2026
0e6611a
More updates
bekadavis9 Jan 15, 2026
51b108c
Fix TRY macro error check
rroelke Jan 15, 2026
4cddfc2
Use TRY for tiledb_array_consolidate so that Catch2 prints error
rroelke Jan 15, 2026
cff1000
Update test.
bekadavis9 Jan 16, 2026
89ab72d
Address comments.
bekadavis9 Jan 21, 2026
cd4a640
Fix maybe-unintialized std::optional false positive in TRY macro
rroelke Jan 22, 2026
6fc39a8
Merge remote-tracking branch 'origin' into rd/core-411
bekadavis9 Jan 25, 2026
2646824
Add intercept test, some debug WIP.
bekadavis9 Jan 28, 2026
d759930
Add recycled buffer queue and other safeguards for speedup and to pre…
bekadavis9 Feb 10, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 3 additions & 1 deletion test/src/unit-capi-config.cc
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@
*
* The MIT License
*
* @copyright Copyright (c) 2017-2025 TileDB Inc.
* @copyright Copyright (c) 2017-2026 TileDB Inc.
*
* Permission is hereby granted, free of charge, to any person obtaining a copy
* of this software and associated documentation files (the "Software"), to deal
Expand Down Expand Up @@ -277,6 +277,7 @@ void check_save_to_file() {
<< "\n";
ss << "sm.max_tile_overlap_size 314572800\n";
ss << "sm.mem.consolidation.buffers_weight 1\n";
ss << "sm.mem.consolidation.initial_buffer_size 10485760\n";
ss << "sm.mem.consolidation.reader_weight 3\n";
ss << "sm.mem.consolidation.writer_weight 2\n";
ss << "sm.mem.malloc_trim true\n";
Expand Down Expand Up @@ -653,6 +654,7 @@ TEST_CASE("C API: Test config iter", "[capi][config]") {
all_param_values["sm.query.condition_evaluator"] =
Config::SM_QUERY_CONDITION_EVALUATOR;
all_param_values["sm.query.sparse_unordered_with_dups.reader"] = "refactored";
all_param_values["sm.mem.consolidation.initial_buffer_size"] = "10485760";
all_param_values["sm.mem.consolidation.buffers_weight"] = "1";
all_param_values["sm.mem.consolidation.reader_weight"] = "3";
all_param_values["sm.mem.consolidation.writer_weight"] = "2";
Expand Down
Loading
Loading