Skip to content

Use Vec for low card term aggregations #2737

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
wants to merge 2 commits into from
Closed

Use Vec for low card term aggregations #2737

wants to merge 2 commits into from

Conversation

@PSeitz
Copy link
Collaborator

@PSeitz PSeitz commented Nov 19, 2025 •
edited
Loading

Use Vec for low card term aggregations if:

  • Column is of type String
  • Column has less than 1024 terms (best number unconfirmed)
  • Does not use missing (could be resolved)
  • is a top level aggregation (to safeguard against pre-allocating too much in many sub-buckets).
full
average_u64                                    Memory: 13.8 KB (-2.88%)     Avg: 3.1754ms (-0.18%)      Median: 3.1602ms (-0.63%)      [3.1283ms .. 3.3447ms]
average_f64                                    Memory: 14.3 KB (+0.07%)     Avg: 3.1763ms (-0.81%)      Median: 3.1631ms (-0.90%)      [3.1367ms .. 3.3253ms]
average_f64_u64                                Memory: 15.8 KB (+1.30%)     Avg: 5.8605ms (-0.00%)      Median: 5.8578ms (+0.22%)      [5.7943ms .. 5.9461ms]
stats_f64                                      Memory: 14.7 KB (+0.08%)     Avg: 3.1632ms (-1.02%)      Median: 3.1574ms (-0.80%)      [3.1338ms .. 3.2281ms]
extendedstats_f64                              Memory: 14.4 KB (+1.47%)     Avg: 4.3156ms (-0.69%)      Median: 4.3137ms (-0.47%)      [4.2798ms .. 4.3670ms]
percentiles_f64                                Memory: 17.8 KB (+1.17%)     Avg: 9.2986ms (-0.48%)      Median: 9.2648ms (-0.42%)      [9.1982ms .. 9.6412ms]
terms_few                                      Memory: 27.5 KB (+0.23%)     Avg: 1.1747ms (-46.34%)     Median: 1.1669ms (-46.45%)     [1.1525ms .. 1.2771ms]
terms_many                                     Memory: 6.9 MB (+0.35%)      Avg: 6.0952ms (+0.14%)      Median: 6.0925ms (+0.16%)      [6.0475ms .. 6.2274ms]
terms_many_top_1000                            Memory: 8.0 MB (+17.21%)     Avg: 7.5642ms (+1.04%)      Median: 7.5550ms (+1.08%)      [7.4856ms .. 7.8294ms]
terms_many_order_by_term                       Memory: 6.9 MB (+0.36%)      Avg: 7.4959ms (+0.49%)      Median: 7.4875ms (+0.40%)      [7.4370ms .. 7.5943ms]
terms_many_with_top_hits                       Memory: 58.5 MB (+0.31%)     Avg: 123.9062ms (+3.50%)    Median: 121.3925ms (+2.47%)    [118.3412ms .. 156.1855ms]
terms_many_with_avg_sub_agg                    Memory: 20.8 MB (+0.82%)     Avg: 43.1272ms (-0.63%)     Median: 43.0778ms (-0.77%)     [40.9254ms .. 44.1274ms]
terms_few_with_avg_sub_agg                     Memory: 31.4 KB (-0.48%)     Avg: 8.4319ms (-25.60%)     Median: 8.4168ms (-25.56%)     [8.2111ms .. 8.7222ms]
terms_many_json_mixed_type_with_avg_sub_agg    Memory: 30.2 MB (+0.54%)     Avg: 59.4458ms (-0.23%)     Median: 59.5619ms (-0.29%)     [58.3219ms .. 61.0412ms]
cardinality_agg                                Memory: 3.6 MB               Avg: 17.6700ms (-0.44%)     Median: 17.6734ms (-0.54%)     [17.3332ms .. 18.1094ms]
terms_few_with_cardinality_agg                 Memory: 10.6 MB (-0.00%)     Avg: 66.8774ms (-6.58%)     Median: 66.8457ms (-6.71%)     [66.6454ms .. 67.1766ms]
range_agg                                      Memory: 15.8 KB              Avg: 3.8148ms (-0.45%)      Median: 3.8083ms (-0.34%)      [3.7831ms .. 3.8548ms]
range_agg_with_avg_sub_agg                     Memory: 30.3 KB              Avg: 11.1320ms (-0.33%)     Median: 11.1001ms (-0.47%)     [11.0300ms .. 11.3840ms]
range_agg_with_term_agg_few                    Memory: 43.7 KB              Avg: 16.2399ms (-0.74%)     Median: 16.2198ms (-0.63%)     [16.1433ms .. 16.4861ms]
range_agg_with_term_agg_many                   Memory: 6.9 MB (+0.36%)      Avg: 25.7224ms (-0.39%)     Median: 25.6567ms (-0.45%)     [25.4444ms .. 26.8263ms]
histogram                                      Memory: 16.2 KB (+1.98%)     Avg: 4.9566ms (+0.62%)      Median: 4.9350ms (+0.47%)      [4.8819ms .. 5.3776ms]
histogram_hard_bounds                          Memory: 14.0 KB (-0.97%)     Avg: 3.6590ms (+0.80%)      Median: 3.6547ms (+0.75%)      [3.6147ms .. 3.7391ms]
histogram_with_avg_sub_agg                     Memory: 47.8 KB              Avg: 13.5733ms (-0.19%)     Median: 13.5528ms (-0.21%)     [13.4644ms .. 13.7869ms]
histogram_with_term_agg_few                    Memory: 310.4 KB (+0.05%)    Avg: 24.7707ms (-0.08%)     Median: 24.7557ms (+0.05%)     [24.6134ms .. 24.9984ms]
avg_and_range_with_avg_sub_agg                 Memory: 25.3 KB              Avg: 13.8600ms (-0.04%)     Median: 13.8084ms (-0.15%)     [13.7142ms .. 14.2719ms]
dense
average_u64                                    Memory: 15.1 KB (+1.39%)     Avg: 7.1696ms (+0.81%)      Median: 7.1600ms (+0.91%)      [7.0610ms .. 7.4624ms]
average_f64                                    Memory: 14.5 KB (-0.01%)     Avg: 7.1991ms (+0.92%)      Median: 7.1574ms (+0.66%)      [7.0578ms .. 7.9417ms]
average_f64_u64                                Memory: 16.4 KB (-2.48%)     Avg: 13.8088ms (+0.52%)     Median: 13.8150ms (+0.75%)     [13.6326ms .. 13.9686ms]
stats_f64                                      Memory: 14.7 KB (+2.90%)     Avg: 7.2393ms (+1.62%)      Median: 7.1938ms (+1.14%)      [7.0710ms .. 8.2380ms]
extendedstats_f64                              Memory: 15.1 KB (-1.34%)     Avg: 8.3388ms (+0.75%)      Median: 8.3383ms (+0.93%)      [8.2290ms .. 8.5260ms]
percentiles_f64                                Memory: 19.5 KB (+7.05%)     Avg: 13.4671ms (+1.24%)     Median: 13.4249ms (+1.04%)     [13.2413ms .. 14.1539ms]
terms_few                                      Memory: 28.9 KB (+1.12%)     Avg: 5.1917ms (-16.23%)     Median: 5.1829ms (-16.21%)     [5.0998ms .. 5.3918ms]
terms_many                                     Memory: 6.9 MB (+0.36%)      Avg: 9.9859ms (-1.59%)      Median: 9.9834ms (-1.09%)      [9.9077ms .. 10.1329ms]
terms_many_top_1000                            Memory: 8.0 MB (+17.22%)     Avg: 11.5132ms (-0.72%)     Median: 11.4991ms (-0.36%)     [11.3721ms .. 11.9563ms]
terms_many_order_by_term                       Memory: 6.9 MB (+0.37%)      Avg: 11.4093ms (-1.03%)     Median: 11.4092ms (-0.97%)     [11.3350ms .. 11.5237ms]
terms_many_with_top_hits                       Memory: 58.5 MB (+0.31%)     Avg: 130.5140ms (-1.02%)    Median: 128.7428ms (+1.13%)    [124.6096ms .. 162.7296ms]
terms_many_with_avg_sub_agg                    Memory: 20.8 MB (+0.82%)     Avg: 51.0843ms (-1.23%)     Median: 50.5884ms (-1.87%)     [49.3957ms .. 54.0742ms]
terms_few_with_avg_sub_agg                     Memory: 33.1 KB (-0.46%)     Avg: 14.7029ms (-19.26%)    Median: 14.6638ms (-19.44%)    [14.2017ms .. 15.7396ms]
terms_many_json_mixed_type_with_avg_sub_agg    Memory: 30.2 MB (+0.54%)     Avg: 64.0253ms (+0.22%)     Median: 63.6709ms (-0.04%)     [61.5907ms .. 80.1777ms]
cardinality_agg                                Memory: 3.6 MB               Avg: 21.7676ms (-0.95%)     Median: 21.7326ms (-0.56%)     [21.4455ms .. 22.6784ms]
terms_few_with_cardinality_agg                 Memory: 10.6 MB (-0.00%)     Avg: 76.4266ms (-6.94%)     Median: 76.3464ms (-6.95%)     [75.7922ms .. 77.6580ms]
range_agg                                      Memory: 15.7 KB (-1.03%)     Avg: 7.8789ms (+0.71%)      Median: 7.8572ms (+0.49%)      [7.7254ms .. 8.1233ms]
range_agg_with_avg_sub_agg                     Memory: 31.2 KB              Avg: 18.7227ms (+0.17%)     Median: 18.7301ms (+0.33%)     [18.4624ms .. 19.0455ms]
range_agg_with_term_agg_few                    Memory: 45.1 KB              Avg: 25.2619ms (+0.19%)     Median: 25.2161ms (+0.93%)     [24.9242ms .. 25.8797ms]
range_agg_with_term_agg_many                   Memory: 6.9 MB (+0.36%)      Avg: 35.4822ms (+0.49%)     Median: 35.3800ms (+0.65%)     [34.9936ms .. 36.9570ms]
histogram                                      Memory: 16.1 KB (+2.84%)     Avg: 9.0547ms (+0.93%)      Median: 8.9391ms (+0.12%)      [8.7754ms .. 10.8449ms]
histogram_hard_bounds                          Memory: 13.4 KB (-2.85%)     Avg: 7.7920ms (+1.78%)      Median: 7.7174ms (+1.03%)      [7.5089ms .. 8.8231ms]
histogram_with_avg_sub_agg                     Memory: 47.9 KB              Avg: 20.8664ms (+0.65%)     Median: 20.7675ms (+0.34%)     [20.5748ms .. 22.6784ms]
histogram_with_term_agg_few                    Memory: 311.8 KB (+0.05%)    Avg: 33.5612ms (+0.34%)     Median: 33.4091ms (+0.02%)     [33.2135ms .. 34.9118ms]
avg_and_range_with_avg_sub_agg                 Memory: 28.7 KB (+3.70%)     Avg: 25.2989ms (-0.05%)     Median: 25.2275ms (+0.15%)     [25.0010ms .. 26.4287ms]
sparse
average_u64                                    Memory: 14.2 KB (-1.45%)     Avg: 13.5206ms (-0.22%)    Median: 13.5019ms (-0.25%)    [13.3120ms .. 13.7094ms]
average_f64                                    Memory: 13.3 KB (-1.54%)     Avg: 13.5311ms (-0.38%)    Median: 13.5243ms (-0.30%)    [13.3807ms .. 13.7415ms]
average_f64_u64                                Memory: 15.1 KB (+2.83%)     Avg: 25.4099ms (+0.14%)    Median: 25.3828ms (+0.05%)    [25.2365ms .. 25.7054ms]
stats_f64                                      Memory: 13.6 KB (-1.51%)     Avg: 13.4951ms (-0.76%)    Median: 13.4971ms (-0.83%)    [13.3374ms .. 13.6698ms]
extendedstats_f64                              Memory: 14.4 KB (+1.44%)     Avg: 13.5422ms (-0.64%)    Median: 13.5423ms (-0.51%)    [13.2786ms .. 13.8569ms]
percentiles_f64                                Memory: 16.3 KB (+1.29%)     Avg: 13.4165ms (+0.47%)    Median: 13.3724ms (+0.12%)    [13.1352ms .. 13.7180ms]
terms_few                                      Memory: 27.2 KB (+0.24%)     Avg: 12.9262ms (-0.79%)    Median: 12.9306ms (-0.76%)    [12.7619ms .. 13.1455ms]
terms_many                                     Memory: 1.8 MB (+1.36%)      Avg: 13.7324ms (-0.60%)    Median: 13.7234ms (-0.63%)    [13.6005ms .. 13.8809ms]
terms_many_top_1000                            Memory: 3.0 MB (+13.47%)     Avg: 15.0214ms (-0.06%)    Median: 14.9590ms (-0.43%)    [14.7214ms .. 16.7719ms]
terms_many_order_by_term                       Memory: 1.8 MB (+1.36%)      Avg: 13.9700ms (-0.41%)    Median: 13.9684ms (-0.40%)    [13.8727ms .. 14.1340ms]
terms_many_with_top_hits                       Memory: 13.3 MB (+1.41%)     Avg: 21.1450ms (+0.20%)    Median: 21.0505ms (-0.17%)    [20.9084ms .. 23.8429ms]
terms_many_with_avg_sub_agg                    Memory: 5.7 MB (+3.07%)      Avg: 17.5125ms (-0.33%)    Median: 17.4951ms (-0.40%)    [17.3125ms .. 17.7442ms]
terms_few_with_avg_sub_agg                     Memory: 31.7 KB (-0.48%)     Avg: 14.6634ms (-2.55%)    Median: 14.6555ms (-2.50%)    [14.4712ms .. 14.8489ms]
terms_many_json_mixed_type_with_avg_sub_agg    Memory: 5.6 MB (+3.11%)      Avg: 24.7218ms (-0.50%)    Median: 24.6940ms (-0.50%)    [24.5165ms .. 25.0219ms]
cardinality_agg                                Memory: 896.3 KB             Avg: 19.2530ms (-0.76%)    Median: 19.2654ms (-0.51%)    [19.0069ms .. 19.4433ms]
terms_few_with_cardinality_agg                 Memory: 680.4 KB (-0.04%)    Avg: 23.5404ms (-1.48%)    Median: 23.3843ms (-2.14%)    [23.1825ms .. 26.0060ms]
range_agg                                      Memory: 15.8 KB              Avg: 12.9648ms (-0.48%)    Median: 12.9395ms (-0.70%)    [12.7562ms .. 13.8627ms]
range_agg_with_avg_sub_agg                     Memory: 30.1 KB              Avg: 15.0666ms (-0.59%)    Median: 15.0575ms (-0.62%)    [14.7832ms .. 15.3740ms]
range_agg_with_term_agg_few                    Memory: 43.7 KB              Avg: 15.5305ms (-0.52%)    Median: 15.5281ms (-0.50%)    [15.4049ms .. 15.6824ms]
range_agg_with_term_agg_many                   Memory: 1.8 MB (+1.36%)      Avg: 16.6593ms (-0.55%)    Median: 16.6632ms (-0.69%)    [16.4955ms .. 16.8260ms]
histogram                                      Memory: 14.1 KB (+3.21%)     Avg: 13.1321ms (-0.48%)    Median: 13.1422ms (-0.57%)    [12.9767ms .. 13.2924ms]
histogram_hard_bounds                          Memory: 12.7 KB (-1.39%)     Avg: 13.1151ms (+0.21%)    Median: 13.1015ms (+0.12%)    [12.9443ms .. 13.3672ms]
histogram_with_avg_sub_agg                     Memory: 34.3 KB              Avg: 15.2114ms (-0.14%)    Median: 15.1723ms (-0.42%)    [14.9686ms .. 16.6574ms]
histogram_with_term_agg_few                    Memory: 192.7 KB (+0.08%)    Avg: 16.1567ms (-0.11%)    Median: 16.1472ms (-0.22%)    [15.9662ms .. 16.3148ms]
avg_and_range_with_avg_sub_agg                 Memory: 24.8 KB              Avg: 26.7548ms (+0.09%)    Median: 26.7751ms (+0.16%)    [26.5586ms .. 26.9363ms]
multivalue
average_u64                                    Memory: 15.1 KB              Avg: 10.0380ms (+1.54%)     Median: 10.0303ms (+1.51%)     [9.8721ms .. 10.4650ms]
average_f64                                    Memory: 15.1 KB (-1.36%)     Avg: 10.0250ms (+1.42%)     Median: 9.9913ms (+1.08%)      [9.8316ms .. 10.4795ms]
average_f64_u64                                Memory: 18.3 KB (-2.22%)     Avg: 19.5115ms (+0.69%)     Median: 19.4985ms (+0.55%)     [19.3221ms .. 20.0693ms]
stats_f64                                      Memory: 16.0 KB (+5.50%)     Avg: 10.0202ms (+1.24%)     Median: 9.9645ms (+0.73%)      [9.7970ms .. 10.7373ms]
extendedstats_f64                              Memory: 15.4 KB (-2.62%)     Avg: 11.2154ms (+2.03%)     Median: 11.1890ms (+1.90%)     [11.0563ms .. 11.9582ms]
percentiles_f64                                Memory: 18.7 KB (-1.10%)     Avg: 16.3808ms (+0.91%)     Median: 16.3653ms (+1.25%)     [16.0730ms .. 16.7402ms]
terms_few                                      Memory: 244.3 KB             Avg: 8.7805ms (-10.94%)     Median: 8.7674ms (-10.77%)     [8.6650ms .. 8.9529ms]
terms_many                                     Memory: 6.9 MB (+0.36%)      Avg: 12.9970ms (-0.30%)     Median: 13.0031ms (-0.15%)     [12.8939ms .. 13.1285ms]
terms_many_top_1000                            Memory: 8.0 MB (+17.21%)     Avg: 14.4775ms (-0.21%)     Median: 14.4390ms (-0.26%)     [14.3153ms .. 14.8209ms]
terms_many_order_by_term                       Memory: 6.9 MB (+0.36%)      Avg: 14.4496ms (-0.32%)     Median: 14.4183ms (-0.30%)     [14.2888ms .. 15.0792ms]
terms_many_with_top_hits                       Memory: 58.5 MB (+0.31%)     Avg: 138.9345ms (+1.95%)    Median: 135.9366ms (+0.73%)    [132.2174ms .. 177.4881ms]
terms_many_with_avg_sub_agg                    Memory: 20.8 MB (+0.82%)     Avg: 63.7324ms (+1.67%)     Median: 63.7939ms (+2.40%)     [61.0947ms .. 64.9547ms]
terms_few_with_avg_sub_agg                     Memory: 245.7 KB             Avg: 21.0859ms (-16.28%)    Median: 20.9947ms (-16.30%)    [20.6561ms .. 22.4729ms]
terms_many_json_mixed_type_with_avg_sub_agg    Memory: 30.2 MB (+0.54%)     Avg: 73.1627ms (+1.55%)     Median: 73.2830ms (+2.55%)     [71.1376ms .. 75.4244ms]
cardinality_agg                                Memory: 3.6 MB               Avg: 24.9000ms (-0.65%)     Median: 24.8633ms (-0.97%)     [24.2386ms .. 25.5935ms]
terms_few_with_cardinality_agg                 Memory: 10.8 MB (-0.00%)     Avg: 85.9587ms (-4.32%)     Median: 85.8148ms (-4.12%)     [85.2174ms .. 87.0632ms]
range_agg                                      Memory: 16.0 KB (-2.00%)     Avg: 10.7008ms (+0.10%)     Median: 10.6706ms (+0.03%)     [10.5738ms .. 11.3715ms]
range_agg_with_avg_sub_agg                     Memory: 31.5 KB              Avg: 26.4729ms (-0.44%)     Median: 26.4102ms (-0.25%)     [26.2589ms .. 27.4311ms]
range_agg_with_term_agg_few                    Memory: 247.1 KB             Avg: 33.0424ms (+0.30%)     Median: 32.9898ms (+0.47%)     [32.6962ms .. 34.3517ms]
range_agg_with_term_agg_many                   Memory: 6.9 MB (+0.36%)      Avg: 42.0736ms (-0.32%)     Median: 41.9440ms (-0.24%)     [41.6499ms .. 43.2043ms]
histogram                                      Memory: 15.5 KB (-2.72%)     Avg: 11.8248ms (+0.25%)     Median: 11.8071ms (+0.14%)     [11.6120ms .. 12.3114ms]
histogram_hard_bounds                          Memory: 14.0 KB (+0.04%)     Avg: 10.6006ms (+0.41%)     Median: 10.5876ms (+0.55%)     [10.3574ms .. 11.0011ms]
histogram_with_avg_sub_agg                     Memory: 48.1 KB              Avg: 27.4836ms (+0.22%)     Median: 27.4044ms (+0.16%)     [27.1847ms .. 28.4091ms]
histogram_with_term_agg_few                    Memory: 468.4 KB (+0.03%)    Avg: 41.4490ms (+0.17%)     Median: 41.4112ms (+0.50%)     [41.1392ms .. 42.4159ms]
avg_and_range_with_avg_sub_agg                 Memory: 28.0 KB              Avg: 35.9135ms (+0.50%)     Median: 35.7273ms (+0.07%)     [35.4482ms .. 37.0724ms]

@fulmicoton fulmicoton closed this Dec 1, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@PSeitz @fulmicoton @PSeitz-dd