update

Author: adriangb

datafusion/sqllogictest/test_files/information_schema.slt

@@ -275,6 +275,7 @@ datafusion.explain.show_statistics false
 datafusion.optimizer.allow_symmetric_joins_without_pruning true
 datafusion.optimizer.default_filter_selectivity 20
 datafusion.optimizer.enable_distinct_aggregation_soft_limit true
+datafusion.optimizer.enable_dynamic_filter_pushdown true
 datafusion.optimizer.enable_round_robin_repartition true
 datafusion.optimizer.enable_topk_aggregation true
 datafusion.optimizer.expand_views_at_output false
@@ -375,6 +376,7 @@ datafusion.explain.show_statistics false When set to true, the explain statement
 datafusion.optimizer.allow_symmetric_joins_without_pruning true Should DataFusion allow symmetric hash joins for unbounded data sources even when its inputs do not have any ordering or filtering If the flag is not enabled, the SymmetricHashJoin operator will be unable to prune its internal buffers, resulting in certain join types - such as Full, Left, LeftAnti, LeftSemi, Right, RightAnti, and RightSemi - being produced only at the end of the execution. This is not typical in stream processing. Additionally, without proper design for long runner execution, all types of joins may encounter out-of-memory errors.
 datafusion.optimizer.default_filter_selectivity 20 The default filter selectivity used by Filter Statistics when an exact selectivity cannot be determined. Valid values are between 0 (no selectivity) and 100 (all rows are selected).
 datafusion.optimizer.enable_distinct_aggregation_soft_limit true When set to true, the optimizer will push a limit operation into grouped aggregations which have no aggregate expressions, as a soft limit, emitting groups once the limit is reached, before all rows in the group are read.
+datafusion.optimizer.enable_dynamic_filter_pushdown true When set to true attempts to push down dynamic filters generated by operators into the file scan phase. For example, for a query such as `SELECT * FROM t ORDER BY timestamp DESC LIMIT 10`, the optimizer will attempt to push down the current top 10 timestamps that the TopK operator references into the file scans. This means that if we already have 10 timestamps in the year 2025 any files that only have timestamps in the year 2024 can be skipped / pruned at various stages in the scan.
 datafusion.optimizer.enable_round_robin_repartition true When set to true, the physical plan optimizer will try to add round robin repartitioning to increase parallelism to leverage more CPU cores
 datafusion.optimizer.enable_topk_aggregation true When set to true, the optimizer will attempt to perform limit operations during aggregations, if possible
 datafusion.optimizer.expand_views_at_output false When set to true, if the returned type is a view type then the output will be coerced to a non-view. Coerces `Utf8View` to `LargeUtf8`, and `BinaryView` to `LargeBinary`.

datafusion/sqllogictest/test_files/regexp/regexp_count.slt

@@ -341,4 +341,4 @@ statement ok
 drop table t_stringview;
 
 statement ok
-drop table empty_table;
+drop table empty_table;

datafusion/sqllogictest/test_files/regexp/regexp_like.slt

@@ -277,4 +277,4 @@ statement ok
 drop table strings
 
 statement ok
-drop table dict_table
+drop table dict_table

datafusion/sqllogictest/test_files/regexp/regexp_match.slt

@@ -198,4 +198,4 @@ NULL
 query B
 select null !~* 'abc';
 ----
-NULL
+NULL

datafusion/sqllogictest/test_files/regexp/regexp_replace.slt

@@ -126,4 +126,4 @@ select
 from (values ('a'), ('b')) as tbl(col);
 ----
 NULL NULL NULL
-NULL NULL NULL
+NULL NULL NULL