Daft UDF Dynamic Scaling Requirement

[dujl]

Currently, Daft UDF does not support dynamic scaling, which becomes a bottleneck in long-chain pipeline execution. Its fixed instance count causes insufficient resources during peak loads (leading to task backlogs) and idle resources during low loads, resulting in low resource utilization, wasted cluster resources and limited pipeline efficiency.

We expect Daft UDF to support dynamic scaling, which can automatically adjust the number of UDF instances in real time based on the actual load of long-chain pipelines. It should expand instances when the load rises to avoid resource shortage and shrink instances when the load drops to release idle resources, thus maximizing resource utilization and improving pipeline efficiency.

To achieve this, four key steps are proposed: 1. Build a real-time load monitoring system to collect core indicators for scaling decisions. 2. Formulate scientific scaling strategies, defining trigger thresholds and step sizes. 3. Integrate scaling logic into Daft’s scheduling module to ensure stable execution without affecting existing tasks. 4. Conduct comprehensive testing and optimization to ensure the function is stable and meets business needs.

[srilman]

@dujl could you maybe provide an example as to when you see that UDFs don't scale up? I ask because in the new @daft.func UDF decorator, there is a parameter max_concurrency that does support dynamic scaling up to a max number of instances. Are you using this or the older @daft.udf?