Skip to content

fix: reduce UDF metrics #2624

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 5 commits into from
May 30, 2025
Merged

fix: reduce UDF metrics #2624

merged 5 commits into from
May 30, 2025

Conversation

tmenjo
Copy link
Contributor

@tmenjo tmenjo commented May 12, 2025

chore: remove unused metrics in reduce UDF

The metric reduce_pnf_partitions_inflight (ReducePartitionsInFlight) had been no longer visited. This commit removes it.

doc: reduce_pbq_write_time that have not been documented in metrics

The metric reduce_pbq_write_time have been visited, but not documented yet. This commit adds it to the Metrics page.

chore: redefine ReduceForwardTime and ReduceProcessTime

There is no metric visited in reduce UDF meaning latency from an upstream ISB to a PBQ, nor from a PBQ to a downstream ISB. I'd say such metrics are useful to monitor pipeline's healthiness and performance.

The two metrics ReduceForwardTime and ReduceProcessTime have been defined but not visited. They look just right, so this commit redefines them. Now ReduceForwardTime means latency from an upstream ISB to a PBQ, and ReduceProcessTime from a PBQ to a downstream ISB.

fix: visit missing latency mertics in reduce UDF

This commit has reduce UDF visit the following metrics:

  • forwarder_read_processing_time
  • forwarder_write_processing_time
  • forwarder_ack_processing_time
  • reduce_data_forward_forward_time (ReduceForwardTime, redefined by the previous commit)
  • reduce_pnf_process_time (ReduceProcessTime, ditto)

Close #2512

doc: two latency metrics not visited in reduce UDF

The metrics forwarder_udf_processing_time and forwarder_forward_chunk_processing_time are visited only in map UDF, not in reduce UDF. This commit states that.

Copy link

codecov bot commented May 12, 2025

Codecov Report

Attention: Patch coverage is 74.19355% with 8 lines in your changes missing coverage. Please review.

Project coverage is 71.22%. Comparing base (9b341ef) to head (48a646f).
Report is 32 commits behind head on main.

Files with missing lines Patch % Lines
pkg/reduce/pnf/pnf.go 20.00% 8 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main    #2624      +/-   ##
==========================================
+ Coverage   71.11%   71.22%   +0.10%     
==========================================
  Files         397      397              
  Lines       63371    63399      +28     
==========================================
+ Hits        45066    45155      +89     
+ Misses      17190    17127      -63     
- Partials     1115     1117       +2     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@tmenjo tmenjo force-pushed the reduce-udf-metrics branch from 4a7590c to 48a646f Compare May 12, 2025 05:37
@tmenjo tmenjo marked this pull request as ready for review May 12, 2025 06:28
@tmenjo tmenjo requested review from whynowy and vigith as code owners May 12, 2025 06:28
@whynowy
Copy link
Member

whynowy commented May 12, 2025

We are implementing a Rust version, maybe we should continue from there? @vigith @yhl25

@vigith
Copy link
Member

vigith commented May 12, 2025

We are implementing a Rust version, maybe we should continue from there? @vigith @yhl25

If this is complete, we can merge it in. We can refer this and make sure Rust based has all the right metrics.

@tmenjo tmenjo mentioned this pull request May 30, 2025
@whynowy whynowy merged commit 6bf5247 into numaproj:main May 30, 2025
26 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Latency metrics not visited or not documented in Reduce UDF
3 participants