Skip to content

Conversation

mroeschke
Copy link
Contributor

Description

Follow up to #19980

Checklist

  • I am familiar with the Contributing Guidelines.
  • New or existing tests cover these changes.
  • The documentation is up to date with these changes.

@mroeschke mroeschke added the improvement Improvement / enhancement to an existing function label Sep 26, 2025
@mroeschke mroeschke requested a review from a team as a code owner September 26, 2025 19:16
@mroeschke mroeschke added the non-breaking Non-breaking change label Sep 26, 2025
@mroeschke mroeschke requested a review from AyodeAwe September 26, 2025 19:16
@mroeschke mroeschke added the cudf-polars Issues specific to cudf-polars label Sep 26, 2025
Copy link
Contributor

@bdice bdice left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks fine. Please report the before/after times.

@mroeschke mroeschke requested a review from a team as a code owner September 29, 2025 17:07
@github-actions github-actions bot added the Python Affects Python cuDF API. label Sep 29, 2025
@GPUtester GPUtester moved this to In Progress in cuDF Python Sep 29, 2025


@pytest.fixture(autouse=True)
@pytest.fixture(scope="session", autouse=True)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we add a note about why session scope here is important here?

@mroeschke
Copy link
Contributor Author

Looks fine. Please report the before/after times.

Here's the raw pytest run times comparing the CUDA 12.9 cudf_polars wheels test CI job f4554de from this branch to the same job in #20141 as <minute>.<seconds> (the similar CUDA 13.01 job shows a similar magnitude decrease)

Before (#20141)

Polars version in-memory streaming streaming + small block streaming + distributed
1.28.1 1.33 1.33 1.47 2.04
1.29.0 1.34 1.33 1.47 2.10
1.30.0 1.35 1.36 1.49 2.15
1.31.0 1.39 1.37 1.51 2.10
1.32.3 1.45 1.48 2.05 2.23

After (f4554de)

Polars version in-memory streaming streaming + small block streaming + distributed
1.28.1 1.04 1.04 1.13 0.32
1.29.0 1.05 1.05 1.13 0.33
1.30.0 1.05 1.05 1.13 0.34
1.31.0 1.05 1.05 1.14 0.33
1.32.3 1.08 1.09 1.17 0.37

The overall job time has decreased from ~43 minutes to ~22 minutes.

@vyasr
Copy link
Contributor

vyasr commented Oct 1, 2025

That's great! Glad to see parallelism getting us enough improvement here to keep this as is.

@mroeschke
Copy link
Contributor Author

/merge

@rapids-bot rapids-bot bot merged commit 449173c into rapidsai:branch-25.12 Oct 2, 2025
133 of 135 checks passed
@mroeschke mroeschke deleted the ci/cudf_polars/xdist_wheels branch October 2, 2025 20:16
@github-project-automation github-project-automation bot moved this from In Progress to Done in cuDF Python Oct 2, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
cudf-polars Issues specific to cudf-polars improvement Improvement / enhancement to an existing function non-breaking Non-breaking change Python Affects Python cuDF API.
Projects
Status: Done
Development

Successfully merging this pull request may close these issues.

3 participants