Run cudf-polars conda unit tests with more than 1 process

[mroeschke]

Description

Now that cudf-polars uses managed memory by default, the prior comment here should no longer be applicable and we should be able to run these tests with more than 1 process for a hopeful improvement in runtime.

Probably depends on #20042 so each xdist process doesn't set the initial_pool_size of the memory resource to 80% of the available device memory.

Checklist

• I am familiar with the Contributing Guidelines.
• New or existing tests cover these changes.
• The documentation is up to date with these changes.

[copy-pr-bot]

Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually.

Contributors can view more details about this message here.

[mroeschke]

/ok to test 59d144d

[TomAugspurger]

As mentioned in #19895 (comment), we'll want to update

cudf/python/cudf_polars/tests/test_config.py

Line 104 in be76be8

def test_cudf_polars_enable_disable_managed_memory(monkeypatch, enable_managed_memory):

to clear cudf-polars memory pool cache by calling

default_memory_resource.cache_clear()

somewhere near the end of that test. Otherwise, there will be a memory pool with 50% of GPU memory sitting around doing nothing. That should perhaps be a fixture so that we know it runs, even if the test fails somewhere after creating that memory resource.

There might still be some risk in two tests running that at the same time, so that should perhaps rerun on failures a few times.

I can push those changes here if you'd like.

[mroeschke]

/ok to test 29b3b41

[TomAugspurger]

That one passed. @mroeschke do you remember how consistently you saw the test suite OOM previously?

[mroeschke]

do you remember how consistently you saw the test suite OOM previously?

Each run was OOMing even with just in memory executor (before limiting the initial pool size to 1GB). I'm going to rerun these test in an "ideal" setup (8 processes, no limiting of initial pool size) to see if test_cudf_polars_enable_disable_managed_memory was the only blocker to running these test in multiple processes.

[mroeschke]

/ok to test 4c51a08

[mroeschke]

I put this fixture in the experimental directory since currently these are the only tests (with the CI script) that purposefully test with the distributed executor. Is that OK @TomAugspurger given your thoughts on reorganizing the cudf_polars test suite in the future?

[TomAugspurger]

Yep, that sounds good to me.

[TomAugspurger]

Yep, that sounds good to me.

[mroeschke]

Just documenting some runtime observations for a follow up running the cudf polars wheel tests with multiple processes

• With 6 processes, the runtime of the 4 cudf polars test variants were essentially equivalent except for the distributed variant where the runtime increased by 1-3 minutes
• With 8 processes, the runtime of the for the in memory, streaming, stream + small block size decreased by 40 sec - 1.5 minutes while the streaming + distributed variant increased by ~40 seconds (overall net decrease)

So in the follow up, I'll probably use 8 processes for these unit test with conda (and wheels) and maybe disable running the distributed variant with multiple processes

[mroeschke]

/merge