[BUG] Degraded to_parquet performance on lustre

[ayushdg]

Describe the bug
When bumping from Rapids 25.10 to 26.08 (it is a big jump). We see degraded performance from to_parquet. when reading/writing from many GPUs to a lustre based distributed filesystem.

In one case the avg time to write a dataframe went from ~0.3s to ~1.5s and the standard deviation went from 0.02s to 7.5s.
On end to end deduplication workflows this has a 20-40% impact on overall runtime.
Side note: The issue isn't as apparent when testing on a single GPU, I think the combination of multiple GPUs all hitting the filesystem at the same time with these new default options is what makes perf worse.

Steps/Code to reproduce bug

1. On a machine with 8 or more GPUs, setup 1 process per GPU each writing a dataframe to disk (lustre).
2. The specific schema I used was int64, list<int64>[260]

Expected behavior
Based on the pattern it seems related to the interaction of many threads with the lustre filesystem and I explores disabling the changes in rapidsai/kvikio#863.

Setting KVIKIO_AUTO_DIRECT_IO_WRITE=0 helps performance go back to 25.10 levels.

So either the auto direct write feature doesn't play well with lustre or it's interaction with #23231 severely degrades performance on lustre like filesystems where multiple threads per GPU and multiple GPUs are simultaneously writing to the filesystem.

Environment overview (please complete the following information)

• Environment location: [Bare-metal, Docker, Cloud(specify cloud provider)]
• Method of cuDF install: [conda, Docker, or from source]
  • If method of install is [Docker], provide docker pull & docker run commands used

Environment details
Please run and paste the output of the cudf/print_env.sh script here, to gather any other relevant environment details

Additional context
Add any other context about the problem here.