Skip to content

Additional ext4-backed mount point for high-throughput I/O caching (besides /kaggle/working) #1506

@bilzard

Description

@bilzard

🚀 Feature

Provide an additional mount point backed by a physical device (ext4), separate from /kaggle/working, to use as a high-throughput I/O cache (e.g., /tmp).

Motivation

  • The container root (/) appears to be provided via an overlay filesystem; write/metadata performance on overlay can be slower or less predictable than on a host-backed volume such as /kaggle/working.
  • Some applications default to /tmp for I/O-intensive caches; on overlay this default can become an I/O bottleneck.
  • /kaggle/working is relatively fast and can be an alternatives to /tmp, but its capacity is limited to ~20 GB per session, which is sometimes insufficient to host checkpoint of LLMs (~100GB).
  • An additional fast endpoint for caching has the potential to better utilize the instance’s compute resources (e.g., T4, P100) that are often bottlenecked by disk I/O.

Additional context

I benchmarked sequential I/O and observed both writes and reads to be slower on /tmp (overlay) than on /kaggle/working (host-backed). Write throughput on /tmp is especially unstable; it sometimes gets ~5–10× lower in my tests.

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions