Skip to content

⚡ Bolt: Streaming File Loading Optimization#26

Open
erayfirat wants to merge 1 commit intomainfrom
bolt-streaming-io-optimization-7318918625458296556
Open

⚡ Bolt: Streaming File Loading Optimization#26
erayfirat wants to merge 1 commit intomainfrom
bolt-streaming-io-optimization-7318918625458296556

Conversation

@erayfirat
Copy link
Owner

💡 What: Replaced the file loading mechanism in app.py. Instead of reading the entire streamlit.UploadedFile content into memory (read().decode()) and wrapping it in StringIO, we now use io.TextIOWrapper to stream and decode the file content on-the-fly.

🎯 Why: The previous approach required allocating memory for:

  1. The raw bytes of the file.
  2. The decoded string of the file (which can be larger).
  3. The StringIO buffer.
    For large proteomics files (hundreds of MBs to GBs), this caused significant memory pressure and potentially Out-Of-Memory errors.

📊 Impact:

  • Memory: Reduces peak memory usage during file loading by avoiding full-file in-memory duplication. The file is now processed in chunks.
  • Speed: Reduces initial load time by avoiding the synchronous read() and decode() of the entire file before parsing begins.

🔬 Measurement:

  • Verified with tests/test_streaming_io.py which confirms pyteomics can read from io.TextIOWrapper correctly.
  • Integration tests pass, ensuring no regression in app functionality.

PR created automatically by Jules for task 7318918625458296556 started by @erayfirat

Replaced inefficient `read().decode()` with `io.TextIOWrapper` for MGF and mzTab loading.
This avoids loading the entire file into memory as bytes and then as a string, significantly reducing memory footprint for large datasets.

Added regression test `tests/test_streaming_io.py`.
Updated `.gitignore` to exclude `__pycache__`.

Co-authored-by: erayfirat <59361860+erayfirat@users.noreply.github.com>
@google-labs-jules
Copy link
Contributor

👋 Jules, reporting for duty! I'm here to lend a hand with this pull request.

When you start a review, I'll add a 👀 emoji to each comment to let you know I've read it. I'll focus on feedback directed at me and will do my best to stay out of conversations between you and other bots or reviewers to keep the noise down.

I'll push a commit with your requested changes shortly after. Please note there might be a delay between these steps, but rest assured I'm on the job!

For more direct control, you can switch me to Reactive Mode. When this mode is on, I will only act on comments where you specifically mention me with @jules. You can find this option in the Pull Request section of your global Jules UI settings. You can always switch back!

New to Jules? Learn more at jules.google/docs.


For security, I will only act on instructions from the user who triggered this task.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant