Implement resumable downloads

[ericcurtin]

So if a download gets interrupted, we do not have to start again from scratch

[sourcery-ai]

The pull request #408 has too many files changed.

The GitHub API will only let us fetch up to 300 changed files, and this pull request has 521.

[ericcurtin]

@copilot fix the builds:

failing checks
CI / test (pull_request)
CI / test (pull_request)Failing after 1m
Code scanning results / CodeQL
Code scanning results / CodeQLFailing after 24s — 10 new alerts including 10 high severity security vulnerabilities
Integration Tests / integration-test (push)
Integration Tests / integration-test (push)Failing after 2m

[gemini-code-assist]

Summary of Changes

Hello @ericcurtin, I'm Gemini Code Assist¹! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request introduces a significant enhancement by implementing resumable downloads for models. This feature ensures that if a download is interrupted, it can be seamlessly continued from the point of failure rather than restarting from scratch. The changes involve modifying the model pulling logic to detect incomplete layers, calculate appropriate resume offsets, and update the local storage to handle appending data to partial files, thereby improving the robustness and user experience of model fetching.

Highlights

• Resumable Downloads Implemented: The core functionality for resumable downloads has been added, allowing interrupted model downloads to continue from where they left off, preventing data loss and improving efficiency.
• Enhanced Progress Tracking: Progress reporting now accurately reflects already downloaded bytes when resuming, providing a more precise and user-friendly experience during partial downloads.
• Robust Local Storage for Partial Downloads: The local storage mechanism has been updated to intelligently detect incomplete layer files, append new data to them, and perform integrity checks (hash verification) upon successful completion of the download.
• Vendored go-containerregistry Library: The go-containerregistry library has been vendored into the project's pkg/go-containerregistry directory, and all internal references updated to point to this local copy, enabling custom modifications for resumable download features.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature	Command	Description
Code Review	/gemini review	Performs a code review for the current pull request in its current state.
Pull Request Summary	/gemini summary	Provides a summary of the current pull request in its current state.
Comment	@gemini-code-assist	Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help	/gemini help	Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign up here.

You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.

Footnotes

1. Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

[Copilot]

@ericcurtin I've opened a new pull request, #409, to work on those changes. Once the pull request is ready, I'll request review from you.

[gemini-code-assist]

Code Review

This pull request introduces resumable downloads, a valuable feature for improving the reliability of model pulling. The implementation works by checking for partially downloaded layers and using Range headers to resume. This is enabled by vendoring a forked version of go-containerregistry and modifying the local store to handle appending to incomplete files.

My review has identified a critical race condition in PullModel that could lead to data corruption, as well as high-severity issues concerning the handling of corrupted partial downloads and insecure file permissions. I've also included a medium-severity comment about potential disk space leaks from stale partial files and another about some unused code. Addressing these points will make the feature more robust and secure.

[ericcurtin]

Hopefully this is a short-lived forked as we upstream the changes from:

pkg/go-containerregistry/pkg/v1/remote/

[ericcurtin]

Upstream version here:

google/go-containerregistry#2144