proposal: proxy.golang.org: declare package directories to exclude from proxy

[dnnrly]

Proposal Details

Problem Statement

This relates to an issue with false-positive vulnerability findings associated with github.com/tdewolff/minify, discussed in this issue. For the record, Minify is a fantastic project and the findings are a symptom of the tooling not understanding the project structure, and not the project itself!

In brief, the project includes some Javascript files that are used in benchmark comparisons under the _benchmarks directory. This directory is present in the zip file that contains all of the code in that repository at that version. Security scanning tools can scan these zip files and compare the contents to a database of known vulnerabilities. In the case of Minify, the JS files have CVEs registered against them so trigger findings in certain scanners.

Proposal

I don't have a complete solution worked out but I wonder whether it might be possible to declare certain paths be excluded from the cache/proxy in go.mod. The proxy would then remove these files before creating the zip and placing it in the cache.

Here's an example of what a go.mod file might look like, I'll continue to use Minify as the base so we have a concrete example that we can follow. It's only a guess at what the specification could look like so feedback would be greatly appreciated.

module github.com/dnnrly/minify/v2

go 1.18

exclude_paths (
        Dockerfile // exclude an individual file
        _benchmarks/ // exclude a whole directory
        tests/oss-fuzz-build.sh // exclude a single files in a sub-directory
        .* // exclude files and directories with a simple pattern match
        **/*.md // exclude files with globbing
)

require (
        github.com/djherbis/atime v1.1.0
        github.com/fsnotify/fsnotify v1.8.0
        github.com/matryer/try v0.0.0-20161228173917-9ac251b645a2
        github.com/tdewolff/argp v0.0.0-20240625173203-87b04d5d3e52
        github.com/tdewolff/parse/v2 v2.7.19
        github.com/tdewolff/test v1.0.11-0.20240106005702-7de5f7df4739
)

require (
        filippo.io/edwards25519 v1.1.0 // indirect
        github.com/cheekybits/is v0.0.0-20150225183255-68e9c0620927 // indirect
        github.com/go-sql-driver/mysql v1.8.1 // indirect
        github.com/jmoiron/sqlx v1.4.0 // indirect
        github.com/pelletier/go-toml v1.9.5 // indirect
        github.com/tdewolff/minify/v2 v2.21.3 // indirect
        golang.org/x/sys v0.25.0 // indirect
)

For each of the paths in the exclude_paths block, the proxy would perform the equivalent of rm -rf <path> before the zip is generated.

Discussion

This proposed example would solve the immediate problem of excluding files and directories from the package that is cached by proxies. As stated above, this is a only the start of the conversation so please feel free provide constructive feedback. I am entirely open to the possibility that there is an existing solution or workaround that would solve the problem stated above.

Some pros to this solution:

• This could be implemented by any proxy implementation fairly easily
• It would allow package maintainers to reduce the possibility of false-positives for many scanning solutions
• It would allow package maintainers to reduce the size of packages by excluding assets that are not useful to imports (for example images), consequently reducing storage costs for proxies

Some cons to this solution:

• Each proxy would be expected to implement this solution independently, which is an opportunity for bugs to creep in
• It is possible for package maintainers to break their package by accidentally excluding something important

Other considerations:

• This proposal focuses on solving the issue in the proxy but shouldn't exclude other solutions
• I have not directly considered the behaviour of go get and other related sub-commands that connect to package repositories directly

[dnnrly]

Hi @seankhliao, you closed this issue as completed. Do you have a reference to the issue that this duplicates?

[dominikh]

This issue was closed as a duplicate of #42965: