feat: multi-ecosystem support (12 GHSA ecosystems) + src/ refactor - #6
Merged
Conversation
Generalize the feed fetchers to cover every supported purl type:
- FEED_ECOSYSTEM_MAP maps purl type <-> OSV/GHSA ecosystem string <-> OSV
zip dir (empirically verified against the OSV bucket + real advisory data).
- Single shared jq program with per-type name canonicalization matching
canon_purl_name (pypi PEP503, maven colon->slash, swift URL strip,
composer/nuget/githubactions lowercase, others identity).
- fetch_osv loops the OSV zips; fetch_ghsa does ONE sparse clone + ONE
parallel pass and splits by pkg:<type>/ prefix (npm keeps legacy filenames).
- Deterministic output: LC_ALL=C sort, and each parallel worker writes to its
own temp file to avoid shared-pipe line tearing.
update-feeds.yml counts every data/*.purl and uses a 60-minute timeout.
Docker: FEED_ECOSYSTEMS build-arg (default npm = unchanged image) prunes the
baked-in feeds; add an -all image variant to the docker workflows.
Includes generated data/{ghsa,osv}-<eco>.purl for all ecosystems; npm feeds
stay byte-compatible.
Add the first per-ecosystem extension on top of the registry seam from Tasks 1-4: a Cargo.lock row in ECOSYSTEM_REGISTRY and a shared TOML "[[package]]" lockfile parser (analyze_toml_pkg_lock) that will be reused by poetry.lock/uv.lock/pdm.lock in later tasks. Adds a real cargo-generated rust-project fixture (time 0.1.45 vulnerable, serde and libc safe) plus the shared test-vulnerabilities-multi.purl feed, and wires both the multi-eco feed test and a real-feed smoke test into CI.
…k, pdm.lock, Pipfile.lock)
Add the RubyGems version comparator (compare_versions_gem) verified segment-by-segment against real Gem::Version ordering, wire it into compare_versions_eco, and add a Gemfile.lock parser (analyze_gemfile_lock) that captures only GEM-section specs, skips GIT/PATH-sourced specs and 6-space dependency constraints, and strips native-gem platform suffixes (nokogiri (1.16.5-arm64-darwin) -> 1.16.5). Adds a real bundle-lock-derived ruby-project fixture, 24 comparator harness assertions, the shared multi-ecosystem test feed lines, and CI wiring mirroring the rust/go/python steps.
Add a composer.lock parser (analyze_composer_lock) that walks JSON brace/bracket depth precisely (quoted-string contents skipped) so "name"/"version" are only captured as DIRECT fields of a package object, one level below the "packages"/"packages-dev" arrays - any nested subtable (source, dist, require, autoload, authors, support, funding, ...) sits deeper and is excluded, so a nested author's "name" can never clobber the package name. Package names are lowercased to match the composer feed canon (vendor/package); versions have a leading "v" stripped (composer routes through the plain semver comparator via the existing "*" branch of compare_versions_eco - no new comparator needed) and "dev-*" branch aliases are skipped silently. Adds a hand-crafted but real-shaped php-project fixture (composer wasn't available to generate one) covering both packages and packages-dev, one vulnerable package (guzzlehttp/guzzle 7.4.0, matched against the real GHSA-cwmx-hcrq-mhc3 range), two safe packages (monolog/monolog, and psr/log tagged "v3.0.0" to exercise the v-strip), and a packages-dev entry (phpunit/phpunit) proving the dev section is parsed. Wires the shared multi-ecosystem test feed and CI steps mirroring the rust/go/ruby steps.
Add a faithful port of Apache Maven's ComparableVersion ordering (compare_versions_maven) wired into the comparator dispatch, plus gradle.lockfile and pom.xml parsers registered under the maven purl type. - Comparator reproduces the ComparableVersion Item tree (int/string/nested list), qualifier ranking (alpha<beta<milestone<rc<snapshot<''<sp<unknown), single-letter aliases, trailing-null trimming and the list-padding compare. - gradle.lockfile: group:artifact:version=configs lines (skips comments and the empty= sentinel). pom.xml: <dependency> blocks, literal versions only (property/parent-inherited versions skipped), exclusions guarded. - Canonical name is groupId:artifactId, matching the feed's maven canon key. - Combined fixture (gradle.lockfile + pom.xml in one dir), shared test feed rows, and CI wiring (inline assertion + guarded real-feed smoke).
Add a SemVer-2 + 4th-part-Revision wrapper (compare_versions_nuget) wired into the comparator dispatch, plus a packages.lock.json parser registered under the nuget purl type. - Comparator wraps the frozen npm compare_versions: strips build metadata, compares the Major.Minor.Patch.Revision quad numerically (missing Revision defaults to 0), then falls back to the go comparator's semver-2 dot-split pre-release identifier loop, lower-cased first since NuGet compares pre-release labels case-insensitively (unlike strict SemVer 2.0.0). - packages.lock.json: JSON depth-tracking state machine (modeled on composer.lock's parser) walking dependencies -> framework -> package -> fields; Direct/Transitive entries are captured, Project entries are skipped by construction (no resolved field), names are lower-cased, and identical name/version pairs collapse across multi-targeted frameworks. csproj (tier 2) is not parsed. - Hand-written fixture (two target frameworks, proving cross-framework dedupe), shared test feed rows, and CI wiring (inline assertion + guarded real-feed smoke), verified against the real GHSA nuget feed.
Add a pubspec.lock parser (analyze_pubspec_lock) registered under the pub purl type; no new version comparator needed (pub falls through to the existing npm-semver `*` dispatch branch). - pubspec.lock is YAML: a line-anchored state machine tracks the current 2-space-indented package-name key and its 4-space "source:"/"version:" fields, keyed on exact indent-width regex anchors (the same technique Gemfile.lock's parser uses to tell a 4-space spec line from a 6-space dependency-constraint line). Only source: hosted packages resolve to a checkable name+version pair; source: git (VCS-pinned) and source: sdk (the flutter/dart pseudo-packages) are skipped by construction, since emission only fires once "hosted" was actually seen for that block. Stops at the top-level sdks: key. - Hand-written fixture (dart unavailable in this environment): dio@4.0.6 vulnerable (picked from the real GHSA pub feed, <5.0.0 range), http/ async/collection safe, one git-sourced entry and one sdk-sourced (flutter) entry both proving the skip logic, plus shared test feed rows and CI wiring verified against the real GHSA pub feed.
# Conflicts: # script.sh
Contributor
|
🎉 This PR is included in version 1.11.0 🎉 The release is available on GitHub release Your semantic-release bot 📦🚀 |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Résumé
Deux chantiers en un, sur 19 commits :
Refactor multi-fichiers.
script.sh(4 761 lignes) est éclaté en modulessrc/concaténés parbuild.shen unscript.shunique, qui reste l'artefact distribué (curl|bash, Homebrew, Docker, workflow réutilisable). Le premier commit est identique octet pour octet à l'original (modulo un newline final) — vérifiable parcmp. Un job CIverify-buildrefuse toute dérive entresrc/etscript.sh.Support des 12 écosystèmes GHSA : npm (existant), PyPI, Go, Maven, crates.io, RubyGems, Packagist, NuGet, Pub, Hex, Swift, GitHub Actions.
Ce qui change
eco:nom), avec un wildcard*pour les feeds custom sans écosystème → fin des collisions inter-langages (commons-ionpm vs maven), comportement des feeds custom préservé.ECOSYSTEM_REGISTRYd'où dérivent découverte, dispatch,--lockfile-typeset recherche GitHub. Fluxdetect-then-load: seuls les feeds des écosystèmes détectés sont chargés.packaging, 0 divergence sur ~2 000 paires), Maven ComparableVersion (port du code maven-3.9.x), RubyGems (vsGem::Version), Go semver, NuGet 4-composants. Harnais de tests table-driven (156 cas).data/ghsa.purl/osv.purlrestent npm (noms inchangés) ; nouveauxdata/{ghsa,osv}-<eco>.purl. Jamais de fichier combiné — la séparation borne les téléchargements et évite les gros bundles. Régénérés par le cron 12 h deupdate-feeds.yml.FEED_ECOSYSTEMS(défautnpm→ image identique à aujourd'hui ;allou liste). Tags-allpubliés en plus.Dockerfile.liteinchangé.Correctif inclus
Bug latent scoped-npm : le parseur PURL coupait au premier
@, ignorant silencieusement toutes les lignespkg:npm/@scope/name@…— 1 948 / 9 060 advisories GHSA (21,5 %) perdues. Corrigé (coupe au dernier@) + test de régression. Les CI utilisateurs peuvent voir apparaître de nouvelles vulnérabilités : c'est voulu.Rétrocompatibilité
Projets npm-only : comportement et sortie console identiques (hors advisories scoped récupérées). CLI,
.package-checker.config.json, URLs rawdata/ghsa.purl/osv.purl, colonnes CSV existantes (colonneecosystemajoutée en fin),reusable-check.yml: inchangés.Tests
test-all-bugs.sh35/35,test-version-compare.sh156/156,metadata-collision11/11,test-feed-generation.sh6/6verify-build(anti-dérive src/ ↔ script.sh)Revue finale : 0 bloquant, prête à merger.