Skip to content

Add support to mine maven Package-URLs #660 #678

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
wants to merge 29 commits into
base: main
Choose a base branch
from
Open

Add support to mine maven Package-URLs #660 #678

wants to merge 29 commits into from

Conversation

JonoYang
Copy link
Member

@JonoYang JonoYang commented Sep 1, 2025 •
edited by pombredanne
Loading

@pombredanne pombredanne changed the title 660 purl next maven Add support to mine maven Package-URLs #660 Sep 2, 2025
@pombredanne pombredanne mentioned this pull request Sep 2, 2025
Open
JonoYang
JonoYang commented Sep 4, 2025
View reviewed changes
minecode_pipeline/pipes/__init__.py Outdated
@@ -26,3 +27,28 @@ def write_data_to_file(path, data):
path.parent.mkdir(parents=True, exist_ok=True)
with open(path, encoding="utf-8", mode="w") as f:
f.write(saneyaml.dump(data))


def write_purls_to_repo(repo, package, packages, push_commit=False):
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@AyanSinhaMahapatra I've moved your write_packageurls_to_file and write_purls_to_repo functions to this file and I see that we have overlapping functionality here, especially with the code in https://github.com/aboutcode-org/purldb/blob/main/minecode_pipeline/pipes/pypi.py#L135

I think we should use a common function to do these actions

@JonoYang
Clone repo per package #660
b0a3d6e 
Signed-off-by: Jono Yang <[email protected]>
ziadhany
ziadhany reviewed Sep 5, 2025
View reviewed changes
minecode_pipeline/pipes/__init__.py Outdated


def write_packageurls_to_file(repo, base_dir, packageurls):
purl_file_rel_path = os.path.join(base_dir, PURLS_FILENAME)
purl_file_rel_path = os.path.join(base_dir, hashid.PURLS_FILENAME)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@JonoYang @AyanSinhaMahapatra I think we should avoid computing the package URL path and just use this function. ppath = hashid.get_package_purls_yml_file_path(package)

@JonoYang
Add methods to fetch maven index properties and increments #660
d9c8b8d 
Signed-off-by: Jono Yang <[email protected]>
@JonoYang
Merge branch 'main' into 660-purl-next-maven
ac7f6d8
@JonoYang
Move files to minecode_pipelines #660
1bfb2d8 
Signed-off-by: Jono Yang <[email protected]>
@JonoYang
Move tests over to minecode_pipelines #660
dda2bca 
Signed-off-by: Jono Yang <[email protected]>
@JonoYang
Create purls.yml path without leading segment #660
74bd21c 
    * leading segment will be the repo itself, not a directory within the repo

Signed-off-by: Jono Yang <[email protected]>
@JonoYang
Use last incremental from index properties #660
810956f 
Signed-off-by: Jono Yang <[email protected]>
@JonoYang
Move code to proper place #660
0d441f9 
Signed-off-by: Jono Yang <[email protected]>
@JonoYang
Merge branch 'main' into 660-purl-next-maven
e5ed41a
@JonoYang
Use new common functions for writing and commiting purls #660
eb4a4fb 
    * Create function to get checkpoint from checked out settings repo

Signed-off-by: Jono Yang <[email protected]>
This was referenced Sep 17, 2025
Closed
Closed
@JonoYang
Remove repo clone step #660
a225502 
    * This is done in the collect_packages_from_maven step

Signed-off-by: Jono Yang <[email protected]>
@JonoYang
Update code style #660
d126ea0 
Signed-off-by: Jono Yang <[email protected]>
@JonoYang
Revert change to scio dep version #660
113bce5 
Signed-off-by: Jono Yang <[email protected]>
@JonoYang
Remove unnecessary logger #660
78045ef 
Signed-off-by: Jono Yang <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
1 more reviewer

@ziadhany ziadhany ziadhany left review comments

Reviewers whose approvals may not affect merge requirements
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@JonoYang @ziadhany