[node] support git sources #382

Ian2020 · 2023-11-03T11:40:53Z

Fixes #381 and I hope will help #237 and #182.

I've added a new set of tests to cover more types of git references: https and ssh both with a commit hash and without. The tests cover all versions of the lockfile format v1,2,3.

In my testing I found that it doesn't appear to be necessary to patch package-lock.json (npm.py:444), so that has been removed. If there's another reason that is needed let me know but the tests all pass without it.

refi64 · 2023-11-03T16:29:50Z

I think this has some overlap with #351 😅 see the discussion there.

jwillikers · 2023-11-12T13:22:15Z

Attempting to use this for the Stretchly flatpak, and I get the following error when building:

Downloading sources
Fetching git repo https://[email protected]/Jelmerro/dbus-final.git, ref refs/heads/master
remote: Total 0 (delta 0), reused 0 (delta 0), pack-reused 0
Fetching git repo https://github.com/hovancik/stretchly.git, ref refs/tags/v1.15.0
remote: Total 0 (delta 0), reused 0 (delta 0), pack-reused 0
Starting build of net.hovancik.Stretchly
Cache hit for dbus-glib, skipping build
Cache miss, checking out last cache hit
========================================================================
Building module stretchly in /home/jordan/Projects/net.hovancik.Stretchly/.flatpak-builder/build/stretchly-11
========================================================================
Note: switching to '3e43f60d80bbcdf0dfa0b59b838097d6af4d17ba'.

You are in 'detached HEAD' state. You can look around, make experimental
changes and commit them, and you can discard any commits you make in this
state without impacting any branches by switching back to a branch.

If you want to create a new branch to retain commits you create, you may
do so (now or later) by using -c with the switch command. Example:

  git switch -c <new-branch-name>

Or undo this operation with:

  git switch -

Turn off this advice by setting config variable advice.detachedHead to false

HEAD is now at 3e43f60 update deps
jq: error: Could not open file /var/home/jordan/Projects/net.hovancik.Stretchly/.flatpak-builder/build/stretchly-11/package.json: No such file or directory
/var/home/jordan/Projects/net.hovancik.Stretchly/.flatpak-builder/build/stretchly-11/flatpak-node/patch/app.sh: line 2: /var/home/jordan/Projects/net.hovancik.Stretchly/.flatpak-builder/build/stretchly-11/app/package.json.new: No such file or directory
mv: cannot stat '/var/home/jordan/Projects/net.hovancik.Stretchly/.flatpak-builder/build/stretchly-11/app/package.json.new': No such file or directory
Error: module stretchly: Child process exited with code 1

Ian2020 · 2023-11-12T13:37:48Z

There's something up with the paths here:

Building module stretchly in /home/jordan/Projects/net.hovancik.Stretchly/.flatpak-builder/build/stretchly-11

...but later the jq is looking in /var/home/... rather than /home/...

jq: error: Could not open file /var/home/jordan/Projects/net.hovancik.Stretchly/.flatpak-builder/build/stretchly-11/package.json: No such file or directory

Not sure where that /var prefix has come from.

jwillikers · 2023-11-12T13:48:15Z

There's something up with the paths here:
Building module stretchly in /home/jordan/Projects/net.hovancik.Stretchly/.flatpak-builder/build/stretchly-11
...but later the jq is looking in /var/home/... rather than /home/...
jq: error: Could not open file /var/home/jordan/Projects/net.hovancik.Stretchly/.flatpak-builder/build/stretchly-11/package.json: No such file or directory
Not sure where that /var prefix has come from.

That's a thing on Fedora rpm-ostree systems. /home is a symlink to /var/home. This is documented here.

jwillikers · 2023-11-12T14:12:15Z

The manifest and everything can be found in this MR.

Ian2020 · 2023-11-12T14:14:34Z

Thanks, I'll take a look.

Ian2020 · 2023-11-13T14:55:41Z

Ok a few things going on here:

The inclusion of generated-sources.json in net.hovancik.Stretchly.yaml needs to come AFTER the stretchly source code is checked out, that's why package.json can't be found to be patched by generated-sources. This should fix the error you've reported but you'll next hit a ENOTCACHED error because...
We don't patch refs that looks like "github:user/project" and in Stretchly you have exactly this: "github:Jelmerro/dbus-final". I can't find in the package.json spec where this is allowed but it seems to be common searching across Github. So I think I have to add support for that to this PR.
The Stretchly project has two package.json files: one in / one in /app. Both would have to be patched for its flatpak build to succeed with git refs. The electron-builder cmd in the postinstall script seems to use the /app one. @jwillikers do you know why there are two package.json files?

jwillikers · 2023-11-13T16:42:25Z

Ok a few things going on here:

* The inclusion of `generated-sources.json` in [net.hovancik.Stretchly.yaml](https://github.com/flathub/net.hovancik.Stretchly/blob/master/net.hovancik.Stretchly.yaml) needs to come AFTER the stretchly source code is checked out, that's why package.json can't be found to be patched by generated-sources. This should fix the error you've reported but you'll next hit a ENOTCACHED error because...

* We don't patch refs that looks like "github:user/project" and in Stretchly you have exactly this: "github:Jelmerro/dbus-final". I can't find in the [package.json spec](https://docs.npmjs.com/cli/v9/configuring-npm/package-json) where this is allowed but it seems to be common searching across Github. So I think I have to add support for that to this PR.

* The Stretchly project has two `package.json` files: one in `/` one in `/app`. Both would have to be patched for its flatpak build to succeed with git refs. The `electron-builder` cmd in the postinstall script seems to use the `/app` one. @jwillikers do you know why there are two `package.json` files?

I've changed that order.
As for why there are two package.json files, I'm not sure.
@hovancik Could you shed some light on why there is app/package.json in addition to package.json?

hovancik · 2023-11-14T08:55:13Z

@jwillikers that's the structure from electron-builder: https://www.electron.build/tutorials/two-package-structure.html

I've tried in the past to remove 2 package structure, as they say it's not needed anymore but it did not work for me.

Ian2020 · 2023-11-16T14:23:30Z

We don't patch refs that looks like "github:user/project" and in Stretchly you have exactly this: "github:Jelmerro/dbus-final". I can't find in the package.json spec where this is allowed but it seems to be common searching across Github. So I think I have to add support for that to this PR.

Ok I've added support for these 'shortcut' git urls to the PR. But we still need to overcome the 2 package structure in stretchly somehow.

hovancik · 2023-11-16T15:07:09Z

@Ian2020 should I try to remove 2package structure? It's been some time since last time I tried so I can try again if that helps

Ian2020 · 2023-11-16T18:27:20Z

@hovancik yes I think that would be simplest if you can, thanks.

hovancik · 2023-11-19T16:42:52Z

@Ian2020 done, new version is out.

hovancik · 2023-11-20T12:10:07Z

@Ian2020 actually scratch that, it does not work, will need to look into this more.

Ian2020 · 2023-11-20T12:49:00Z

Ok, despite that I can now successfully build Stretchly v1.15.1 as a flatpak. I created generated-sources.yaml using this PR against v1.15.1 of stretchly. Then copied that file into branch update-71b455c of https://github.com/flathub/net.hovancik.Stretchly and updated the source tag in the yaml to v1.15.1 and it builds a flatpak that I could run @hovancik @jwillikers

So I think this PR is doing what it should do for Stretchly to build as a flatpak, I hope you can sort the other issues in Stretchly. Let me know if you think there's anything more needed here.

jwillikers · 2023-11-20T14:45:21Z

@Ian2020 One thing, I noticed that running flatpak-node-generator from this PR doesn't work when attempting to run it from a relative directory. I usually run flatpak-node-generator npm stretchly/package-lock.json --electron-node-headers with Stretchly checked out in the stretchly subdirectory in the Flatpak repository. I had to change into the stretchly subdirectory, generate the sources, and then move the generated file up one directory to make things work.

hovancik · 2023-11-20T15:14:54Z

@Ian2020 can you possibly try with this PR? hovancik/stretchly#1401

Seems like it was local issue only but would be nice to confirm build is ok after this change

jwillikers · 2023-11-20T15:46:38Z

@Ian2020 can you possibly try with this PR? hovancik/stretchly#1401

Seems like it was local issue only but would be nice to confirm build is ok after this change

@hovancik I just tested it and that commit builds and runs fine.

hovancik · 2023-11-20T16:01:15Z

Thanks, we are all good then, I guess.

As part of the update, I had to refactor quite a bit of how the build process works. flatpak-node-generator is quite bugged: - Output `generated-source.json` expects the package to be in the main directory, so I've split the build into modules but still had to add `flatpak-node` exclusion to build - I had to patch locally flatpak/flatpak-builder-tools#382 and use that because flatpak/flatpak-builder-tools#381 - I've also run into flatpak/flatpak-builder-tools#377 I've also: - Made copying of addr2line dependencies more reliable - Merged the startup scripts into one - Added permission and setup for discord so that rich presence works

proletarius101 · 2024-06-18T04:47:18Z

Hi @hfiguiere. Since you mentioned that you prefer to build from source, could this PR be merged?

hfiguiere · 2024-06-21T02:14:50Z

I don't maintain that module and I don't have the knowledge space.

node/flatpak_node_generator/providers/__init__.py

Submodules outside the sanctioned organisations are not allowed, so install the fixed flatpak-node-generator through its fork directly. See flatpak/flatpak-builder-tools#382 (comment)

Ian2020 · 2025-05-07T09:56:47Z

if you can fix the conflict, rebase and clear up the history it'd be nice.

@bbhtt can do, probably be next week now

refi64 · 2025-05-12T14:58:57Z

will take a look again sometime this week

Ian2020 · 2025-05-15T10:15:04Z

if you can fix the conflict, rebase and clear up the history it'd be nice.

Done.

node/flatpak_node_generator/providers/npm.py

node/flatpak_node_generator/providers/__init__.py

node/flatpak_node_generator/providers/npm.py

node/tests/data/packages/git/package-lock.v1.json

node/tests/test_providers.py

node/flatpak_node_generator/providers/npm.py

See flatpak/flatpak-builder-tools#382 It seemed close to a merge so left it to do it manually But we can just use it and revert later

bbhtt · 2025-07-19T17:23:47Z

@refi64 do you want to have another look at this?

refi64

LGTM now, thanks for this

bbhtt · 2025-07-24T01:02:09Z

Please squash the code review fixes commit with the previous one. You also should rebase over master

Fix [flatpak#377](flatpak#377) and simplify the LocalSource check. Don't process the root package as a dependency. Assumes an entry without a resolved is always a local path.

Co-authored-by: Danilo Bargen <[email protected]> Co-authored-by: Jordan Williams <[email protected]>

Ian2020 · 2025-07-24T14:54:36Z

Please squash the code review fixes commit with the previous one. You also should rebase over master

Done. LocalSource check fix also pulled into own commit as suggested by @refi64 in comment.

bbhtt · 2025-07-24T15:48:09Z

Done. LocalSource check fix also pulled into own commit as suggested by @refi64 in #382 (comment).

Thanks, let's merge this before it bitrots again.

If you want to help maintain the node generator, feel free to open an issue asking for access. I'd like to see more people maintaining it, assuming others agree to this.

We've been waiting on flatpak/flatpak-builder-tools#382 now its part of flatpak-node-generator we no longer need to use our own branch of the tool.

Ian2020 force-pushed the gitrefs branch 2 times, most recently from 3cd5b3e to f28defd Compare November 16, 2023 14:20

p2004a mentioned this pull request Mar 3, 2024

Update launcher to the latest version flathub/info.beyondallreason.bar#45

Merged

hfiguiere added npm node labels Apr 5, 2024

dbrgn reviewed Jul 31, 2024

View reviewed changes

node/flatpak_node_generator/providers/__init__.py Outdated Show resolved Hide resolved

refi64 self-assigned this May 12, 2025

Ian2020 force-pushed the gitrefs branch from 886ff68 to 09b922a Compare May 15, 2025 10:10

Ian2020 requested a review from a team as a code owner May 15, 2025 10:10

github-actions bot removed the npm label May 15, 2025

bbhtt reviewed May 15, 2025

View reviewed changes

node/flatpak_node_generator/providers/npm.py Show resolved Hide resolved

bbhtt reviewed May 15, 2025

View reviewed changes

node/flatpak_node_generator/providers/npm.py Show resolved Hide resolved

Ian2020 force-pushed the gitrefs branch from 09b922a to 5630eec Compare May 16, 2025 09:28

Beryesa mentioned this pull request May 21, 2025

Update to 0.15.0 flathub/com.authormore.penpotdesktop#20

Merged

refi64 reviewed May 22, 2025

View reviewed changes

bbhtt removed the Need Rebase The PR need rebase before merging label May 26, 2025

bbhtt force-pushed the master branch 2 times, most recently from 8c5a0ec to ea9bfa2 Compare June 4, 2025 06:07

k8ieone mentioned this pull request Jun 27, 2025

Beta 5 flathub/io.github.suchnsuch.Tangent#9

Merged

Beryesa added a commit to flathub/com.authormore.penpotdesktop that referenced this pull request Jun 27, 2025

Use git sources support PR temporarily

bb56d63

See flatpak/flatpak-builder-tools#382 It seemed close to a merge so left it to do it manually But we can just use it and revert later

Beryesa added a commit to flathub/com.authormore.penpotdesktop that referenced this pull request Jun 27, 2025

Use git sources support PR temporarily

1d7628b

See flatpak/flatpak-builder-tools#382 It seemed close to a merge so left it to do it manually But we can just use it and revert later

Beryesa added a commit to flathub/com.authormore.penpotdesktop that referenced this pull request Jun 27, 2025

Use git sources support PR temporarily

7809aa8

See flatpak/flatpak-builder-tools#382 It seemed close to a merge so left it to do it manually But we can just use it and revert later

refi64 reviewed Jul 22, 2025

View reviewed changes

Ian2020 and others added 2 commits July 24, 2025 15:11

node: Correct LocalSource check in npm provider

93dbe62

Fix [flatpak#377](flatpak#377) and simplify the LocalSource check. Don't process the root package as a dependency. Assumes an entry without a resolved is always a local path.

node: Add support for git sources

672b2cb

Co-authored-by: Danilo Bargen <[email protected]> Co-authored-by: Jordan Williams <[email protected]>

Ian2020 force-pushed the gitrefs branch from 336cf6c to 672b2cb Compare July 24, 2025 14:51

bbhtt merged commit 2cc8dd0 into flatpak:master Jul 24, 2025
6 checks passed

Uh oh!

[node] support git sources #382

[node] support git sources #382

Uh oh!

Conversation

Ian2020 commented Nov 3, 2023

Uh oh!

refi64 commented Nov 3, 2023

Uh oh!

jwillikers commented Nov 12, 2023

Uh oh!

Ian2020 commented Nov 12, 2023

Uh oh!

jwillikers commented Nov 12, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

jwillikers commented Nov 12, 2023

Uh oh!

Ian2020 commented Nov 12, 2023

Uh oh!

Ian2020 commented Nov 13, 2023

Uh oh!

jwillikers commented Nov 13, 2023

Uh oh!

hovancik commented Nov 14, 2023

Uh oh!

Ian2020 commented Nov 16, 2023

Uh oh!

hovancik commented Nov 16, 2023

Uh oh!

Ian2020 commented Nov 16, 2023

Uh oh!

hovancik commented Nov 19, 2023

Uh oh!

hovancik commented Nov 20, 2023

Uh oh!

Ian2020 commented Nov 20, 2023

Uh oh!

jwillikers commented Nov 20, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

hovancik commented Nov 20, 2023

Uh oh!

jwillikers commented Nov 20, 2023

Uh oh!

hovancik commented Nov 20, 2023

Uh oh!

proletarius101 commented Jun 18, 2024

Uh oh!

hfiguiere commented Jun 21, 2024

Uh oh!

Uh oh!

Ian2020 commented May 7, 2025

Uh oh!

refi64 commented May 12, 2025

Uh oh!

Ian2020 commented May 15, 2025

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

bbhtt commented Jul 19, 2025

Uh oh!

refi64 left a comment

Choose a reason for hiding this comment

Uh oh!

bbhtt commented Jul 24, 2025

Uh oh!

Ian2020 commented Jul 24, 2025

Uh oh!

bbhtt commented Jul 24, 2025

jwillikers commented Nov 12, 2023 •

edited

Loading

jwillikers commented Nov 20, 2023 •

edited

Loading