node: Add yarn2 support #252

catsout · 2021-12-28T02:05:49Z

todo:

unit test

need to manually add $FLATPAK_BUILDER_BUILDDIR/flatpak-node/yarn2-setup.sh to build-commands

gasinvein · 2022-07-12T11:26:30Z

Isn't lockfile v2 format just normal YAML?

proletarius101 · 2022-07-20T09:00:28Z

I think you've managed to use this PR in https://sourcegraph.com/github.com/flathub/net.cozic.joplin_desktop@5420d1b2d6c0debe599222af99d048f10de2ac1f/-/blob/net.cozic.joplin_desktop.yml?L62. But just wondering how you get yarn2-setup.sh? Is that file introduced by this PR? Can I use it right now?

catsout · 2022-07-20T09:34:54Z

@proletarius101
yarn2-setup.sh is introduced by 'generated-sources.json'
Just run it before yarn install
You can use it right now, I have used it for a long time, also please me let know the result.

proletarius101 · 2022-07-20T12:56:16Z

@catsout I found there might be some errors: https://buildbot.flathub.org/#/builders/10/builds/12201

catsout · 2022-07-22T14:39:43Z

@proletarius101
I fixed error above, but when I try to build the standardnotes, I meet an error with react-native-keychain
It's a yarn 1.x bug yarnpkg/yarn#685, which cause yarn pack include .yarn/.yarnrc.yml.
So cache file isn't reproducible.
We need to patch lock file to solve this problem, like npm's git patch.

proletarius101 · 2022-07-22T15:10:15Z

@catsout Could you please push your fix so that I can check, or will you fix it yourself?

catsout · 2022-07-23T02:29:19Z

@proletarius101
You only need the yarn.lock on the top folder of project.
flatpak-node-generator -r isn't needed.

proletarius101 · 2022-07-23T08:58:11Z

flatpak-node-generator -r isn't needed.

True. But do you know why? I thought -r was designed for that usage.

I confirm that this PR works for standard notes.

proletarius101 · 2022-09-05T17:26:57Z

Hi @gasinvein @catsout any updates? :)

proletarius101 · 2022-09-06T12:39:12Z

f0d317a causes the following error: https://buildbot.flathub.org/#/builders/10/builds/13033

➤ YN0000: ┌ Link step
➤ YN0007: │ husky@npm:3.1.0 must be built because it never has been before or the last one failed
➤ YN0007: │ styled-components@npm:5.3.5 [76447] must be built because it never has been before or the last one failed
➤ YN0007: │ electron@npm:17.4.2 must be built because it never has been before or the last one failed
➤ YN0007: │ keytar@npm:7.9.0 must be built because it never has been before or the last one failed
➤ YN0007: │ styled-components@npm:5.3.5 [2efff] must be built because it never has been before or the last one failed
➤ YN0007: │ core-js@npm:3.23.3 must be built because it never has been before or the last one failed
➤ YN0007: │ node-sass@npm:6.0.1 must be built because it never has been before or the last one failed
➤ YN0007: │ core-js-pure@npm:3.23.3 must be built because it never has been before or the last one failed
➤ YN0007: │ detox@npm:19.7.1 [7a3f4] must be built because it never has been before or the last one failed
➤ YN0007: │ react-native-sodium-jsi@npm:1.2.0 [7a3f4] must be built because it never has been before or the last one failed
➤ YN0007: │ styled-components@npm:5.3.5 [7a3f4] must be built because it never has been before or the last one failed
➤ YN0007: │ electron@npm:11.5.0 must be built because it never has been before or the last one failed
➤ YN0007: │ dtrace-provider@npm:0.8.8 must be built because it never has been before or the last one failed
➤ YN0007: │ node-sass@npm:7.0.1 must be built because it never has been before or the last one failed
➤ YN0007: │ core-js@npm:2.6.12 must be built because it never has been before or the last one failed
➤ YN0009: │ electron@npm:11.5.0 couldn't be built successfully (exit code 1, logs can be found here: /tmp/xfs-cdf77bf5/build.log)
➤ YN0009: │ electron@npm:17.4.2 couldn't be built successfully (exit code 1, logs can be found here: /tmp/xfs-8a1b771e/build.log)
➤ YN0000: └ Completed in 4m 43s
➤ YN0000: Failed with errors in 5m 41s
FB: host_command_exited_cb 1777080 256

afc6911 doesn't

catsout · 2022-10-25T07:43:50Z

Seems I forgot name, _ = name.rsplit('@', 1) at afc6911

muelli · 2023-02-28T13:15:18Z

I'm trying to run this on https://github.com/mifi/lossless-cut/blob/37e49d5682a4dada498f3fe7ec3d2d2cf530a7df/yarn.lock but it fails:

python3 -m flatpak_node_generator yarn ~/vcs/lossless-cut/package.json --output ~/vcs/no.mifi.losslesscut/generated-sources.json
Reading packages from lockfiles...
Traceback (most recent call last):
  File "/usr/lib/python3.8/runpy.py", line 194, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/usr/lib/python3.8/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "dev/flatpak-builder-tools/node/flatpak_node_generator/__main__.py", line 3, in <module>
    main()
  File "dev/flatpak-builder-tools/node/flatpak_node_generator/main.py", line 277, in main
    asyncio.run(_async_main())
  File "/usr/lib/python3.8/asyncio/runners.py", line 44, in run
    return loop.run_until_complete(main)
  File "/usr/lib/python3.8/asyncio/base_events.py", line 616, in run_until_complete
    return future.result()
  File "dev/flatpak-builder-tools/node/flatpak_node_generator/main.py", line 198, in _async_main
    packages.update(lockfile_provider.process_lockfile(lockfile))
  File "dev/flatpak-builder-tools/node/flatpak_node_generator/providers/yarn.py", line 145, in process_lockfile
    lock_dict: Dict[str, Any] = self.parse_lockfile(lockfile)
  File "dev/flatpak-builder-tools/node/flatpak_node_generator/providers/yarn.py", line 64, in parse_lockfile
    assert level <= len(parent_entries) - 1
AssertionError

catsout · 2023-02-28T13:50:03Z

@muelli
You need https://github.com/mifi/lossless-cut/raw/37e49d5682a4dada498f3fe7ec3d2d2cf530a7df/yarn.lock.
I rebased to new master to fix esbuild version.

catsout · 2023-02-28T14:03:36Z

example: https://github.com/flathub/net.cozic.joplin_desktop/blob/master/net.cozic.joplin_desktop.yml

muelli · 2023-03-01T14:29:34Z

I'm still experiencing the error.

I'm on c278045 now. And I'm calling like this: python3 -m flatpak_node_generator yarn ~/vcs/lossless-cut/package.json --output ~/vcs/no.mifi.losslesscut/generated-sources.json. Does it actually work for you?

catsout · 2023-03-01T14:35:08Z

Yes, it works.

wget https://github.com/mifi/lossless-cut/raw/37e49d5682a4dada498f3fe7ec3d2d2cf530a7df/yarn.lock

❯ flatpak-node-generator yarn yarn.lock                                                  
Reading packages from lockfiles...
1036 packages read.
Generating packages [1036/1036] esbuild @ 0.16.17                                             
Wrote 1064 source(s).

❯ git show --summary 
commit c278045d7cc73437945b3dfb919fc7aea5d43cb5 (HEAD -> yarn2, origin/yarn2)

Note that updating the generated sources is more complicated at the moment until flatpak-node-generator starts supporting Yarn version 2+. I had to use the code in pull request flatpak/flatpak-builder-tools#252 which adds support for Yarn version 2+. The exact commit that I used is: flatpak/flatpak-builder-tools@eb43ec2. A newer update of this pull request also removes the need to execute "$FLATPAK_BUILDER_BUILDDIR/flatpak-node/yarn2-setup.sh" before running "yarn install" so I removed that line from the build-commands. Commands that I used to update the generated sources are: wget https://raw.githubusercontent.com/mifi/lossless-cut/refs/tags/v3.64.0/yarn.lock pipx run "flatpak_node_generator @ git+https://github.com/flatpak/flatpak-builder-tools.git@refs/pull/252/head#subdirectory=node" yarn yarn.lock --output generated-sources.json rm yarn.lock

tjanez · 2024-11-11T10:53:50Z

@catsout , I've used the current latest commit in this PR (eb43ec2) to successfully generate generated-sources.json for LosslessCut version 3.64.0. See flathub/no.mifi.losslesscut#73.

However, the Flatpak build process now fails at yarn install step with countless errors of the form:

➤ YN0001: │ RequestError: getaddrinfo EAI_AGAIN registry.yarnpkg.com
    at ClientRequest.<anonymous> (/run/build/lossless-cut/.yarn/releases/yarn-4.4.0.cjs:147:14258)
    at Object.onceWrapper (node:events:632:26)
    at ClientRequest.emit (node:events:529:35)
    at u.emit (/run/build/lossless-cut/.yarn/releases/yarn-4.4.0.cjs:142:14855)
    at TLSSocket.socketErrorListener (node:_http_client:501:9)
    at TLSSocket.emit (node:events:517:28)
    at emitErrorNT (node:internal/streams/destroy:151:8)
    at emitErrorCloseNT (node:internal/streams/destroy:116:3)
    at process.processTicksAndRejections (node:internal/process/task_queues:82:21)
    at GetAddrInfoReqWrap.onlookup [as oncomplete] (node:dns:107:26)

The LosslessCut project currently uses Yarn 4.4.0.

Do you have any suggestion on which commands to use in build-commands step to make this work?

catsout · 2024-12-26T09:11:04Z

Sorry for the late reply.
You can find a complete example at https://github.com/flathub/net.cozic.joplin_desktop/blob/6cda355601b2507924327b2982553af1976205a7/net.cozic.joplin_desktop.yml.
Currently, it requires loading a Yarn plugin.

catsout · 2024-12-26T09:17:47Z

  - name: app_example
    buildsystem: simple
    build-options:
      append-path: /usr/lib/sdk/node20/bin
      env:
        XDG_CACHE_HOME: /run/build/app_example/flatpak-node/cache
        npm_config_nodedir: /usr/lib/sdk/node20
        YARN_ENABLE_INLINE_BUILDS: '1'
        YARN_ENABLE_TELEMETRY: '0'
        YARN_ENABLE_NETWORK: '0'
        YARN_ENABLE_GLOBAL_CACHE: '0'
        YARN_GLOBAL_FOLDER: /run/build/app_example/flatpak-node/yarn-mirror/global
        TMPDIR: /run/build/app_example/flatpak-tmp
    build-commands:
      - yarn config
      - yarn plugin import $FLATPAK_BUILDER_BUILDDIR/flatpak-node/flatpak-yarn.js
      - yarn convertToZip $(which yarn)
      - yarn install

Since anki now uses yarn 4, some workarounds are required. flatpak-builder-tools upstream does not support yarn >=2, but there is an open PR to add support [1]. That gets most of the way there, but there is still a minor issue that I assume was introduced between yarn 2 and 4 not included in that PR, so I forked it. Anki itself also needs minor changes (yarn-4-fixes.patch) since yarn 4 removed the --offline and --ignore-scripts options, but anki's offline build still uses them. They are now instead set by the flatpak-builder-tools script. Anki uses the YARN_BINARY environment variable, but yarn will interpret any environment variables starting with YARN_ as yarn configuration, so that will need to be renamed. [1] flatpak/flatpak-builder-tools#252

Since anki now uses yarn 4, some workarounds are required. The flatpak environment does not include yarn 4, so it is added to the sources. flatpak-builder-tools upstream does not support yarn >=2, but there is an open PR to add support [1]. That gets most of the way there, but there is still a minor issue that I assume was introduced between yarn 2 and 4 not included in that PR, so I forked from that submitted a PR to the developer of that PR [2]. Anki itself also needs minor changes (yarn-4-fixes.patch) since yarn 4 removed the --offline and --ignore-scripts options, but anki's offline build still uses them. They are now instead set by the flatpak-builder-tools script. Anki uses the YARN_BINARY environment variable, but yarn will interpret any environment variables starting with YARN_ as yarn configuration, so that will need to be renamed. [1] flatpak/flatpak-builder-tools#252 [2] catsout/flatpak-builder-tools#1

tjanez · 2025-04-07T12:26:12Z

@catsout, thanks for your help!

It appears that this PR doesn't yet support projects using Yarn4 as discovered by @holubv in flathub/no.mifi.losslesscut#73 (comment), @RangHo in flathub/social.whalebird.WhalebirdDesktop#23 (comment) and @Oppzippy in flathub/net.ankiweb.Anki#198 (comment).

In a way, the situation with building Flatpak apps written in JavaScript from source is pretty unfavorable at the moment.
This PR has been open since 2021 and it doesn't look it will get merged any time soon. And also if it gets merged, it will currently only cover projects using Yarn v2 (which appears to be discontinued) and v3 while projects using Yarn v4 (currently the latest major version) will not supported.

Probably many Flatpak maintainers will find it easier (and currently the only viable option) to just repackage the prebuilt upstream provided binaries...

MoralCode · 2025-06-08T12:07:08Z

seems like https://github.com/flathub/net.ankiweb.Anki/blob/7ffe4cc7eff554bb41b76d41243effa5e7c30a1f/net.ankiweb.Anki.yaml#L145C7-L146C134 is also pretty helpful (lockfileFilename got removed in yarn4 and this is a patch to the flatpak-yarn.js plugin here to change that)

Helps #361 v2 is WIP #252

catsout marked this pull request as draft December 28, 2021 14:35

catsout force-pushed the yarn2 branch 3 times, most recently from 159ca9f to ff204fd Compare December 30, 2021 04:53

catsout force-pushed the yarn2 branch 2 times, most recently from d22a790 to 8aaa053 Compare July 12, 2022 08:37

catsout marked this pull request as ready for review July 12, 2022 08:44

catsout force-pushed the yarn2 branch from 8aaa053 to afc6911 Compare July 22, 2022 22:53

catsout force-pushed the yarn2 branch from afc6911 to f0d317a Compare August 12, 2022 23:14

catsout force-pushed the yarn2 branch from f0d317a to a596720 Compare October 24, 2022 23:46

catsout force-pushed the yarn2 branch 3 times, most recently from 799eb2a to 0894c91 Compare October 25, 2022 07:58

catsout force-pushed the yarn2 branch from 0894c91 to c278045 Compare February 28, 2023 13:49

benni347 mentioned this pull request Jul 15, 2024

flatpak-node-generator #388

Closed

tjanez mentioned this pull request Nov 11, 2024

Update to version 3.64.0 flathub/no.mifi.losslesscut#73

Open

tjanez mentioned this pull request Nov 25, 2024

Update to 3.61.1 flathub/no.mifi.losslesscut#62

Open

RangHo mentioned this pull request Feb 25, 2025

Whalebird v6.2.0 flathub/social.whalebird.WhalebirdDesktop#23

Open

Oppzippy mentioned this pull request Apr 2, 2025

Update Anki to 25.02 flathub/net.ankiweb.Anki#198

Closed

dynst mentioned this pull request Apr 29, 2025

486 add linux to flat hub asgardex/asgardex-desktop#487

Open

bbhtt added the Need Rebase The PR need rebase before merging label May 8, 2025

bbhtt requested a review from a team as a code owner June 4, 2025 05:45

bbhtt force-pushed the master branch 2 times, most recently from 8c5a0ec to ea9bfa2 Compare June 4, 2025 06:07

muelli mentioned this pull request Jun 8, 2025

Is this still maintained? flathub/no.mifi.losslesscut#81

Closed

catsout added 3 commits June 28, 2025 02:20

node: Add yarn2 support

af4432f

yarn: update js plugin and remove setup.sh

46e5ab4

add chromium-headless-shell and update webkit to 22.04

255d509

catsout force-pushed the yarn2 branch from eb43ec2 to 255d509 Compare June 28, 2025 02:44

github-actions bot added node and removed yarn labels Jun 28, 2025

bbhtt added a commit that referenced this pull request Jul 25, 2025

node: Raise an error unless yarn lockfile is v1

6f390d7

Helps #361 v2 is WIP #252

node: Add yarn2 support #252

Are you sure you want to change the base?

node: Add yarn2 support #252

Uh oh!

Conversation

catsout commented Dec 28, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

gasinvein commented Jul 12, 2022

Uh oh!

proletarius101 commented Jul 20, 2022

Uh oh!

catsout commented Jul 20, 2022

Uh oh!

proletarius101 commented Jul 20, 2022

Uh oh!

catsout commented Jul 22, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

proletarius101 commented Jul 22, 2022

Uh oh!

catsout commented Jul 23, 2022

Uh oh!

proletarius101 commented Jul 23, 2022

Uh oh!

proletarius101 commented Sep 5, 2022

Uh oh!

proletarius101 commented Sep 6, 2022

Uh oh!

catsout commented Oct 25, 2022

Uh oh!

muelli commented Feb 28, 2023

Uh oh!

catsout commented Feb 28, 2023

Uh oh!

catsout commented Feb 28, 2023

Uh oh!

muelli commented Mar 1, 2023

Uh oh!

catsout commented Mar 1, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

tjanez commented Nov 11, 2024

Uh oh!

catsout commented Dec 26, 2024

Uh oh!

catsout commented Dec 26, 2024

Uh oh!

tjanez commented Apr 7, 2025

Uh oh!

MoralCode commented Jun 8, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

10 participants

catsout commented Dec 28, 2021 •

edited

Loading

catsout commented Jul 22, 2022 •

edited

Loading

catsout commented Mar 1, 2023 •

edited

Loading