SDK: Fix so libraries dangling symlinks - #294
Conversation
98b91e4 to
772fcd6
Compare
sergio-costas
left a comment
sergio-costas
left a comment
There was a problem hiding this comment.
It looks fine, and even has a check to notify for any left dangling symlink.
|
Before merging this... what happens with the review tools? It seems that they do ensure that no soft link points outside the snap... |
|
So maybe the solution is to add all those links in Core24... |
Oh I didn't check review tools output... But if they don't work with this, then... The only option would be to copy the actual libraries in the sdk too and ensure that they don't get removed as duplicate... |
I still think that they should go in Core24: if you think about it, it IS being used to compile, when you build any snap... |
|
I really dislike having duplicated libraries. |
Yeah, but I feel that's the way... Since I'm pretty sure that core doesn't want to include dev packages contents. While us being a dev sdk we need to provide everything required to build. So if core is missing something, we have to. Implies more maintenance though, I agree. |
|
So... do we copy the required files from CoreXX into the right folder wherever there is a dangling pointer...? |
|
AFAIK is the only way we can... And ensure they get not deleted by duplicates checking tool |
|
Moving from #296:
What do you mean with "build itself from scratch", what is "itself"? In Libtool's case the -dev package (libc6-dev) was added to build-packages. |
Exactly: since Core24 doesn't have the soft link |
|
In fact... why is libpciaccess in gnome-46-2404-sdk if it is already in Core24? |
|
Ok, confirmed my suspicions: just by removing the broken soft link, the The problem is that the So if we remove a library in the gnome SDK because it is already in Core24, we must also remove the corresponding .pc file to avoid this. |
That makes sense. We never built libc6 though. |
|
@3v1n0 So, in my opinion, the true solution is to find which .pc file correspond to each duplicated library in Core24 and remove it. It's not easy, I now, but with some python magic... ;-) |
|
In fact... just checking the "Libs" and "Libs.private" parts should be enough to know which libraries are affected by a .pc file, so if they aren't available in gnome-XX-YY-sdk, then that .pc file can be removed. |
Well, I'm unsure either... These are build dependencies of dependencies we have in the SDK IIRC, I mean if any of these libraries headers/pc files are part of the gnome-sdk, then we should provide them too. For example for the pciaccess one: Since we have stage packages such as It's fine to remove them from the runtime snap, but not from the sdk. |
|
But then, what we have to do is to ensure that using the gnome SDK requires to install the corresponding stage-packages that are provided by CoreXX. |
|
Anyway... you don't necessarily need those dependencies to build something with those libraries... and if you need them for whatever reason, the build error will tell you what you have to add to the |
|
Ok, let's take a step back and see the whole thing... Core24 (and other Cores) are built just by "dumping" an Ubuntu base system, as seen here: https://github.com/canonical/core-base/blob/277049e8bf67fdeadb9afe92428220e02cf95142/snapcraft.yaml#L13 The point is that, when building new snaps against CoreXX, the development files aren't there. Where are they get? And the answer is that Also, there is a list of packages that must not be staged, because they are known to be included from Core... https://github.com/canonical/snapcraft/blob/0157e5b15e66af34a4b12d397e5f56a30eebb13d/snapcraft_legacy/internal/repo/_deb.py#L51 But all that I found at this point is only available in So the point, in my opinion, is to find where those required -dev files are defined, and send a patch for snapcraft to add |
|
Ok, it seems that the list of blocked packages is in craft-parts: https://github.com/canonical/craft-parts/blob/1f163c0ec0691182cf08da52cc1cfc7160618241/craft_parts/packages/deb.py#L57-L283 So... Should we add there |
|
Sorry, my connection is not only slow but today also flaky,
```
\rRetrieving image: metadata: 13% (93.34kB/s)\r \rRetrieving image:
metadata: 14% (96.15kB/s)\r \rRetrieving image: metadata: 14%
(96.15kB/s)\r \r' * Command standard error output: b'Error: Failed
instance creation: read tcp 192.168.0.5:49524->185.125.190.40:443: read:
connection reset by peer\n'
```
I'll have to leave that for tomorrow or give you the honor. :P
Message ID: ***@***.***>
|
|
I'll do it. |
Can we make a CI job that handles this a bit like autopkgtests? So whenever we push to main (at least) or a PR is approved, we check if some snaps are building fine. |
|
I'm preparing a patch for the runtime that allows to easily build it from a local SDK snap. |
|
Good, then can we also use it in CI? |
|
Yes, that's my idea. |
libraries .so files provided by -dev packages normally point to the versioned so file, but if these files are provided by the core snap or another base snap then we should instead use them, if they're not provided by the SDK itself. So, ensure that all the symlinks in the lib directory are actually pointing to a library file, trying to resolve them with the core-provided libraries as first try.
While this is fine for all the *.so.* files, we should not remove the *.so files by mistake, since these are dev-only files
This reverts commit 385bffb since it's not the right fix: .so files should be included if they are dependencies of dev libraries we stage, but we should instead also stage their dependencies instead
This reverts commit df43897.
The GNOME SDK should contain all the required dev-dependencies of the libraries we ship, including the core files dependencies. Now, even though core includes already some of the runtime libraries it does not include their *.so counterparts, and then that's now our duty to provide them all and in a working state. Since we can't really symlink to external paths in a snap, we should provide all the libraries here, since there's no way to fix the *.so files that are used at link time. Dropping them would be wrong though, since there may be libraries in the SDK that requires them at link time only, and it's not up to final applications to manage that.
If a dangling *.so file is found, it means that a package it links has not been staged, so fail in such case
We already have code to remove duplicates and these libraries are already dropped as part of the previous lines
…krb5support0. They provide the targets of the -dev's symlinks.
7995923 to
ac23645
Compare
|
Tests passes, so let's go with this in! Ideally we should also try to build apps with the SDK in CI, but this is fine for now I think. |
|
@3v1n0 That's a great idea, too. |
Yeah, I won't ask you to go further this road unless you want to though (in case with a super simple app only), unless you're enjoying with fighting with CI even more |
On riscv64, the build used to fail with: ``` :: [ +2 ms] /usr/bin/ld: plugins/handy_window/libhandy/libhandy_flutter.so: undefined reference to `__frexp' :: [ +2 ms] /usr/bin/ld: plugins/handy_window/libhandy/libhandy_flutter.so: undefined reference to `__ldexp' :: [ +2 ms] clang++: error: linker command failed with exit code 1 (use -v to see invocation) ``` A first solution to the problem was to stage `libc6` package, but it was not a good solution as it led to runtime issue (not being able to start the installer). The reason for that issue is the fact that libm on riscv64 uses __frexp and others as links to libc, contrary to amd64 where those symbols are only present in libc. Some packages in the snap were pulling static versions of those libraries (libc.a, libm.a) and the linker tried to statically link to libm.a which was missing those symbols as they were dynamically pointing to the libc.so ones. The right solution to that issue seems to be making sure that we don't stage libc.a/libm.a/libc.so/libm.so on riscv64. This commits removed the previously staged-package libc6 and instead uses the previously described solution of not staging the libc/libm files at all. See also: - ubuntu/gnome-sdk#240 - ubuntu/gnome-sdk#294 - ubuntu/gnome-sdk#296
On riscv64, the build used to fail with: ``` :: [ +2 ms] /usr/bin/ld: plugins/handy_window/libhandy/libhandy_flutter.so: undefined reference to `__frexp' :: [ +2 ms] /usr/bin/ld: plugins/handy_window/libhandy/libhandy_flutter.so: undefined reference to `__ldexp' :: [ +2 ms] clang++: error: linker command failed with exit code 1 (use -v to see invocation) ``` A first solution to the problem was to stage `libc6` package, but it was not a good solution as it led to runtime issue (not being able to start the installer). The reason for that issue is the fact that libm on riscv64 uses __frexp and others as links to libc, contrary to amd64 where those symbols are only present in libc. Some packages in the snap were pulling static versions of those libraries (libc.a, libm.a) and the linker tried to statically link to libm.a which was missing those symbols as they were dynamically pointing to the libc.so ones. The right solution to that issue seems to be making sure that we don't stage libc.a/libm.a/libc.so/libm.so on riscv64. This commits removed the previously staged-package libc6 and instead uses the previously described solution of not staging the libc/libm files at all. See also: - ubuntu/gnome-sdk#240 - ubuntu/gnome-sdk#294 - ubuntu/gnome-sdk#296
On riscv64, the build used to fail with: ``` :: [ +2 ms] /usr/bin/ld: plugins/handy_window/libhandy/libhandy_flutter.so: undefined reference to `__frexp' :: [ +2 ms] /usr/bin/ld: plugins/handy_window/libhandy/libhandy_flutter.so: undefined reference to `__ldexp' :: [ +2 ms] clang++: error: linker command failed with exit code 1 (use -v to see invocation) ``` A first solution to the problem was to stage `libc6` package, but it was not a good solution as it led to runtime issue (not being able to start the installer). The reason for that issue is the fact that libm on riscv64 uses __frexp and others as links to libc, contrary to amd64 where those symbols are only present in libc. Some packages in the snap were pulling static versions of those libraries (libc.a, libm.a) and the linker tried to statically link to libm.a which was missing those symbols as they were dynamically pointing to the libc.so ones. The right solution to that issue seems to be making sure that we don't stage libc.a/libm.a/libc.so/libm.so on riscv64. This commits removed the previously staged-package libc6 and instead uses the previously described solution of not staging the libc/libm files at all. See also: - ubuntu/gnome-sdk#240 - ubuntu/gnome-sdk#294 - ubuntu/gnome-sdk#296
On riscv64, the build used to fail with: ``` :: [ +2 ms] /usr/bin/ld: plugins/handy_window/libhandy/libhandy_flutter.so: undefined reference to `__frexp' :: [ +2 ms] /usr/bin/ld: plugins/handy_window/libhandy/libhandy_flutter.so: undefined reference to `__ldexp' :: [ +2 ms] clang++: error: linker command failed with exit code 1 (use -v to see invocation) ``` A first solution to the problem was to stage `libc6` package, but it was not a good solution as it led to runtime issue (not being able to start the installer). The reason for that issue is the fact that libm on riscv64 uses __frexp and others as links to libc, contrary to amd64 where those symbols are only present in libc. Some packages in the snap were pulling static versions of those libraries (libc.a, libm.a) and the linker tried to statically link to libm.a which was missing those symbols as they were dynamically pointing to the libc.so ones. The right solution to that issue seems to be making sure that we don't stage libc.a/libm.a/libc.so/libm.so on riscv64. This commits removed the previously staged-package libc6 and instead uses the previously described solution of not staging the libc/libm files at all. See also: - ubuntu/gnome-sdk#240 - ubuntu/gnome-sdk#294 - ubuntu/gnome-sdk#296
On riscv64, the build used to fail with: ``` :: [ +2 ms] /usr/bin/ld: plugins/handy_window/libhandy/libhandy_flutter.so: undefined reference to `__frexp' :: [ +2 ms] /usr/bin/ld: plugins/handy_window/libhandy/libhandy_flutter.so: undefined reference to `__ldexp' :: [ +2 ms] clang++: error: linker command failed with exit code 1 (use -v to see invocation) ``` A first solution to the problem was to stage `libc6` package, but it was not a good solution as it led to runtime issue (not being able to start the installer). The reason for that issue is the fact that libm on riscv64 uses __frexp and others as links to libc, contrary to amd64 where those symbols are only present in libc. Some packages in the snap were pulling static versions of those libraries (libc.a, libm.a) and the linker tried to statically link to libm.a which was missing those symbols as they were dynamically pointing to the libc.so ones. The right solution to that issue seems to be making sure that we don't stage libc.a/libm.a/libc.so/libm.so on riscv64. This commits removed the previously staged-package libc6 and instead uses the previously described solution of not staging the libc/libm files at all. See also: - ubuntu/gnome-sdk#240 - ubuntu/gnome-sdk#294 - ubuntu/gnome-sdk#296
3 participants
libraries .so files provided by -dev packages normally point to the
versioned so file, but if these files are provided by the core snap or
another base snap then we should instead use them, if they're not
provided by the SDK itself.
So, ensure that all the symlinks in the lib directory are actually
pointing to a library file, trying to resolve them with the
core-provided libraries as first try.
This fixes the compile issue we have with gnome-boxes as per the linking issues:
This fixes also these other cases:
UDENG-6680