[bug] replace_requires prevents transitive dependencies from matching

[cheshirekow]

Describe the bug

Based on my understanding of the feature from the blog post and the documentation the [replace_requires] section should replace requirements everywhere they show up in the dependency graph. However, it seems that if a package shows up transitively in the graph, replacement will cause the "parent" dependency to fail matching. Perhaps this is due to (from the documentation):

References listed under this section work as a literal replacement of requires in recipes, and is done as the very first step before any other processing of recipe requirements, without processing them or checking for conflicts.

This issue arises when trying to use [replace_requires] to test a candidate version for an upstream dependency of a dependent pair:

project_a
├── project_b
└───┴ project_c

I've created a Minimal Working Example to demonstrate this issue. A summary of the MWE setup is:

1. package-a exists at version 1.0
2. package-b exists at version 1.0 and depends on package-a/1.0
3. Using a profile with [replace-requires] specifying a replacement package-a/1.0: package-a/1.1@dev/fix-a, package-b/1.0 is no longer installable because it's requirements do not match

The error message in this case is:

ERROR: Missing binary: package-b/1.0:237a411982f9012227dd778400a9e4199de8c61e

package-b/1.0: WARN: Can't find a 'package-b/1.0' package binary '237a411982f9012227dd778400a9e4199de8c61e' for the configuration:
[settings]
arch=x86_64
os=Linux
[requires]
package-a/1.Y.Z@dev/fix-a

ERROR: Missing prebuilt package for 'package-b/1.0'. You can try:
    - List all available packages using 'conan list "package-b/1.0:*" -r=remote'
    - Explain missing binaries: replace 'conan install ...' with 'conan graph explain ...'
    - Try to build locally from sources using the '--build=package-b/1.0' argument

More Info at 'https://docs.conan.io/2/knowledge/faq.html#error-missing-prebuilt-package'

How to reproduce it

For convenience I have attached this set of files as a single diff: conan-replace-requires-mwe.txt. The test setup can be recreated by the command:

patch -p1 < conan-replace-requires-mwe.txt

package-a/conanfile.py:

from conan import ConanFile, conan_version
from conan.tools.files import copy

class PackageAConan(ConanFile):
    name = "package-a"
    settings = "os", "arch"

package-b/conanfile.py:

from conan import ConanFile, conan_version
from conan.tools.files import copy

class PackageBConan(ConanFile):
    name = "package-b"
    settings = "os", "arch"
    requires = "package-a/1.0"

host-profile.txt: `

[settings]
os=Linux
arch=x86_64

[replace_requires]
package-a/1.0: package-a/1.1@dev/fix-a

repro.sh:

conan create package-a --version 1.0
conan create package-a --version 1.1 --user dev --channel fix-a
conan create package-b --version 1.0
conan install --no-remote --requires "package-b/1.0"
conan install --no-remote --profile:host host-profile.txt --requires "package-b/1.0"

[AbrilRBS]

Hi @cheshirekow thanks a lot for taking the time to create this issue.

I would not say that this is a bug, but expected behaviour.
From the error message and your reproduction steps, we see that you're:

• First building package-b without the replacement in conan create package-b --version 1.0
• And then trying to install it with the replacement conan install --no-remote --profile:host host-profile.txt --requires "package-b/1.0", which gets you the error you mention, ERROR: Missing prebuilt package for 'package-b/1.0'. You can try

This would be expected, because you have only created a binary for package-b when it requires package-a/1.0.

Note that for Conan, different user/channels are not binary compatible, and so you see that it's looking in your cache for a package-b that has

[requires]
package-a/1.Y.Z@dev/fix-a

The fix for this would be to either first create the package for package-b with the replacement, or calling conan install with the --build=missing argument added to let Conan automatically build the missing binary for you.

Let me know if this helps, happy to clear things up further if needed

[cheshirekow]

Thank you @AbrilRBS for replying. You mentioned:

different user/channels are not binary compatible

But as I understand, the intent of [replace_requires] in the profile is that it allows me to assert that a package version is binary compatible. The example from the blogpost using zlibng demonstrates this replacement.

Are you saying that the use of the user/channel is special and doesn't propagate through [replace_requires] the way that the version number does?

The fix for this would be to either first create the package for package-b with the replacement, or calling conan install with the --build=missing argument

In the real world, I do not have access to the source for package-b and I cannot build it myself. Once I complete testing on package-a/1.1@dev/fix-a I will coordinate with the maintainers for package-a and package-b to release new versions of their respective projects with this fix in place.

[cheshirekow]

From what you said about user/channel, I tried the following:

diff --git a/host-profile.txt b/host-profile.txt
index 5bd3015..19d05a4 100644
--- a/host-profile.txt
+++ b/host-profile.txt
@@ -3,4 +3,4 @@ os=Linux
 arch=x86_64
 
 [replace_requires]
-package-a/1.0: package-a/1.1@dev/fix-a
+package-a/1.0: package-a/1.1+dev-fix-a
diff --git a/repro.sh b/repro.sh
index 8b91958..87e5865 100755
--- a/repro.sh
+++ b/repro.sh
@@ -1,5 +1,5 @@
 conan create package-a --version 1.0
-conan create package-a --version 1.1 --user dev --channel fix-a
+conan create package-a --version 1.1+dev-fix-a
 conan create package-b --version 1.0
 conan install --no-remote --requires "package-b/1.0"
 conan install --no-remote --profile:host host-profile.txt --requires "package-b/1.0"

And this works as expected.

So it seems that user/channel somehow propagates differently than version through [replace_requires].

I find this to be surprising as I undertand user/channel as a recommended mechanism for releasing forked/modified/experimental package versions. It seems odd that these packages are not supported for [replace_requires] while different-named packages are supported.

More specifically, it is odd that I can replace foo/1.2 with bar/4.7 but I can't replace foo/1.2 with foo/1.2@user/channel.

[AbrilRBS]

So it seems that user/channel somehow propagates differently than version through [replace_requires].

The key point to understand here is that the graph expansion is decoupled from the binary existence checks.

When you replace a reference from the graph, this does not change what binaries the already built packages have been built.

So what happens when you replace just the version:

• The graph moves from say package-a/1.0 to package-a/1.1
• But package-b/1.0 has in its conaninfo.txt (Visible via conan list package-b/1.0:* too)

  [requires]
  package-a/1.Y.Z
  

• This works because both 1.1 and 1.0 match the expected binary as shown in the pattern

Now, if you change the user-channel:

• The graph moves from say package-a/1.0 to package-a/1.0@user/channel
• But package-b/1.0 still has in its conaninfo.txt

  [requires]
  package-a/1.Y.Z
  

• And this does not match the pattern in the requires, because as you've seen in the message error, it's looking for package-a/1.Y.Z@dev/fix-a.

Let me know if this clarifies your question as to why changing the user/channel does fail

[cheshirekow]

Thank you @AbrilRBS for providing that explanation. I think I understand why this happens and I can imagine why this might make sense as a design choice.

However, this behavior appears to subvert the documented nature of [replace_requires]. In the documentation it says:

This can be useful when a package can be changed by a similar one like zlib and zlibng.

So let's look at an MWE which does that:

package-a/conanfile.py:

from conan import ConanFile

class PackageAConan(ConanFile):
    name = "package-a"
    version = "1.0"
    settings = "os", "arch"
    requires = "zlib/1.3.1"

package-b/conanfile.py:

from conan import ConanFile

class PackageBConan(ConanFile):
    name = "package-b"
    version = "1.0"
    settings = "os", "arch"
    requires = ["zlib/1.3.1", "package-a/1.0"]

profile-1.txt:

[settings]
arch=x86_64
os=Linux
compiler=gcc
compiler.version=11
build_type=Release

profile-2.txt:

[settings]
arch=x86_64
os=Linux
compiler=gcc
compiler.version=11
build_type=Release

[replace_requires]
zlib/1.3.1: zlib-ng/2.2.5

repro.sh:

set -x

conan create --profile profile-1.txt package-a
conan create --profile profile-1.txt package-b
conan install --no-remote --profile profile-1.txt --requires "package-b/1.0"
conan install --no-remote --profile profile-2.txt --requires "package-b/1.0"

Which results in an error:

package-a/1.0: WARN: Can't find a 'package-a/1.0' package binary '63c6217bb7518f8e46125712ca8d00a80eaa1ce1' for the configuration:
[settings]
arch=x86_64
os=Linux
[requires]
zlib-ng/2.Y.Z

Note also that the documentation includes the following "Other Examples":

[replace_requires]
dep/*: dep/1.1               # To override dep/[>=1.0 <2] in recipes to a specific version dep/1.1)
dep/*: dep/*@system          # To override a dep/1.3 in recipes to dep/1.3@system
dep/*: dep/[>=1 <2]          # To override every dep requirement in recipes to a specific version range
dep/*@*/*: dep/*@system/*    # To override "dep/1.3@comp/stable" in recipes to the same version with other user but same channel
dep/1.1: dep/1.1@system      # To replace exact reference in recipes by the same one in the system
dep/1.1@*: dep/1.1@*/stable  # To replace dep/[>=1.0 <2]@comp version range in recipes by 1.1 version in stable chanel

Based on what you've described, it seems that most of these won't work with dependency diamonds encountered transitively.

Perhaps I am unduly influenced by the choice of example used in the documentation. zlib-ng is a binary-compatible drop-in replacement for zlib (if shared=True) so using this as an example seems to suggest that with a dependency structure like:

project_a
├── project_b
└───┴ zlib

then one should be able to install pre-build binaries package-a and package-b with zlib-ng as a replacement for zlib which they were built against (not true). Furthermore, the zlib-ng package versions in conancenter do not track zlib versions, so in order for this to actually be possible one would not expect the binary-compatibility-version-checking that conan seems to have in the presence of [replace_requires].

Anyway, I appreciate the explanations and this has helped me work-around the real-world-issue that I actually have. For posterity, I requested the upstream team to produce package-a/1.0-dev-fix-a instead of package-a/1.0@dev/fix-a and that allowed me to adequately test the package using [replace_requires].

I might humbly suggest that the documentation be updated to an example that is actually possible in the presence of a dependency diamond, or to include a discussion of how conan behaves in the presence of a dependency diamond.

[memsharded]

Hi @cheshirekow

it seems that what you are seeing there is not an issue of the replace_requires, but from the package_id computation.

The replace_requires does it task, and replaces the zlib package for the other zlib-ng.
But API-compatibility with the package doesn't necessarily mean ABI compatibility. And Conan requires that binaries that uses a different, replaced dependency, do re-build its binaries.

The replace_requires does its effect at the graph resolution stage, which is independent of the required binaries computation. It is like a "versioning" mechanism, similar to version-ranges or overrides, that allows to define and resolve to different "recipes" and versions in the dependency graph. It was never intended to implement transparent binary replacement, these replacements are already difficult to accomplish at the API level, so forcing a different binary of consumers for completely different dependencies has always been expected in the design.

I agree that some clarifications about this in the docs would be good and could help to scope the feature, thanks for the feedback!