Importing a source-repository-package cannot be overridden

[parsonsmatt]

Describe the bug

I have a cabal.project and a ghc94.cabal.project that I am using to get compatible with GHC 9.4 on the work codebase. To simplify the code, I'm using the import feature in Cabal 3.8 with ghc94.cabal.project to ensure that there is as little duplication as possible and to track what work needs to be done to integrate everything.

Unfortunately, because I have two different source-repository-package in each file for the same package, I get a conflict, and cabal doesn't try to override.

-- cabal.project

packages: .

source-repository-package
  type: git
  location: https://github.com/avieth/bytebuild.git
  tag: 036aa84fb037e62c4e7da4523c818541481b9a07
  --sha256: 9yiWSMGyi9vxVAV27YctWFYTDL0Uaa1j7TfyUdgMYtY=
  subdir: .

-- ghc94.cabal.project
packages: .

import: ./cabal.project

-- https://github.com/byteverse/bytebuild/pull/29
source-repository-package
    type: git
    location: https://github.com/parsonsmatt/bytebuild
    tag: e8dee26a4cc379a1971b428187fad51824de8e38

To Reproduce

Put the two cabal.project files in a directory somewhere with a .cabal file that depends on bytebuild.

$ cabal v2-build --project-file ghc94.cabal.project

Expected behavior

cabal clearly knows that I depend on bytebuild. And in the imported cabal.project, I satisfy that dependency with a source-repository-package. In the importing ghc94.cabal.project, I would expect the new source-repository-package to override the one in the import.

System information

• Operating system: Ubuntu
• cabal, ghc versions

λ cabal --version
cabal-install version 3.8.1.0
compiled using version 3.8.1.0 of the Cabal library 
λ ghc --version
The Glorious Glasgow Haskell Compilation System, version 9.4.2

[ulysses4ever]

A similar issue with imported constraints being not overridable was mentioned in the original ticket about "extended project files". @gbaz do you think overriding for either (constraints or source-repository-packages) may come with cabal 3.10?

[parsonsmatt]

Huh, that's surprising. I think I'd expect the naive approach to be something like:

type Dependencies = [Map PackageName DepInfo]

getDependency :: PackageName -> Dependencies -> Maybe DepInfo
getDependency packageName = 
    listToMaybe . mapMaybe (Map.lookup packageName)

where imports are snoced to the list, and so have lower priority.

[gbaz]

Overriding for constraints is certainly something we'd want. Overriding for source-repository-packages feels different to me. Cabal actually has no notion that these are repositories that are used to satisfy particular dependencies. It just treats them all as places that also have "local" packages that are part of the project. In the phase where they're resolved, there's no real way for cabal to see that the two provide the same package -- they're different urls, which could point to anything. It just so happens that they both provide the same package, but that's only something that can be observed once the build-plan construction is underway.

In this case, perhaps refactoring so there's a "core" file with everything but the source-repository-package, and then two different ones that import it, each of which provides a different one, is an appropriate solution?

[philderbeast]

Overriding for constraints is certainly something we'd want.

I understand that currently constraints are additive. If I want to import something like lts-18.28/cabal.config then override one of the constraints I can't. Instead I download the config file and remove the constraints I wish to override. Is there a better way?

[ulysses4ever]

@philderbeast there's none currently. See Gershom's comment I linked in my previous message.

[parsonsmatt]

Overriding for constraints is certainly something we'd want. Overriding for source-repository-packages feels different to me.

I think you're describing a difficulty in implementing this particular feature, rather than a reason to not want it.

In this case, perhaps refactoring so there's a "core" file with everything but the source-repository-package, and then two different ones that import it, each of which provides a different one, is an appropriate solution?

This is a viable work-around for right now, but it doesn't obviate the feature request.

The workflow I want is to:

1. Create a new.cabal.project for forward compatibility work
2. import the old cabal.project as the basis
3. As I discover problems with the old cabal.project, provide overrides for working with the new version in the new cabal.project

To answer "What needs to be done to be fully forward compatible?" I can simply see: are there any entries in new.cabal.project?

The workflow that this work-around requires is:

1. Create a new.cabal.project and a core.cabal.project. cabal.project imports from core, and new imports from core as well.
2. Copy everything into core.cabal.project.
3. As I discover problems with the existing cabal.project, move that declaration into cabal.project and provide the new one in new.

To answer the above question, I need to check both current and new - there shouldn't be anything in either file, it should all be safely in core.

This is a much more cumbersome workflow.

[Mikolaj]

I think you're describing a difficulty in implementing this particular feature, rather than a reason to not want it.

It sounds so to me as well. Given that, if somebody proposes a good method to implement it, that would move things forward.

[gbaz]

I think you're describing a difficulty in implementing this particular feature, rather than a reason to not want it.

Well I'm having difficulty conceptualizing the syntax and semantics of such a feature -- implementation can follow from specification. In particular, it would be ad-hoc and not always correct for any future import of a src-repository-package in one repo with one hash to override an existing one in the same repo with a different hash. So we need not a way of overriding but of removing a stanza, at which point we probably need a grammar for removing arbitrary stanzas. And I do not know of a lightweight grammar that makes that sort of reference easily possible, and certainly not one that would feel natural to end-users.

[philderbeast]

Overriding for constraints is certainly something we'd want.

I understand that currently constraints are additive. If I want to import something like lts-18.28/cabal.config then override one of the constraints I can't. Instead I download the config file and remove the constraints I wish to override. Is there a better way?

A reasonable expectation is that local overrides remote. For instance, if I cabal unpack to add a hackage package to my project then I can alter it in all but the version number and the local package will be used. When building stack-2.10.0 from source using cabal the cabal.project cannot import stackage nightly as-is. We don't even make an exception for a local package to escape its own version from the imported constraints.

> cabal build all --disable-tests
Resolving dependencies...
Error: cabal: Could not resolve dependencies:
[__0] next goal: stack (user goal)
[__0] rejecting: stack-2.10.0 (constraint from project config
stackage/ghc-9.2.4/cabal.config requires ==2.9.1)

[philderbeast]

On using the build of stack with cabal as test bed, disregarding the stack ==2.9.1 constraint from stackage nightly by downloading nightly and commenting out that constraint line. We're left with a constraint conflict with one other package:

> cabal build all --disable-tests
Resolving dependencies...
Error: cabal: Could not resolve dependencies:
[__0] trying: stack-2.10.0 (user goal)
[__1] next goal: pantry (dependency of stack)
[__1] rejecting: pantry-0.6.0 (constraint from project config
stackage/ghc-9.2.4/cabal.config requires ==0.5.7)

I can get this to work as-is now (save for commenting the stack constraint) by adding allow-older: pantry to the cabal.project which is saying let pantry-0.5.7 beat pantry-0.6.0 solving for pantry version.

cabal build all --disable-tests
Resolving dependencies...
Build profile: -w ghc-9.2.4 -O1
In order, the following will be built (use -v for more details):
 - pantry-0.5.7 (lib) (requires build)
 - stack-2.10.0 (lib:stack, exe:stack) (configuration changed)
...

This doesn't work the other way, I can't have allow-newer: pantry pick pantry-0.6.0. This seems unreasonable and unfair. With allow-newer the more local thing, the package I'm building, doesn't even get to talk over unfamiliar versions gatecrashing the party.

[philderbeast]

@parsonsmatt maybe we could give priority to shallower source-repository-package commit hashes in the project import tree as I have been exploring for package version equality constraints in #9510?