Overriding specific repos in `repos.yaml` may not work as intended

[CodeGat]

See PR #104

Background

After the merging of the above PR, build-ci is erroneously picking develop as the default ref for builtin - see https://github.com/ACCESS-NRI/access-spack-packages/actions/runs/22743783447/job/65963073688?pr=396#step:12:151. This is a last resort if it can't get any other value from the repos.yaml. See https://github.com/ACCESS-NRI/build-ci/blob/v3/.github/workflows/ci.yml#L336.

I suspect that this is because the :: in repos.builtin:: is being interpreted literally as a key called builtin:, rather than spack interpreting it as it's "override" operator. I say this because when comparing the raw repos.yaml against the scope-merged spack config get repos before the above PR was merged, we get:

# repos.yaml
repos::
  access_spack_packages:
    git: https://github.com/ACCESS-NRI/access-spack-packages.git
    branch: api-v2
    destination: $spack/../access-spack-packages
  builtin:
    git: https://github.com/spack/spack-packages.git
    # gnu packages: bump (#3390)
    commit: 383e969358c951abe17623696083e6f862c4488e
---
# spack config get repos
repos:
  access_spack_packages:
    git: https://github.com/ACCESS-NRI/access-spack-packages.git
    branch: api-v2
    destination: $spack/../access-spack-packages
  builtin:
    git: https://github.com/spack/spack-packages.git
    # gnu packages: bump (#3390)
    commit: 383e969358c951abe17623696083e6f862c4488e

Note that the repos:: key in repos.yaml is correctly converted to repos: in spack config get repos - this means its special syntax was interpreted correctly, and it's scoping rules applied.

Now note the current main:

# repos.yaml
repos:
  access_spack_packages::
    git: https://github.com/ACCESS-NRI/access-spack-packages.git
    branch: api-v2
    destination: $spack/../access-spack-packages
  builtin::
    git: https://github.com/spack/spack-packages.git
    # gnu packages: bump (#3390)
    commit: 383e969358c951abe17623696083e6f862c4488e
---
# spack config get repos
repos:
  'access_spack_packages:':
    git: https://github.com/ACCESS-NRI/access-spack-packages.git
    branch: api-v2
    destination: $spack/../access-spack-packages
  'builtin:':
    git: https://github.com/spack/spack-packages.git
    # gnu packages: bump (#3390)
    commit: 383e969358c951abe17623696083e6f862c4488e

I interpret this to mean that the names of repositories are being interpreted literally as builtin: (and access_spack_packages:, rather than interpreting the special :: syntax.

Therefore I wonder if the :: syntax does not apply to that type of key.

Ways Forward

• build-ci can handle this bug with an updated yq statement, meaning it won't default to develop.
• But I am wondering if the :: operator is being applied correctly at all in this case, given the above. In that case, we may need to find a different place to put the ::, if at all.

[CodeGat]

I've found a small wrinkle in spack config get compared spack config blame that might explain it.

spack config get is not correctly interpreting the :: all the time, whereas spack config blame is.

In the current main of spack-config (the one with builtin::):

[root@72f118266fb2 /]# spack config get repos

repos:
  'access_spack_packages:':
    git: https://github.com/ACCESS-NRI/access-spack-packages.git
    branch: api-v2
    destination: $spack/../access-spack-packages
  'builtin:':
    git: https://github.com/spack/spack-packages.git
    # gnu packages: bump (#3390)
    commit: 383e969358c951abe17623696083e6f862c4488e

[root@72f118266fb2 /]# spack config blame repos

---                                 repos:
/opt/spack/etc/spack/repos.yaml:3     access_spack_packages:
/opt/spack/etc/spack/repos.yaml:4       git: https://github.com/ACCESS-NRI/access-spack-packages.git
/opt/spack/etc/spack/repos.yaml:5       branch: api-v2
/opt/spack/etc/spack/repos.yaml:6       destination: $spack/../access-spack-packages
/opt/spack/etc/spack/repos.yaml:7     builtin:
/opt/spack/etc/spack/repos.yaml:8       git: https://github.com/spack/spack-packages.git
/opt/spack/etc/spack/repos.yaml:10      commit: 383e969358c951abe17623696083e6f862c4488e

Note that the access_spack_packages and builtin are correctly parsed without the :: in spack config blame, but aren't in spack config get.

Alternatively, in the spack-config commit before (the one with repo::), the yaml is parsed correctly for both.

[root@72f118266fb2 spack-config]# spack config get repos

repos:
  access_spack_packages:
    git: https://github.com/ACCESS-NRI/access-spack-packages.git
    branch: api-v2
    destination: $spack/../access-spack-packages
  builtin:
    git: https://github.com/spack/spack-packages.git
    # gnu packages: bump (#3390)
    commit: 383e969358c951abe17623696083e6f862c4488e

[root@72f118266fb2 spack-config]# spack config blame repos

---                                 repos:
/opt/spack/etc/spack/repos.yaml:3     access_spack_packages:
/opt/spack/etc/spack/repos.yaml:4       git: https://github.com/ACCESS-NRI/access-spack-packages.git
/opt/spack/etc/spack/repos.yaml:5       branch: api-v2
/opt/spack/etc/spack/repos.yaml:6       destination: $spack/../access-spack-packages
/opt/spack/etc/spack/repos.yaml:7     builtin:
/opt/spack/etc/spack/repos.yaml:8       git: https://github.com/spack/spack-packages.git
/opt/spack/etc/spack/repos.yaml:10      commit: 383e969358c951abe17623696083e6f862c4488e

[scheibelp]

In general output like 'builtin:': should be fine (spack derives additional meaning from :: but because it isn't native yaml, it can display strange - but it should be understood)

I think this is doing a proper override, but the reason for the behavior you are seeing is:

• the default destination is based on a hash of the git property
• two configured repo entries have the same git (the one you have here, and one in a different config scope that chooses branch: develop)
• If I set something like this up and then do spack repo update builtin for example, the repo then updates to be the commit in the context of my env

As for how to address it:

• For now, can you set up alternative destinations, or use spack repo update builtin?
• For later, I should bring up the possibility of generating new destination if the repo entry lists a commit or branch

[CodeGat]

Thanks @scheibelp ! I'll look into those solutions you raised :)

Just regarding the differing output for spack config blame vs spack config get in #105 (comment), is that expected? I often pipe spack config get through to yq for processing related to CI, and was wondering if (in the first code block of that comment) the 'access_spack_packages:': from spack config get was supposed to be access_spack_packages: - like how repo: is consistent across the second code example.

[scheibelp]

Just regarding the differing output for spack config blame vs spack config get in #105 (comment), is that expected?

I don't think it's expected in the sense that it's explicitly documented, but

• It's useful to have a command that displays ::
• The output of config blame I assume it's less important from a parsing perspective to generate one output over the other
• IMO if we want to guarantee dumping yaml without :: (or "key:":), then that should be an option on config get, which doesn't exist right now, although --json does do this for JSON output