invalid-publisher error when using reusable workflow

[janosh]

My publish step sits in a resuable workflow .github/workflows/pytest-release.yml.

When setting up trusted publishing, should the workflow name be the file in the actual repo (test.yml in my case) or the reusable workflow file being exectued (pytest-release.yml)? The docs sound like the former which throws

Error: Trusted publishing exchange failure:
Token request failed: the server refused the request for the following reasons:

• invalid-publisher: valid token, but no corresponding publisher

[webknjaz]

Hello, reusable workflows aren't fully supported on the PyPI side at the moment. The only known case when this might work is when the reusable workflows are in the same repository.
This is tracked @ pypi/warehouse#11096 (comment).

[webknjaz]

@woodruffw should we improve the hint that PyPI returns to also suggest checking whether the workflow is reusable?

[woodruffw]

@woodruffw should we improve the hint that PyPI returns to also suggest checking whether the workflow is reusable?

IMO this would be a (small) mismatch: reusable workflows are a GitHub concept not an OIDC one, while these error messages are mostly limited generic to OIDC configuration problems or errors (and not specific publishers).

We could do it, but I think my preference would be for linking the troubleshooting page everywhere instead and having users diagnose from there -- that way we don't introduce individual trusted publisher implementation details and don't give them a potential red herring 🙂

[janosh]

The error message could state that the workflow was not found in the publishing repo itself which currently is a must. Seems generic enough. Other git hosters may have the same concept of reusable workflows.

Does it make sense to wait for reusable workflow support btw? Would be a shame to have to revert legacy publishing everywhere.

[woodruffw]

Other git hosters may have the same concept of reusable workflows.

Sorry, this is a source of confusion: trusted publishers are not necessarily git-based, or even source repositories at all: they might be arbitrary CI or cloud platforms (such as Google Cloud, which we're currently working on supporting).

In general, IMO PyPI should avoid responding with recommendations that it isn't confident in (e.g. the error might be because of a reusable workflow, or it might be something completely different). This unfortunately inconveniences some users, but keeps us from throwing red herrings at others.

[janosh]

Oh, thanks for clearing that up.

Another thing I was uncertain about: Is the need for the workflow file to be in the same repo a security measure?

[woodruffw]

Another thing I was uncertain about: Is the need for the workflow file to be in the same repo a security measure?

Nope! That's purely a technical limitation of the current approach. It's tracked as an item to fix as part of the issue linked above.

[mikegerber]

I stumbled upon this, and having the reusable workflow in the same repo also causes this error, it seems.

[woodruffw]

I stumbled upon this, and having the reusable workflow in the same repo also causes this error, it seems.

Can you share your trusted publisher configuration and the repository + workflow you're attempting to use it with?

[mikegerber]

I stumbled upon this, and having the reusable workflow in the same repo also causes this error, it seems.

Can you share your trusted publisher configuration and the repository + workflow you're attempting to use it with?

I've since somewhat reverted it, but I think this was it:

Trusted publisher configuration (not sure if the second one is necessary):

This was my attempt:
https://github.com/qurator-spk/setuptools_ocrd/tree/5415329d46c78981190b37389b89ab46b97a53c2/.github/workflows

release.yml should use release-github+pypi.yml. I put secrets: inherit into release.yml, should that also go into the (supposedly) reusable workflow release-github+pypi.yml?

It works beautifully non-reusable, see current master branch if necessary. Ideally I'd like to use the workflow in a few more projects where I want to publish to GitHub + PyPI.

[mikegerber]

Could gh-action-pypi-publish give more debugging info - if possible? Eg. the values of Publisher - Repository - Workflow - Environment name that should be used in the Trusted publisher configuration?

[webknjaz]

@mikegerber v1.8.9 has more debug info

[webknjaz]

release.yml should use release-github+pypi.yml. I put secrets: inherit into release.yml, should that also go into the (supposedly) reusable workflow release-github+pypi.yml?

@mikegerber
You don't use a relative in-repo path + you reuse the workflow from whatever master points to, which will not be the same commit most of the time. Maybe that's what prevents it from working.
Try uses: ./.github/workflows/release-github+pypi.yml — this makes sure that both workflows come from the same commit.

[virtuald]

PR #186 adds a warning to tell users that reusable workflows don't work.

... why don't they work? Is there a manual workaround using "The Manual Way" referenced at https://docs.pypi.org/trusted-publishers/using-a-publisher/ ?

[webknjaz]

There's no workaround. It has to be implemented in Warehouse.

[woodruffw]

To add to what @webknjaz said: the limitations with reusable workflows are currently documented here: https://docs.pypi.org/trusted-publishers/troubleshooting/#reusable-workflows-on-github

Adding support for reusable workflows is on my list of things to do, but there's no current timeline for it.

[woodruffw]

This keeps snaring people despite documentation, unfortunately, so maybe it's time to add a check to the action for it 😅

@webknjaz How would you feel about a check similar to the current third-party PR one? In effect, "fail if the underlying action event is workflow_call."

[webknjaz]

@woodruffw maybe not fail hard but show a big explainer for this specific case? I don't remember if workflow_call is exposed as an event. I think it's the event of the calling workflow that is visible to us. Also, some people managed to get it to work when the reusable workflow is within the same repo and all the secrets are inherited, it seems. I don't want to break them. But if the upload fails, showing something that is impossible to miss would be a good idea. For some reason I thought we already had that warning in the error messages calling out the reusable workflows special case explicitly but it seems like it's not there.

[woodruffw]

Makes sense! I'll try and set aside some time to add that warning.

[janosh]

Adding support for reusable workflows is on my list of things to do, but there's no current timeline for it.

would you accept a bounty for this feature? happy to use GH Sponsors

[woodruffw]

would you accept a bounty for this feature? happy to use GH Sponsors

This is a very thoughtful offer, thank you! Unfortunately, I can't really personally take bounties for this kind of work -- OSS engineering is my full time job, and I get paid a salary for it. I might do this eventually as a side project, but to do it professionally I'll need one of my clients (or a new client) to fund it.

(I am happy to discuss that further if you'd like, but I also completely understand if that kind of funding isn't what you're interested in.)

[alecbcs]

Also, some people managed to get it to work when the reusable workflow is within the same repo and all the secrets are inherited, it seems. I don't want to break them.

@webknjaz you mentioned some folks were able to successfully setup trusted publishers when using reusable workflows from the same repository. I'm trying to do the same, but running into errors even when inheriting secrets. Do you have any examples of a successful configuration here?

For reference my top level workflow is,
https://github.com/LLNL/repligit/blob/main/.github/workflows/ci.yml

Which is calling this reusable workflow after unit-tests have all passed,
https://github.com/LLNL/repligit/blob/main/.github/workflows/publish.yml

And I've set both as trusted publishers on PyPI for the project.

[webknjaz]

No, I don't. Until PyPI implements support, there's no point in wasting time on hacks. Plus, they'll likely break. For now, it's strictly discouraged. I think some people had it working before attestations were a thing. But if you want to disable useful features, you might as well go back to API tokens, while on it 🤷‍♂️

[alecbcs]

Thank you for the quick response. It's a bit disappointing that simply trying to order unit tests before a publish step forces you back to an API token, but it makes sense that we're better off waiting for official support from PyPI.

[webknjaz]

@alecbcs you can still have tests and the rest/most of the jobs in reusable workflows. Check out sphinx-contrib/sphinxcontrib-towncrier for how I do this.

[webknjaz]

P.S. Once PyPI supports this, I was thinking of providing a reusable workflow similar to this action so that people don't have to have diverging copies of basically the same thing all over GitHub...
P.P.S. Looking at your workflow, I think you may also be interested in my other creation for your last job: https://github.com/marketplace/actions/alls-green#why.

[webknjaz]

As for the publish workflow you have — I think that when you bundle it with testing, the job order should be different. You can build the dists first and then test those dists (currently, you build possibly diverging dists in your test jobs additionally), and only then, you could publish what you tested, not untested dists.
I also have another helper action for testing from sdist instead of a Git checkout that facilitates such a CI/CD structure and more: https://github.com/marketplace/actions/checkout-python-sdist.

[FelixS90]

Also, some people managed to get it to work when the reusable workflow is within the same repo and all the secrets are inherited, it seems. I don't want to break them.

@webknjaz you mentioned some folks were able to successfully setup trusted publishers when using reusable workflows from the same repository. I'm trying to do the same, but running into errors even when inheriting secrets. Do you have any examples of a successful configuration here?

For reference my top level workflow is, https://github.com/LLNL/repligit/blob/main/.github/workflows/ci.yml

Which is calling this reusable workflow after unit-tests have all passed, https://github.com/LLNL/repligit/blob/main/.github/workflows/publish.yml

And I've set both as trusted publishers on PyPI for the project.

I observed the same with https://github.com/ZEISS/pylibczirw. Publishing from a reusable workflow in the same repo without inheriting secrets used to work last year whereas with a recent try yesterday turned out to not work anymore ... It is a bit annoying not to know what caused this to change ...

[webknjaz]

not to know what caused this to change ...

Attestations.