Roadmap for PEP 458

[di]

This is a meta-issue to document the roadmap for PyPI's support for PEP 458. This top-level comment will be updated as the roadmap progresses. Comments on this issue should be limited to a discussion of this roadmap only, e.g. whether there are steps missing, discovered to be necessary or resolved.

Roadmap

• PEP 458 is accepted

  • Defer PEP 458 python/peps#931
  • PEP 458: update dead or outdated references python/peps#1178
  • PEP 458: fix technical choices and remove ambiguity python/peps#1203
  • PEP 458: Change title to clarify intent python/peps#1247
  • PEP 458: add hash algorithm transition plan python/peps#1253
  • PEP 458: Add sponsor and update status python/peps#1261
  • PEP 458: Update abstract python/peps#1268
  • PEP 458: Update Discussions-To header python/peps#1269
  • PEP 458: Add Post-History header python/peps#1270
  • PEP 458: Add non-goals section python/peps#1280
  • PEP 458: Allow compression of json metadata python/peps#1281
  • PEP 458: Add clarification python/peps#1284
  • PEP 458: use "OpenPGP" instead of "GPG" python/peps#1287
  • PEP 458: update list of authors python/peps#1295
  • PEP 458: Mark as Accepted python/peps#1306

• Key generation and signing ceremony for PyPI

  • Runbook: https://github.com/psf/psf-tuf-runbook
  • Announcement: https://pyfound.blogspot.com/2020/10/key-generation-and-signing-ceremony-for.html
  • Recording: https://www.youtube.com/watch?v=jjAq7S49eow

• Pre-generate and serve simple index metadata #8487

  • Create utility to render and store transactional snapshot of simple detail page for a project #8586
  • Serve hashed simple index pages directly from storage via CDN
  • Serve simple index metadata

• Initial TUF services

  • [WIP] Breaking up of initial TUF services #8955

• Updates to python-tuf

  • Support abstract files and directories theupdateframework/python-tuf#1009
    • Implement filesystem abstraction secure-systems-lab/securesystemslib#232
  • Support abstract files and directories theupdateframework/python-tuf#1009
    • Port to securesystemslib with abstract files and directories (securesystemslib PR 232) theupdateframework/python-tuf#1024
  • Reading some attributes of a delegated targets role after loading a repository throws a KeyError theupdateframework/python-tuf#574
  • load_repository() does not load correctly Targets objects of delegated roles theupdateframework/python-tuf#1045
    • Load correctly the delegated Targets objects hierarchy theupdateframework/python-tuf#1052
  • load_repository does not load target file info for delegated targets metadata theupdateframework/python-tuf#1046
    • Load full target file info for delegated targets metadata theupdateframework/python-tuf#1049
  • support updating individual metadata upon addition of target file theupdateframework/python-tuf#1048
    • Add simple TUF role metadata model theupdateframework/python-tuf#1112
  • Support custom signing implementations in Metadata.sign method theupdateframework/python-tuf#1263
    • Add the Signer interface secure-systems-lab/securesystemslib#319
    • Make new api compatible with the Signing interface theupdateframework/python-tuf#1272
  • python-tuf 1.0.0 release: https://github.com/theupdateframework/python-tuf/projects/2

• Integrate with python-tuf

  • Support for bumping snapshots, bin roles, adding targets
    • [WIP] warehouse: TUF initialization #7488
    • TUF Initialization using python-tuf 2.0.0  #10870

• Populate top-level TUF roles

• Bring TUF keys online

  • HSMs containing the signing keys need to be distributed
  • Each keyholder needs to use their HSM to sign the top-level TUF targets
  • Create the online bits?

Downstream issues unblocked once this roadmap is complete:

• Implement PEP-458: Secure PyPI downloads with signed repository metadata pypa/pip#8585

---

This is likely incomplete, cc @ewdurbin @woodruffw @trishankatdatadog @JustinCappos @mnm678 @joshuagl @jku @pradyunsg @brainwane for your input & awareness.

[di]

(Sorry, cc @kairoaraujo as well!)

[di]

#8586 has been merged, next step on our end here is setting up the CDN to serve the hashed pages from storage.

[di]

tuf==1.0.0 has been released: https://pypi.org/project/tuf/1.0.0/

[brainwane]

I may be mistaken, but it looks like movement on PEP 458 has slowed -- what can we do to help get it moving again? Should I be watching https://github.com/jku/repository-playground or helping test #10870 or #8955 or #7488? Thanks!

[ofek]

Please let me know if I can help.