Extensive CPAN breakage from two code removal projects

[jkeenan]

We are experiencing considerable "Blead Breaks CPAN (BBC)" breakage on blead. This has already been reported on the Perl 5 Porters mailing list (thread starting here, and as already reported as breakage of individual CPAN distributions: Mojolicious; CPAN; and DBIx::Class.

The two most likely sources of this breakage are commits which implemented two long-scheduled removals of deprecated code. I'll refer to these as "apostrophe in name removal" (#22303) and "given/when/switch/smartmatch" (#22370). I believe a strong case can be made for reverting at least one of these two changes -- but first let me sketch the scope of the breakage.

Blead Breakage as of Aug 16 2024

Yesterday I built and installed an unthreaded perl on Debian GNU/Linux 12 (bookworm). This was built at v5.41.2-289-gc85d5a6718:

commit c85d5a67185efe2ff72c5c523fa070bbc7ccecdf (HEAD -> blead, origin/blead, origin/HEAD, revert-apostrophe-20240816)
Author:     David Mitchell <davem@iabyn.nospamdeletethisbit.com>
AuthorDate: Thu Aug 8 11:19:46 2024 +0100
Commit:     David Mitchell <davem@iabyn.nospamdeletethisbit.com>
CommitDate: Thu Aug 15 19:31:22 2024 +0100

    ExtUtils::ParseXS: rename most object fields

If this had been a monthly dev release, I would have next installed Task::CPAN::Reporter against that perl and proceeded to attempt to install over 600 CPAN distributions, mostly in "CPAN River" order, and generated reports for CPANtesters.org. When I did this after the perl-5.41.2 release in July, out of 607 distros I got only 19 which were logged as anything other than test PASS in ~/.cpanreporter/reports-sent.db.

This was not a monthly dev release, and I did not want results from this run reported to CPANtesters. So I merely attempted to install the same 600+ distributions against c85d5a6 and logged all the output from ./bin/cpan. The process stalled several times, had to be restarted and never completed entirely. I was, however, able to glean a list of failing distros from the top of the CPAN river (though listed here in AUTHOR/distro order):

ANDK/CPAN-Checksums-2.14.tar.gz
BJOERN/XML-SAX-Expat-0.51.tar.gz
BPS/HTTP-Server-Simple-0.52.tar.gz
BTROTT/Digest-BubbleBabble-0.02.tar.gz
CHANSEN/Unicode-UTF8-0.62.tar.gz
CHOCOLATE/autobox-v3.0.1.tar.gz
DAGOLDEN/MooseX-Types-Stringlike-0.003.tar.gz
DAGOLDEN/Path-Iterator-Rule-1.015.tar.gz
DAGOLDEN/Types-Path-Tiny-0.006.tar.gz
DBOOK/PPIx-Utils-0.003.tar.gz
DCANTRELL/Data-Compare-1.29.tar.gz
DMUEY/File-Copy-Recursive-0.45.tar.gz
DOY/Dist-CheckConflicts-0.11.tar.gz
DOY/Eval-Closure-0.14.tar.gz
DROLSKY/DateTime-1.65.tar.gz
DROLSKY/DateTime-Format-Builder-0.83.tar.gz
DROLSKY/DateTime-Format-ISO8601-0.16.tar.gz
DROLSKY/DateTime-Format-Strptime-1.79.tar.gz
DROLSKY/DateTime-HiRes-0.04.tar.gz
DROLSKY/DateTime-Locale-1.43.tar.gz
DROLSKY/DateTime-TimeZone-2.62.tar.gz
DROLSKY/List-AllUtils-0.19.tar.gz
DROLSKY/List-SomeUtils-0.59.tar.gz
DROLSKY/Log-Dispatch-2.71.tar.gz
DROLSKY/Module-Implementation-0.09.tar.gz
DROLSKY/MooseX-StrictConstructor-0.21.tar.gz
DROLSKY/Package-DeprecationManager-0.18.tar.gz
DROLSKY/Params-Validate-1.31.tar.gz
DROLSKY/Params-ValidationCompiler-0.31.tar.gz
DROLSKY/Specio-0.48.tar.gz
DROLSKY/URI-FromHash-0.05.tar.gz
ELLIOTJS/PPIx-Utilities-1.001000.tar.gz
ETHER/Any-Moose-0.27.tar.gz
ETHER/B-Hooks-EndOfScope-0.28.tar.gz
ETHER/Class-Load-0.25.tar.gz
ETHER/Class-Load-XS-0.10.tar.gz
ETHER/Data-Visitor-0.32.tar.gz
ETHER/Devel-Declare-0.006022.tar.gz
ETHER/JSON-Any-1.40.tar.gz
ETHER/Module-Runtime-Conflicts-0.003.tar.gz
ETHER/Moose-2.2207.tar.gz
ETHER/MooseX-ConfigFromFile-0.14.tar.gz
ETHER/MooseX-Getopt-0.76.tar.gz
ETHER/MooseX-LazyRequire-0.11.tar.gz
ETHER/MooseX-MethodAttributes-0.32.tar.gz
ETHER/MooseX-Role-Parameterized-1.11.tar.gz
ETHER/MooseX-Role-WithOverloading-0.17.tar.gz
ETHER/MooseX-SimpleConfig-0.11.tar.gz
ETHER/MooseX-Types-0.50.tar.gz
ETHER/MooseX-Types-Path-Class-0.09.tar.gz
ETHER/MooseX-Types-Path-Tiny-0.012.tar.gz
ETHER/namespace-autoclean-0.29.tar.gz
ETHER/Package-Stash-0.40.tar.gz
ETHER/Package-Stash-XS-0.30.tar.gz
ETHER/Plack-Test-ExternalServer-0.02.tar.gz
ETHER/Test-CleanNamespaces-0.24.tar.gz
ETHER/Test-Kwalitee-1.28.tar.gz
ETHER/Try-Tiny-0.31.tar.gz
FREW/Syntax-Keyword-Junction-0.003008.tar.gz
GAAS/HTTP-Negotiate-6.01.tar.gz
GAAS/WWW-RobotRules-6.02.tar.gz
GARU/Clone-0.46.tar.gz
GETTY/HTTP-Body-1.23.tar.gz
GWILLIAMS/DateTime-Format-W3CDTF-0.08.tar.gz
HAARG/MooseX-Emulate-Class-Accessor-Fast-0.009032.tar.gz
ILMARI/Devel-OverloadInfo-0.007.tar.gz
ISHIGAKI/Module-CPANTS-Analyse-1.02.tar.gz
JJNAPIORK/Catalyst-Runtime-5.90131.tar.gz
KAZEBURO/Apache-LogFormat-Compiler-0.36.tar.gz
KAZEBURO/Cookie-Baker-0.12.tar.gz
KAZEBURO/HTTP-Entity-Parser-0.25.tar.gz
KENTNL/HTML-Tree-5.07.tar.gz
KENTNL/Test-File-ShareDir-1.001002.tar.gz
KMACLEOD/libxml-perl-0.08.tar.gz
LEEJO/CGI-4.66.tar.gz
LEONT/Const-Fast-0.014.tar.gz
LEONT/Software-License-0.104006.tar.gz
MANWAR/XML-XPath-1.48.tar.gz
MARKF/Test-utf8-1.02.tar.gz
MATEU/MooX-Types-MooseLike-0.29.tar.gz
MICHIELB/File-DesktopEntry-0.22.tar.gz
MICHIELB/File-MimeInfo-0.35.tar.gz
MIROD/XML-Twig-3.52.tar.gz
MIYAGAWA/HTTP-Server-Simple-PSGI-0.16.tar.gz
MIYAGAWA/Plack-1.0051.tar.gz
MIYAGAWA/Plack-Middleware-MethodOverride-0.20.tar.gz
MIYAGAWA/Plack-Middleware-ReverseProxy-0.16.tar.gz
MLEHMANN/Async-Interrupt-1.26.tar.gz
MLEHMANN/JSON-XS-4.03.tar.gz
MLEHMANN/Types-Serialiser-1.01.tar.gz
MSCHOUT/Log-Dispatch-FileRotate-1.38.tar.gz
MSCHWERN/URI-Find-20160806.tar.gz
MSTROUT/Package-Variant-1.003002.tar.gz
NEILB/Text-Table-Tiny-1.03.tar.gz
NIGELM/HTML-Formatter-2.16.tar.gz
OALDERS/HTML-Parser-3.83.tar.gz
OALDERS/HTTP-Cookies-6.11.tar.gz
OALDERS/HTTP-Daemon-6.16.tar.gz
OALDERS/HTTP-Message-6.46.tar.gz
OALDERS/libwww-perl-6.77.tar.gz
OALDERS/LWP-MediaTypes-6.04.tar.gz
OALDERS/LWP-Protocol-https-6.14.tar.gz
OALDERS/Net-HTTP-6.23.tar.gz
OALDERS/PPI-1.278.tar.gz
OALDERS/Test-WWW-Mechanize-PSGI-0.39.tar.gz
OALDERS/URI-5.28.tar.gz
OALDERS/WWW-Mechanize-2.18.tar.gz
PETDANCE/Test-Perl-Critic-1.04.tar.gz
PETDANCE/Test-WWW-Mechanize-1.60.tar.gz
PEVANS/Devel-MAT-0.53.tar.gz
PHAYLON/syntax-0.004.tar.gz
PLICEASE/Alien-Build-Plugin-Download-GitLab-0.01.tar.gz
PLICEASE/URI-ws-0.03.tar.gz
REHSACK/SQL-Statement-1.414.tar.gz
RIBASUSHI/DBIx-Class-0.082843.tar.gz
RIBASUSHI/namespace-clean-0.27.tar.gz
RJBS/App-Cmd-0.336.tar.gz
RJBS/Config-MVP-2.200013.tar.gz
RJBS/Config-MVP-Reader-INI-2.101465.tar.gz
RJBS/CPAN-Uploader-0.103018.tar.gz
RJBS/Dist-Zilla-6.032.tar.gz
RJBS/Getopt-Long-Descriptive-0.114.tar.gz
RJBS/Log-Dispatch-Array-1.005.tar.gz
RJBS/Log-Dispatchouli-3.007.tar.gz
RJBS/MooseX-OneArgNew-0.007.tar.gz
RJBS/MooseX-SetOnce-0.203.tar.gz
RJBS/MooseX-Types-Perl-0.101344.tar.gz
RJBS/Perl-PrereqScanner-1.100.tar.gz
RJBS/Pod-Elemental-0.103006.tar.gz
RJBS/Role-HasMessage-0.007.tar.gz
RJBS/Role-Identifiable-0.009.tar.gz
RJBS/Test-Fatal-0.017.tar.gz
SAMTREGAR/HTML-Template-2.97.tar.gz
SCHUBIGER/DateTime-Format-Natural-1.18.tar.gz
SIMBABQUE/HTML-Form-6.11.tar.gz
SKAJI/Mouse-v2.5.11.tar.gz
SPROUT/Sub-Delete-1.00002.tar.gz
SRI/Mojolicious-9.37.tar.gz
SWEETKID/Plack-Middleware-FixMissingBodyInRedirect-0.12.tar.gz
SWEETKID/Plack-Middleware-RemoveRedundantBody-0.09.tar.gz
SZABGAB/Test-Class-0.52.tar.gz
TARAO/Test-MockTime-HiRes-0.08.tar.gz
TJMATHER/XML-DOM-1.46.tar.gz
TOBYINK/Data-Perl-0.002011.tar.gz
TOBYINK/MooX-HandlesVia-0.001009.tar.gz
TOBYINK/Type-Tiny-2.004000.tar.gz
WYANT/PPIx-QuoteLike-0.023.tar.gz
WYANT/PPIx-Regexp-0.088.tar.gz
YANICK/Dancer-1.3521.tar.gz
YANICK/File-ShareDir-Tarball-0.2.2.tar.gz

Analysis

Caveats: I have not thoroughly analyzed the reasons for the breakage on a distro-by-distro basis. I suspect that most are due to "apostrophe in name removal" and a smaller number are due to given/when/switch/smartmatch. Some of these distros could have broken in the past month even before the "apostrophe" changes were applied to blead.

Nonetheless, that's a lot of patching and CPAN-releasing to be done. Some of the distros listed there have been mainstays in production for more than 25 years. And trying to conduct two major code removals in one production cycle -- not to mention in a single monthly development cycle -- is a recipe for a lot of resentment directed toward the Porters. Yes, there have been times when we made changes which we knew would generate "a lot" of CPAN breakage (say, up to 100 distros) and we organized to quickly submit patches upstream. But here we have breakage of over 150 distros -- and that's just at the top of the CPAN river and from a presumably quite diligent subset of all CPAN authors.

Suggestions

• Delay the perl-5.41.3 release scheduled for Tuesday, August 20 until we have a more thorough understanding of the scope of the breakage, have come to a consensus as to how to proceed, and have a plan to communicate with affected CPAN authors.

• Consider reverting the commits associated with one or the other or both of the two removal projects. Complicating factor: Some CPAN authors have already begun to issue new release to deal with the breakage, and some of those distros that ship with core have already been synched into blead. (I have a branch which can serve as a proof-of-concept for reversion of the "apostrophe" problem.)

[karenetheridge]

Try-Tiny contained some tests for smartmatch which would fail on 5.41.3, and it's at the very top of the cpan river; I've done a release for this to fix the tests, so if you try again you should see a substantial improvement (certainly everything in the Moose ecosystem depends on Try::Tiny, so many of those distributions should disappear from the failure list now).

[jkeenan]

Try-Tiny contained some tests for smartmatch which would fail on 5.41.3, and it's at the very top of the cpan river; I've done a release for this to fix the tests, so if you try again you should see a substantial improvement (certainly everything in the Moose ecosystem depends on Try::Tiny, so many of those distributions should disappear from the failure list now).

@karenetheridge, thanks very much for that. I also see that @atoomic has released a new version of Clone which was the missing prerequisite most often cited in the cpan logs. I have now kicked off a new screen session testing 530 CPAN distributions.

[jkeenan]

Try-Tiny contained some tests for smartmatch which would fail on 5.41.3, and it's at the very top of the cpan river; I've done a release for this to fix the tests, so if you try again you should see a substantial improvement (certainly everything in the Moose ecosystem depends on Try::Tiny, so many of those distributions should disappear from the failure list now).

@karenetheridge, thanks very much for that. I also see that @atoomic has released a new version of Clone which was the missing prerequisite most often cited in the cpan logs. I have now kicked off a new screen session testing 530 CPAN distributions.

Swift action on the part of several CPAN distribution maintainers has greatly alleviated the problems we were facing just two days ago. As a consequence, I no longer feel that we need to consider reverting either of the two big code removal projects or postponing Tuesday's scheduled release of perl-5.41.3, nor do we need a code freeze in advance of that release.

I say this on the basis of data gathered yesterday in comparison to similar data gathered last month and earlier this month. I often try to install a monthly dev release, then install Task::CPAN::Reporter against that perl, then try to install 500+ distributions, generating CPANtesters reports along the way. The result for each distro is logged in ~/.cpanreporter/reports-sent.db. This plain-text file will grow indefinitely, but if you rename the file on a regular basis and compress it, you get something more useful. So yesterday I set out to compare two files, one (reports-sent.db.before.20240808.gz) which held results generated between the perl-5.41.2 release in July and smaller CPANtesters runs done up till August 8, the other (reports-sent.db) which held the results of a fresh run begun yesterday after better versions of Try-Tiny and Clone had been uploaded.

The distros I test include (i) approximately 500 from the top of the CPAN River; (ii) most of my own CPAN distros; (iii) a list of reverse dependencies on Variable::Magic chosen because that module is particularly sensitive to changes in Perl's internals. Note: When testing the CPAN River, I do not customarily test distributions that are shipped with core. Hence, the data reported below does not detect or report the current failures in the CPAN distribution itself. These tests were conducted on FreeBSD-13; there are a handful of distros which FAIL on that platform but PASS on Linux.

Month-to-Month Comparisons

• Worsened: These 14 distros PASSed in July (earlier) but had a worse result yesterday (later). They are presumably adversely affected by either the "apostrophe" removal or the "switch/smartmatch" removal and need our continued attention.

{
  "common-sense"            => { earlier => "test PASS", later => "make UNKNOWN" },
  "Crypt-CBC"               => { earlier => "test PASS", later => "test DISCARD" },
  "Crypt-PBKDF2"            => { earlier => "test PASS", later => "test DISCARD" },
  "DBIx-Class"              => { earlier => "test PASS", later => "test FAIL" },
  "Dist-Zilla"              => { earlier => "test PASS", later => "test DISCARD" },
  "JSON-XS"                 => { earlier => "test PASS", later => "test DISCARD" },
  "Modern-Perl"             => { earlier => "test PASS", later => "test FAIL" },
  "Perl-PrereqScanner"      => { earlier => "test PASS", later => "test DISCARD" },
  "Sub-Delete"              => { earlier => "test PASS", later => "test FAIL" },
  "Syntax-Keyword-Junction" => { earlier => "test PASS", later => "test FAIL" },
  "Test-utf8"               => { earlier => "test PASS", later => "test FAIL" },
  "Type-Tiny"               => { earlier => "test PASS", later => "test FAIL" },
  "Types-Path-Tiny"         => { earlier => "test PASS", later => "test DISCARD" },
  "Types-Serialiser"        => { earlier => "test PASS", later => "test DISCARD" },
}

• Non-PASS, but previous status unknown: These 44 distros were not tested in July or early August but were tested yesterday. They failed to get a PASS, but whether that represents a new problem caused by either of the two code removals is not clear.

{
  "AnyEvent-AIO"                         => "test DISCARD",
  "CBOR-XS"                              => "test DISCARD",
  "common-header"                        => "test UNKNOWN",
  "Coro"                                 => "test DISCARD",
  "DBD-mysql"                            => "PL UNKNOWN",
  "DBD-Pg"                               => "PL DISCARD",
  "DBICx-TestDatabase"                   => "test DISCARD",
  "DBIx-Class-DynamicDefault"            => "test DISCARD",
  "DBIx-Class-InflateColumn-Object-Enum" => "test DISCARD",
  "DBIx-Class-TimeStamp"                 => "test DISCARD",
  "Devel-Events"                         => "test FAIL",
  "Devel-Events-Objects"                 => "test DISCARD",
  "Devel-FindRef"                        => "test DISCARD",
  "Dios"                                 => "test FAIL",
  "Exporter-Tidy"                        => "test FAIL",
  "IO-AIO"                               => "test DISCARD",
  "Kavorka"                              => "test DISCARD",
  "Lexical-Accessor"                     => "test DISCARD",
  "Lexical-Types"                        => "test FAIL",
  "Lexical-TypeTiny"                     => "test DISCARD",
  "Logger-Fq"                            => "make UNKNOWN",
  "Moops"                                => "test DISCARD",
  "MooseX-Lexical-Types"                 => "test DISCARD",
  "MooseX-Params"                        => "test FAIL",
  "MooX-late"                            => "test DISCARD",
  "MRO-Magic"                            => "test FAIL",
  "Net--RabbitMQ"                        => "test FAIL",
  "Parse-Keyword"                        => "test FAIL",
  "Pcore"                                => "make DISCARD",
  "Pinto"                                => "test DISCARD",
  "Proc-Fork"                            => "test DISCARD",
  "Protocol-HTTP2"                       => "test DISCARD",
  "Ref-Store"                            => "make DISCARD",
  "Return-Type"                          => "test DISCARD",
  "Scope-Upper"                          => "make UNKNOWN",
  "Sub-HandlesVia"                       => "test DISCARD",
  "Sub-Op"                               => "make UNKNOWN",
  "subs-auto"                            => "test FAIL",
  "Tapper-Schema"                        => "test DISCARD",
  "Test-Fixture-DBIC-Schema"             => "test DISCARD",
  "Tie-RefHash-Weak"                     => "test FAIL",
  "TryCatch"                             => "test DISCARD",
  "Variable-Temp"                        => "test DISCARD",
  "X11-Protocol"                         => "test FAIL",
} 

Tuesday's dev release will trigger a new round of "testing all of CPAN" on the part of some of our heavy-duty CPANtesters. I fully expect that there will be a lot of new failures for distributions farther down the CPAN River, e.g., distros that, for whatever reason, bundled their own copies of Test::More. Such failures are of concern not just for themselves, but also for the fact that their reverse dependencies cease to be reachable in dependency-order automated testing. So while the pre-5.41.3 crisis is mostly past, we will soon have a lot of storm damage to clean up.

[Tux]

common::sense is part of LEHMANNWARE. Most of hist distributions are protected by Canary::Stability which causes a FAIL on any perl 5.022 and up. Most recent update was April 2020. Do not expect any fixes for modern perl.
I think this is like a 'I don't care what happens after 5.022 and you are not allowed to update anything from me that depends on this'.
https://metacpan.org/dist/Canary-Stability/source/Stability.pm#L145
https://metacpan.org/module/Canary::Stability/requires

[jkeenan]

common::sense is part of LEHMANNWARE. Most of hist distributions are protected by Canary::Stability which causes a FAIL on any perl 5.022 and up. Most recent update was April 2020. Do not expect any fixes for modern perl. I think this is like a 'I don't care what happens after 5.022 and you are not allowed to update anything from me that depends on this'. https://metacpan.org/dist/Canary-Stability/source/Stability.pm#L145 https://metacpan.org/module/Canary::Stability/requires

It should be noted that Canary-Stability continues to PASS on CPANtesters; see:
http://fast-matrix.cpantesters.org/?dist=Canary-Stability;perl=5.41.3;reports=1#sl=7,1

A small patch to common::sense gets it working on blead:
https://rt.cpan.org/Ticket/Display.html?id=154926.

[Tux]

Marc disgusts RT, so I do not think he'll notice. The fact that Canary::Stability itself works does not take away the fact that using it will cause build failures. See https://metacpan.org/release/MLEHMANN/JSON-XS-4.03/source/Makefile.PL#L3

[karenetheridge]

re File::HomeDir::Tiny and ' as a package separator: I did garu/App-cpanminus-reporter#42 and dagolden/Path-Tiny#294 and genehack/app-gitgot#78

[Leont]

I have several distributions that I know are broken by the smartmatch change that haven't been detected by the usual mechanisms yet. I'm worried the problem is larger than we might think it is.

[jkeenan]

I have several distributions that I know are broken by the smartmatch change that haven't been detected by the usual mechanisms yet. I'm worried the problem is larger than we might think it is.

Can you name them? (I've been keeping an off-list log and would like to add them if needed.)

[karenetheridge]

Now that 5.41.3 is out (and 5.41.4 is just a few days away), have we got an updated list of known-affected distributions?

[jkeenan]

Now that 5.41.3 is out (and 5.41.4 is just a few days away), have we got an updated list of known-affected distributions?

Just ask, "Is this particular distribution dependent on DBIx::Class, Type::Tiny, or common::sense?" (Type::Tiny's author has been working on a fix, but as yet there has been no CPAN release.)