Feature request: feature parity with debsums (locales purge, and generating missing checksums)

[mbideau]

Is your feature request related to a problem? Please describe.

I would like to be able to exclude all the locales files like debsumsdo, especially with #155 feature.
I would also like to be able to know which package lacks checksums, like debsums -g do.

Describe the solution you'd like

For excluding locale, may be port the code of debsums.
For being able to generate packages that lacks checksums, add a flag like debsums.

Describe alternatives you've considered

I currently use debsums but that is slower.
As an example this is what I get:

❯ dpkg -l | wc -l
2706

❯ time sudo LC_ALL=C.UTF8 debsums --all --changed --generate=missing,keep --deb-path=/var/cache/apt/archives >/tmp/debsums-changed.txt
real	1m47,072s
user	0m0,015s
sys	0m0,009s

❯ cat /tmp/debsums-changed.txt
/etc/bash.bashrc
/etc/default/bluetooth
/etc/cryptsetup-initramfs/conf-hook
/etc/davfs2/secrets
/etc/davfs2/davfs2.conf
/etc/gdm3/daemon.conf
/etc/initramfs-tools/initramfs.conf

❯ time sudo target/release/paketkoll --config-files include check >/tmp/paketkoll-changed.txt
real	0m32,938s
user	0m0,005s
sys	0m0,013s

❯ cat /tmp/paketkoll-changed.txt
aspell-en: /var/lib/aspell/en-common.rws type mismatch (expected unknown non-regular file, actual file)
aspell-en: /var/lib/aspell/en-variant_0.rws type mismatch (expected unknown non-regular file, actual file)
aspell-en: /var/lib/aspell/en-variant_1.rws type mismatch (expected unknown non-regular file, actual file)
aspell-en: /var/lib/aspell/en-variant_2.rws type mismatch (expected unknown non-regular file, actual file)
aspell-en: /var/lib/aspell/en-w_accents-only.rws type mismatch (expected unknown non-regular file, actual file)
aspell-en: /var/lib/aspell/en-wo_accents-only.rws type mismatch (expected unknown non-regular file, actual file)
aspell-en: /var/lib/aspell/en.compat type mismatch (expected unknown non-regular file, actual file)
aspell-en: /var/lib/aspell/en_AU-variant_0.rws type mismatch (expected unknown non-regular file, actual file)
aspell-en: /var/lib/aspell/en_AU-variant_1.rws type mismatch (expected unknown non-regular file, actual file)
aspell-en: /var/lib/aspell/en_AU-w_accents-only.rws type mismatch (expected unknown non-regular file, actual file)
aspell-en: /var/lib/aspell/en_AU-wo_accents-only.rws type mismatch (expected unknown non-regular file, actual file)
aspell-en: /var/lib/aspell/en_CA-variant_0.rws type mismatch (expected unknown non-regular file, actual file)
aspell-en: /var/lib/aspell/en_CA-variant_1.rws type mismatch (expected unknown non-regular file, actual file)
aspell-en: /var/lib/aspell/en_CA-w_accents-only.rws type mismatch (expected unknown non-regular file, actual file)
aspell-en: /var/lib/aspell/en_CA-wo_accents-only.rws type mismatch (expected unknown non-regular file, actual file)
aspell-en: /var/lib/aspell/en_GB-ise-w_accents-only.rws type mismatch (expected unknown non-regular file, actual file)
aspell-en: /var/lib/aspell/en_GB-ise-wo_accents-only.rws type mismatch (expected unknown non-regular file, actual file)
aspell-en: /var/lib/aspell/en_GB-ize-w_accents-only.rws type mismatch (expected unknown non-regular file, actual file)
aspell-en: /var/lib/aspell/en_GB-ize-wo_accents-only.rws type mismatch (expected unknown non-regular file, actual file)
aspell-en: /var/lib/aspell/en_GB-variant_0.rws type mismatch (expected unknown non-regular file, actual file)
aspell-en: /var/lib/aspell/en_GB-variant_1.rws type mismatch (expected unknown non-regular file, actual file)
aspell-en: /var/lib/aspell/en_US-w_accents-only.rws type mismatch (expected unknown non-regular file, actual file)
aspell-en: /var/lib/aspell/en_US-wo_accents-only.rws type mismatch (expected unknown non-regular file, actual file)
bash: /etc/bash.bashrc checksum mismatch (expected md5:89269e1298235f1b12b4c16e4065ad0d, actual md5:e60c15481abcc5a3585fd14bcb5f616e)
bluez: /etc/default/bluetooth checksum mismatch (expected md5:189d66a6a4f14ed849f0d26e60c73f67, actual md5:1e25cf981f29147187311b94f8e0ea89)
cryptsetup-initramfs: /etc/cryptsetup-initramfs/conf-hook checksum mismatch (expected md5:9a127a9edfca239740eba9197b0b901f, actual md5:794f983c71cee3f01dfb84ceb8488425)
davfs2: /etc/davfs2/davfs2.conf checksum mismatch (expected md5:b2972a9687a7f7138d660be5fe57e5fe, actual md5:b906e68cb6abc3c81595b6f0bb188bf3)
davfs2: /etc/davfs2/secrets checksum mismatch (expected md5:5c95e0dd2bc89ba9c1a7e5a8761a1655, actual md5:74144cfe02071635d1238caea8d22f58)
fwupd: /var/cache/app-info missing or inaccessible file/directory/...
fwupd: /var/cache/app-info/xmls missing or inaccessible file/directory/...
gdm3: /etc/gdm3/daemon.conf checksum mismatch (expected md5:c6aca2790519798ce963b8b3254e22b3, actual md5:b844c2271c3897a90000c110648169d3)
initramfs-tools-core: /etc/initramfs-tools/initramfs.conf checksum mismatch (expected md5:7cd32817dcb81991da2e2a928a32fdb0, actual md5:199a4b3b43026549cc11e93a3f99dc7b)
media-types: /etc/mime.types checksum mismatch (expected md5:f4631d08bcc92bf2dde274696d7b4b35, actual md5:e8937e06f21a0edb49813f91567be8e6)
nfs-common: /var/lib/nfs/state type mismatch (expected unknown non-regular file, actual file)
nfs-kernel-server: /var/lib/nfs/etab type mismatch (expected unknown non-regular file, actual file)
nfs-kernel-server: /var/lib/nfs/rmtab type mismatch (expected unknown non-regular file, actual file)

I don't have package that misses checksums, so I can't test debsums -g.

[VorpalBlade]

This is really 3 issues in one, so here goes.

Ignoring files

I was going to say you can do this already with --ignore <REGEX> but apparently that is only supported by paketkoll check-unexpected not the plain check.

In that case I believe it ties into directly how we walk the file system for check-unexpected so we don't even walk into the ignored directory trees. I don't quite know how that would work for check, it would need a different implementation there since it works quite differently.

I don't really want to add and maintain a list of which files localepurge is likely to remove, since I believe it is multiple, I seem to remember it will remove files from /usr/share/doc as well for example.

Aspell & NFS differences

I'm not sure what is going on with aspell etc there, my best interpretation is that:

• paketkoll detected a file that isn't in md5sums but is listed as installed
• paketkoll makes the assumption that means the file is not a regular file (instead it might be a symlink, directory, fifo or something even more exotic)
• paketkoll then found a regular file instead and reports that.

The good news is that if I install aspell-en on my Raspeberry Pi 5 I can reproduce that specific weirdness. And it appears to be exactly what I said above, md5sums is incomplete compared to the lists file. However, aspell-en has a weird post-install script that calls update-dictcommon-aspell. Maybe that is related?

I then went and looked at the source package and found this:

~/tmp/aspell-en-2020.12.07-0 $ grep -R var/lib .
./debian/changelog:  * Exclude /var/lib/aspell from dh_md5sums. closes: #324111, #346410, #374949
./debian/rules:DEB_DH_MD5SUMS_ARGS += -Xvar/lib/aspell
[...]

Well, it appears this was intentional.

I don't know why debsums did not report anything there. Maybe it doesn't detect this weird case? I think the goal should be to detect as much discrepancies as possible (rather have false positives than false negatives). But I could definitely see having a "--ignore-unexpected-type" flag that is opt-in.

I have not looked at the nfs packages, but my guess is that the situation is similar.

Generating missing checksums

I haven't seen this on Debian for many years to be honest. All packages nowdays seem to have checksums? Also do you know how debsums does this? Is it from installed files or from downloading the packages again? I would strongly prefer the latter, as we can't possibly know what files have been changed, the installed system state might not be trustworthy.

Also my understanding of the man page is that debsums would write out the missing data, and thus "fix" the package for any future uses. So it would only be needed once.

It seems like a very niche feature to me nowdays. You even say so yourself, you don't have any of them. So I am likely not going to implement this.