Regenerate PDFs for PDFs which failed preservation.

[tpendragon]

Summary or User Story

A bunch of PDFs failed to preserve because of we're not sure why, but it seems constrained to that couple day timeline and these resources. We should re-generate their derivatives

Acceptance Criteria

• The affected PDFs are regenerated.

First step

Get the IDs of all the PDFs.

#7023 has a script for finding the bad PDFs:

csp = ChangeSetPersister.default
qs = csp.query_service
pos = qs.custom_queries.find_by_property(property: :preserved_object_id, value: [], model: PreservationObject, created_at: 2.months.ago..Time.now); nil
bad_pos = pos.select do |po|
  resource = Wayfinder.for(po).preserved_resource
  (resource.try(:preservation_targets) || []).any? do |file_metadata|
    node = po.binary_node_for(file_metadata)
    node && node.checksum[0].md5 != file_metadata.checksum[0].md5
  end
end; nil

#7025 is a blocker for this ticket.

Resulting in PreserveResourceJob failing

Backtrace

line 84 of [PROJECT_ROOT]/app/services/preserver.rb: Preserver#preserve_binary_node
line 42 of [PROJECT_ROOT]/app/services/preserver.rb: block in Preserver#preserve_binary_content
line 40 of [PROJECT_ROOT]/app/services/preserver.rb: Array#each

View full backtrace and more info at honeybadger.io

[hackartisan]

We investigated https://figgy.princeton.edu/catalog/36ec07a6-0ba1-46a4-abae-be7b303004f6. We found that the binary node on the preservation object had a checksum that was different from the checksum stored on the fileset's primary file. The binary node checksum is set when the preservation object is initialized. see:

figgy/app/services/preserver.rb

Line 98 in fde186a

preservation_object.binary_nodes += [file_metadata] unless file_metadata.persisted?

We found also that the checksum on the preservation object was the same as the checksum reported by google:

file = Valkyrie::StorageAdapter.find_by(id: preservation_object.binary_nodes.first.file_identifiers.first)
file.io.file.data

We downloaded and compared the 2 files; the one in figgy was good and the one in gcs could not open. Comparing the head of the two files, they looked the same at the beginning.

The fact that the checksum in the preservation object binary node did not match the checksum in the stored for the local primary file keeps preservation from proceeding, even when forced.

To resolve this item I did the following:

r = ChangeSetPersister.default.query_service.find_by(id: id)
preservation_object = Wayfinder.for(r).preservation_object
bnode = preservation_object.binary_nodes.first
bnode.checksum = MultiChecksum.for(Valkyrie::StorageAdapter.find_by(id: r.primary_file.file_identifiers.first))
preservation_object.binary_nodes = [bnode]
ChangeSetPersister.default.query_service.find_by(id: preservation_object.id)
ChangeSetPersister.default.persister.save(resource: preservation_object)
PreserveResourceJob.perform_now(id: r.id, lock_tokens: r.optimistic_lock_token, force_preservation: true)

We'll need to figure out a safe way to make this repeatable. There are some 250 resources failing in the way described by this ticket. They can't all be part of the same resource, because that resource only has 70-some pages. What needs to be confirmed before we can do the above? How much of that confirmation can we automate? Maybe we only need some sort of confirmation at the resource level, not the fileset level?

[hackartisan]

More analysis, I exported all the data from the honeybadger error and pulled the ids from it. there were 963 unique ids (as opposed to 255 jobs retrying -- I guess there were failures at some other point that aren't currently in the queue) CSV has been uploaded here https://docs.google.com/spreadsheets/d/1KB_y-uslUPU7Ot9yKQYbzJ1L6we4Cb3bvSEORcUuoZc/edit?gid=1006393144#gid=1006393144

Some interesting things about these resources:

• the file sets were all created late the night of 8/21/2025, eastern time
• the preservation objects were all created between 1/14 and 1/21 2025 (so far)
• not every fileset in a given resource is affected, but there do tend to be multiple filesets in a given resource
• preservation objects for different file sets may have the same md5s as one another (and the gcs object md5 matches that preservation object)
• they are all generated pdf pages

Here's the code I used to generate the csv

require 'json'
ids = JSON.parse(File.read(Rails.root.join("tmp/preservation_failure_ids.json")))
qs = ChangeSetPersister.default.query_service

resources = ids.map { |id| qs.find_by(id: id) }

path = Rails.root.join("tmp/preservation_failures.csv")
headers = [
  "class",
  "id",
  "created_at",
  "updated_at",
  "link",
  "primary_file label",
  "primary_file created_at",
  "primary_file updated_at",
  "parent id",
  "parent title",
  "parent link",
  "primary_file md5",
  "primary_file md5 hex encoded",
  "gcs file id",
  "gcs file md5",
  "gcs file created_at",
  "preservation object md5",
  "preservation object md5 hex encoded",
  "preservation object created_at"
]

rows = resources.map do |r|
  w = Wayfinder.for(r)
  p = w.parent
  po = w.preservation_object
  gcs_data = Valkyrie::StorageAdapter.find_by(id: po.binary_nodes.first.file_identifiers.first).io.file.data
  [
    r.class,
    r.id,
    r.created_at,
    r.updated_at,
    "https://figgy.princeton.edu/catalog/#{r.id}",
    r.primary_file.label,
    r.primary_file.created_at,
    r.primary_file.updated_at,
    p.id,
    p.title,
    "https://figgy.princeton.edu/catalog/#{p.id}",
    r.primary_file.checksum.first.md5,
    Base64.strict_encode64([r.primary_file.checksum.first.md5].pack("H*")),
    gcs_data[:file].id,
    gcs_data[:file].md5,
    gcs_data[:file].created_at,
    po.binary_nodes.first.checksum.first.md5,
    Base64.strict_encode64([po.binary_nodes.first.checksum.first.md5].pack("H*")),
    po.created_at
  ]
end

CSV.open(path, "w") do |csv|
  csv << headers
  rows.each do |row|
    csv << row
  end
end

[hackartisan]

Okay I think these PreserveResourceJobs happened because there was a metadata update on the resource record. I followed one up through the logs and it originated from a PreserveResourceJob on the parent resource, which was enqueued from a web machine. This makes me think it came from ChangeSetPersister::PreserveResource.

[hackartisan]

Next question, why weren't there existing preservation objects from when the files were originally generated? The preservation code retrieves those if they exist, but the preservation objects are all from the past week even though these files were generated in August.

[hackartisan]

Okay, I don't think August 22nd is actually relevant here, I think these are just related to each other so they were ingested at the same time, and now they're being edited at the same time

Here's the narrative.

A while back a bunch of pdfs were ingested. They were marked complete at some point. The tiff file sets were generated for them, but not preserved at that time (don't know why not -- is this something having to do with the fact that they are generated from pdfs?)

This past week, someone came to do metadata updates to these resources. saving the resources enqueued PreserveResourceJobs. In some cases, multiple PreserveResourceJobs were enqueued. I see this from the logs although I'm not sure why. In the case I looked at, there are several PATCH 302 and one PATCH 200 for a multi-volume work. the file set I was looking at ended up with 2 PreserveResourceJobs enqueued.

The first job creates the preservation object, in this case fetching a bad version of the file, calculating its checksum from that, uploading that bad file to gcs, and saving everything "successfully".

The second job pulls the preservation object, uses its checksum that was calculated earlier, then tries to send the good version of the file to gcs. gcs rejects it because the good file's checksum doesn't match the one we told it to compare against (the one we pulled from the preservation object). So the bad file stays up there and we have the job failure.