crawler: adapt Lidl crawler to their new manual upload process - #105
Merged
Merged
Conversation
Since 2026-07-14, Lidl price list ZIPs are uploaded manually with ad-hoc names (often without a year), sometimes with CSVs nested in a subdirectory or wrapped in a ZIP inside the ZIP, and are sometimes linked only from the /cijene/cijene-u-trgovinama sub-page. * scrape both index pages and treat dates parsed from ZIP names as hints only (day/month match, lenient separators, optional year) * verify candidate ZIPs against the dates embedded in the CSV filenames, which remain machine-generated and stable * unwrap one level of ZIP-in-ZIP and handle nested directories Fixes #104 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #104.
Since 2026-07-14 Lidl appears to have switched to manually uploading their price lists. The ZIP names became ad-hoc and inconsistent (
securehub-download (Cijene_14.07.).zip,cijene za 16.07.zip, ...), the CSVs moved into a subdirectory inside the ZIP, some days the payload is a ZIP inside a ZIP, and some files are only linked from the/cijene/cijene-u-trgovinamasub-page rather than the main price list page. Each of these independently broke the crawler.The crawler now scrapes both pages, treats dates parsed from ZIP names as hints only (day/month match, lenient separators, optional year), verifies candidates against the dates embedded in the CSV filenames (which are still machine-generated and stable), and handles the nested directory / double-ZIP packaging.
Verified against all upload variants seen from 14.07 through 17.07, plus a byte-identical regression check on the old format (13.07).
Note: files now disappear from Lidl's site after a few days, so 14.07 and 15.07 likely can't be backfilled retroactively — the back-filling support discussed in #90 remains the long-term fix for that class of problem.