[Bug] Many, many hours of artwork and duration backfill activity at every startup

[nano9g]

Describe the bug
I've imported my 20+ year scrobble history from Last.fm and the Koito database now has 346,631 listens, 79,167 tracks, 39,149 albums, and 17,242 artists. Every time the Koito container is started, it spends over nine hours trying to backfill album/artist images and track durations for the exact same tracks and albums.

Many of the FetchMissingAlbumImages are also for tracks that where the album is empty, so the request ends up being for FetchMissingAlbumImages: Attempting to fetch missing album image | title=(null).

This seems like it puts unnecessary load on the MusicBrainz and album art servers, especially if Koito is restarted several times in succession, since it’s extremely unlikely to find any new data in a short timeframe.

In my case, it’s trying to backfill data for:

• 8,791 album images
• 1,334 artist images
• 31,726 track durations

To Reproduce

1. Have a large listening history
2. Restart Koito
3. Observe a huge amount of activity from FetchMissingAlbumImages, FetchMissingArtistImages, and BackfillTrackDurationsFromMusicBrainz in the log
4. Even if this work is allowed to run to completion, Koito will do it all again at the next startup

The KOITO_FETCH_IMAGES_DURING_IMPORT variable doesn't seem to affect this behavior (not that it seems like it should from the description).

Expected behavior
Koito shouldn't spend hours trying to backfill data for the same items every time it starts up.

There are probably multiple ways to address this, but my thought is that the functions responsible for backfilling artwork and durations could flag items where one attempt has been made and avoid retrying them at startup after that. Then maybe have a weekly-by-default job to re-check (job scheduling could be modified by an environment variable).

For FetchMissingAlbum/ArtistImages, it might also make sense to re-check on a lazy-load when that specific album/artist is accessed by the web UI.

Finally, FetchMissingAlbumImages probably shouldn't be making requests if the album is empty.

Screenshots/Logs

Here's a snippet from the beginning of the logs. It goes on like this for a long time. Let me know if you need more. :)

koito  | {"level":"debug","time":1782905115337,"message":"Engine: Enabling console logging"}
koito  | 2026-07-01T11:25:15Z INF > Koito v0.3.2 |
koito  | 2026-07-01T11:25:15Z DBG > Engine: Starting application initialization |
koito  | 2026-07-01T11:25:15Z DBG > Engine: Checking config directory: /etc/koito |
koito  | 2026-07-01T11:25:15Z INF > Engine: Using config directory: /etc/koito |
koito  | 2026-07-01T11:25:15Z DBG > Engine: Checking import directory: /etc/koito/import |
koito  | 2026-07-01T11:25:15Z DBG > Engine: Initializing database connection |
koito  | 2026-07-01T11:25:15Z INF > Engine: Using SQLite database driver |
koito  | 2026/07/01 11:25:15 goose: no migrations to run. current version: 2
koito  | 2026-07-01T11:25:15Z INF > Engine: Database connection established |
koito  | 2026-07-01T11:25:15Z DBG > Engine: Checking for default user |
koito  | 2026-07-01T11:25:15Z DBG > Engine: Initializing MusicBrainz client |
koito  | 2026-07-01T11:25:15Z INF > Engine: MusicBrainz client initialized |
koito  | 2026-07-01T11:25:15Z DBG > Engine: Initializing image sources |
koito  | 2026-07-01T11:25:15Z INF > Engine: Image sources initialized |
koito  | 2026-07-01T11:25:15Z INF > Engine: Using default CORS policy |
koito  | 2026-07-01T11:25:15Z DBG > Engine: Setting up HTTP server |
koito  | 2026-07-01T11:25:15Z INF > Engine: Beginning startup tasks... |
koito  | 2026-07-01T11:25:15Z DBG > Engine: Checking import configuration |
koito  | 2026-07-01T11:25:15Z INF > Engine: Pruning orphaned images |
koito  | 2026-07-01T11:25:15Z INF > Engine: Listening on :4110 |
koito  | 2026-07-01T11:25:15Z DBG > Importer: Checking for import files... |
koito  | 2026-07-01T11:25:15Z INF > Engine: Checking image cache migration status |
koito  | 2026-07-01T11:25:15Z INF > Engine: Running duration backfill task |
koito  | 2026-07-01T11:25:15Z INF > Engine: Attempting to fetch missing artist images |
koito  | 2026-07-01T11:25:15Z INF > Engine: Attempting to fetch missing album images |
koito  | 2026-07-01T11:25:15Z INF > FetchMissingArtistImages: Starting backfill of missing artist images |
koito  | 2026-07-01T11:25:15Z INF > Engine: Initialization finished |
koito  | 2026-07-01T11:25:15Z DBG > Fetching artist images to backfill from ID | ID=0
koito  | 2026-07-01T11:25:15Z INF > FetchMissingAlbumImages: Starting backfill of missing album images |
koito  | 2026-07-01T11:25:15Z DBG > Fetching album images to backfill from ID | ID=0
koito  | 2026-07-01T11:25:15Z INF > BackfillTrackDurationsFromMusicBrainz: Starting backfill of track durations from MusicBrainz |
koito  | 2026-07-01T11:25:15Z DBG > Fetching tracks to backfill from ID | ID=0
koito  | 2026-07-01T11:25:15Z DBG > FetchMissingArtistImages: Attempting to fetch missing artist image | title="Uppermost & Fūji"
koito  | 2026-07-01T11:25:15Z DBG > GetArtistImage: Subsonic image fetching is disabled |
koito  | 2026-07-01T11:25:15Z DBG > Sending request to ImageSrc: GET https://api.deezer.com/search/artist?q=Uppermost+%26+F%C5%ABji |
koito  | 2026-07-01T11:25:15Z DBG > Adding ImageSrc request to queue |
koito  | 2026-07-01T11:25:15Z DBG > BackfillTrackDurationsFromMusicBrainz: Backfilling duration from MusicBrainz | mbz_id=5a72b196-7952-3b6a-8fd8-2a102df237c5 title="En Hand I Himlen (The Sound Of Arrows Remix)"
koito  | 2026-07-01T11:25:15Z DBG > Adding MusicBrainz request to queue |
koito  | 2026-07-01T11:25:15Z DBG > FetchMissingAlbumImages: Attempting to fetch missing album image | title="Underworld vs. The Misterons Live Jam 12.12.07"
koito  | 2026-07-01T11:25:15Z DBG > Attempting to find album image from CoverArtArchive |
koito  | 2026-07-01T11:25:15Z DBG > Attempting to find album image from Deezer |
koito  | 2026-07-01T11:25:15Z DBG > Finding album image for Underworld vs. The Misterons Live Jam 12.12.07 from artist(s) [Underworld] |
koito  | 2026-07-01T11:25:15Z DBG > Sending request to ImageSrc: GET https://api.deezer.com/search/album?q=artist%3A%22Underworld%22album%3A%22Underworld+vs.+The+Misterons+Live+Jam+12.12.07%22 |
koito  | 2026-07-01T11:25:15Z DBG > Adding ImageSrc request to queue |
koito  | 2026-07-01T11:25:15Z DBG > GetArtistImage: Could not find artist image from Deezer | error="GetArtistImages: artist image not found"
koito  | 2026-07-01T11:25:15Z DBG > GetArtistImage: LastFM image fetching is disabled |
koito  | 2026-07-01T11:25:15Z INF > FetchMissingArtistImages: Failed to fetch artist image | name="Uppermost & Fūji"
koito  | 2026-07-01T11:25:15Z DBG > FetchMissingArtistImages: Attempting to fetch missing artist image | title="Let’s Eat Grandma"
koito  | 2026-07-01T11:25:15Z DBG > Sending request to ImageSrc: GET https://api.deezer.com/search/album?q=Underworld+vs.+The+Misterons+Live+Jam+12.12.07 |
koito  | 2026-07-01T11:25:15Z DBG > Adding ImageSrc request to queue |
koito  | 2026-07-01T11:25:15Z DBG > GetArtistImage: Subsonic image fetching is disabled |
koito  | 2026-07-01T11:25:15Z DBG > Sending request to ImageSrc: GET https://api.deezer.com/search/artist?q=Let%E2%80%99s+Eat+Grandma |
koito  | 2026-07-01T11:25:15Z DBG > Adding ImageSrc request to queue |
koito  | 2026-07-01T11:25:15Z ERR > MusicBrainz request failed | error="received non-ok status from MusicBrainz: 404 Not Found"
koito  | 2026-07-01T11:25:15Z ERR > BackfillTrackDurationsFromMusicBrainz: Failed to fetch track from MusicBrainz | error="GetTrack: getEntity: received non-ok status from MusicBrainz: 404 Not Found" title="En Hand I Himlen (The Sound Of Arrows Remix)"
koito  | 2026-07-01T11:25:15Z DBG > BackfillTrackDurationsFromMusicBrainz: Backfilling duration from MusicBrainz | mbz_id=555421e4-bb6f-3848-8cd7-2602c73a2ab4 title="Going Nowhere (Sebastian Remix)"
koito  | 2026-07-01T11:25:15Z DBG > Adding MusicBrainz request to queue |
koito  | 2026-07-01T11:25:15Z DBG > GetArtistImage: Could not find artist image from Deezer | error="GetArtistImages: artist image not found"
koito  | 2026-07-01T11:25:15Z DBG > GetArtistImage: LastFM image fetching is disabled |
koito  | 2026-07-01T11:25:15Z INF > FetchMissingArtistImages: Failed to fetch artist image | name="Let’s Eat Grandma"
koito  | 2026-07-01T11:25:15Z DBG > FetchMissingArtistImages: Attempting to fetch missing artist image | title="夢想家 bliss"
koito  | 2026-07-01T11:25:15Z DBG > GetArtistImage: Subsonic image fetching is disabled |
koito  | 2026-07-01T11:25:15Z DBG > Sending request to ImageSrc: GET https://api.deezer.com/search/artist?q=%E5%A4%A2%E6%83%B3%E5%AE%B6+bliss |
koito  | 2026-07-01T11:25:15Z DBG > Adding ImageSrc request to queue |
koito  | 2026-07-01T11:25:15Z DBG > GetAlbumImage: Could not find artist image from Deezer | error="GetAlbumImages: album image not found"
koito  | 2026-07-01T11:25:15Z INF > FetchMissingAlbumImages: Failed to fetch album image | name="Underworld vs. The Misterons Live Jam 12.12.07"
koito  | 2026-07-01T11:25:15Z DBG > FetchMissingAlbumImages: Attempting to fetch missing album image | title="French Express Vol. 1"
koito  | 2026-07-01T11:25:15Z DBG > Attempting to find album image from CoverArtArchive |
koito  | 2026-07-01T11:25:15Z DBG > Attempting to find album image from Deezer |
koito  | 2026-07-01T11:25:15Z DBG > Finding album image for French Express Vol. 1 from artist(s) [StewRat] |
koito  | 2026-07-01T11:25:15Z DBG > Sending request to ImageSrc: GET https://api.deezer.com/search/album?q=artist%3A%22StewRat%22album%3A%22French+Express+Vol.+1%22 |
koito  | 2026-07-01T11:25:15Z DBG > Adding ImageSrc request to queue |
koito  | 2026-07-01T11:25:16Z DBG > GetArtistImage: Could not find artist image from Deezer | error="GetArtistImages: artist image not found"
koito  | 2026-07-01T11:25:16Z DBG > GetArtistImage: LastFM image fetching is disabled |
koito  | 2026-07-01T11:25:16Z INF > FetchMissingArtistImages: Failed to fetch artist image | name="夢想家 bliss"
koito  | 2026-07-01T11:25:16Z DBG > FetchMissingArtistImages: Attempting to fetch missing artist image | title="Tycho & Cautious Clay"
koito  | 2026-07-01T11:25:16Z DBG > GetArtistImage: Subsonic image fetching is disabled |
koito  | 2026-07-01T11:25:16Z DBG > Sending request to ImageSrc: GET https://api.deezer.com/search/artist?q=Tycho+%26+Cautious+Clay |
koito  | 2026-07-01T11:25:16Z DBG > Adding ImageSrc request to queue |
koito  | 2026-07-01T11:25:16Z DBG > Sending request to ImageSrc: GET https://api.deezer.com/search/album?q=French+Express+Vol.+1 |
koito  | 2026-07-01T11:25:16Z DBG > Adding ImageSrc request to queue |
koito  | 2026-07-01T11:25:16Z DBG > GetArtistImage: Could not find artist image from Deezer | error="GetArtistImages: artist image not found"
koito  | 2026-07-01T11:25:16Z DBG > GetArtistImage: LastFM image fetching is disabled |
koito  | 2026-07-01T11:25:16Z INF > FetchMissingArtistImages: Failed to fetch artist image | name="Tycho & Cautious Clay"
koito  | 2026-07-01T11:25:16Z DBG > FetchMissingArtistImages: Attempting to fetch missing artist image | title="Mall Talk Collective"
koito  | 2026-07-01T11:25:16Z DBG > GetArtistImage: Subsonic image fetching is disabled |
koito  | 2026-07-01T11:25:16Z DBG > Sending request to ImageSrc: GET https://api.deezer.com/search/artist?q=Mall+Talk+Collective |
koito  | 2026-07-01T11:25:16Z DBG > Adding ImageSrc request to queue |
koito  | 2026-07-01T11:25:16Z DBG > GetAlbumImage: Could not find artist image from Deezer | error="GetAlbumImages: album image not found"
koito  | 2026-07-01T11:25:16Z INF > FetchMissingAlbumImages: Failed to fetch album image | name="French Express Vol. 1"
koito  | 2026-07-01T11:25:16Z DBG > FetchMissingAlbumImages: Attempting to fetch missing album image | title="Shiny Disco Club Presents: Millenium Disco (vol.1)"
koito  | 2026-07-01T11:25:16Z DBG > Attempting to find album image from CoverArtArchive |
koito  | 2026-07-01T11:25:16Z DBG > Attempting to find album image from Deezer |
koito  | 2026-07-01T11:25:16Z DBG > Finding album image for Shiny Disco Club Presents: Millenium Disco (vol.1) from artist(s) [Xinobi] |
koito  | 2026-07-01T11:25:16Z DBG > Sending request to ImageSrc: GET https://api.deezer.com/search/album?q=artist%3A%22Xinobi%22album%3A%22Shiny+Disco+Club+Presents%3A+Millenium+Disco+%28vol.1%29%22 |
koito  | 2026-07-01T11:25:16Z DBG > Adding ImageSrc request to queue |
koito  | 2026-07-01T11:25:16Z DBG > GetArtistImage: Could not find artist image from Deezer | error="GetArtistImages: artist image not found"
koito  | 2026-07-01T11:25:16Z DBG > GetArtistImage: LastFM image fetching is disabled |
koito  | 2026-07-01T11:25:16Z INF > FetchMissingArtistImages: Failed to fetch artist image | name="Mall Talk Collective"
koito  | 2026-07-01T11:25:16Z DBG > FetchMissingArtistImages: Attempting to fetch missing artist image | title="Ｒｕｎｎｅｒｓ Ｃｌｕｂ ９５"
koito  | 2026-07-01T11:25:16Z DBG > GetArtistImage: Subsonic image fetching is disabled |
koito  | 2026-07-01T11:25:16Z DBG > Sending request to ImageSrc: GET https://api.deezer.com/search/artist?q=%EF%BC%B2%EF%BD%95%EF%BD%8E%EF%BD%8E%EF%BD%85%EF%BD%92%EF%BD%93+%EF%BC%A3%EF%BD%8C%EF%BD%95%EF%BD%82+%EF%BC%99%EF%BC%95 |
koito  | 2026-07-01T11:25:16Z DBG > Adding ImageSrc request to queue |
koito  | 2026-07-01T11:25:16Z DBG > Sending request to ImageSrc: GET https://api.deezer.com/search/album?q=Shiny+Disco+Club+Presents%3A+Millenium+Disco+%28vol.1%29 |
koito  | 2026-07-01T11:25:16Z DBG > Adding ImageSrc request to queue |
koito  | 2026-07-01T11:25:16Z ERR > MusicBrainz request failed | error="received non-ok status from MusicBrainz: 404 Not Found"
koito  | 2026-07-01T11:25:16Z ERR > BackfillTrackDurationsFromMusicBrainz: Failed to fetch track from MusicBrainz | error="GetTrack: getEntity: received non-ok status from MusicBrainz: 404 Not Found" title="Going Nowhere (Sebastian Remix)"
koito  | 2026-07-01T11:25:16Z DBG > BackfillTrackDurationsFromMusicBrainz: Backfilling duration from MusicBrainz | mbz_id=cc92f191-621d-4465-b638-d46a11dbe380 title="All Possible States (Reprise)"
koito  | 2026-07-01T11:25:16Z DBG > Adding MusicBrainz request to queue |
koito  | 2026-07-01T11:25:16Z DBG > GetArtistImage: Could not find artist image from Deezer | error="GetArtistImages: artist image not found"
koito  | 2026-07-01T11:25:16Z DBG > GetArtistImage: LastFM image fetching is disabled |
koito  | 2026-07-01T11:25:16Z INF > FetchMissingArtistImages: Failed to fetch artist image | name="Ｒｕｎｎｅｒｓ Ｃｌｕｂ ９５"
koito  | 2026-07-01T11:25:16Z DBG > FetchMissingArtistImages: Attempting to fetch missing artist image | title="The Altered Beast"
koito  | 2026-07-01T11:25:16Z DBG > GetArtistImage: Subsonic image fetching is disabled |
koito  | 2026-07-01T11:25:16Z DBG > Sending request to ImageSrc: GET https://api.deezer.com/search/artist?q=The+Altered+Beast |
koito  | 2026-07-01T11:25:16Z DBG > Adding ImageSrc request to queue |
koito  | 2026-07-01T11:25:16Z DBG > GetAlbumImage: Could not find artist image from Deezer | error="GetAlbumImages: album image not found"
koito  | 2026-07-01T11:25:16Z INF > FetchMissingAlbumImages: Failed to fetch album image | name="Shiny Disco Club Presents: Millenium Disco (vol.1)"
koito  | 2026-07-01T11:25:16Z DBG > FetchMissingAlbumImages: Attempting to fetch missing album image | title="Rekids One"
koito  | 2026-07-01T11:25:16Z DBG > Attempting to find album image from CoverArtArchive |
koito  | 2026-07-01T11:25:16Z DBG > Attempting to find album image from Deezer |
koito  | 2026-07-01T11:25:16Z DBG > Finding album image for Rekids One from artist(s) [Rekid] |
koito  | 2026-07-01T11:25:16Z DBG > Sending request to ImageSrc: GET https://api.deezer.com/search/album?q=artist%3A%22Rekid%22album%3A%22Rekids+One%22 |
koito  | 2026-07-01T11:25:16Z DBG > Adding ImageSrc request to queue |
koito  | 2026-07-01T11:25:17Z DBG > GetArtistImage: Could not find artist image from Deezer | error="GetArtistImages: artist image not found"
koito  | 2026-07-01T11:25:17Z DBG > GetArtistImage: LastFM image fetching is disabled |
koito  | 2026-07-01T11:25:17Z INF > FetchMissingArtistImages: Failed to fetch artist image | name="The Altered Beast"
koito  | 2026-07-01T11:25:17Z DBG > FetchMissingArtistImages: Attempting to fetch missing artist image | title="Lord Leopard & Little Boots"
koito  | 2026-07-01T11:25:17Z DBG > GetArtistImage: Subsonic image fetching is disabled |
koito  | 2026-07-01T11:25:17Z DBG > Sending request to ImageSrc: GET https://api.deezer.com/search/artist?q=Lord+Leopard+%26+Little+Boots |
koito  | 2026-07-01T11:25:17Z DBG > Adding ImageSrc request to queue |
koito  | 2026-07-01T11:25:17Z DBG > Sending request to ImageSrc: GET https://api.deezer.com/search/album?q=Rekids+One |
koito  | 2026-07-01T11:25:17Z DBG > Adding ImageSrc request to queue |
koito  | 2026-07-01T11:25:17Z ERR > MusicBrainz request failed | error="received non-ok status from MusicBrainz: 404 Not Found"
koito  | 2026-07-01T11:25:17Z ERR > BackfillTrackDurationsFromMusicBrainz: Failed to fetch track from MusicBrainz | error="GetTrack: getEntity: received non-ok status from MusicBrainz: 404 Not Found" title="All Possible States (Reprise)"
koito  | 2026-07-01T11:25:17Z DBG > BackfillTrackDurationsFromMusicBrainz: Backfilling duration from MusicBrainz | mbz_id=7ac7a050-fd9f-428e-989e-783f807ffe90 title="All Possible States"
koito  | 2026-07-01T11:25:17Z DBG > Adding MusicBrainz request to queue |
koito  | 2026-07-01T11:25:17Z DBG > GetArtistImage: Could not find artist image from Deezer | error="GetArtistImages: artist image not found"

Version:

• Koito version: v0.3.2

Additional context
Thank you for all your hard work on Koito! I'd be happy to provide any additional information that would be helpful.

[noisesfromspace]

I'm still importing my 250k listens but could it be that you have your export in the /exports directory and every startup it sees the file and tries to backfill all the missing stuff every restart?

[nano9g]

I'm still importing my 250k listens but could it be that you have your export in the /exports directory and every startup it sees the file and tries to backfill all the missing stuff every restart?

That's a good thought, but the import definitely finished because a) I saw the messages in the log about the completion and b) the JSON file was moved from import to import_complete. My import directory is now empty. I've also allowed the backfill processes to complete multiple times (BackfillTrackDurationsFromMusicBrainz: Backfill complete, and similar messages from the two artwork functions), but if I restart Koito it redoes all the artwork and duration fetching for items it wasn't able to backfill.