Skip to content

fix: improve import speed - #335

Open
jgonet wants to merge 6 commits into
gabehf:mainfrom
jgonet:jgonet/no-import-images
Open

fix: improve import speed#335
jgonet wants to merge 6 commits into
gabehf:mainfrom
jgonet:jgonet/no-import-images

Conversation

@jgonet

@jgonet jgonet commented Aug 2, 2026

Copy link
Copy Markdown

Description

These changes modify few things:

  • we drop KOITO_FETCH_IMAGES_DURING_IMPORT. Imported data is typically large and it's better to finish importing quickly and then backfill the images. I don't think Koito needs this configuration var, simpler config out of the box is better.
  • we change SkipCacheImage to also avoid doing requests for image URL. Same reason as above.
  • we order backfill, image pruning, and cache migration after the import. This avoids us having to restart Koito to kick-off the backfill process. All four processes are dependent on each other so having them ordered make more sense to me.

Each commit is isolated and the patchset should read better going one-by-one instead of looking at full diff.

Part of my changes to improve importer speed. I have one branch that wasn't cleaned up at all and I don't intend to upstream (jgonet@b50cc94) but it dramatically improves import speed – calculating stats for each imported item takes more time with more imported items.
Other venues not implemented: when importing, it would be great to batch entries with ~100-300 items chunks.

Note

This is my first contact with Go, so treat those changes with suspicion. Feel free to take any changes or implement it yourself, however you see fit. While I used LLMs for virtually everything, I heavily guided it onto this approach.
My import on current version of Koito took few hours which didn't sound right and motivated these changes.

Note

While implementing, I found possible bugs:

  • Subsonic album search looks up 10 albums, filters by artists and takes first one anyway
  • CoverArtArchive has incorrect condition around "CoverArtArchive with Release Group MBID" error – I suspect you wanted to have else if in OK status code check.

Type of Change

  • Bug fix (non-breaking change which fixes an issue)
  • New feature (non-breaking change which adds functionality)
  • Breaking change (fix or feature that would cause existing functionality to not work as expected)
  • Documentation update

How Has This Been Tested?

  • Unit Tests
  • Manual Testing – I used LLMs to generate perf benchmarks on my real import data.

Checklist

  • I have performed a self-review of my own code
  • I have made corresponding changes to the documentation
  • I have ensured my changes generate no new warnings

AI/LLM Usage

  • I disclose that 100% of the code in this PR is written by an LLM. (please estimate)
  • I fully understand all changes made by LLM-generated code.
  • I have not and will not use an LLM to write any part of this PR description or PR comments.

jgonet added 6 commits August 2, 2026 15:52
It's better to have fast import and backfill
images after it.
We should avoid doing any image requests when importing.
We don't need to restart to kick it off.
@jgonet jgonet mentioned this pull request Aug 2, 2026
12 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant