Lidarr post processing

[jsaddiction]

As you know, I have been lurking around looking for a way to force some specific tags to fill the void where lidarr fails in metadata tagging. I pushed a couple of PRs your way on the old "lidarr-extended" version of these scripts. I have learned a few things since then and would like to share my findings for your reference (some of which you may already know).

1. Every day each file (audio track) is scanned and it's metadata, if different than what lidarr expects, is then replaced. This process is not terribly long. For me around an hour for 76k tracks. If any post processing scripts are enabled using the "Track Retag" handle, the total execution time of the task is multiplied.
2. I was attempting to assign last.FM tags as genre using beets and this caused the "Rescan Library" task to take days. Since the task is called every day, Lidarr is always viewed as "busy" from the perspective of your scripts and is never allowed to download anything new.
3. Since any of my scripts or beets changes the genre tag on a track, this causes an infinite loop of Lidarr arguing with a script on the basis of which genres should be set.
4. Python takes around 400ms to start it's interpreter so just that multiplied by the number of tracks takes too long.
5. beets, written in python, is subject to this limitation. Adding lastgenre plugin or any other plugin that makes http requests are subject to even more delays.

I noticed that you still have some left over dependencies "pylast" in your setup scripts that are not being used within the requisite beets plugins configuration. You were smart not to carry over some of the edits I made.

What next?
In my never ending quest for genres to resemble something of use, I am rewriting the beets-lastgenre plugin in Rust with a few extra features. Firstly, It is backed by a database that stores a whitelist, genre tree, configuration and previous results from last.FM api calls. So far, the application consumes around 9ms to examine a track, and write genres which were stored in the database making no web api requests. In the even the track has never been seen before, the request takes up the most time and is something around the 200ms execution time. I plan to expose this at some point.

What's better?
I don't think that having two applications arguing about what tags should exist on a track is a good idea. This means there are two writes and two reads for every track in the library every day, unless of course last.FM and musicbrainz agree on the tagging for that track. Maybe these thoughts and actions would be better served on the upstream at lidarr to implement a more customized way to handle these tags. I've looked through their code and it just doesn't make much since to me. Lidarr seems to only collect genres for artist and album. Not sure yet how these are applied to each track but the album genres are not direct copies of artist genres thusly there must be some calculation that applies something to each track. I just want something that is parsed correctly and makes since when displayed by my music players.

Are you getting any reports of these tasks taking forever? Do any of your scripts apply tags to tracks? Have you noticed that track tags are rewritten? I have tried just about everything to prevent this behavior but Lidarr assumes ownership and thinks it's the authoritative source to tagging.

[RandomNinjaAtk]

So for me, TLDR: I'm not doing any post processing anymore. (not using any of the scripts for lidarr connect feature, except for Plex notify)

I believe Lidarr should not overwrite the tags if you do post-process or tag the files after import if the settings are set to not keep the files in sync.

However, I think because of tagging and file modifications, it causes lidarr to freak out and my guess is that is why when it rescans, it takes forever... Likely due to the fact that lidarr will essentially re-read the tags and match the files, which is an intensive process.

Hypothetically (untested), your best bet is to likely pre-process tag the files first before import, and not have lidarr tag them at all. So disable tagging in lidarr, and only have lidarr rename/organize the files.

This way the files are never modified after the fact and likely won't cause an issue with lidarr. But I'm making some assumptions.... without any testing to validate...

Edit:
Since not post-processing the files, my rescans are less than a minute for 120K+ tracks. I believe I updated AutoConfig to not automatically add these scripts anymore as a default configuration for that very reason. The best solution would be for Lidarr to improve it's tagging, but honestly the Dev's are so busy and it's not technically a problem, it will require someone else to put in the effort and make that happen....

[jsaddiction]

So I have just about exhausted my knowledge on the subject. One iteration of my script retained the mod time and ensured the file size were left intact after setting the genre tag. (I read somewhere in the lidarr code that this was checked). These files were still being picked up by the scanner and subsequently retagged. Taking a look around the lidarr db, I can't see how else lidarr could know that a file needs a tag read and replace performed on it. I have tried a number of different configs on lidarr to prevent this behavior but apparently lidarr and I are just not cooperating.

I might just have to learn C# just for this very issue.. I would love if lidarr had a way that you could prevent it from writing tags other than what is required. In reality, I don't know why lidarr would need anything other than the typical MB ids. Everything else could be "relational" in the database to keep it in check. In addition, the tagger lidarr uses, though quick, sometimes hangs or fails silently. Their PR / Issues list is quite long with lots of "tagging" issues to deal with. I would expect a division of responsibility would be in order. For instance package beets with lidarr and allow it to handle the tagging as a preprocess and then only update the MB ids of what is required. The user would then be left to his own demise with the config.

Could you tell me more on how you have things set with respect to tagging? I'd like to have the most "organized" library as I can to keep the OCD at bay. Also with that many tracks I imagine you have at least 1k genres. What applications do you use to play the music? They all seem to have different ways to deal with multiple genres (no one either reads or writes tags in the correct manner)

[RandomNinjaAtk]

I use Plexamp for playback, the tagging in lidarr is fine for basic organization and such. Lidarr just does not provide extended artist tags, which is really my biggest disappointment with it other than not supporting lyrics. So every track will have the same albumartist and artist for each song for the most part. The one exception for the artist tag, would be if the primary artist was different than the album artist, that would actually show.

It's not ideal, but it works. I would probably try my hypothetical untested solution if it weren't for how bad tagging is with beets (Personal Opinion). The tags beets tags files with are very messy and cause way more problems with sorting and organization in Plex/emby/jellyfin/kodi apps (in my experience, if you look at the tags, it looks like hot garbage...), which is my primary use-case.

The only app that I feel can truely get it right, with some minor tweaks, is Picard, but they do not have a headless way of using it via cli...

Edit:

At this point, I want a fully automated solution that just works without headaches, and just having Lidarr handle it, even with it's faults (tagging, lyrics, etc), is just easier to live with and accept that it is what it is for now.

[jsaddiction]

Agreed! I have tried just about everything there is and lidarr is just about the best out there albeit still very lacking in a lot of respects. Tagging for me is probably the primary concern. My OCD kicks in when I have that song by that artist I can't remember the name.. So my library ingress could be from multiple places, by artist, by album, by genre, etc. Usually there is only one detail I can remember. Having hot garbage for tags makes this impossible. A very close second place is audio quality. I am no audiophile but I do like a good quality library. Maybe it's time to set out on an adventure and create this all powerful all knowing music managing device!! I can dedicate some time to a big project like that but I don't have any experience in developing something big. However, I think the demand is high enough that more developers would want to join the programming party.

[RandomNinjaAtk]

You should work on improving lidarr, if your going to work on anything, instead of building yet another tool.... It’s already a pretty great tool and the team can always use the help.

Also pretty sure the Arr team are pretty friendly and would gladly give you pointers to help you get started.