Input effects without RT priority

[my4ng]

EasyEffects Version

7.2.3

What package are you using?

Fedora

Distribution

Fedora 41

Describe the bug

Unlike the output effects that pipewire use with realtime priority via rtkit, the input effects only have normal priority. This is especially problematic as the Deep Noise Remover effect causes high latency cracking sound and even crashes easyeffects when the system is under heavy load:

...
Feb 01 08:45:16 fedora-my4ng easyeffects[51679]: 2025-02-01T08:45:16.358Z | WARN |  deep_filter_ladspa | DF 866dd930b3bb | Underrun detected (RTF: 1.04). Processing too slow!
Feb 01 08:45:16 fedora-my4ng easyeffects[51679]: 2025-02-01T08:45:16.443Z | WARN |  deep_filter_ladspa | DF 866dd930b3bb | Underrun detected (RTF: 1.04). Processing too slow!
Feb 01 08:45:16 fedora-my4ng easyeffects[51679]: 2025-02-01T08:45:16.494Z | WARN |  deep_filter_ladspa | DF 866dd930b3bb | Underrun detected (RTF: 1.42). Processing too slow!
Feb 01 08:45:18 fedora-my4ng easyeffects[51679]: 2025-02-01T08:45:18.088Z | WARN |  deep_filter_ladspa | DF 866dd930b3bb | Underrun detected (RTF: 1.13). Processing too slow!
Feb 01 08:45:18 fedora-my4ng easyeffects[51679]: 2025-02-01T08:45:18.529Z | WARN |  deep_filter_ladspa | DF 866dd930b3bb | Underrun detected (RTF: 6.79). Processing too slow!
Feb 01 08:45:18 fedora-my4ng easyeffects[51679]: thread '<unnamed>' panicked at 'DF 866dd930b3bb | Processing too slow! Please upgrade your CPU. Try to decrease 'Max DF processing threshold (dB)'.', ladspa/src/lib.rs:444:17
Feb 01 08:45:18 fedora-my4ng easyeffects[51679]: note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
...

Output from htop while recording and playing back at the same time:

The top process with 20 priority is for the input effects and the bottom one with -51 priority is for the output effects. I do NOT believe that the main computation is being done in a pipewire process since its highest CPU load is only 0.6 at the same time.

No pipewire RT config is changed and the rtkit maximum RT priority is set to 50. This occurs with both inactivity timeout on and off.

Expected Behavior

The input effects should have the same RT priority as the output effects.

Debug Log

No response

Additional Information

No response

[wwmm]

All the configuration for realtime priorities is done by pipewire. And there isn't the pipeline concept in Pipewire like there is in gstreamer and similar frameworks.

On pipewire we just say "put the filter in your realtime thread". There isn't a difference between our pipelines in this regard. From pipewire point of view it is just one more filter it has to handle with realtime priorities. As far as I remember it handles all of them in only one realtime thread.

[wwmm]

The thread with normal priority and high cpu load shown in your top image is most likely one of the gtk threads doing gtk work. We definitely do not want to put that thread with realtime priority. Some users did that in the past and the result is the system killing easyeffects.

[wwmm]

Thre thread with normal priority and high cpu load shown in your top image is most likely one of the gtk threads doing gtk work.

Or maybe it is the main app thread if easyeffects is running in the background.

[my4ng]

@wwmm Thank you for the quick response! I was just having a search on the issues, and found #3224 which got me confused:

What would you think about moving most of the processing to outside the realtime thread?

I updated our master branch now moving the fft call to the main thread.

This seems to suggest that the computation heavy FFT call was moved outside the realtime thread? Does that mean easyeffects can control whether a plugin use the realtime thread or not?

The thread with normal priority and high cpu load shown in your top image is most likely one of the gtk threads doing gtk work. We definitely do not want to put that thread with realtime priority. Some users did that in the past and the result is the system killing easyeffects.

Mhmm I am not sure about that. I didn't have the easyeffects window open, and that thread was definitely not there during only playbacks.

EDIT: also that process is ONLY active during recording.

[wwmm]

This seems to suggest that the computation heavy FFT call was moved outside the realtime thread? Does that mean easyeffects can control whether a plugin use the realtime thread or not?

No. We ask for all of them to be handled with realtime priorities. The fft was moved away from the realtime thread in the spectrum plugin because it was a very heavy fft. Too heavy to be in a realtime thread. The remaining operations are still in pipewire's realtime thread.

[my4ng]

I have just done some more testing. It seems if I disable the 'Deep Noise Remover' effect, that process's CPU load drop to 0 and then back up to about 25% (more than anything else on the system) once turned back on, so I am pretty confident that process is responsible for the deep filtering computation. Again, I have no idea why it is not in the RT thread.

[wwmm]

Again, I have no idea why it is not in the RT thread.

I will try to do some tests with it later. But one possibility is that the deepnoise library is creating its own threads to do heavy work. Some plugins do that when they have to do something that is too heavy for realtime threads.

[my4ng]

I will try to do some tests with it later. But one possibility is that the deepnoise library is creating its own threads to do heavy work. Some plugins do that when they have to do something that is too heavy for realtime threads.

That would definitely make sense. However, I wonder if it is possible to use something like ananicy to at least get some niceness (e.g. -10) on that process. I know there used to be crashes with that when setting RT, but keeping it at normal priority should be fine?

[wwmm]

However, I wonder if it is possible to use something like ananicy to at least get some niceness (e.g. -10) on that process. I know there used to be crashes with that when setting RT, but keeping it at normal priority should be fine?

Setting the niceness value is usually fine. But if it is indeed the plugin creating threads there is nothing we can do from our side. Either the plugin code does the niceness configuration or the user does it manually through the thread PID.

It would be similar to what happens with PipeWire's realtime thread. We have no control at all over its configuration.

[wwmm]

or the user does it manually through the thread PID.

Or through a service like ananicy. Although it would probably set all EasyEffects threads to that niceness value. What usually is not a problem like it would be if the whole application got realtime priority.

[wwmm]

I will try to do some tests with it later.

I did some tests using the microphone pipeline "monitor button". This way only the mic pipeline is active. The realtime thread managed by PipeWire is working. But I've noticed that as soon as the deepfilter is added to the pipeline an additional thread becomes active with noticeable cpu usage. Considering that the other plugins did not do this it really feels like the deepfilter library is creating an additional thread.

[wwmm]

Considering that the other plugins did not do this it really feels like the deepfilter library is creating an additional thread.

And it indeed does not get realtime priority. Maybe because PipeWire is disabling the propagation of this permission to children threads. Or maybe the deepfilter library enforces the priority reset on thread fork because it is not something that should be in a realtime thread.

[wwmm]

As I was not the one that implemented the support for deepfilter on EasyEffects I did not have its code in my memory. The library is initialized in our main thread

easyeffects/src/deepfilternet.cpp

Line 92 in 9377aa2

util::idle_add([&, this] {

. What is necessary in several cases because the initialization of these plugins is usually too heavy to be in the audio processing realtime thread. So my guess is that the deepfilter inherited the priority from our main thread.

Looking at how much CPU this deepfilter thread eats putting giving it realtime priority does not feel like a good idea. And EasyEffects can't do it anyway. We do not have permission for this. Unless we try direct interaction with realtimekit to get it. Try to see if setting niceness in ananicy improves the situation.

[my4ng]

Looking at how much CPU this deepfilter thread eats putting giving it realtime priority does not feel like a good idea. And EasyEffects can't do it anyway. We do not have permission for this. Unless we try direct interaction with realtimekit to get it. Try to see if setting niceness in ananicy improves the situation.

I've added a ananicy rule for easyeffects with a niceness of -12. This of course changes the priority for all normal easyeffects process, but this shouldn't be a problem since most are just sleeping all the time:

Now the deep noise remover process also runs at priority 8, and only very seldomly hit with the "too slow" warning when CPU at 100% utilisation with stress.

[wwmm]

Now the deep noise remover process also runs at priority 8, and only very seldomly hit with the "too slow" warning when CPU at 100% utilisation with stress

Interesting. I guess this is the best that can be done for now. Unless someone makes the deep noise remover library more efficient.