More Accurate VocalsResync #766
Conversation
|
you sure this doesnt get triggered frequently and only happens in a noticable delay |
Yes, I achieved high-precision audio offset detection by smoothing the audio time offset. |
Unless any issues are discovered, this should be the final version of the modifications.
|
Alright, unless any issues are discovered, no further modifications should be needed at this point. |
|
Here's a summary: Based on my testing, I’ve found that with my algorithm, any audio desync exceeding 7.1 ms becomes noticeably perceptible. The smoothed audio offset measurement I implemented effectively prevents false positives in detection. However, one limitation remains: I haven’t found a way to fully eliminate the slight desync sometimes introduced by resyncVocals() itself. As a result, the system may need to readjust multiple times in quick succession before stabilizing. In my tests, realignment typically completes within one second, after which the audio remains perfectly synced with no audible issues until the next desync event occurs. I’ve also done my best to optimize the overall behavior. |
Alright, to prevent cases where a sudden large audio offset could cause the smoothed value to instantly exceed the threshold and trigger readjustment, I've made further modifications. I also took the opportunity to tweak a few other things.
|
alright, asked a friend that always had vocal resync problems to test this |
|
|
By the way, I actually have a demo video for this on Discord. Although it was recorded during the initial version, it's still quite similar to how it works now. |
|
I've been working on a new approach for the audio sync issue. The idea is to use imperceptibly small adjustments to the audio playback speed. This keeps the audio drift within a ±1ms range during stable periods, while avoiding the sync disruptions caused by hard resets of the audio position. What are your thoughts? @NexIsDumb Regarding my previous implementation, I've observed that even within the ±7ms threshold, there can still be subtle, barely perceptible audio drift. |
|
mhh then sure, why not trying, lets see if its actually possible and better |
|
@NexIsDumb Excuse me for the interruption. I've shared the script that synchronizes audio by adjusting playback speed on Discord. Could you ask your friends to test it? If it works, I'll integrate it into the source code. |
|
i havent been answering here cuz that friend of mine hasnt still unfortunately answered :( i'll prob check it myself then |
|
alrighty, my friend checked and they say its really good, they pulled on sept 12 |
Awesome! I'll migrate it to the source code and push the update later. |
tysm! |
|
you know what, at this point, imma make a ping inside the discord for asking everyone to test it so we can be sure 100% |
ok |
|
I've now integrated the variable-speed audio sync feature into the source code, but I still need to check for any potential issues. |
|
Testing complete. I can confirm the HScript calls and core code are functioning properly. Additionally, I can't think of anything else that needs testing at this point. |
great!
not really, lemme ask to some other friends of mine to test this too |
Using time-stretching for minor corrections (subtle drift), and hard playback resets for high-latency scenarios. This approach minimizes audible pitch distortion while maintaining high-precision audio playback.
|
Alright, if no issues arise, this should be the final update. Time-stretching correction is used for minor adjustments (within ±16ms) Direct playback reset is applied for higher latency scenarios (beyond ±16ms) |
|
@NexIsDumb Alright, it's now confirmed that no further modifications will be made unless any issues are reported.🔥🔥🔥 |
|
@HEIHUAa okay so the only report ive received is that the vocals get fucky if the timeScale isnt 1 |
|
Alright, I'll try to address this issue. |
@NexIsDumb Could you tell me what the specific issue is? I tested it and didn't seem to find any problems.😳😳😳 |
|
Could it be that modifying the timeScale itself causes pitch changes in all audio? |
|
lemme ask |
|
the vocals just loop instead of the song finishing for reference the song is supposed to end here |
Alright, I'll definitely check it out in a few hours. |
|
@NexIsDumb Alright, I've attempted to fix this issue. On my end, I can't seem to reproduce the problem, but I've kept the trace statements in place. If the issue persists, please try to include the output results when recording a video. |
|
lemme ask again, tysm |
|
I wanna merge this, is it good now to merge? |
|
Woah what the fuck |
|
still not, i have a friend testing it still cuz it has some issues |
|
🔥🔥🔥 |
3 participants
I've found that the current detection mechanism for ResyncVocals seems to have issues—it rarely triggers resynchronization when the audio timing desyncs. That's why I've rewritten a better version that ensures vocals can be effectively resynchronized.