StreamAudioSource.SampleCallbackAsync() crashes on Unity 6 and 6.2

[dudziakl]

Bug Report

The issue applies to the Android build. PC is working fine. I didn't try iOS.

Overview

After the change of the SampleCallback in StreamingAudioSource from BufferCallback() to SampleCallackAsync(), there is a serious crash that closes the app on Android. It happens on Unity 6/6.2. Older Unity versions like 2021.3.45f2, 2022.3.62f2 are working fine.

private async Task GenerateSpeechAsync(string text, CancellationToken cancellationToken)
        {
            try
            {
                if (string.IsNullOrWhiteSpace(text)) { return; }
                var request = new SpeechRequest(input: text, model: Model.TTS_1, voice: voice, responseFormat: SpeechResponseFormat.PCM);
                var stopwatch = Stopwatch.StartNew();
                using var speechClip = await openAI.AudioEndpoint.GetSpeechAsync(
                    request,
                    partialClip => streamAudioSource.SampleCallbackAsync(partialClip.AudioSamples), //<-- it crashes in this line
                    cancellationToken)
                    .ConfigureAwait(true);
                var playbackTime = speechClip.Length - (float)stopwatch.Elapsed.TotalSeconds + 0.1f;

To Reproduce

I have used Sample project with version 8.8.7. Use any sample scene except Realtime because this one works as expected.

Steps to reproduce the behavior:

1. Open the sample project and change the platform to Android
2. [optional] Install Android Logcat plugin if you want to see the logs.
3. Deploy project to device
4. On the title screen select either Chat, Assistant or Response button.
5. Say anything or write "hello" to request OpenAI to respond.
6. Observe that text response will be displayed and audio will be played for a second then app will crash and exit.
7. [Optional] Check Logcat logs for more details

Expected behavior

Should not crash and work as on PC.

Screenshots

Additional context

Sample crash log com.openai.unity-crash.txt

[StephenHodgson]

relative logs:

(BuildId: f0419d1b3f1d6617)
(BuildId: f0419d1b3f1d6617)
(tlsf_free+128) (BuildId: f0419d1b3f1d6617)
(DynamicHeapAllocator::Deallocate(void*)+960) (BuildId: f0419d1b3f1d6617)
(DualThreadAllocator<DynamicHeapAllocator>::TryDeallocate(void*)+164) (BuildId: f0419d1b3f1d6617)
(MemoryManager::TryDeallocateWithLabel(void*, MemLabelId, char const*, int)+392) (BuildId: f0419d1b3f1d6617)
(MemoryManager::Deallocate(void*, MemLabelId, char const*, int)+16) (BuildId: f0419d1b3f1d6617)
(BuildId: f0419d1b3f1d6617)
(UnsafeUtility_CUSTOM_FreeTracked(void*, NativeCollection::Allocator)+68) (BuildId: f0419d1b3f1d6617)
(BuildId: 4b7e13512f68870a001630e5d90caef48a5c9137)
(BuildId: 4b7e13512f68870a001630e5d90caef48a5c9137)
(BuildId: 4b7e13512f68870a001630e5d90caef48a5c9137)
(BuildId: 4b7e13512f68870a001630e5d90caef48a5c9137)
(BuildId: 4b7e13512f68870a001630e5d90caef48a5c9137)
(BuildId: 4b7e13512f68870a001630e5d90caef48a5c9137)
(BuildId: 4b7e13512f68870a001630e5d90caef48a5c9137)
(BuildId: 4b7e13512f68870a001630e5d90caef48a5c9137)
(BuildId: 4b7e13512f68870a001630e5d90caef48a5c9137)
(scripting_method_invoke(ScriptingMethodPtr, ScriptingObjectPtr, ScriptingArguments&, ScriptingExceptionPtr*, bool)+168) (BuildId: f0419d1b3f1d6617)
(ScriptingInvocation::Invoke(ScriptingExceptionPtr*, bool)+136) (BuildId: f0419d1b3f1d6617)
(DownloadHandlerScript::InvokeReceiveData(ScriptingObjectPtr, ScriptingArrayPtr, int)+96) (BuildId: f0419d1b3f1d6617)
(DownloadHandlerScriptCached::InvokeReceiveDataForCurrentData(ScriptingObjectPtr)+168) (BuildId: f0419d1b3f1d6617)
(UnityWebRequestManager::InvokeScriptHandlers()+48) (BuildId: f0419d1b3f1d6617)
(BuildId: f0419d1b3f1d6617)
(ExecutePlayerLoop(NativePlayerLoopSystem*)+132) (BuildId: f0419d1b3f1d6617)
(ExecutePlayerLoop(NativePlayerLoopSystem*)+232) (BuildId: f0419d1b3f1d6617)
(PlayerLoop()+320) (BuildId: f0419d1b3f1d6617)
(UnityPlayerLoop()+788) (BuildId: f0419d1b3f1d6617)
(nativeRender(_JNIEnv*, _jobject*)+84) (BuildId: f0419d1b3f1d6617)

[StephenHodgson]

@dudziakl I wasn't able to reproduce this. Can you share with me your project settings for android build?

Edit:
I was able to repro it.

Here's what I'm using for the com.openai.unity project:

[StephenHodgson]

@dudziakl I was able to pinpoint the issue. So each partial speech clip has to be disposed of after use, and the way the Func<T> is used by downstream consumers requires them to use the async/await keywords. I removed them thinking it'd save some overhead with out having to implement the whole state machine, but that's actually what messed it up.

com.openai.unity/OpenAI/Packages/com.openai.unity/Runtime/Audio/AudioEndpoint.cs

Lines 105 to 112 in 316efef

	try
	{
	await partialClipCallback(partialClip).ConfigureAwait(false);
	}
	finally
	{
	partialClip.Dispose();
	}

So you can easily fix on your end (and I've implemented fixes in the existing samples)

wherever you're invoking the partial clip callbacks, you MUST also implement async/await keywords in the callback lambda function

using var speechClip = await openAI.AudioEndpoint.GetSpeechAsync(
    request,
    async partialClip => await streamAudioSource.SampleCallbackAsync(partialClip.AudioSamples),
    cancellationToken).ConfigureAwait(true);

It is either this or have the caller dispose of the partial clips, but I'm not sure everyone would know to do that.

[dudziakl]

Thanks @StephenHodgson, I have tried and it didn't work for me😔 Did you change anything else in the project settings?

[StephenHodgson]

I've made changes and fixes in #436 to address and fix the issue

[dudziakl]

Works fine! Thank you