Add support for an inbound audio processor

[Lazarus404]

Contributing to Twilio

All third party contributors acknowledge that any contributions they provide will be made under the same open source license that the open source project is provided under.

• I acknowledge that all my contributions will be made under the project's license.

Pull Request Details

Inbound stream audio processing hooks

Description

We're looking to add a means to apply an AudioProcessor to the inbound stream, so that we can apply custom noise filtering before it is passed to the speaker / headset.

Burndown

Before review

• Updated CHANGELOG.md if necessary
• Added unit tests if necessary
• Updated affected documentation
• Verified locally with npm test
• Manually sanity tested running locally
• Ready for review

[charliesantos]

Thank you for this work, Jahred. We got similar requests from other customers for remote audio processor. But since the remote audio track can be accessed already, we have not prioritized adding this in the audio processor API. Having said that, we will consider the benefits of adding this in the official audio processor API.

Regarding accessing the remote audio track, is the following something that would work for you?

// Raised when the remote audio track is attached to the audio element
call.on('audio', (remoteAudio) => {
  // Raised when there's media available
  remoteAudio.addEventListener('canplay', () => {
    const mediaStream = remoteAudio.srcObject;
    console.log('remote media stream', mediaStream);
  });
});

[AleksadraMT]

Thank you for this work, Jahred. We got similar requests from other customers for remote audio processor. But since the remote audio track can be accessed already, we have not prioritized adding this in the audio processor API. Having said that, we will consider the benefits of adding this in the official audio processor API.

Regarding accessing the remote audio track, is the following something that would work for you?

// Raised when the remote audio track is attached to the audio element
call.on('audio', (remoteAudio) => {
  // Raised when there's media available
  remoteAudio.addEventListener('canplay', () => {
    const mediaStream = remoteAudio.srcObject;
    console.log('remote media stream', mediaStream);
  });
});

Hello charliesantos!

Thank you for provided example! For sure like that we can get remote stream, apply the processor, create a new audio source and replug it.

However, with this MR we want to achieve the same pattern as for the local stream and be able just plug the processor and replace the stream on the Twilio side without the need to to: check if the call is open and set timeout for that; have completely different logic for outbound and inbound streams; have all nessesary tests are running on the Twilio side to be sure that the stream have a proper fallback in a case of the error; etc.

We'd appreciate if you take this change into the consideration! Thank you in advance.

[charliesantos]

Thank you for the additional details @AleksadraMT . We'll take this into consideration.