[Feature Request] Bot SDK for Audio/Video Calls

[Luna2026-a11y]

Summary

Add a dedicated SDK/API to allow bots to programmatically join, listen, and speak in Mattermost Calls (audio/video). This would enable AI assistants, transcription services, and voice agents to participate in real-time conversations.

Problem Statement

Currently, Mattermost Calls supports:

• ✅ Human-to-human audio/video calls via WebRTC
• ✅ Recording calls (enterprise)
• ✅ Transcription with live captions (enterprise)
• ✅ Bot API for posting recordings/transcriptions

But there is NO way for a bot to:

• Join a call as a participant
• Receive real-time audio streams
• Send audio to other participants

This is a significant limitation for organizations wanting to integrate AI voice assistants into their Mattermost workflow.

Use Cases

1. AI Voice Assistant

User: Starts a call in channel
Bot: Joins automatically
User: "Hey Luna, what's the status of the deployment?"
Bot: listens via STT → processes → responds via TTS in the call

2. Real-time Transcription & Summarization

• Bot joins call as silent participant
• Generates live transcript in channel
• Provides post-call summary with action items

3. Voice-Enabled Customer Support

• Customer starts call
• AI bot receives call, triages issue
• Escalates to human if needed with full context

4. Accessibility

• Real-time captioning for hearing-impaired users
• Voice-to-text for speech-impaired users
• Screen reader integration for video

Proposed Solution

Bot WebRTC Client SDK

Provide a Go/TypeScript SDK that handles:

1. Authentication - Bot session → Call join
2. Signaling - WebSocket connection to RTCD
3. Media - WebRTC peer connection setup
4. Audio I/O - Receive audio frames, send audio frames

// Example Go SDK usage
client, _ := calls.NewBotClient(calls.BotConfig{
    BotToken: "xxx",
    SiteURL:  "https://mattermost.example.com",
})

// Join active call in channel
call, _ := client.JoinCall(context.Background(), "channelID")

// Receive audio stream
go func() {
    for frame := range call.AudioIn() {
        // frame is PCM audio []byte
        transcript := sttService.Transcribe(frame)
        // Process transcript...
    }
}()

// Send audio response
audioOut := ttsService.Synthesize("Hello, how can I help?")
call.SendAudio(audioOut)

REST API Extension

POST /api/v4/calls/{call_id}/bot/join
POST /api/v4/calls/{call_id}/bot/leave
GET  /api/v4/calls/{call_id}/bot/audio/stream  (WebSocket)

Server-Side Bot Participant

Add a new participant type bot alongside user:

• No UI required
• Can join calls programmatically
• Appears as "Luna (Bot)" in participant list
• Has mute/permission controls like regular participants

Implementation Suggestions

Phase 1: Bot Audio Receive (Listen-only)

• Bot joins call as silent participant
• Receives mixed audio stream via WebRTC
• No send capability

Phase 2: Bot Audio Send (Talk)

• Bot can send audio frames
• Server mixes bot audio with other participants
• Push-to-talk or voice activity detection

Phase 3: Video Support

• Bot can receive video streams
• Bot can send video (avatar, screen share)

Phase 4: Native Integration

• First-class bot account type
• Plugin hooks for call events
• WebSocket events for bots

Benefits

Stakeholder	Benefit
Enterprise	AI voice assistants integrated directly in Mattermost
Customers	Better accessibility, voice-driven workflows
Developers	New integration capabilities
Mattermost	Competitive advantage over Slack (limited bot call access)

Technical Considerations

Audio Format

• Standardize on Opus for WebRTC compatibility
• Provide PCM conversion utilities in SDK
• Sample rate: 48kHz, mono channel

Security

• Bot tokens with scoped permissions (calls:join, calls:audio:send)
• Rate limiting per bot
• Audit logging for bot actions

Scalability

• Bot participants counted against call limits
• Server-side audio mixing handles bot streams
• RTCD handles additional peer connections

Alternatives Considered

1. Post-call processing only

• Limited to recording + transcription
• No real-time interaction
• Already supported

2. External SIP gateway

• Requires additional infrastructure (Asterisk/FreeSWITCH)
• Complex setup for users
• Not self-contained

3. Headless browser automation

• Fragile, resource-intensive
• Not a production solution
• Browser dependencies

---

Would you be interested in this feature? Please react with 👍 if this would benefit your organization.

[pawlsclick]

In OpenClaw v2026.4.23

Browser Talk has WebRTC realtime sessions through talk.realtime.session
OpenAI Realtime and Google Gemini Live are registered realtime voice providers

So the way this works in OpenClaw is basically:

• Browser Talk asks the Gateway for a realtime session via talk.realtime.session
• Gateway mints a short-lived provider session/secret for OpenAI Realtime or Gemini Live
• Browser connects directly to the realtime provider over WebRTC mic/audio flows browser ↔ provider for low latency
• OpenClaw stays in the loop for agent reasoning / tools / orchestration when the voice session needs heavier agent help, it can bounce that through OpenClaw