Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions .env.example
Original file line number Diff line number Diff line change
Expand Up @@ -138,6 +138,9 @@ ASR_QWEN_BASE_URL=
ASR_AZURE_API_KEY=
ASR_AZURE_BASE_URL=https://{region}.api.cognitive.microsoft.com

# FunASR (local, WAV input only, no API key required)
# ASR_FUNASR_BASE_URL=http://localhost:8000/v1

# Lemonade ASR (local, WAV input only, no API key required)
# ASR_LEMONADE_BASE_URL=http://localhost:13305/v1

Expand Down
26 changes: 24 additions & 2 deletions README-zh.md
Original file line number Diff line number Diff line change
Expand Up @@ -33,7 +33,7 @@
<p align="center">
<a href="./README.md">English</a> | <a href="./README-zh.md">简体中文</a>
<br/>
<a href="https://open.maic.chat/">在线体验</a> · <a href="#-快速开始">快速开始</a> · <a href="#lemonade-local-ai">Lemonade</a> · <a href="#-功能特性">功能特性</a> · <a href="#-使用场景">使用场景</a> · <a href="#-openclaw-集成">OpenClaw</a>
<a href="https://open.maic.chat/">在线体验</a> · <a href="#-快速开始">快速开始</a> · <a href="#lemonade-local-ai">Lemonade</a> · <a href="#funasr-local-asr">FunASR</a> · <a href="#-功能特性">功能特性</a> · <a href="#-使用场景">使用场景</a> · <a href="#-openclaw-集成">OpenClaw</a>
</p>


Expand Down Expand Up @@ -137,7 +137,7 @@ providers:
- us.anthropic.claude-opus-4-8
```

支持的服务商:**OpenAI**、**Azure OpenAI**、**Anthropic**、**Amazon Bedrock**、**Google Gemini**、**DeepSeek**、**通义千问 Qwen**、**Kimi**、**MiniMax**、**Grok (xAI)**、**OpenRouter**、**豆包**、**腾讯混元 / TokenHub**、**小米 MiMo**、**智谱 GLM**、**Ollama**(本地)、**Lemonade**(本地 LLM / 图像 / TTS / ASR)以及任何兼容 OpenAI API 的服务。
支持的服务商:**OpenAI**、**Azure OpenAI**、**Anthropic**、**Amazon Bedrock**、**Google Gemini**、**DeepSeek**、**通义千问 Qwen**、**Kimi**、**MiniMax**、**Grok (xAI)**、**OpenRouter**、**豆包**、**腾讯混元 / TokenHub**、**小米 MiMo**、**智谱 GLM**、**Ollama**(本地)、**Lemonade**(本地 LLM / 图像 / TTS / ASR)、**FunASR**(本地 ASR)以及任何兼容 OpenAI API 的服务。

Amazon Bedrock 快速示例:

Expand All @@ -164,6 +164,28 @@ ASR_LEMONADE_BASE_URL=http://localhost:13305/v1
IMAGE_LEMONADE_BASE_URL=http://localhost:13305/v1
```

<a id="funasr-local-asr"></a>

### 可选:FunASR(本地语音识别)

OpenMAIC 可以通过 FunASR 的 OpenAI 兼容服务完成本地转写。内置 provider 支持 SenseVoiceSmall、Paraformer 和 Fun-ASR-Nano,无需 API Key。

```bash
python -m pip install torch torchaudio
python -m pip install "funasr==1.4.0" fastapi uvicorn python-multipart
# NVIDIA GPU 上运行 Fun-ASR-Nano 时再安装 vLLM
python -m pip install vllm
funasr-server --device cuda --model fun-asr-nano
```

将 OpenMAIC 指向该服务:

```env
ASR_FUNASR_BASE_URL=http://localhost:8000/v1
```

纯 CPU 环境可运行 `funasr-server --device cpu --model sensevoice`。生产部署方式参见 [FunASR 部署指南](https://github.com/modelscope/FunASR#deploy)。

OpenAI 快速示例:

```env
Expand Down
26 changes: 24 additions & 2 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -33,7 +33,7 @@
<p align="center">
<a href="./README.md">English</a> | <a href="./README-zh.md">简体中文</a>
<br/>
<a href="https://open.maic.chat/">Live Demo</a> · <a href="#-quick-start">Quick Start</a> · <a href="#lemonade-local-ai">Lemonade</a> · <a href="#-features">Features</a> · <a href="#-use-cases">Use Cases</a> · <a href="#-openclaw-integration">OpenClaw</a>
<a href="https://open.maic.chat/">Live Demo</a> · <a href="#-quick-start">Quick Start</a> · <a href="#lemonade-local-ai">Lemonade</a> · <a href="#funasr-local-asr">FunASR</a> · <a href="#-features">Features</a> · <a href="#-use-cases">Use Cases</a> · <a href="#-openclaw-integration">OpenClaw</a>
</p>


Expand Down Expand Up @@ -137,7 +137,7 @@ providers:
- us.anthropic.claude-opus-4-8
```

Supported providers: **OpenAI**, **Azure OpenAI**, **Anthropic**, **Amazon Bedrock**, **Google Gemini**, **DeepSeek**, **Qwen**, **Kimi**, **MiniMax**, **Grok (xAI)**, **OpenRouter**, **Doubao**, **Tencent Hunyuan/TokenHub**, **Xiaomi MiMo**, **GLM (Zhipu)**, **Ollama** (local), **Lemonade** (local LLM / image / TTS / ASR), and any OpenAI-compatible API.
Supported providers: **OpenAI**, **Azure OpenAI**, **Anthropic**, **Amazon Bedrock**, **Google Gemini**, **DeepSeek**, **Qwen**, **Kimi**, **MiniMax**, **Grok (xAI)**, **OpenRouter**, **Doubao**, **Tencent Hunyuan/TokenHub**, **Xiaomi MiMo**, **GLM (Zhipu)**, **Ollama** (local), **Lemonade** (local LLM / image / TTS / ASR), **FunASR** (local ASR), and any OpenAI-compatible API.

Amazon Bedrock quick example:

Expand All @@ -164,6 +164,28 @@ ASR_LEMONADE_BASE_URL=http://localhost:13305/v1
IMAGE_LEMONADE_BASE_URL=http://localhost:13305/v1
```

<a id="funasr-local-asr"></a>

### Optional: FunASR (Local Speech Recognition)

OpenMAIC can transcribe locally through FunASR's OpenAI-compatible server. The built-in provider supports SenseVoiceSmall, Paraformer, and Fun-ASR-Nano and requires no API key.

```bash
python -m pip install torch torchaudio
python -m pip install "funasr==1.4.0" fastapi uvicorn python-multipart
# Add vLLM for Fun-ASR-Nano on NVIDIA GPUs
python -m pip install vllm
funasr-server --device cuda --model fun-asr-nano
```

Point OpenMAIC at the server:

```env
ASR_FUNASR_BASE_URL=http://localhost:8000/v1
```

Use `funasr-server --device cpu --model sensevoice` for a CPU-only setup. See the [FunASR deployment guide](https://github.com/modelscope/FunASR#deploy) for production options.

OpenAI quick example:

```env
Expand Down
1 change: 1 addition & 0 deletions components/settings/asr-settings.tsx
Original file line number Diff line number Diff line change
Expand Up @@ -281,6 +281,7 @@ export function ASRSettings({ selectedProviderId }: ASRSettingsProps) {
} else {
switch (selectedProviderId) {
case 'openai-whisper':
case 'funasr-asr':
case 'lemonade-asr':
endpointPath = '/audio/transcriptions';
break;
Expand Down
26 changes: 18 additions & 8 deletions lib/audio/asr-providers.ts
Original file line number Diff line number Diff line change
Expand Up @@ -185,8 +185,16 @@ export async function transcribeAudio(
case 'azure-asr':
return await transcribeAzureASR(config, audioBuffer);

case 'funasr-asr':
return await transcribeWavOpenAICompatibleASR(config, audioBuffer, 'funasr-asr', 'FunASR');

case 'lemonade-asr':
return await transcribeLemonadeASR(config, audioBuffer);
return await transcribeWavOpenAICompatibleASR(
config,
audioBuffer,
'lemonade-asr',
'Lemonade',
);

default:
if (isCustomASRProvider(config.providerId)) {
Expand All @@ -197,29 +205,31 @@ export async function transcribeAudio(
}

/**
* Lemonade ASR implementation (OpenAI-compatible multipart transcription).
* WAV-only OpenAI-compatible multipart transcription.
*
* Lemonade currently supports WAV input and JSON response format.
* Used by local providers whose transcription endpoint accepts WAV and JSON.
*/
async function transcribeLemonadeASR(
async function transcribeWavOpenAICompatibleASR(
config: ASRModelConfig,
audioBuffer: Buffer | Blob,
providerId: 'funasr-asr' | 'lemonade-asr',
providerName: string,
): Promise<ASRTranscriptionResult> {
const baseUrl = (config.baseUrl || ASR_PROVIDERS['lemonade-asr'].defaultBaseUrl || '').replace(
const baseUrl = (config.baseUrl || ASR_PROVIDERS[providerId].defaultBaseUrl || '').replace(
/\/$/,
'',
);

const audioBlob = await toAudioBlob(audioBuffer);
if (!(await isWavAudio(audioBlob))) {
throw new Error(
'Lemonade ASR currently supports WAV input only. Recordings should be converted to WAV before upload.',
`${providerName} ASR currently supports WAV input only. Recordings should be converted to WAV before upload.`,
);
}

const formData = new FormData();
formData.set('file', audioBlob, 'audio.wav');
formData.set('model', config.modelId || ASR_PROVIDERS['lemonade-asr'].defaultModelId);
formData.set('model', config.modelId || ASR_PROVIDERS[providerId].defaultModelId);
formData.set('response_format', 'json');
if (config.language && config.language !== 'auto') {
formData.set('language', config.language);
Expand All @@ -236,7 +246,7 @@ async function transcribeLemonadeASR(
if (errorText.includes('audio is empty') || errorText.includes('too short')) {
return { text: '' };
}
throw new Error(`Lemonade ASR API error: ${errorText || response.statusText}`);
throw new Error(`${providerName} ASR API error: ${errorText || response.statusText}`);
}

const data = await response.json();
Expand Down
16 changes: 16 additions & 0 deletions lib/audio/constants.ts
Original file line number Diff line number Diff line change
Expand Up @@ -1250,6 +1250,22 @@ export const ASR_PROVIDERS: Record<BuiltInASRProviderId, ASRProviderConfig> = {
supportedFormats: ['webm'], // MediaRecorder format
},

'funasr-asr': {
id: 'funasr-asr',
name: 'FunASR',
requiresApiKey: false,
defaultBaseUrl: 'http://localhost:8000/v1',
icon: '/logos/funasr.png',
models: [
{ id: 'sensevoice', name: 'SenseVoiceSmall' },
{ id: 'paraformer', name: 'Paraformer' },
{ id: 'fun-asr-nano', name: 'Fun-ASR-Nano' },
],
defaultModelId: 'sensevoice',
supportedLanguages: ['auto', 'zh', 'en', 'ja', 'ko', 'yue'],
supportedFormats: ['wav'],
},

'lemonade-asr': {
id: 'lemonade-asr',
name: 'Lemonade ASR',
Expand Down
1 change: 1 addition & 0 deletions lib/audio/provider-display.ts
Original file line number Diff line number Diff line change
Expand Up @@ -18,6 +18,7 @@ const ASR_PROVIDER_NAME_KEYS: Record<string, string> = {
'browser-native': 'settings.providerBrowserNative',
'qwen-asr': 'settings.providerQwenASR',
'azure-asr': 'settings.providerAzureASR',
'funasr-asr': 'settings.providerFunASRASR',
'lemonade-asr': 'settings.providerLemonadeASR',
};

Expand Down
1 change: 1 addition & 0 deletions lib/audio/types.ts
Original file line number Diff line number Diff line change
Expand Up @@ -157,6 +157,7 @@ export type BuiltInASRProviderId =
| 'openai-whisper'
| 'browser-native'
| 'qwen-asr'
| 'funasr-asr'
| 'lemonade-asr'
| 'azure-asr';

Expand Down
2 changes: 1 addition & 1 deletion lib/audio/wav-utils.ts
Original file line number Diff line number Diff line change
Expand Up @@ -77,7 +77,7 @@ export async function normalizeASRUploadAudio(
providerId: string,
audioBlob: Blob,
): Promise<{ blob: Blob; fileName: string }> {
if (providerId !== 'lemonade-asr') {
if (providerId !== 'lemonade-asr' && providerId !== 'funasr-asr') {
return { blob: audioBlob, fileName: 'recording.webm' };
}
return { blob: await audioBlobToWav(audioBlob), fileName: 'recording.wav' };
Expand Down
1 change: 1 addition & 0 deletions lib/i18n/locales/ar-SA.json
Original file line number Diff line number Diff line change
Expand Up @@ -1317,6 +1317,7 @@
"providerOpenAIWhisper": "OpenAI ASR (gpt-4o-mini-transcribe)",
"providerBrowserNative": "التعرّف على الكلام المدمج في المتصفح",
"providerQwenASR": "Qwen ASR (سحابة علي بابا بايليان)",
"providerFunASRASR": "FunASR (محلي)",
"providerLemonadeASR": "Lemonade ASR (محلي)",
"providerAzureASR": "Azure STT",
"providerUnpdf": "unpdf (مُدمج)",
Expand Down
1 change: 1 addition & 0 deletions lib/i18n/locales/en-US.json
Original file line number Diff line number Diff line change
Expand Up @@ -1317,6 +1317,7 @@
"providerOpenAIWhisper": "OpenAI ASR (gpt-4o-mini-transcribe)",
"providerBrowserNative": "Browser Native ASR",
"providerQwenASR": "Qwen ASR (Alibaba Cloud Bailian)",
"providerFunASRASR": "FunASR (Local)",
"providerLemonadeASR": "Lemonade ASR (Local)",
"providerAzureASR": "Azure STT",
"providerUnpdf": "unpdf (Built-in)",
Expand Down
1 change: 1 addition & 0 deletions lib/i18n/locales/es-MX.json
Original file line number Diff line number Diff line change
Expand Up @@ -1317,6 +1317,7 @@
"providerOpenAIWhisper": "OpenAI ASR (gpt-4o-mini-transcribe)",
"providerBrowserNative": "ASR nativo del navegador",
"providerQwenASR": "Qwen ASR (Alibaba Cloud Bailian)",
"providerFunASRASR": "FunASR (local)",
"providerLemonadeASR": "Lemonade ASR (local)",
"providerAzureASR": "Azure STT",
"providerUnpdf": "unpdf (integrado)",
Expand Down
1 change: 1 addition & 0 deletions lib/i18n/locales/ja-JP.json
Original file line number Diff line number Diff line change
Expand Up @@ -1317,6 +1317,7 @@
"providerOpenAIWhisper": "OpenAI ASR (gpt-4o-mini-transcribe)",
"providerBrowserNative": "ブラウザネイティブASR",
"providerQwenASR": "Qwen ASR(Alibaba Cloud百錬)",
"providerFunASRASR": "FunASR(ローカル)",
"providerLemonadeASR": "Lemonade ASR(ローカル)",
"providerAzureASR": "Azure STT",
"providerUnpdf": "unpdf(組み込み)",
Expand Down
1 change: 1 addition & 0 deletions lib/i18n/locales/ko-KR.json
Original file line number Diff line number Diff line change
Expand Up @@ -1317,6 +1317,7 @@
"providerOpenAIWhisper": "OpenAI ASR (gpt-4o-mini-transcribe)",
"providerBrowserNative": "브라우저 기본 ASR",
"providerQwenASR": "Qwen ASR (Alibaba Cloud Bailian)",
"providerFunASRASR": "FunASR (로컬)",
"providerLemonadeASR": "Lemonade ASR (로컬)",
"providerAzureASR": "Azure STT",
"providerUnpdf": "unpdf (내장)",
Expand Down
1 change: 1 addition & 0 deletions lib/i18n/locales/pt-BR.json
Original file line number Diff line number Diff line change
Expand Up @@ -1317,6 +1317,7 @@
"providerOpenAIWhisper": "OpenAI ASR (gpt-4o-mini-transcribe)",
"providerBrowserNative": "ASR Nativo do Navegador",
"providerQwenASR": "Qwen ASR (Alibaba Cloud Bailian)",
"providerFunASRASR": "FunASR (Local)",
"providerLemonadeASR": "Lemonade ASR (Local)",
"providerAzureASR": "Azure STT",
"providerUnpdf": "unpdf (Nativo)",
Expand Down
1 change: 1 addition & 0 deletions lib/i18n/locales/ru-RU.json
Original file line number Diff line number Diff line change
Expand Up @@ -1317,6 +1317,7 @@
"providerOpenAIWhisper": "OpenAI ASR (gpt-4o-mini-transcribe)",
"providerBrowserNative": "Встроенный ASR браузера",
"providerQwenASR": "Qwen ASR (Alibaba Cloud Bailian)",
"providerFunASRASR": "FunASR (Локальный)",
"providerLemonadeASR": "Lemonade ASR (Локальный)",
"providerAzureASR": "Azure STT",
"providerUnpdf": "unpdf (встроенный)",
Expand Down
1 change: 1 addition & 0 deletions lib/i18n/locales/zh-CN.json
Original file line number Diff line number Diff line change
Expand Up @@ -1317,6 +1317,7 @@
"providerOpenAIWhisper": "OpenAI ASR (gpt-4o-mini-transcribe)",
"providerBrowserNative": "浏览器原生 ASR",
"providerQwenASR": "Qwen ASR(阿里云百炼)",
"providerFunASRASR": "FunASR(本地)",
"providerLemonadeASR": "Lemonade ASR(本地)",
"providerAzureASR": "Azure STT",
"providerUnpdf": "unpdf(内置)",
Expand Down
1 change: 1 addition & 0 deletions lib/i18n/locales/zh-TW.json
Original file line number Diff line number Diff line change
Expand Up @@ -1250,6 +1250,7 @@
"providerBrowserNativeTTS": "瀏覽器原生 TTS",
"providerOpenAIWhisper": "OpenAI ASR (gpt-4o-mini-transcribe)",
"providerBrowserNative": "瀏覽器原生 ASR",
"providerFunASRASR": "FunASR(本機)",
"providerLemonadeASR": "Lemonade ASR(本機)",
"providerAzureASR": "Azure STT(語音轉文字)",
"providerQwenASR": "Qwen ASR(阿里雲百煉)",
Expand Down
3 changes: 2 additions & 1 deletion lib/server/provider-config.ts
Original file line number Diff line number Diff line change
Expand Up @@ -99,6 +99,7 @@ const ASR_ENV_MAP: Record<string, string> = {
ASR_OPENAI: 'openai-whisper',
ASR_QWEN: 'qwen-asr',
ASR_AZURE: 'azure-asr',
ASR_FUNASR: 'funasr-asr',
ASR_LEMONADE: 'lemonade-asr',
};

Expand Down Expand Up @@ -424,7 +425,7 @@ function buildConfig(yamlData: YamlData): ServerConfig {
keylessProviders: new Set(['voxcpm-tts', 'lemonade-tts']),
}),
asr: loadEnvSection(ASR_ENV_MAP, yamlData.asr, {
keylessProviders: new Set(['lemonade-asr']),
keylessProviders: new Set(['funasr-asr', 'lemonade-asr']),
}),
pdf: applyAliDocMindFallback(
loadEnvSection(PDF_ENV_MAP, yamlData.pdf, {
Expand Down
1 change: 1 addition & 0 deletions lib/store/settings.ts
Original file line number Diff line number Diff line change
Expand Up @@ -501,6 +501,7 @@ const getDefaultAudioConfig = () => ({
'browser-native': { apiKey: '', baseUrl: '', enabled: true },
'qwen-asr': { apiKey: '', baseUrl: '', enabled: false },
'azure-asr': { apiKey: '', baseUrl: '', enabled: false },
'funasr-asr': { apiKey: '', baseUrl: '', enabled: false },
'lemonade-asr': { apiKey: '', baseUrl: '', enabled: false },
} as Record<ASRProviderId, { apiKey: string; baseUrl: string; enabled: boolean }>,
});
Expand Down
5 changes: 5 additions & 0 deletions packages/docs/content/docs/configuration.ar.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -104,10 +104,15 @@ ASR_QWEN_BASE_URL= # تجاوز اختياري
ASR_AZURE_API_KEY=
ASR_AZURE_BASE_URL=https://{region}.api.cognitive.microsoft.com

# FunASR (محلي، لا يحتاج إلى مفتاح)
ASR_FUNASR_BASE_URL=http://localhost:8000/v1

# Lemonade ASR (محلي، لا يحتاج إلى مفتاح)
ASR_LEMONADE_BASE_URL=http://localhost:13305/v1
```

شغّل خدمة FunASR المتوافقة مع OpenAI باستخدام `funasr-server --device cuda`. تستقبل ملفات WAV على `/v1/audio/transcriptions` وتدعم SenseVoiceSmall وParaformer وFun-ASR-Nano.

ASR المدمج في المتصفح لا يحتاج إلى إعدادات خادم.

يمكن أيضًا إضافة موفري ASR مخصصين متوافقين مع OpenAI من الإعدادات، مع إدخال عنوان URL الأساسي، النموذج، واللغات المدعومة. تحفظ هذه الإعدادات في إعدادات العميل.
Expand Down
5 changes: 5 additions & 0 deletions packages/docs/content/docs/configuration.ja.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -104,10 +104,15 @@ ASR_QWEN_BASE_URL= # 任意の上書き
ASR_AZURE_API_KEY=
ASR_AZURE_BASE_URL=https://{region}.api.cognitive.microsoft.com

# FunASR(ローカル、key 不要)
ASR_FUNASR_BASE_URL=http://localhost:8000/v1

# Lemonade ASR(ローカル、key 不要)
ASR_LEMONADE_BASE_URL=http://localhost:13305/v1
```

`funasr-server --device cuda` で OpenAI 互換サービスを起動します。`/v1/audio/transcriptions` で WAV を受け付け、モデル選択から SenseVoiceSmall、Paraformer、Fun-ASR-Nano を利用できます。

ブラウザのネイティブ ASR はサーバー設定不要です。

カスタム OpenAI 互換 ASR provider も設定画面から追加可能で、Base URL、モデル、対応言語を指定します。これらはクライアント設定に保存されます。
Expand Down
5 changes: 5 additions & 0 deletions packages/docs/content/docs/configuration.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -104,10 +104,15 @@ ASR_QWEN_BASE_URL= # optional override
ASR_AZURE_API_KEY=
ASR_AZURE_BASE_URL=https://{region}.api.cognitive.microsoft.com

# FunASR (local, no key required)
ASR_FUNASR_BASE_URL=http://localhost:8000/v1

# Lemonade ASR (local, no key required)
ASR_LEMONADE_BASE_URL=http://localhost:13305/v1
```

Start the OpenAI-compatible FunASR service with `funasr-server --device cuda`. It accepts WAV uploads at `/v1/audio/transcriptions` and provides SenseVoiceSmall, Paraformer, and Fun-ASR-Nano through the model selector.

Browser-native ASR requires no server configuration.

You can also add a custom OpenAI-compatible ASR provider in Settings by entering its Base URL, model, and supported languages; this configuration is stored in client settings.
Expand Down
5 changes: 5 additions & 0 deletions packages/docs/content/docs/configuration.ru.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -104,10 +104,15 @@ ASR_QWEN_BASE_URL= # опциональное переопределен
ASR_AZURE_API_KEY=
ASR_AZURE_BASE_URL=https://{region}.api.cognitive.microsoft.com

# FunASR (локально, key не нужен)
ASR_FUNASR_BASE_URL=http://localhost:8000/v1

# Lemonade ASR (локально, key не нужен)
ASR_LEMONADE_BASE_URL=http://localhost:13305/v1
```

Запустите OpenAI-совместимый сервис командой `funasr-server --device cuda`. Он принимает WAV на `/v1/audio/transcriptions` и поддерживает SenseVoiceSmall, Paraformer и Fun-ASR-Nano.

Встроенный в браузер ASR не требует серверной настройки.

Кастомные OpenAI-совместимые ASR провайдеры можно добавить в настройках, указав Base URL, модели и поддерживаемые языки. Эти настройки сохраняются в клиенте.
Expand Down
Loading
Loading