Lightweight local “Me vs Other Speaker” diarization using voice profile

[homeraman]

Hi,

First of all, thank you for Murmure. I discovered the project recently and I find the philosophy very strong: local-first, privacy-focused, lightweight, open-source, and usable in real workflows.

I am an osteopath in France, and I am currently exploring a local workflow for clinical note-taking during consultations. My use case is not classic dictation only, but a real conversation between a practitioner and a patient.

Today, my ideal workflow would be:

• record or dictate during the consultation;
• transcribe locally;
• optionally send the transcript to a local LLM through Ollama;
• generate structured clinical notes;
• keep patient data local and avoid cloud transcription.

One limitation for this kind of use case is speaker separation.

I was wondering if a lightweight diarization feature could be possible in Murmure, but maybe not as full generic diarization at first. My intuition is that a simpler approach could be enough for many medical / therapeutic use cases:

Instead of trying to identify every speaker in a complex meeting, Murmure could have a local “voice profile” mode.

Example:

1. The user records a short local voice profile with the same microphone used in real conditions.
2. Murmure stores only a local speaker embedding / voice fingerprint.
3. During transcription, Murmure detects whether a spoken segment is probably “me” or “not me”.
4. The transcript could then be labeled as:

Practitioner: …
Patient: …

For my use case, this would already be extremely useful. I do not need a perfect meeting diarization system with 5 or 6 speakers. I mostly need to know whether the speaker is the practitioner or the patient.

This could help a lot for:

• doctors;
• osteopaths;
• psychologists;
• therapists;
• interviews;
• podcasts;
• clinical note-taking;
• any 1-to-1 professional conversation.

Technically, I may be wrong because I am not a developer, but I was thinking about a lightweight local pipeline such as:

• VAD to split speech segments;
• speaker embeddings through a CPU-friendly model;
• comparison with the enrolled local user voice profile;
• cosine similarity threshold;
• fallback to “unknown speaker” if confidence is low.

Some possible directions I have seen mentioned recently are:

• sherpa-onnx;
• WeSpeaker;
• SpeechBrain speaker embeddings;
• pyannote-style segmentation, but maybe too heavy for CPU;
• lighter ONNX-based speaker verification instead of full diarization.

The key idea would be to keep it aligned with Murmure’s philosophy:

• local only;
• no cloud;
• low resource usage;
• no long-term audio storage;
• privacy-first;
• optional feature;
• useful for medical and professional workflows.

I would love to know if this kind of feature seems realistic within Murmure’s architecture, especially since Murmure already uses Parakeet for local ASR and Ollama for local LLM post-processing.

Do you think a “local voice profile / me vs other speaker” mode would be technically possible, or would it be too complex / too heavy for the project?

Thanks again for your work. Murmure seems very promising for privacy-sensitive professional use cases.

[Kieirra]

Replied on linkedIn