Skip to content

Use Gemini in video and audio transcript #359

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 29 commits into from
Feb 7, 2025
Merged

Use Gemini in video and audio transcript #359

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
29 commits
Select commit Hold shift + click to select a range
6a32a71
chore: replace ffmpeg with vertex ai
MrOrz Feb 1, 2025
c71a35b
refactor: Replace OpenAI Whisper with Gemini for audio/video transcri…
MrOrz Feb 1, 2025
cd470a4
chore(package.json): upgrade media-manager to 0.3.2 to use file's goo…
MrOrz Feb 1, 2025
6965eaa
fix(graphql): rename audio test file suffix
MrOrz Feb 1, 2025
f373912
fix(graphql): adjust transcript prompt and mimeType
MrOrz Feb 1, 2025
f6fea20
test(graphql/util): use an easier case for CI test
MrOrz Feb 2, 2025
61d7afd
test(graphql): add test for video transcript
MrOrz Feb 2, 2025
3f522be
chore(graphql): add langfuse and fix variable not used lint error
MrOrz Feb 2, 2025
28991a1
refactor: remove ffmpeg from CI and built images
MrOrz Feb 2, 2025
dc93797
refactor(util): unify langfuse env setup
MrOrz Feb 2, 2025
274fc60
fix(graphql): try making Langfuse receive data
MrOrz Feb 2, 2025
77be026
test(graphql): simplify transcript assertions to allow gemini-1.5-pro…
MrOrz Feb 2, 2025
c4b2d12
refactor(graphql): setting Langfuse trace io and generation output pr…
MrOrz Feb 2, 2025
930419c
refactor(graphql): set maxOutputToken to cut hallucinated content
MrOrz Feb 2, 2025
92cf1ff
refactor(graphql): format transcript prompt to markdown
MrOrz Feb 2, 2025
714c85a
fix: Restore model parameter in transcript generation trace end method
MrOrz Feb 3, 2025
4601be6
refactor: Replace single model with multiple models for transcript ge…
MrOrz Feb 3, 2025
acfddf7
style: Format console warning message for better readability
MrOrz Feb 3, 2025
c610e6b
feat: Initialize VertexAI instance for transcript generation models
MrOrz Feb 3, 2025
2caa58c
feat: Update google-auth-library to version 9.15.1 and refactor imports
MrOrz Feb 3, 2025
a05695c
fix: Update transcript model version in util.js
MrOrz Feb 3, 2025
0641d8d
fix(graphql): use generally available gemini 2.0-flash for transcripts
MrOrz Feb 6, 2025
abd7c8b
feat: Add onUploadStop callback to uploadMedia function parameters
MrOrz Feb 6, 2025
093c88d
feat(graphql): ensure media file is uploaded before calling LLM
MrOrz Feb 6, 2025
a2a391e
fix(graphql): typo
MrOrz Feb 6, 2025
af44e54
fix(util): seems that gemini model version cannot be omitted on verte…
MrOrz Feb 6, 2025
3ac49a0
fix(graphql): handle both existing and new file scenario in media upload
MrOrz Feb 6, 2025
1736ed1
fix(util): make transcript test pass
MrOrz Feb 6, 2025
a88e2b7
fix(graphql): specify location to avoid the busy, default us-central1
MrOrz Feb 6, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 0 additions & 2 deletions .github/workflows/ci.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -21,8 +21,6 @@ jobs:
id-token: 'write'

steps:
- name: Setup ffmpeg
uses: FedericoCarboni/setup-ffmpeg@v2
- uses: actions/checkout@v3
with:
submodules: true
Expand Down
1 change: 0 additions & 1 deletion Dockerfile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -18,7 +18,6 @@ RUN npm prune --production

#########################################
FROM node:18-alpine3.18
RUN apk update && apk add ffmpeg

WORKDIR /srv/www
EXPOSE 5000 5500
Expand Down
Loading
Loading