Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add support for Replicate.ai models #584

Merged
merged 7 commits into from
May 22, 2024
Merged

Add support for Replicate.ai models #584

merged 7 commits into from
May 22, 2024

Conversation

muhammed-shihebi
Copy link
Collaborator

What does this PR do?

  1. It adds a new service called replicate that acts as a proxy between the front end and the Replicate.ai API.
  2. The service supports streaming, chat, and completion requests.
  3. The service uses chat templates when available from hf, otherwise it uses a basic chat template to communicate with completion models in a chat setting.
  4. Replicate.ai is made available in the prompting interface together with OpenAI and local models.
  5. The PR updates the interface for better model source separation and better error handling.

Note 1: All currently available models from Replicate.ai that support streaming are hardcoded in the service. They were scraped from their website. Currently, they don't have an endpoint to get a list of available models.

Note 2: Model chat templates are pulled from hf. The templates are also hardcoded in the service to avoid problems with models with restricted access in hf like Llama 2.

@HaritzPuerto HaritzPuerto merged commit 4abea25 into master May 22, 2024
38 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants