From 887839609cb8725d9f8ec70390056abf5d0f5e5e Mon Sep 17 00:00:00 2001 From: Jinan Zhou Date: Fri, 22 Nov 2024 00:22:46 +0000 Subject: [PATCH] pushed docker image, updated documentation --- README.md | 1 + docs/source/distributions/index.md | 1 + .../remote_hosted_distro/index.md | 1 + llama_stack/templates/nutanix/doc_template.md | 73 ++++++++++++------- 4 files changed, 51 insertions(+), 25 deletions(-) diff --git a/README.md b/README.md index 78e56ac0af..2ba36c1fde 100644 --- a/README.md +++ b/README.md @@ -69,6 +69,7 @@ A Distribution is where APIs and Providers are assembled together to provide a c | TGI | [llamastack/distribution-tgi](https://hub.docker.com/repository/docker/llamastack/distribution-tgi/general) | [Guide](https://llama-stack.readthedocs.io/en/latest/getting_started/distributions/self_hosted_distro/tgi.html) | | Together | [llamastack/distribution-together](https://hub.docker.com/repository/docker/llamastack/distribution-together/general) | [Guide](https://llama-stack.readthedocs.io/en/latest/getting_started/distributions/remote_hosted_distro/together.html) | | Fireworks | [llamastack/distribution-fireworks](https://hub.docker.com/repository/docker/llamastack/distribution-fireworks/general) | [Guide](https://llama-stack.readthedocs.io/en/latest/getting_started/distributions/remote_hosted_distro/fireworks.html) | +| Nutanix | [distribution-nutanix](https://hub.docker.com/repository/docker/jinanz/distribution-nutanix/general) | [Guide](https://llama-stack.readthedocs.io/en/latest/getting_started/distributions/remote_hosted_distro/nutanix.html) | ## Installation diff --git a/docs/source/distributions/index.md b/docs/source/distributions/index.md index 4b66a5fc85..d7452c276a 100644 --- a/docs/source/distributions/index.md +++ b/docs/source/distributions/index.md @@ -39,6 +39,7 @@ If so, we suggest: - **Do you have an API key for a remote inference provider like Fireworks, Together, etc.?** If so, we suggest: - [distribution-together](./remote_hosted_distro/together.md) - [distribution-fireworks](./remote_hosted_distro/fireworks.md) + - [distribution-nutanix](./remote_hosted_distro/nutanix.md) - **Do you want to run Llama Stack inference on your iOS / Android device** If so, we suggest: - [iOS](./ondevice_distro/ios_sdk.md) diff --git a/docs/source/distributions/remote_hosted_distro/index.md b/docs/source/distributions/remote_hosted_distro/index.md index d2c9282fc3..45e07c7705 100644 --- a/docs/source/distributions/remote_hosted_distro/index.md +++ b/docs/source/distributions/remote_hosted_distro/index.md @@ -13,6 +13,7 @@ Remote-Hosted distributions are available endpoints serving Llama Stack API that |-------------|----------|-----------|---------|---------|---------|------------| | Together | [https://llama-stack.together.ai](https://llama-stack.together.ai) | remote::together | meta-reference | remote::weaviate | meta-reference | meta-reference | | Fireworks | [https://llamastack-preview.fireworks.ai](https://llamastack-preview.fireworks.ai) | remote::fireworks | meta-reference | remote::weaviate | meta-reference | meta-reference | +| Nutanix | [https://llamastack-preview.nutanix.ai](https://llamastack-preview.nutanix.ai) | remote::nutanix | meta-reference | meta-reference | meta-reference | meta-reference | ## Connecting to Remote-Hosted Distributions diff --git a/llama_stack/templates/nutanix/doc_template.md b/llama_stack/templates/nutanix/doc_template.md index e67529771d..be68fa7f91 100644 --- a/llama_stack/templates/nutanix/doc_template.md +++ b/llama_stack/templates/nutanix/doc_template.md @@ -1,40 +1,63 @@ # Nutanix Distribution -The `llamastack/distribution-nutanix` distribution consists of the following provider configurations. +```{toctree} +:maxdepth: 2 +:hidden: +self +``` -| **API** | **Inference** | **Agents** | **Memory** | **Safety** | **Telemetry** | -|----------------- |--------------- |---------------- |-------------------------------------------------- |---------------- |---------------- | -| **Provider(s)** | remote::nutanix | meta-reference | meta-reference | meta-reference | meta-reference | +The `llamastack/distribution-{{ name }}` distribution consists of the following provider configurations. +{{ providers_table }} -### Start the Distribution (Hosted remote) +{% if run_config_env_vars %} +### Environment Variables -> [!NOTE] -> This assumes you have an hosted Nutanix AI endpoint and an API Key. +The following environment variables can be configured: -1. Clone the repo -``` -git clone git@github.com:meta-llama/llama-stack.git -cd llama-stack -``` +{% for var, (default_value, description) in run_config_env_vars.items() %} +- `{{ var }}`: {{ description }} (default: `{{ default_value }}`) +{% endfor %} +{% endif %} -2. Config the model name +{% if default_models %} +### Models -Please adjust the `NUTANIX_SUPPORTED_MODELS` variable at line 29 in `llama_stack/providers/adapters/inference/nutanix/nutanix.py` according to your deployment. +The following models are available by default: -3. Build the distrbution -``` -pip install -e . -llama stack build --template nutanix --image-type conda -``` +{% for model in default_models %} +- `{{ model.model_id }} ({{ model.provider_model_id }})` +{% endfor %} +{% endif %} -4. Edit the yaml file -``` -vim -``` -5. Serve and enjoy! +### Prerequisite: API Keys +Make sure you have a Nutanix AI Endpoint deployed and a API key. + + +## Running Llama Stack with Nutanix + +You can do this via Conda (build code) or Docker. + +### Via Docker + +```bash +llama stack build --template nutanix --image-type docker + +LLAMA_STACK_PORT=1740 +llama stack run nutanix \ + --port $LLAMA_STACK_PORT \ + --env NUTANIX_API_KEY=$NUTANIX_API_KEY ``` -llama stack run ntnx --port 174 + +### Via Conda + +```bash +llama stack build --template nutanix --image-type conda + +LLAMA_STACK_PORT=1740 +llama stack run ./run.yaml \ + --port $LLAMA_STACK_PORT \ + --env NUTANIX_API_KEY=$NUTANIX_API_KEY ```