From 396513dd579766ee0692d32bb6c4df4e213b7af7 Mon Sep 17 00:00:00 2001 From: jroddev Date: Wed, 29 Jan 2025 15:52:44 +1100 Subject: [PATCH 1/3] Update ai.md Make some changes required to use podman instead of docker --- docs/ai.md | 34 ++++++++++++++++++++++++++++++++-- 1 file changed, 32 insertions(+), 2 deletions(-) diff --git a/docs/ai.md b/docs/ai.md index 7de2c66..bdfdfbb 100644 --- a/docs/ai.md +++ b/docs/ai.md @@ -17,9 +17,10 @@ GPU Acceleration for both Nvidia and AMD are included out of the box and usually Since Alpaca doesn't expose any API, if you need other applications than Alpaca to interact with your ollama instance (for example an IDE) you should consider installing it [in a docker container](https://hub.docker.com/r/ollama/ollama). -To do so, first configure docker to use the nvidia drivers (that come preinstalled with Bluefin) with: +To do so, first configure docker to use the nvidia drivers (that come preinstalled with Bluefin). This is only required with docker and not with podman. ```bash +# Only required for Docker, Not required for Podman sudo nvidia-ctk runtime configure --runtime=docker sudo systemctl restart docker ``` @@ -27,6 +28,7 @@ sudo systemctl restart docker Then, choose a folder where to install the ollama container (for example `~/Containers/ollama`) and inside it create a new file named `docker-compose.yaml` with the following content: ```yaml +# docker compose --- services: ollama: @@ -36,7 +38,7 @@ services: ports: - 11434:11434 volumes: - - ./ollama_v:/root/.ollama + - ./ollama_v:/root/.ollama:z deploy: resources: reservations: @@ -45,10 +47,38 @@ services: - gpu ``` +```yaml +# podman-compose +--- +services: + ollama: + image: ollama/ollama + container_name: ollama + restart: unless-stopped + ports: + - 11434:11434 + volumes: + - ./ollama_v:/root/.ollama:z + devices: + - nvidia.com/gpu=all + deploy: + resources: + reservations: + devices: + - capabilities: + - gpu + +``` + + Finally, open a terminal in the folder containing the file just created and start the container with ```bash docker compose up -d +# or +sudo podman-compose up -d +# podman needs to be run as root for nvidia gpu passthrough until this is fixed +# https://github.com/containers/podman/issues/19338 ``` and your ollama instance should be up and running at `http://127.0.0.1:11434`! From b70bca67c5f381f097fb0b13d2ef49e93a124ca1 Mon Sep 17 00:00:00 2001 From: jroddev Date: Wed, 29 Jan 2025 15:55:12 +1100 Subject: [PATCH 2/3] Update ai.md --- docs/ai.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/ai.md b/docs/ai.md index bdfdfbb..8b3ae06 100644 --- a/docs/ai.md +++ b/docs/ai.md @@ -17,7 +17,7 @@ GPU Acceleration for both Nvidia and AMD are included out of the box and usually Since Alpaca doesn't expose any API, if you need other applications than Alpaca to interact with your ollama instance (for example an IDE) you should consider installing it [in a docker container](https://hub.docker.com/r/ollama/ollama). -To do so, first configure docker to use the nvidia drivers (that come preinstalled with Bluefin). This is only required with docker and not with podman. +To do so, first configure docker to use the nvidia drivers (that come preinstalled with Bluefin). ```bash # Only required for Docker, Not required for Podman @@ -38,7 +38,7 @@ services: ports: - 11434:11434 volumes: - - ./ollama_v:/root/.ollama:z + - ./ollama_v:/root/.ollama deploy: resources: reservations: From 2ff32a705c4df1babe290f14e298366a11cfa3c5 Mon Sep 17 00:00:00 2001 From: jroddev Date: Thu, 30 Jan 2025 19:33:53 +1100 Subject: [PATCH 3/3] Update ai.md split podman from the docker section, also add a quadlet version --- docs/ai.md | 82 ++++++++++++++++++++++++++++++++++++++++++------------ 1 file changed, 64 insertions(+), 18 deletions(-) diff --git a/docs/ai.md b/docs/ai.md index 8b3ae06..df1d69c 100644 --- a/docs/ai.md +++ b/docs/ai.md @@ -15,20 +15,56 @@ GPU Acceleration for both Nvidia and AMD are included out of the box and usually ### Ollama API -Since Alpaca doesn't expose any API, if you need other applications than Alpaca to interact with your ollama instance (for example an IDE) you should consider installing it [in a docker container](https://hub.docker.com/r/ollama/ollama). +Since Alpaca doesn't expose any API, if you need other applications than Alpaca to interact with your ollama instance (for example an IDE) you should consider installing it in a [container](https://hub.docker.com/r/ollama/ollama). -To do so, first configure docker to use the nvidia drivers (that come preinstalled with Bluefin). +#### Quadlet (recommended) +`~/.local/share/systemd/ollama.container` +``` +[Unit] +Description=Ollama Service +After=network.target local-fs.target + +[Container] +Image=ollama/ollama:latest +ContainerName=ollama +AutoUpdate=yes +PublishPort=11434:11434 +Volume=./ollama_v:/root/.ollama:z +Device=/dev/nvidia*:ro +Deploy=resources.reservations.devices.capabilities=gpu + +[Service] +RestartUnlessStopped=yes +TimeoutStartSec=60s + +[Install] +WantedBy=multi-user.target +``` -```bash -# Only required for Docker, Not required for Podman -sudo nvidia-ctk runtime configure --runtime=docker -sudo systemctl restart docker +```sh +❯ systemctl --user daemon-reload + +# start Ollama podlet for current session +❯ systemctl --user start ollama +❯ systemctl --user status ollama + +# start Ollama podlet automatically after reboot +❯ systemctl --user enable ollama + +# connect to ollama +❯ ollama list + +# download and run model https://ollama.com/search +❯ ollama run ``` -Then, choose a folder where to install the ollama container (for example `~/Containers/ollama`) and inside it create a new file named `docker-compose.yaml` with the following content: + +#### Podman Compose +> **NOTE:** Podman needs to be run with sudo for nvidia gpu passthrough until [this](https://github.com/containers/podman/issues/19338) issue is fixed. +> +Create this `podman-compose.yaml file ```yaml -# docker compose --- services: ollama: @@ -38,17 +74,34 @@ services: ports: - 11434:11434 volumes: - - ./ollama_v:/root/.ollama + - ./ollama_v:/root/.ollama:z + devices: + - nvidia.com/gpu=all deploy: resources: reservations: devices: - capabilities: - gpu + ``` +`❯ sudo podman-compose up -d` + + + +#### Docker Compose + +To do so, first configure docker to use the nvidia drivers (that come preinstalled with Bluefin). + +```bash +sudo nvidia-ctk runtime configure --runtime=docker +sudo systemctl restart docker +``` + +Then, choose a folder where to install the ollama container (for example `~/Containers/ollama`) and inside it create a new file named `docker-compose.yaml` with the following content: + ```yaml -# podman-compose --- services: ollama: @@ -58,16 +111,13 @@ services: ports: - 11434:11434 volumes: - - ./ollama_v:/root/.ollama:z - devices: - - nvidia.com/gpu=all + - ./ollama_v:/root/.ollama deploy: resources: reservations: devices: - capabilities: - gpu - ``` @@ -75,10 +125,6 @@ Finally, open a terminal in the folder containing the file just created and star ```bash docker compose up -d -# or -sudo podman-compose up -d -# podman needs to be run as root for nvidia gpu passthrough until this is fixed -# https://github.com/containers/podman/issues/19338 ``` and your ollama instance should be up and running at `http://127.0.0.1:11434`!