Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

vgpu: Improve reliability of enabling sriov #37

Merged
merged 6 commits into from
Mar 12, 2025
Merged

vgpu: Improve reliability of enabling sriov #37

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
2408b38
Make enabling sriov more reliable
jovial Jan 10, 2025
aa10d8a
Add retries to mdev unit
jovial Jan 10, 2025
486a47e
Take2: Fix failure to start on dependency failure
jovial Jan 10, 2025
b75712d
Improve comment
jovial Jan 10, 2025
31cc166
Add comment explaing why we rebind the nvidia driver
jovial Mar 12, 2025
4a2f6e8
Fix line length on comments
jovial Mar 12, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
12 changes: 8 additions & 4 deletions roles/vgpu/templates/nvidia-mdev.service.j2
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,15 +4,19 @@ Before=docker.service
{% if vgpu_definition.mig_devices is defined %}
After=nvidia-mig-manager.service
Requires=nvidia-mig-manager.service
{% else %}
After=nvidia-sriov-{{ vgpu_definition.pci_address }}.service
Requires=nvidia-sriov-{{ vgpu_definition.pci_address }}.service
{% endif %}

[Service]
Restart=on-failure
RestartSec=30
Type=oneshot
User=root
ExecStartPre=/bin/sleep 5
{% if vgpu_definition.mig_devices is not defined %}
# Workaround lack of UpheldBy/RestartMode=direct in systemd<254 to ensure unit is
# started when the dependency fails, see:
# https://unix.stackexchange.com/questions/213185/restarting-systemd-service-on-dependency-failure
ExecStartPre=/usr/bin/systemctl is-active nvidia-sriov-{{ vgpu_definition.pci_address }}.service
{% endif %}
ExecStart=-/usr/sbin/mdevctl start --uuid %i
RemainAfterExit=yes

Expand Down
13 changes: 10 additions & 3 deletions roles/vgpu/templates/nvidia-sriov.service.j2
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,12 +6,19 @@ After=local-fs.target {{ vgpu_systemd_device[vgpu_definition.pci_address] }}
Wants={{ vgpu_systemd_device[vgpu_definition.pci_address] }}

[Service]
Restart=on-failure
RestartSec=30
Type=oneshot
User=root
# NOTE(wszumski): There is a race in the driver initialization where if we run this too early, then
# the mdev_support_devices entry doesn't show up in sysfs. I was unable to get this to show up again
# without a reboot.
# NOTE(wszumski): There is a race in the driver initialization where if we run
# this too early, then the mdev_support_devices entry doesn't show up in sysfs.
# I was unable to get this to show up again without a reboot.
ExecStartPre=/bin/sleep 5
# NOTE(wszumski): The sriov-manage script will unbind the nvidia driver to
# initialize the virtual functions. If it fails part way through, the driver
# can be left unbound, and subsequent executions of sriov-mange will fail. This
# ensures that the nvidia driver is always bound before we run sriov-manage.
ExecStart=/bin/bash -c "echo '{{ vgpu_definition.pci_address }}' > /sys/bus/pci/drivers/nvidia/bind || true"
ExecStart=/usr/lib/nvidia/sriov-manage -e {{ vgpu_definition.pci_address }}
RemainAfterExit=yes

Expand Down