Skip to content

ci/triton resilience and eager ml tools tests #1609

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
NJManganelli wants to merge 7 commits into
base: master
Choose a base branch
from
Open

ci/triton resilience and eager ml tools tests #1609

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
7 commits
Select commit Hold shift + click to select a range
5a8dc4f
ci: don't start the triton server for the job that never uses it
Jul 26, 2026
a08d0c6
ci: tolerate an unreachable nvcr.io when starting the triton server
Jul 26, 2026
c7b0688
ci: widen the triton start backoff to 15s/1m/5m/20m
Jul 27, 2026
c409feb
Revert "ci: don't start the triton server for the job that never uses…
Jul 27, 2026
c76a944
test(ml_tools): cover the eager wrapper path without a dask client
Jul 27, 2026
3cedfdc
ci: apply the retry and skip flag to both triton server steps
Jul 27, 2026
ed33cb9
test(ml_tools): name the wrapper tests by the array type they exercise
Jul 27, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
36 changes: 34 additions & 2 deletions .github/workflows/ci.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -89,7 +89,23 @@ jobs:
- name: Start triton server with example model
if: startsWith( matrix.os, 'ubuntu' ) && matrix.python-version != '3.14t'
run: |
docker run -d --rm -p 8000:8000 -p 8001:8001 -p 8002:8002 -v ${{ github.workspace }}/tests/samples/triton_models_test:/models nvcr.io/nvidia/tritonserver:25.11-pyt-python-py3 tritonserver --model-repository=/models
# Pulls from nvcr.io intermittently time out. Back off 15s, 1m, 5m, 20m
# between attempts: long enough to ride out a transient registry outage,
# bounded so a sustained one costs ~26 minutes. On exhaustion report the
# server as unavailable so the triton tests skip rather than failing the
# job. A server that starts but misbehaves still fails the tests.
backoff=(15 60 300 1200)
for attempt in 0 1 2 3 4; do
if docker run -d --rm -p 8000:8000 -p 8001:8001 -p 8002:8002 -v ${{ github.workspace }}/tests/samples/triton_models_test:/models nvcr.io/nvidia/tritonserver:25.11-pyt-python-py3 tritonserver --model-repository=/models; then
exit 0
fi
if [ "${attempt}" -lt 4 ]; then
echo "::warning::triton server start failed (attempt $((attempt + 1))/5); retrying in ${backoff[$attempt]}s"
sleep "${backoff[$attempt]}"
fi
done
echo "::warning::triton server unavailable after 5 attempts; triton tests will skip"
echo "TRITON_UNAVAILABLE=1" >> "$GITHUB_ENV"
- name: Test with pytest (without dask Client)
run: |
python -m pytest --cov-report=xml --cov=coffea --ignore=tests/test_taskvine_dask.py --ignore=tests/test_taskvine_virtual.py -m "not dask_client" -n 4
Expand Down Expand Up @@ -166,7 +182,23 @@ jobs:
- name: Start triton server with example model
if: startsWith( matrix.os, 'ubuntu' ) && matrix.python-version != '3.14t'
run: |
docker run -d --rm -p 8000:8000 -p 8001:8001 -p 8002:8002 -v ${{ github.workspace }}/tests/samples/triton_models_test:/models nvcr.io/nvidia/tritonserver:25.11-pyt-python-py3 tritonserver --model-repository=/models
# Pulls from nvcr.io intermittently time out. Back off 15s, 1m, 5m, 20m
# between attempts: long enough to ride out a transient registry outage,
# bounded so a sustained one costs ~26 minutes. On exhaustion report the
# server as unavailable so the triton tests skip rather than failing the
# job. A server that starts but misbehaves still fails the tests.
backoff=(15 60 300 1200)
for attempt in 0 1 2 3 4; do
if docker run -d --rm -p 8000:8000 -p 8001:8001 -p 8002:8002 -v ${{ github.workspace }}/tests/samples/triton_models_test:/models nvcr.io/nvidia/tritonserver:25.11-pyt-python-py3 tritonserver --model-repository=/models; then
exit 0
fi
if [ "${attempt}" -lt 4 ]; then
echo "::warning::triton server start failed (attempt $((attempt + 1))/5); retrying in ${backoff[$attempt]}s"
sleep "${backoff[$attempt]}"
fi
done
echo "::warning::triton server unavailable after 5 attempts; triton tests will skip"
echo "TRITON_UNAVAILABLE=1" >> "$GITHUB_ENV"
- name: Test with pytest (with dask Client)
run: |
python -m pytest --cov-report=xml --cov=coffea --ignore=tests/test_taskvine_dask.py --ignore=tests/test_taskvine_virtual.py -m "dask_client" -n 4
Expand Down
Loading
Loading