Sdk tests with papermill #2448

yehudit1987 · 2024-10-27T10:51:42Z

What this PR does / why we need it:
This PR creates E2E tests for katib examples to run with papermill.

Which issue(s) this PR fixes (optional, in fixes #<issue number>(, fixes #<issue_number>, ...) format, will close the issue(s) when PR gets merged):
Fixes #2417

Checklist:

Docs included if any changes are user facing

review-notebook-app · 2024-10-27T10:51:47Z

Check out this pull request on

See visual diffs & provide feedback on Jupyter Notebooks.

Powered by ReviewNB

Electronic-Waste · 2024-10-28T02:57:15Z

/rerun-all

Electronic-Waste · 2024-10-28T06:22:56Z

@yehudit1987 Can you please fix these CI errors?

Electronic-Waste · 2024-10-29T12:07:12Z

@yehudit1987 Can you sign your commits with git commit -s? The DCO checks failed due to this reason.

Electronic-Waste · 2024-10-29T12:08:26Z

FYI, you can check this reference: https://github.com/kubeflow/katib/pull/2448/checks?check_run_id=32215445282

Electronic-Waste · 2024-10-31T05:13:54Z

/rerun-all

YosiElias · 2024-10-31T08:51:56Z

/rerun-all

Electronic-Waste · 2024-10-31T09:28:30Z

/rerun-all

Electronic-Waste

Thanks for your great contributions @yehudit1987! 🎉

I left some reviews for you, excluding notebooks. Will soon review other files :)

Btw, @andreyvelich @tenzen-y are busy with other projects now and will be back in the middle of November. Your PR will be merged then.

Electronic-Waste · 2024-10-31T15:33:38Z

.github/workflows/sdk-e2e-tests-with-papermill.yaml

@@ -0,0 +1,28 @@
+name: Run e2e sdk tests with papermill


Suggested change

name: Run e2e sdk tests with papermill

name: E2E Tests with Notebooks

I guess it will be better to make the testcase's name consistent with others :)

Electronic-Waste · 2024-10-31T15:39:23Z

.github/workflows/sdk-e2e-tests-with-papermill.yaml

+  cancel-in-progress: true
+
+jobs:
+  create-katib-notebooks-test:


Suggested change

create-katib-notebooks-test:

e2e:

Electronic-Waste · 2024-10-31T15:55:50Z

test/e2e/v1beta1/scripts/gh-actions/build-load.sh

+    # Loop through each algorithm in the array
+    for algorithm_name in "${ALGORITHM_ARRAY[@]}"; do
+      suggestion_image_name="$(algorithm_name=$algorithm_name yq eval '.runtime.suggestions.[] | select(.algorithmName == env(algorithm_name)) | .image' \
+        manifests/v1beta1/installs/katib-standalone/katib-config.yaml | cut -d: -f1)"
+      suggestion_name="$(basename "$suggestion_image_name")"
+      suggestions+=("$suggestion_name")
+    done
+


I think this loop is redundant with the loop in front of it:

katib/test/e2e/v1beta1/scripts/gh-actions/build-load.sh

Lines 77 to 89 in 706a6f2

# Search for Suggestion Images required for Trial.

for exp_name in "${EXPERIMENT_ARRAY[@]}"; do

exp_path=$(find examples/v1beta1 -name "${exp_name}.yaml")

algorithm_name="$(yq eval '.spec.algorithm.algorithmName' "$exp_path")"

suggestion_image_name="$(algorithm_name=$algorithm_name yq eval '.runtime.suggestions.[] | select(.algorithmName == env(algorithm_name)) | .image' \

manifests/v1beta1/installs/katib-standalone/katib-config.yaml | cut -d: -f1)"

suggestion_name="$(basename "$suggestion_image_name")"

suggestions+=("$suggestion_name")

done

Can we combine these two loops into a unified one by using the ALGORITHM parameters with other e2e tests.

WDYT👀 @yehudit1987 @kubeflow/wg-automl-leads

Electronic-Waste · 2024-10-31T16:20:57Z

.github/workflows/template-notebook-test/action.yaml

+              echo "Papermill failed for notebook: $NOTEBOOK"
+              exit 1
+          }
+        done


Suggested change

done

done

A missing new line here

Electronic-Waste · 2024-10-31T16:21:20Z

test/e2e/v1beta1/scripts/gh-actions/build-load.sh

@@ -172,4 +182,4 @@ fi
 echo -e "\nCleanup Build Cache...\n"
 docker buildx prune -f

-echo -e "\nAll Katib images with ${TAG} tag have been built successfully!\n"
+echo -e "\nAll Katib images with ${TAG} tag have been built successfully!\n"


Suggested change

echo -e "\nAll Katib images with ${TAG} tag have been built successfully!\n"

echo -e "\nAll Katib images with ${TAG} tag have been built successfully!\n"

Electronic-Waste · 2024-10-31T16:21:44Z

test/e2e/v1beta1/scripts/gh-actions/setup-katib.sh

+  kubectl create namespace kubeflow-user-example-com
+fi
+
+exit 0


Suggested change

exit 0

exit 0

Electronic-Waste · 2024-10-31T16:22:23Z

test/e2e/v1beta1/scripts/gh-actions/setup-minikube.sh


 echo "Start to setup Minikube Kubernetes Cluster"
 kubectl version
 kubectl cluster-info
 kubectl get nodes

 echo "Build and Load container images"
-./build-load.sh "$DEPLOY_KATIB_UI" "$TUNE_API" "$TRIAL_IMAGES" "$EXPERIMENTS" 
+./build-load.sh "$DEPLOY_KATIB_UI" "$TUNE_API" "$TRIAL_IMAGES" "$EXPERIMENTS" "$ALGORITHMS"


Suggested change

./build-load.sh "$DEPLOY_KATIB_UI" "$TUNE_API" "$TRIAL_IMAGES" "$EXPERIMENTS" "$ALGORITHMS"

./build-load.sh "$DEPLOY_KATIB_UI" "$TUNE_API" "$TRIAL_IMAGES" "$EXPERIMENTS" "$ALGORITHMS"

Electronic-Waste · 2024-10-31T16:25:11Z

.github/workflows/template-notebook-test/action.yaml

+    - name: Setup Minikube Cluster
+      shell: bash
+      run: ./test/e2e/v1beta1/scripts/gh-actions/setup-minikube.sh true true "" "" "cmaes"


Could we reuse template-setup-e2e-test? I guess it will be better if we make full use of the existing template :)

Wait for you thoughts👀 @yehudit1987 @kubeflow/wg-automl-leads

That template being use as a pre template to template-e2e-test. We are using the second one for running yaml experiments by calling a shell script that calls a python script. In our case we just need to add to the job a step that run the notebook directly with papermill. I guess we can use template-setup-e2e-test but it will not prevent us from using the new one.

Electronic-Waste

Sorry for the late response @yehudit1987. I left a few comments for you.

And I'm busy with my works now. I'll give reviews on Notebooks later:)

Electronic-Waste · 2024-11-05T08:15:44Z

test/e2e/v1beta1/scripts/gh-actions/setup-minikube.sh

+if [ -x "$(command -v apt-get)" ]; then
+  echo "Upgrading Podman using apt-get..."
+  sudo apt-get update
+  sudo apt-get install -y podman
+elif [ -x "$(command -v dnf)" ]; then
+  echo "Upgrading Podman using dnf..."
+  sudo dnf upgrade podman -y
+else
+  echo "Package manager not found. Skipping upgrade."
+fi


Could you please tell me why we need to use podman?

Electronic-Waste · 2024-11-05T08:16:55Z

.github/workflows/template-notebook-test/action.yaml

It will be better to change the dir name from template-notebook-test to template-e2e-notebook-test to be consistent with other dirs:)

Electronic-Waste · 2024-11-05T08:38:14Z

/rerun-all

Electronic-Waste · 2024-12-02T13:20:05Z

/rerun-all

andreyvelich · 2025-01-21T13:31:52Z

Hi @yehudit1987, do you have time to finalize this PR ?
@saileshd1402 implemented tests as part of this PR: kubeflow/trainer#2274, so we can use the same script for Katib.

yehudit1987 · 2025-01-22T10:37:24Z

Hi @andreyvelich, yes I have been waited for your approval regarding the script decisions as mentioned above. I will finalize this PR.

google-oss-prow · 2025-01-23T08:09:03Z

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please assign tenzen-y for approval. For more information see the Kubernetes Code Review Process.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

google-oss-prow · 2025-01-23T08:09:09Z

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please assign tenzen-y for approval. For more information see the Kubernetes Code Review Process.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

Signed-off-by: Yehudit Kerido <[email protected]>

andreyvelich

Thank you for doing this, I left initial comments.
/assign @Electronic-Waste @helenxie-bit @shashank-iitbhu @kubeflow/wg-training-leads @saileshd1402 Please help with review

andreyvelich · 2025-02-12T23:31:26Z

.github/workflows/sdk-e2e-tests-with-papermill.yaml

@@ -0,0 +1,28 @@
+name: E2E Tests with Notebooks


Let's call this file:

e2e-test-notebooks.yaml

andreyvelich · 2025-02-12T23:31:42Z

.github/workflows/sdk-e2e-tests-with-papermill.yaml

+on:
+  push: {}
+  pull_request: {}
+  workflow_dispatch: {}


Suggested change

on:

push: {}

pull_request: {}

workflow_dispatch: {}

on:

- pull_request

andreyvelich · 2025-02-12T23:32:57Z

.github/workflows/template-e2e-notebook-test/action.yaml

@@ -0,0 +1,54 @@
+name: Notebook test template


You don't need template if we keep all Notebooks test jobs in the single file.

andreyvelich · 2025-02-12T23:35:09Z

.github/workflows/template-e2e-notebook-test/action.yaml

+    - name: Setup Minikube Cluster
+      uses: medyagh/[email protected]
+      with:
+        network-plugin: cni
+        cni: flannel
+        driver: none
+        kubernetes-version: v1.29.2
+        minikube-version: 1.34.0
+        start-args: --wait-timeout=120s


You can re-use setup-e2e-test template for that, similar to this:

katib/.github/workflows/e2e-test-pytorch-mnist.yaml

Lines 20 to 25 in 683608f

- name: Setup Test Env

uses: ./.github/workflows/template-setup-e2e-test

with:

kubernetes-version: ${{ matrix.kubernetes-version }}

python-version: "3.10"

andreyvelich · 2025-02-12T23:35:59Z

.github/workflows/template-e2e-notebook-test/action.yaml

+
+    - name: Setup Minikube
+      shell: bash
+      run: ./test/e2e/v1beta1/scripts/gh-actions/setup-minikube.sh true true "" "" "cmaes"


Can we re-use this template for this action, but passing the appropriate values to it:
https://github.com/kubeflow/katib/blob/683608f6a61e7b10218b7084f310af42334d8e65/.github/workflows/template-e2e-test/action.yaml
We can have one more input for notebooks tests that triggers run-notebook.sh.

andreyvelich · 2025-02-12T23:48:17Z

test/e2e/v1beta1/scripts/gh-actions/run-notebook.sh

+  echo "Options:"
+  echo "  -i  Input notebook (required)"
+  echo "  -o  Output notebook (required)"
+  echo "  -k  Kubeflow Training Operator Python SDK (optional)"


andreyvelich · 2025-02-12T23:49:13Z

test/e2e/v1beta1/scripts/gh-actions/run-notebook.sh

+  case "$opt" in
+    i) NOTEBOOK_INPUT="$OPTARG" ;;            # -i for notebook input path
+    o) NOTEBOOK_OUTPUT="$OPTARG" ;;           # -o for notebook output path
+    k) TRAINING_PYTHON_SDK="$OPTARG" ;;       # -k for training operator python sdk


andreyvelich · 2025-02-12T23:49:32Z

test/e2e/v1beta1/scripts/gh-actions/run-notebook.sh

+  exit 1
+fi
+
+papermill_cmd="papermill $NOTEBOOK_INPUT $NOTEBOOK_OUTPUT -p training_python_sdk $TRAINING_PYTHON_SDK -p namespace $NAMESPACE"


We should install Katib SDK as a first cell in the Notebooks, similar to this:
https://github.com/kubeflow/trainer/blob/release-1.9/examples/pytorch/image-classification/create-pytorchjob.ipynb

andreyvelich · 2025-02-12T23:49:40Z

test/e2e/v1beta1/scripts/gh-actions/run-notebook.sh

+  exit 1
+fi
+
+echo "Notebook execution completed successfully"


Suggested change

echo "Notebook execution completed successfully"

echo "Notebook execution completed successfully"

andreyvelich · 2025-02-12T23:50:44Z

.github/workflows/template-e2e-notebook-test/action.yaml

+      shell: bash
+      run: |
+        python -m pip install --upgrade pip
+        pip install papermill kubeflow-katib jupyter ipykernel


kubeflow-katib SDK should be installed from source.

Hey @andreyvelich, thanks for the review!
Great points, I’ve fixed them. Let me know if anything else is needed.

Signed-off-by: Yehudit Kerido <[email protected]>

andreyvelich · 2025-02-19T11:52:51Z

/ok-to-test
/rerun-all

google-oss-prow bot added the size/XXL label Oct 27, 2024

google-oss-prow bot requested review from anencore94 and Electronic-Waste October 27, 2024 10:51

yehudit1987 marked this pull request as draft October 28, 2024 09:45

google-oss-prow bot added the do-not-merge/work-in-progress label Oct 28, 2024

yehudit1987 force-pushed the sdk-tests-with-papermill branch from 963d367 to 6633aa5 Compare October 29, 2024 19:29

yehudit1987 marked this pull request as ready for review October 29, 2024 19:31

google-oss-prow bot removed the do-not-merge/work-in-progress label Oct 29, 2024

Electronic-Waste reviewed Oct 31, 2024

View reviewed changes

yehudit1987 marked this pull request as draft November 3, 2024 10:58

google-oss-prow bot added the do-not-merge/work-in-progress label Nov 3, 2024

yehudit1987 marked this pull request as ready for review November 3, 2024 11:16

google-oss-prow bot removed the do-not-merge/work-in-progress label Nov 3, 2024

google-oss-prow bot requested a review from Electronic-Waste November 3, 2024 11:16

Electronic-Waste reviewed Nov 5, 2024

View reviewed changes

yehudit1987 marked this pull request as draft November 5, 2024 09:06

google-oss-prow bot added do-not-merge/work-in-progress size/XL and removed size/XXL labels Nov 5, 2024

yehudit1987 marked this pull request as ready for review November 5, 2024 13:00

google-oss-prow bot removed the do-not-merge/work-in-progress label Nov 5, 2024

google-oss-prow bot added the size/XXL label Dec 2, 2024

yehudit1987 closed this Jan 23, 2025

yehudit1987 force-pushed the sdk-tests-with-papermill branch from 2c0ce60 to 59af784 Compare January 23, 2025 07:58

google-oss-prow bot added size/XS and removed size/XXL labels Jan 23, 2025

yehudit1987 reopened this Jan 23, 2025

google-oss-prow bot added size/XL and removed size/XS labels Jan 23, 2025

Yehudit Kerido added 2 commits January 23, 2025 10:18

sdk-tests-with-papermill

f2d1e7a

Signed-off-by: Yehudit Kerido <[email protected]>

sdk tests with papermill

683608f

Signed-off-by: Yehudit Kerido <[email protected]>

yehudit1987 force-pushed the sdk-tests-with-papermill branch from 70d149a to 683608f Compare January 23, 2025 08:19

yehudit1987 marked this pull request as ready for review January 23, 2025 08:35

google-oss-prow bot removed the do-not-merge/work-in-progress label Jan 23, 2025

google-oss-prow bot requested a review from Electronic-Waste January 23, 2025 08:35

andreyvelich reviewed Feb 12, 2025

View reviewed changes

sdk tests with papermill

c3c4faa

Signed-off-by: Yehudit Kerido <[email protected]>

google-oss-prow bot added size/XXL and removed size/XL labels Feb 17, 2025

yehudit1987 marked this pull request as draft February 17, 2025 14:52

google-oss-prow bot added the do-not-merge/work-in-progress label Feb 17, 2025

sdk tests with papermill

ee43826

Signed-off-by: Yehudit Kerido <[email protected]>

yehudit1987 marked this pull request as ready for review February 17, 2025 15:03

google-oss-prow bot removed the do-not-merge/work-in-progress label Feb 17, 2025

google-oss-prow bot added the ok-to-test label Feb 19, 2025

	name: Run e2e sdk tests with papermill
	name: E2E Tests with Notebooks

	# Search for Suggestion Images required for Trial.
	for exp_name in "${EXPERIMENT_ARRAY[@]}"; do

	exp_path=$(find examples/v1beta1 -name "${exp_name}.yaml")
	algorithm_name="$(yq eval '.spec.algorithm.algorithmName' "$exp_path")"

	suggestion_image_name="$(algorithm_name=$algorithm_name yq eval '.runtime.suggestions.[] \| select(.algorithmName == env(algorithm_name)) \| .image' \
	manifests/v1beta1/installs/katib-standalone/katib-config.yaml \| cut -d: -f1)"
	suggestion_name="$(basename "$suggestion_image_name")"

	suggestions+=("$suggestion_name")

	done

	echo -e "\nAll Katib images with ${TAG} tag have been built successfully!\n"
	echo -e "\nAll Katib images with ${TAG} tag have been built successfully!\n"

	./build-load.sh "$DEPLOY_KATIB_UI" "$TUNE_API" "$TRIAL_IMAGES" "$EXPERIMENTS" "$ALGORITHMS"
	./build-load.sh "$DEPLOY_KATIB_UI" "$TUNE_API" "$TRIAL_IMAGES" "$EXPERIMENTS" "$ALGORITHMS"

	- name: Setup Test Env
	uses: ./.github/workflows/template-setup-e2e-test
	with:
	kubernetes-version: ${{ matrix.kubernetes-version }}
	python-version: "3.10"

	echo "Notebook execution completed successfully"
	echo "Notebook execution completed successfully"

Sdk tests with papermill #2448

Are you sure you want to change the base?

Sdk tests with papermill #2448

Conversation

yehudit1987 commented Oct 27, 2024

review-notebook-app bot commented Oct 27, 2024

Electronic-Waste commented Oct 28, 2024

Electronic-Waste commented Oct 28, 2024 • edited Loading

Electronic-Waste commented Oct 29, 2024

Electronic-Waste commented Oct 29, 2024

Electronic-Waste commented Oct 31, 2024

YosiElias commented Oct 31, 2024

Electronic-Waste commented Oct 31, 2024

Electronic-Waste left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Electronic-Waste left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Electronic-Waste commented Nov 5, 2024

Electronic-Waste commented Dec 2, 2024

andreyvelich commented Jan 21, 2025

yehudit1987 commented Jan 22, 2025

google-oss-prow bot commented Jan 23, 2025

google-oss-prow bot commented Jan 23, 2025

andreyvelich left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

andreyvelich commented Feb 19, 2025

Electronic-Waste commented Oct 28, 2024 •

edited

Loading