Testing matrix support for python version 3.9/3.10, linux/mac in ci p…

…ipeline (microsoft#197) # Description Testing matrix support for sdk cli tests, windows still need more time to pass. Bypass some mac tests on Heyi's suggestion. # All Promptflow Contribution checklist: - [x] **The pull request does not introduce [breaking changes]** - [ ] **CHANGELOG is updated for new features, bug fixes or other significant changes.** - [x] **I have read the [contribution guidelines](../CONTRIBUTING.md).** ## General Guidelines and Best Practices - [x] Title of the pull request is clear and informative. - [x] There are a small number of commits, each of which have an informative message. This means that previously merged commits do not appear in the history of the PR. For more information on cleaning up the commits in your PR, [see this page](https://github.com/Azure/azure-powershell/blob/master/documentation/development-docs/cleaning-up-commits.md). ### Testing Guidelines - [x] Pull request includes test coverage for the included changes.
Malagurti · Aug 30, 2023 · f0fd77e · f0fd77e
1 parent 23cb98f
commit f0fd77e
Show file tree

Hide file tree

Showing 40 changed files with 288 additions and 173 deletions.
diff --git a/.github/CODEOWNERS b/.github/CODEOWNERS
@@ -1 +1 @@
-* @microsoft/prompt-flow-approvers
+* @microsoft/prompt-flow-approvers
diff --git a/.github/ISSUE_TEMPLATE/bug_report.md b/.github/ISSUE_TEMPLATE/bug_report.md
@@ -12,7 +12,7 @@ A clear and concise description of the bug.
 
 **How To Reproduce the bug**
 Steps to reproduce the behavior, how frequent can you experience the bug:
-1. 
+1.
 
 **Expected behavior**
 A clear and concise description of what you expected to happen.

diff --git a/.github/PULL_REQUEST_TEMPLATE.md b/.github/PULL_REQUEST_TEMPLATE.md
@@ -12,4 +12,4 @@ Please add an informative description that covers that changes made by the pull
 - [ ] There are a small number of commits, each of which have an informative message. This means that previously merged commits do not appear in the history of the PR. For more information on cleaning up the commits in your PR, [see this page](https://github.com/Azure/azure-powershell/blob/master/documentation/development-docs/cleaning-up-commits.md).
 
 ### Testing Guidelines
-- [ ] Pull request includes test coverage for the included changes.
+- [ ] Pull request includes test coverage for the included changes.
diff --git a/.github/actions/step_create_conda_environment/action.yml b/.github/actions/step_create_conda_environment/action.yml
@@ -4,6 +4,10 @@ inputs:
     required: false
     default: "scripts/building/release-env.yml"
     type: string
+  pythonVersion:
+    required: false
+    default: "3.9"
+    type: string
 runs:
   using: composite
   steps:
@@ -13,11 +17,11 @@ runs:
       miniconda-version: "latest"
       activate-environment: release-env
       environment-file: ${{ inputs.condaEnvironmentFilePath }}
-      python-version: 3.9
+      python-version: ${{ inputs.pythonVersion }}
       auto-activate-base: false
       auto-update-conda: true
   - run: |
       conda info
       conda list
       python --version
-    shell: bash -el {0}
+    shell: bash -el {0}
diff --git a/.github/actions/step_generate_configs/action.yml b/.github/actions/step_generate_configs/action.yml
@@ -15,4 +15,4 @@ runs:
       conda activate release-env
       echo "Generating connection config file..."
       python3 ./scripts/building/generate_connection_config.py \
-          --target_folder ${{ inputs.targetFolder }}
+          --target_folder ${{ inputs.targetFolder }}
diff --git a/.github/actions/step_initialize_build/action.yml b/.github/actions/step_initialize_build/action.yml
diff --git a/.github/actions/step_publish_test_results/action.yml b/.github/actions/step_publish_test_results/action.yml
@@ -0,0 +1,40 @@
+name: step_publish_test_results
+inputs:
+  osVersion:
+    required: false
+    default: "ubuntu-latest"
+    type: string
+  pythonVersion:
+    required: false
+    default: "3.9"
+    type: string
+  coverageThreshold:
+    required: false
+    default: "0.3"
+    type: string
+  token:
+    description: 'A Github PAT'
+    required: true
+runs:
+  using: composite
+  steps:
+  - name: Download Artifacts
+    uses: actions/download-artifact@v3
+    with:
+      path: artifacts
+  - name: Display and Set Environment Variables
+    run: env | sort >> $GITHUB_OUTPUT
+    shell: bash -el {0}
+    id: display_env
+  - name: Publish Test Results
+    uses: EnricoMi/publish-unit-test-result-action@v2
+    with:
+      check_name: "Executor Unit Test Result [${{ steps.display_env.outputs.GITHUB_HEAD_REF }}](https://github.com/microsoft/promptflow/actions/workflows/promptflow-executor-unit-test.yml?query=branch:${{ steps.display_env.outputs.GITHUB_HEAD_REF }}++)"
+      comment_title: "Executor Unit Test Result [${{ steps.display_env.outputs.GITHUB_HEAD_REF }}](https://github.com/microsoft/promptflow/actions/workflows/promptflow-executor-unit-test.yml?query=branch:${{ steps.display_env.outputs.GITHUB_HEAD_REF }}++)"
+      files: "artifacts/**/test-*.xml"
+  - name: Generate Coverage Report
+    uses: orgoro/[email protected]
+    with:
+      coverageFile: "artifacts/Test Results (Python ${{ inputs.pythonVersion }}) (OS ${{ inputs.osVersion }})/coverage.xml"
+      token: ${{ inputs.token }}
+      thresholdAll: ${{ inputs.coverageThreshold }}
diff --git a/.github/actions/step_sdk_setup/action.yml b/.github/actions/step_sdk_setup/action.yml
@@ -75,4 +75,4 @@ runs:
       pip install './dist/promptflow_sdk-0.0.1-py3-none-any.whl[azure]'
       echo "########### pip freeze ###########"
       pip freeze
-    working-directory: ${{ inputs.scriptPath }}
+    working-directory: ${{ inputs.scriptPath }}
diff --git a/.github/labeler.yml b/.github/labeler.yml
@@ -13,4 +13,4 @@ promptflow-tools:
 - src/promptflow-tools/**
 fundamental:
 - scripts/**
-- .github/**
+- .github/**
diff --git a/.github/workflows/promptflow-executor-e2e-test.yml b/.github/workflows/promptflow-executor-e2e-test.yml
@@ -1,5 +1,7 @@
 name: promptflow-executor-e2e-test
 on:
+  schedule:
+    - cron: "40 20 * * *" # Every day starting at 4:40 BJT
   pull_request:
     branches: [ main ]
     paths:
@@ -15,20 +17,31 @@ env:
   testWorkingDirectory: ${{ github.workspace }}/src/promptflow
 jobs:
   executor_e2e_tests:
-    runs-on: ubuntu-latest
+    strategy:
+      fail-fast: false
+      matrix:
+        os: [ubuntu-latest, macos-latest]
+    runs-on: ${{ matrix.os }}
     steps:
     - name: checkout
       uses: actions/checkout@v3
     - name: Display and Set Environment Variables
-      run: env | sort >> $GITHUB_OUTPUT
-      shell: bash -el {0}
+      run: |
+        if [ "ubuntu-latest" == "${{ matrix.os }}" ]; then
+          export pyVersion="3.9";
+        elif [ "macos-latest" == "${{ matrix.os }}" ]; then
+          export pyVersion="3.10";
+        else
+          echo "Unsupported OS: ${{ matrix.os }}";
+          exit 1;
+        fi
+        env | sort >> $GITHUB_OUTPUT
       id: display_env
-    - name: Conda Setup
+      shell: bash -el {0}
+    - name: Conda Setup - ${{ matrix.os }} - Python Version ${{ steps.display_env.outputs.pyVersion }}
       uses: "./.github/actions/step_create_conda_environment"
-    - run: |
-        echo ${{ env.packageSetupType }}
-        echo ${{ env.testWorkingDirectory }}
-      shell: bash -l {0}
+      with:
+        pythonVersion: ${{ steps.display_env.outputs.pyVersion }}
     - name: Build wheel
       uses: "./.github/actions/step_sdk_setup"
       with:
@@ -42,6 +55,9 @@ jobs:
       uses: "./.github/actions/step_generate_configs"
       with:
         targetFolder: ${{ env.testWorkingDirectory }}
+    - name: Get number of CPU cores
+      uses: SimenB/github-actions-cpu-cores@v1
+      id: cpu-cores
     - name: Run Coverage Test
       shell: bash -l {0}
       working-directory: ${{ github.workspace }}
@@ -57,29 +73,34 @@ jobs:
           -t ${{ github.workspace }}/src/promptflow/tests/executor/e2etests \
           -l eastus \
           -m "all" \
-          -n 4 \
+          -n ${{ steps.cpu-cores.outputs.count }} \
           --coverage-config ${{ github.workspace }}/src/promptflow/tests/executor/.coveragerc
     - name: Upload Test Results
       if: always()
       uses: actions/upload-artifact@v3
       with:
-        name: pytest-results
+        name: Test Results (Python ${{ steps.display_env.outputs.pyVersion }}) (OS ${{ matrix.os }})
         path: |
           ${{ github.workspace }}/*.xml
           ${{ github.workspace }}/htmlcov/
-      # Use always() to always run this step to publish test results when there are test failures
+  publish-test-results:
+    name: "Publish Tests Results"
+    needs: executor_e2e_tests
+    runs-on: ubuntu-latest
+    permissions:
+      checks: write
+      pull-requests: write
+      contents: read
+      issues: read
+    if: always()
+
+    steps:
+    - name: checkout
+      uses: actions/checkout@v3
     - name: Publish Test Results
-      if: always()
-      uses: EnricoMi/publish-unit-test-result-action/composite@v2
-      with:
-        check_name: "Executor E2E Test Result [${{ steps.display_env.outputs.GITHUB_HEAD_REF }}](https://github.com/microsoft/promptflow/actions/workflows/promptflow-executor-e2e-test.yml?query=branch:${{ steps.display_env.outputs.GITHUB_HEAD_REF }}++)"
-        comment_title: "Executor E2E Test Result [${{ steps.display_env.outputs.GITHUB_HEAD_REF }}](https://github.com/microsoft/promptflow/actions/workflows/promptflow-executor-e2e-test.yml?query=branch:${{ steps.display_env.outputs.GITHUB_HEAD_REF }}++)"
-        files: |
-          ${{ github.workspace }}/test-results.xml
-    - name: Generate Coverage Report
-      if: (success() || failure())
-      uses: orgoro/[email protected]
+      uses: "./.github/actions/step_publish_test_results"
       with:
-        coverageFile: ${{ github.workspace }}/coverage.xml
-        token: ${{ secrets.GITHUB_TOKEN }}
-        thresholdAll: 0.55
+        osVersion: ubuntu-latest
+        pythonVersion: 3.9
+        coverageThreshold: 0.55
+        token: ${{ secrets.GITHUB_TOKEN }}
diff --git a/.github/workflows/promptflow-executor-unit-test.yml b/.github/workflows/promptflow-executor-unit-test.yml
@@ -1,5 +1,7 @@
 name: promptflow-executor-unit-test
 on:
+  schedule:
+    - cron: "40 19 * * *" # Every day starting at 3:40 BJT
   pull_request:
     branches: [ main ]
     paths:
@@ -15,20 +17,31 @@ env:
   testWorkingDirectory: ${{ github.workspace }}/src/promptflow
 jobs:
   executor_unit_tests:
-    runs-on: ubuntu-latest
+    strategy:
+      fail-fast: false
+      matrix:
+        os: [ubuntu-latest, macos-latest]
+    runs-on: ${{ matrix.os }}
     steps:
     - name: checkout
       uses: actions/checkout@v3
     - name: Display and Set Environment Variables
-      run: env | sort >> $GITHUB_OUTPUT
-      shell: bash -el {0}
+      run: |
+        if [ "ubuntu-latest" == "${{ matrix.os }}" ]; then
+          export pyVersion="3.9";
+        elif [ "macos-latest" == "${{ matrix.os }}" ]; then
+          export pyVersion="3.10";
+        else
+          echo "Unsupported OS: ${{ matrix.os }}";
+          exit 1;
+        fi
+        env | sort >> $GITHUB_OUTPUT
       id: display_env
-    - name: Conda Setup
+      shell: bash -el {0}
+    - name: Conda Setup - ${{ matrix.os }} - Python Version ${{ steps.display_env.outputs.pyVersion }}
       uses: "./.github/actions/step_create_conda_environment"
-    - run: |
-        echo ${{ env.packageSetupType }}
-        echo ${{ env.testWorkingDirectory }}
-      shell: bash -l {0}
+      with:
+        pythonVersion: ${{ steps.display_env.outputs.pyVersion }}
     - name: Build wheel
       uses: "./.github/actions/step_sdk_setup"
       with:
@@ -42,6 +55,9 @@ jobs:
       uses: "./.github/actions/step_generate_configs"
       with:
         targetFolder: ${{ env.testWorkingDirectory }}
+    - name: Get number of CPU cores
+      uses: SimenB/github-actions-cpu-cores@v1
+      id: cpu-cores
     - name: Run Coverage Test
       shell: bash -l {0}
       working-directory: ${{ github.workspace }}
@@ -56,29 +72,34 @@ jobs:
           -t ${{ github.workspace }}/src/promptflow/tests/executor/unittests \
           -l eastus \
           -m "all" \
-          -n 4 \
+          -n ${{ steps.cpu-cores.outputs.count }} \
           --coverage-config ${{ github.workspace }}/src/promptflow/tests/executor/.coveragerc
     - name: Upload Test Results
       if: always()
       uses: actions/upload-artifact@v3
       with:
-        name: pytest-results
+        name: Test Results (Python ${{ steps.display_env.outputs.pyVersion }}) (OS ${{ matrix.os }})
         path: |
           ${{ github.workspace }}/*.xml
           ${{ github.workspace }}/htmlcov/
-      # Use always() to always run this step to publish test results when there are test failures
+  publish-test-results:
+    name: "Publish Tests Results"
+    needs: executor_unit_tests
+    runs-on: ubuntu-latest
+    permissions:
+      checks: write
+      pull-requests: write
+      contents: read
+      issues: read
+    if: always()
+
+    steps:
+    - name: checkout
+      uses: actions/checkout@v3
     - name: Publish Test Results
-      if: always()
-      uses: EnricoMi/publish-unit-test-result-action/composite@v2
-      with:
-        check_name: "Executor Unit Test Result [${{ steps.display_env.outputs.GITHUB_HEAD_REF }}](https://github.com/microsoft/promptflow/actions/workflows/promptflow-executor-unit-test.yml?query=branch:${{ steps.display_env.outputs.GITHUB_HEAD_REF }}++)"
-        comment_title: "Executor Unit Test Result [${{ steps.display_env.outputs.GITHUB_HEAD_REF }}](https://github.com/microsoft/promptflow/actions/workflows/promptflow-executor-unit-test.yml?query=branch:${{ steps.display_env.outputs.GITHUB_HEAD_REF }}++)"
-        files: |
-          ${{ github.workspace }}/test-results.xml
-    - name: Generate Coverage Report
-      if: (success() || failure())
-      uses: orgoro/[email protected]
+      uses: "./.github/actions/step_publish_test_results"
       with:
-        coverageFile: ${{ github.workspace }}/coverage.xml
+        osVersion: ubuntu-latest
+        pythonVersion: 3.9
+        coverageThreshold: 0.3
         token: ${{ secrets.GITHUB_TOKEN }}
-        thresholdAll: 0.3
Original file line number	Diff line number	Diff line change
		@@ -1 +1 @@
		* @microsoft/prompt-flow-approvers
		* @microsoft/prompt-flow-approvers