March CLI Breaking Changes (#1020)

* remove component job, flatten code (#1002)

* Update .gitignore

* Addressing CLI breaking changes

* adding uri_file to path

* adding uri_file type to paths

* adding uri_file and uri_folder

* Updating basic pipeline jobs

* Update hello-pipeline-default-artifacts.yml

* Lochen/cli march (#1009)

* remove component job, flatten code

* add r component sample

* Online deployment changes

* Batch Deployment Changes

* Bug fixes to YAML

* Update REST API examples to Feb API (#1013)

Co-authored-by: Kayla Ames <[email protected]>

* minor fix

* resolve yaml error

* environment name and version as optional in custom containers (#1016)

* minor updates (#1017)

* Lochen/cli march (#1019)

* remove component job, flatten code

* add r component sample

* update cli

* Updates to cifar example

* Adding daily build of CLI for testing

* Updating from dataset to data for create

* Rename folder from asset/datasets to assets/data

* Updating data assets to use data schema directive instead of dataset

* rename description to data asset

* Changing github workflows to use the new data assets

* removing dataset from md files

* Update README.md

* Removing local_path from examples

* Fixing python:latest bug with workaround

* Update prefix.md

* Lochen/cli march (#1027)

* remove component job, flatten code

* add r component sample

* update cli

* update after test

* Updating environments to use latest version

* Updating environment to us latest

* Update setup.sh

* Test

* streamline register from a run (#1024)

* streamline register from a run

* update to MLflow specific

* Lochen/cli march (#1031)

* remove component job, flatten code

* add r component sample

* update cli

* update after test

* add default_compute, dataset

* Lochen/cli march (#1032)

* remove component job, flatten code

* add r component sample

* update cli

* update after test

* add default_compute, dataset

* fix default compute, sweep

* fix default settings on pipeline

* Using the new build for CLI

* Update requirements.txt (#1033)

* Updating to Build # 58490278

* Fixing filenames in the workflows.

* Fixing file name in train.sh

* add default value in pipeline settings.

* update default value

* change data asset

* update job link in readme

* Updating to latest CLI build

* Merge latest changes from main related to cli/jobs/single-step/gpu_perf/ command (#1037)

* pick and choose changes from main
* align, add readme

* Update create-datasets.sh

* Updating to latest build

* Update README.md

* Update README.md

* Fixing failures in CLI

* Dummy commit to re-run tests

* fix to align with resource in test ws

* change 4d to use uri_folder

* upgrade dependencies, use fastparquet to fix pyarrow issue (#1043)

* Upgrading to latest build

* merge datastore uri sample

* fix file script

* Users/anksing/mlflow model mir deploy (#1047)

* Adding example for OBO flow using v2 cli

* Updating sklearn model for MIR deployment

* Updating lightgbm mlflow model for MIR deployment

* Removing obo folder

* Users/anksing/mlflow mir deployment (#1048)

* Adding example for OBO flow using v2 cli

* Renaming model to avoid conflict with existing model with same name

* Removing obo folder

* Update batch-score.sh

Adding az version check.

* Updating model name to avoid conflict (#1049)

* change cifar to directly use command

* add image classification cli example

* Updating to latest build

* Making setup files Unix friendly by removing \r

* Update endpoint.yml

remove the properties config -- we are not exposing in v2

* Update blue-deployment.yml

Removed properties field - not exposing user-configurable properties metadata in V2

* Dummy Commit to force tests

* update pipeline cifar to make it work under cpu-cluster

* update cifar sample to support cpu sku

* change 5c to cpu image

* clean up yaml, change cpu image for cifar

* Updating to latest build

* Update README.md

* formatting python files to ensure smoke test does not fail

* update workflow (#1064)

Co-authored-by: Han Wang <[email protected]>

* Wanhan/add image classification CI (#1065)

* add image classification ci

* fix create component

Co-authored-by: Han Wang <[email protected]>

* fix image classification ci (#1066)

Co-authored-by: Han Wang <[email protected]>

* Update entry.py

* Update top latest build

* resolve merge conflict

* update to v2 dsl.component

* update train step (#1071)

Co-authored-by: Han Wang <[email protected]>

* Updating to latest build

* Update README.md

* Update README.md

* remove pyjokes in source file

* recover sample to test CI

* add component CRUD sample in assets

* Updating to new build

* Update setup.sh

* Changing readme temporarily to show march status

* Examples for workspace connections (#1073)

* Create python-feed-pat.yml

* More examples

* Create README.md

* Update README.md

* update component name, add CI workflow

* update readme for asset component

* Update setup.sh

* Add yml for scheduled start stop of CI

Add yml for scheduled start stop of CI

* Delete instance-schedule.yml

* fix: fix attr overwrite command (#1074)

* fix: fix attr overwrite command
add pipeline job test shell

* fix: fix attr overwrite command

* fix: create data with random version

Co-authored-by: zhangxingzhi <[email protected]>

* Updating to latest build

* Running tests on canary

* Trying to run test cases in master

* Reverting back to eastus

* Testing a new secret file

* Delete test-new-creds.yml

* Running tests on master

* Update job.yml (#1083)

* Run tests in canary

* remove old dataset sample

* change process per node to 1 for cifar in pipeline

* add note to use multi-gpu cards

* Running latest build on eastus

* Update README.md

* fix sample issue to workaround output empty folder case

* Updating to latest build and testing in eastus

* Cleaning up all GitHub Workflows and the README file

* Update setup.sh

* Update dev-requirements.txt

* Fixing formatting issue so that black does not error

* Updating to the RC build

* change component name (#1075)

* change component name

* rename

* add march-cli-preview in pipeline CI

* update registered case

* swith CI to master

* add back component creation

* update 3b to use rw_mount (#1097)

* fix: fix run-pipeline-jobs.sh

* change upload mode to rw_mount

Co-authored-by: Han Wang <[email protected]>
Co-authored-by: Cloga Chen <[email protected]>
Co-authored-by: zhangxingzhi <[email protected]>

* Update README.md

* remove pipeline sample (#1102)

* temp delete 2 samples

* remove reference links

Co-authored-by: Anthony Hu <[email protected]>

* Fixing workflows and getting ready for merge to main

Co-authored-by: Cloga Chen <[email protected]>
Co-authored-by: Kayla Seager <[email protected]>
Co-authored-by: Kayla Ames <[email protected]>
Co-authored-by: Clement Wang <[email protected]>
Co-authored-by: Shivani Santosh Sambare <[email protected]>
Co-authored-by: Paul Shealy <[email protected]>
Co-authored-by: Abraham Omorogbe <[email protected]>
Co-authored-by: Ankit Singhal <[email protected]>
Co-authored-by: Jeff Omhover <[email protected]>
Co-authored-by: Hugo Aponte <[email protected]>
Co-authored-by: Lisa Li (STC) <[email protected]>
Co-authored-by: mx-iao <[email protected]>
Co-authored-by: Anthony Hu <[email protected]>
Co-authored-by: Han Wang <[email protected]>
Co-authored-by: Han Wang <[email protected]>
Co-authored-by: Honglin Du <[email protected]>
Co-authored-by: Roope Astala <[email protected]>
Co-authored-by: Swati Gharse <[email protected]>
Co-authored-by: elliotzh <[email protected]>
Co-authored-by: zhangxingzhi <[email protected]>
Co-authored-by: eniac871 <[email protected]>

Author: 19 people

.github/workflows/amlarc-cli-train-test.yml

@@ -160,14 +160,14 @@ jobs:
 
     ## pipeline
     # cifar-10
-    - name: run cli/jobs/pipelines/cifar-10/job.yml
-      run: |
-        bash .github/amlarc-tool.sh run_cli_job cli/jobs/pipelines/cifar-10/job.yml \
-              jobs.get-data.resources.instance_type=cpu \
-              jobs.train-model.resources.instance_type=gpu \
-              jobs.eval-model.resources.instance_type=gpu
-      continue-on-error: true
-      timeout-minutes: 30
+    # - name: run cli/jobs/pipelines/cifar-10/job.yml
+    #   run: |
+    #     bash .github/amlarc-tool.sh run_cli_job cli/jobs/pipelines/cifar-10/job.yml \
+    #           jobs.get-data.resources.instance_type=cpu \
+    #           jobs.train-model.resources.instance_type=gpu \
+    #           jobs.eval-model.resources.instance_type=gpu
+    #   continue-on-error: true
+    #   timeout-minutes: 30
     # nyc-taxi
     - name: run cli/jobs/pipelines/nyc-taxi/job.yml
       run: |

.github/workflows/cli-assets-component-train.yml

@@ -0,0 +1,29 @@
+name: cli-assets-component-train
+on:
+  workflow_dispatch:
+  schedule:
+    - cron: "0 0/4 * * *"
+  pull_request:
+    branches:
+      - main
+    paths:
+      - cli/assets/component/train.yml
+      - .github/workflows/cli-assets-component-train.yml
+      - cli/setup.sh
+jobs:
+  build:
+    runs-on: ubuntu-latest
+    steps:
+    - name: check out repo
+      uses: actions/checkout@v2
+    - name: azure login
+      uses: azure/login@v1
+      with:
+        creds: ${{secrets.AZ_CREDS}}
+    - name: setup
+      run: bash setup.sh
+      working-directory: cli
+      continue-on-error: true
+    - name: create asset
+      run: az ml component create -f assets/component/train.yml
+      working-directory: cli

.github/workflows/cli-assets-dataset-local-file.yml .github/workflows/cli-assets-data-cloud-file-https.yml

@@ -1,16 +1,14 @@
-name: cli-assets-dataset-local-file
+name: cli-assets-data-cloud-file-https
 on:
   workflow_dispatch:
   schedule:
     - cron: "0 0/4 * * *"
   pull_request:
     branches:
       - main
-      - cli-preview
-      - releases/current
     paths:
-      - cli/assets/dataset/local-file.yml
-      - .github/workflows/cli-assets-dataset-local-file.yml
+      - cli/assets/data/cloud-file-https.yml
+      - .github/workflows/cli-assets-data-cloud-file-https.yml
       - cli/setup.sh
 jobs:
   build:
@@ -27,5 +25,5 @@ jobs:
       working-directory: cli
       continue-on-error: true
     - name: create asset
-      run: az ml dataset create -f assets/dataset/local-file.yml
+      run: az ml data create -f assets/data/cloud-file-https.yml
       working-directory: cli

.github/workflows/cli-assets-dataset-cloud-file.yml .github/workflows/cli-assets-data-cloud-file-wasbs.yml

@@ -1,16 +1,14 @@
-name: cli-assets-dataset-cloud-file
+name: cli-assets-data-cloud-file-wasbs
 on:
   workflow_dispatch:
   schedule:
     - cron: "0 0/4 * * *"
   pull_request:
     branches:
       - main
-      - cli-preview
-      - releases/current
     paths:
-      - cli/assets/dataset/cloud-file.yml
-      - .github/workflows/cli-assets-dataset-cloud-file.yml
+      - cli/assets/data/cloud-file-wasbs.yml
+      - .github/workflows/cli-assets-data-cloud-file-wasbs.yml
       - cli/setup.sh
 jobs:
   build:
@@ -27,5 +25,5 @@ jobs:
       working-directory: cli
       continue-on-error: true
     - name: create asset
-      run: az ml dataset create -f assets/dataset/cloud-file.yml
+      run: az ml data create -f assets/data/cloud-file-wasbs.yml
       working-directory: cli

.github/workflows/cli-assets-data-cloud-file.yml

@@ -0,0 +1,29 @@
+name: cli-assets-data-cloud-file
+on:
+  workflow_dispatch:
+  schedule:
+    - cron: "0 0/4 * * *"
+  pull_request:
+    branches:
+      - main
+    paths:
+      - cli/assets/data/cloud-file.yml
+      - .github/workflows/cli-assets-data-cloud-file.yml
+      - cli/setup.sh
+jobs:
+  build:
+    runs-on: ubuntu-latest
+    steps:
+    - name: check out repo
+      uses: actions/checkout@v2
+    - name: azure login
+      uses: azure/login@v1
+      with:
+        creds: ${{secrets.AZ_CREDS}}
+    - name: setup
+      run: bash setup.sh
+      working-directory: cli
+      continue-on-error: true
+    - name: create asset
+      run: az ml data create -f assets/data/cloud-file.yml
+      working-directory: cli

.github/workflows/cli-assets-dataset-local-folder.yml .github/workflows/cli-assets-data-cloud-folder-https.yml

@@ -1,16 +1,14 @@
-name: cli-assets-dataset-local-folder
+name: cli-assets-data-cloud-folder-https
 on:
   workflow_dispatch:
   schedule:
     - cron: "0 0/4 * * *"
   pull_request:
     branches:
       - main
-      - cli-preview
-      - releases/current
     paths:
-      - cli/assets/dataset/local-folder.yml
-      - .github/workflows/cli-assets-dataset-local-folder.yml
+      - cli/assets/data/cloud-folder-https.yml
+      - .github/workflows/cli-assets-data-cloud-folder-https.yml
       - cli/setup.sh
 jobs:
   build:
@@ -27,5 +25,5 @@ jobs:
       working-directory: cli
       continue-on-error: true
     - name: create asset
-      run: az ml dataset create -f assets/dataset/local-folder.yml
+      run: az ml data create -f assets/data/cloud-folder-https.yml
       working-directory: cli

.github/workflows/cli-assets-dataset-cloud-folder.yml .github/workflows/cli-assets-data-cloud-folder-wasbs.yml

@@ -1,16 +1,14 @@
-name: cli-assets-dataset-cloud-folder
+name: cli-assets-data-cloud-folder-wasbs
 on:
   workflow_dispatch:
   schedule:
     - cron: "0 0/4 * * *"
   pull_request:
     branches:
       - main
-      - cli-preview
-      - releases/current
     paths:
-      - cli/assets/dataset/cloud-folder.yml
-      - .github/workflows/cli-assets-dataset-cloud-folder.yml
+      - cli/assets/data/cloud-folder-wasbs.yml
+      - .github/workflows/cli-assets-data-cloud-folder-wasbs.yml
       - cli/setup.sh
 jobs:
   build:
@@ -27,5 +25,5 @@ jobs:
       working-directory: cli
       continue-on-error: true
     - name: create asset
-      run: az ml dataset create -f assets/dataset/cloud-folder.yml
+      run: az ml data create -f assets/data/cloud-folder-wasbs.yml
       working-directory: cli

.github/workflows/cli-assets-data-cloud-folder.yml

@@ -0,0 +1,29 @@
+name: cli-assets-data-cloud-folder
+on:
+  workflow_dispatch:
+  schedule:
+    - cron: "0 0/4 * * *"
+  pull_request:
+    branches:
+      - main
+    paths:
+      - cli/assets/data/cloud-folder.yml
+      - .github/workflows/cli-assets-data-cloud-folder.yml
+      - cli/setup.sh
+jobs:
+  build:
+    runs-on: ubuntu-latest
+    steps:
+    - name: check out repo
+      uses: actions/checkout@v2
+    - name: azure login
+      uses: azure/login@v1
+      with:
+        creds: ${{secrets.AZ_CREDS}}
+    - name: setup
+      run: bash setup.sh
+      working-directory: cli
+      continue-on-error: true
+    - name: create asset
+      run: az ml data create -f assets/data/cloud-folder.yml
+      working-directory: cli

.github/workflows/cli-assets-data-iris-csv-example.yml

@@ -0,0 +1,29 @@
+name: cli-assets-data-iris-csv-example
+on:
+  workflow_dispatch:
+  schedule:
+    - cron: "0 0/4 * * *"
+  pull_request:
+    branches:
+      - main
+    paths:
+      - cli/assets/data/iris-csv-example.yml
+      - .github/workflows/cli-assets-data-iris-csv-example.yml
+      - cli/setup.sh
+jobs:
+  build:
+    runs-on: ubuntu-latest
+    steps:
+    - name: check out repo
+      uses: actions/checkout@v2
+    - name: azure login
+      uses: azure/login@v1
+      with:
+        creds: ${{secrets.AZ_CREDS}}
+    - name: setup
+      run: bash setup.sh
+      working-directory: cli
+      continue-on-error: true
+    - name: create asset
+      run: az ml data create -f assets/data/iris-csv-example.yml
+      working-directory: cli

.github/workflows/cli-assets-data-local-file.yml

@@ -0,0 +1,29 @@
+name: cli-assets-data-local-file
+on:
+  workflow_dispatch:
+  schedule:
+    - cron: "0 0/4 * * *"
+  pull_request:
+    branches:
+      - main
+    paths:
+      - cli/assets/data/local-file.yml
+      - .github/workflows/cli-assets-data-local-file.yml
+      - cli/setup.sh
+jobs:
+  build:
+    runs-on: ubuntu-latest
+    steps:
+    - name: check out repo
+      uses: actions/checkout@v2
+    - name: azure login
+      uses: azure/login@v1
+      with:
+        creds: ${{secrets.AZ_CREDS}}
+    - name: setup
+      run: bash setup.sh
+      working-directory: cli
+      continue-on-error: true
+    - name: create asset
+      run: az ml data create -f assets/data/local-file.yml
+      working-directory: cli

.github/workflows/cli-assets-data-local-folder.yml

@@ -0,0 +1,29 @@
+name: cli-assets-data-local-folder
+on:
+  workflow_dispatch:
+  schedule:
+    - cron: "0 0/4 * * *"
+  pull_request:
+    branches:
+      - main
+    paths:
+      - cli/assets/data/local-folder.yml
+      - .github/workflows/cli-assets-data-local-folder.yml
+      - cli/setup.sh
+jobs:
+  build:
+    runs-on: ubuntu-latest
+    steps:
+    - name: check out repo
+      uses: actions/checkout@v2
+    - name: azure login
+      uses: azure/login@v1
+      with:
+        creds: ${{secrets.AZ_CREDS}}
+    - name: setup
+      run: bash setup.sh
+      working-directory: cli
+      continue-on-error: true
+    - name: create asset
+      run: az ml data create -f assets/data/local-folder.yml
+      working-directory: cli

.github/workflows/cli-assets-data-public-file-https.yml

@@ -0,0 +1,29 @@
+name: cli-assets-data-public-file-https
+on:
+  workflow_dispatch:
+  schedule:
+    - cron: "0 0/4 * * *"
+  pull_request:
+    branches:
+      - main
+    paths:
+      - cli/assets/data/public-file-https.yml
+      - .github/workflows/cli-assets-data-public-file-https.yml
+      - cli/setup.sh
+jobs:
+  build:
+    runs-on: ubuntu-latest
+    steps:
+    - name: check out repo
+      uses: actions/checkout@v2
+    - name: azure login
+      uses: azure/login@v1
+      with:
+        creds: ${{secrets.AZ_CREDS}}
+    - name: setup
+      run: bash setup.sh
+      working-directory: cli
+      continue-on-error: true
+    - name: create asset
+      run: az ml data create -f assets/data/public-file-https.yml
+      working-directory: cli