Skip to content

DOCS-3204: Add docs for add image to dataset button, incorporate into tutorials where helpful #4244

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 19 commits into from
Apr 23, 2025
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
19 commits
Select commit Hold shift + click to select a range
290a47b
DOCS-3204: Add docs for add image to dataset button, incorporate into…
nathan-contino Apr 22, 2025
ce05262
Merge branch 'main' into DOCS-3204
nathan-contino Apr 22, 2025
b734c56
Add Data API script back, fixed
nathan-contino Apr 22, 2025
80cab48
Apply Naomi's wording suggestions from code review
nathan-contino Apr 22, 2025
e5cb9ed
Implement Naomi feedback
nathan-contino Apr 22, 2025
02b550f
Update docs/data-ai/ai/create-dataset.md
nathan-contino Apr 22, 2025
4dfa5df
Merge branch 'main' into DOCS-3204
nathan-contino Apr 22, 2025
d8f0cf7
Add deleted PNG back, turns out it is actually used
nathan-contino Apr 22, 2025
97c0298
Update model training image, implement Naomi feedback, update Data AP…
nathan-contino Apr 23, 2025
ed3988d
Apply suggestions from code review
nathan-contino Apr 23, 2025
4c28b04
Switch machine learning methods to tabs, trim some extra word where f…
nathan-contino Apr 23, 2025
ff33c8b
Add a newline to break up two separate sentences
nathan-contino Apr 23, 2025
65e3d7a
Implement prettier fixes, need to get in the habit of doing this in V…
nathan-contino Apr 23, 2025
0d24157
Fix broken link
nathan-contino Apr 23, 2025
11846f4
Merge branch 'main' into DOCS-3204
nathan-contino Apr 23, 2025
e3f7345
Add a title for all tablesteps in tflite training instructions
nathan-contino Apr 23, 2025
b887d02
Move orphaned part id instructions from SDK docs, remove binaryID usa…
nathan-contino Apr 23, 2025
7fe4365
Trim example line length to prevent horizontal scrolling
nathan-contino Apr 23, 2025
da5eacf
Increase header level by 1 to improve page flow and logical consistency
nathan-contino Apr 23, 2025
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file modified assets/components/camera/example_camera_image.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
14 changes: 14 additions & 0 deletions assets/scss/_styles_project.scss
Original file line number Diff line number Diff line change
Expand Up @@ -3369,6 +3369,20 @@ div.tablestep > table td, div.tablestep table td {
padding-left: 1rem;
}

/* ensure that, up to a reasonable 4 levels of nesting, including tablesteps, we use distinct ordered list labeling schemes */
.tablestep ol, ol ol {
list-style-type: upper-alpha;
}

.tablestep ol ol, ol ol ol {
list-style-type: lower-roman;
}

.tablestep ol ol ol, ol ol ol ol {
list-style-type: lower-alpha;
}
/* end ordered list labeling styling */

.td-content > .code-toolbar, .td-content * .code-toolbar {
margin-top: 1rem;
}
Expand Down
Binary file removed assets/services/data/data_view.png
Binary file not shown.
Binary file removed assets/services/data/delete_all.png
Binary file not shown.
Binary file modified assets/services/ml/train-model.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
528 changes: 268 additions & 260 deletions docs/data-ai/ai/create-dataset.md

Large diffs are not rendered by default.

7 changes: 3 additions & 4 deletions docs/data-ai/ai/run-inference.md
Original file line number Diff line number Diff line change
Expand Up @@ -43,7 +43,7 @@ Then, from the **Select model** dropdown, select the name of the ML model servic

### Test your changes

You can test a deployed vision service by clicking on the **Test** area of its configuration panel or from the [**CONTROL** tab](/manage/troubleshoot/teleoperate/default-interface/#viam-app).
You can test a deployed vision service by clicking on the **Test** area of its configuration panel or from the [**CONTROL** page](/manage/troubleshoot/teleoperate/default-interface/#viam-app).

The camera stream shows when the vision service identifies something.
Try pointing the camera at a scene similar to your training data.
Expand All @@ -59,7 +59,7 @@ Start by setting the value to 0.8.
This reduces your output by filtering out anything below a threshold of 80% confidence.
You can adjust this attribute as necessary.

Click the **Save** button in the top right corner of the page to save your configuration, then close and reopen the **Test** panel of the vision service configuration panel.
Click the **Save** button in the top right corner of the page to save your configuration, then close and reopen the **TEST** panel of the vision service configuration panel.
Now if you reopen the panel, you will only see classifications or detections with a confidence value higher than the `default_minimum_confidence` attribute.

{{< /expand>}}
Expand All @@ -69,8 +69,7 @@ For more detailed information, including optional attribute configuration, see t
## Use an SDK

You can also run inference using a Viam SDK.
You can use the [`Infer`](/dev/reference/apis/services/ml/#infer)
method of the ML Model API to make inferences.
Use the [`Infer`](/dev/reference/apis/services/ml/#infer) method of the ML Model API to make inferences.

For example:

Expand Down
62 changes: 36 additions & 26 deletions docs/data-ai/ai/train-tflite.md
Original file line number Diff line number Diff line change
Expand Up @@ -26,53 +26,46 @@ date: "2024-12-03"
---

Many machines have cameras through which they can monitor their environment.
With machine leaning, you can train models on patterns within that visual data.
You can collect data from the camera stream and label any patterns within the images.

If a camera is pointed at a food display, for example, you can label the image of the display with `full` or `empty`, or label items such as individual `pizza_slice`s.

Using a model trained on such images, machines can make inferences about their environments.
Your machines can then automatically trigger alerts or perform other actions.
If a food display is empty, the machine could, for example, alert a supervisor to restock the display.

Common use cases for this are **security**, **quality assurance**, and **food service** applications.

With machine leaning (ML), you can train models on patterns within image data.
Follow this guide to use your image data to train an ML model, so that your machine can make inferences about its environment.

## Prerequisites

{{% expand "A running machine connected to the Viam app. Click to see instructions." %}}
{{% expand "a machine connected to the Viam app" %}}

{{% snippet "setup.md" %}}

{{% /expand%}}
{{% /expand %}}

{{% expand "A dataset with labels. Click to see instructions." %}}
{{% expand "a dataset with labels" %}}

Follow the guide to [create a dataset](/data-ai/ai/create-dataset/) if you haven't already.
Follow the guide to [create a dataset](/data-ai/ai/create-dataset/).

{{% /expand%}}

## Train a machine learning (ML) model
## Train a machine learning model

Now that you have a dataset with your labeled images, you are ready to train a machine learning model.
Now that you have a dataset that contains your labeled images, you are ready to train a machine learning model.

{{< table >}}
{{% tablestep number=1 %}}
**Train an ML model**
**Find your training dataset**

In the Viam app, navigate to your list of [**DATASETS**](https://app.viam.com/data/datasets) and select the one you want to train on.

Click **Train model** and follow the prompts.
{{% /tablestep %}}
{{% tablestep number=2 %}}
**Train an ML model**

Click **Train model** and follow the prompts.
You can train a TFLite model using **Built-in training**.

Click **Next steps**.
{{<imgproc src="/services/ml/train-model.png" resize="1200x" declaredimensions=true style="width:500px" alt="The shapes dataset." class="imgzoom fill shadow" >}}

{{<imgproc src="/tutorials/data-management/shapes-dataset.png" resize="1200x" declaredimensions=true style="width:500px" alt="The shapes dataset." class="imgzoom fill shadow" >}}
Click **Next steps**.

{{% /tablestep %}}
{{% tablestep number=2 %}}
{{% tablestep number=3 %}}
**Fill in the details for your ML model**

Enter a name for your new model.
Expand All @@ -91,7 +84,7 @@ Click **Train model**.
{{< imgproc src="/tutorials/data-management/train-model.png" alt="The data tab showing the train a model pane" style="width:500px" resize="1200x" class="imgzoom fill shadow" >}}

{{% /tablestep %}}
{{% tablestep number=3 %}}
{{% tablestep number=4 %}}
**Wait for your model to train**

The model now starts training and you can follow its process on the [**TRAINING** tab](https://app.viam.com/training).
Expand All @@ -101,7 +94,7 @@ Once the model has finished training, it becomes visible on the [**MODELS** tab]
You will receive an email when your model finishes training.

{{% /tablestep %}}
{{% tablestep number=4 %}}
{{% tablestep number=5 %}}
**Debug your training job**

From the [**TRAINING** tab](https://app.viam.com/training), click on your training job's ID to see its logs.
Expand All @@ -128,10 +121,10 @@ As you test it, if you notice faulty predictions or confidence scores, you will

If you trained a _classification_ model, you can test it with the following instructions.

1. Navigate to the [**DATA** tab](https://app.viam.com/data/view) and click on the **Images** subtab.
1. Navigate to the [**DATA** page](https://app.viam.com/data/view) and click on the **Images** subtab.
1. Click on an image to open the side menu, and select the **Actions** tab.
1. In the **Run model** section, select your model and specify a confidence threshold.
1. Click **Run model**
1. Click **Run model**.

If the results exceed the confidence threshold, the **Run model** section shows a label and the responding confidence threshold.

Expand All @@ -141,6 +134,23 @@ You can test both detection models and classifier models using the following res
- [a `tflite_cpu` ML model](/data-ai/ai/deploy/) with the model you just trained
- [an `mlmodel` vision service](/operate/reference/services/vision/mlmodel/) using the `tflite_cpu` model

## Iterate on your ML model

With the right training dataset, your ML model can perform well as soon as training completes.
However, edge cases always exist.
You are unlikely to account for all false positives or false negatives during your first round of training.
Instead, keep track of edge cases as you run your model.
Add images capturing edge cases to your training dataset, annotate them, and re-train your model using the new data.
Using this approach, each subsequent model version becomes more accurate than the last.

To capture images of edge cases and re-train your model using those images, complete the following steps:

1. Add edge case images to your training dataset. You can find edge cases in your existing data on the [**DATA** page](https://app.viam.com/data/) or [capture new images and add them to your training dataset](/data-ai/ai/create-dataset/#capture-images).

1. Visit the **DATASET** tab of the **DATA** page and annotate the image.

1. Repeat the [steps above](/data-ai/ai/train-tflite/#train-a-machine-learning-model) to train and release a new version of your ML model. Your machines will automatically update to the new version of the model soon after release.

## Next steps

Now your machine can make inferences about its environment.
Expand Down
54 changes: 24 additions & 30 deletions docs/data-ai/ai/train.md
Original file line number Diff line number Diff line change
Expand Up @@ -17,7 +17,7 @@ description: "If you want to train models to custom specifications, write a cust
date: "2024-12-04"
---

You can create custom Python training scripts that train ML models to your specifications using PyTorch, Tensorflow, TFLite, ONNX, or any other Machine Learning framework.
You can create custom Python training scripts that train machine learning models to your specifications using PyTorch, TensorFlow, TFLite, ONNX, or any other ML framework.
Once you upload a training script to the [Viam Registry](https://app.viam.com/registry?type=Training+Script), you can use it to build ML models in the Viam Cloud based on your datasets.

You can also use training scripts that are in the registry already.
Expand All @@ -27,9 +27,9 @@ If you wish to do this, skip to [Submit a training job](#submit-a-training-job).

{{% expand "A dataset with data you can train an ML model on. Click to see instructions." %}}

For image data, you can follow the instructions to [Create a dataset](/data-ai/ai/create-dataset/) to create a dataset and label data.
For images, follow the instructions to [Create a dataset](/data-ai/ai/create-dataset/) to create a dataset and label data.

For other data you can use the [Data Client API](/dev/reference/apis/data-client/) from within the training script to get data stored in the Viam Cloud.
For other data, use the [Data Client API](/dev/reference/apis/data-client/) from within the training script to store data in the Viam Cloud.

{{% /expand%}}

Expand Down Expand Up @@ -61,9 +61,9 @@ my-training/
{{% tablestep number=2 %}}
**Add `setup.py` code**

Add the following code to `setup.py` and add additional required packages on line 11:
Add the following code to `setup.py`:

```python {class="line-numbers linkable-line-numbers" data-line="9"}
```python {class="line-numbers linkable-line-numbers"}
from setuptools import find_packages, setup

setup(
Expand All @@ -78,13 +78,7 @@ setup(
```

{{% /tablestep %}}
{{% tablestep number=3 %}}
**Create `__init__.py`**

If you haven't already, create a folder called <file>model</file> and create an empty file inside it called <file>\_\_init\_\_.py</file>.

{{% /tablestep %}}
{{< tablestep number=4 >}}
{{< tablestep number=3 >}}

<p><strong>Add <code>training.py</code> code</strong></p>

Expand Down Expand Up @@ -529,7 +523,7 @@ if __name__ == "__main__":
{{% /expand %}}

{{% /tablestep %}}
{{< tablestep number=5 >}}
{{< tablestep number=4 >}}

<p><strong>Understand template script parsing functionality</strong></p>
<p>When a training script is run, the Viam platform passes the dataset file for the training and the designated model output directory to the script.</p>
Expand All @@ -552,7 +546,7 @@ You can add additional custom command line inputs by adding them to the `parse_a

{{% /expand %}}

{{% expand "Click for more information on parsing annotations from dataset file." %}}
{{% expand "Click for more information on parsing annotations from the dataset file." %}}

When you submit a training job to the Viam Cloud, Viam will pass a `dataset_file` to the training script when you train an ML model with it.
The file contains metadata from the dataset used for the training, including the file path for each data point and any annotations associated with the data.
Expand Down Expand Up @@ -624,46 +618,46 @@ Dataset JSON files for image datasets with bounding box labels and classificatio
}
```

In your training script, you must parse the dataset file for the classification or bounding box annotations from the dataset metadata.
In your training script, you must parse the dataset file to extract classification or bounding box annotations from the dataset metadata.
Depending on if you are training a classification or detection model, the template script contains the `parse_filenames_and_labels_from_json()` and the `parse_filenames_and_bboxes_from_json()` function.

{{% /expand%}}

<p>If the script you are creating does not use an image dataset, you only need the model output directory.</p>

{{% /tablestep %}}
{{% tablestep number=6 %}}
{{% tablestep number=5 %}}
**Add logic to produce the model artifact**

You must fill in the `build_and_compile_model` function.
In this part of the script, you use the data from the dataset and the annotations from the dataset file to build a Machine Learning model.
Fill in the `build_and_compile_model` function.
In this part of the script, you use data and annotations from the dataset file to build an ML model.

As an example, you can refer to the logic from <file>model/training.py</file> from this [example classification training script](https://github.com/viam-modules/classification-tflite) that trains a classification model using TensorFlow and Keras.

{{% /tablestep %}}
{{% tablestep number=7 %}}
{{% tablestep number=6 %}}
**Save the model artifact**

The `save_model()` and the `save_labels()` functions in the template before the `main` logic save the model artifact your training job produces to the `model_output_directory` in the cloud.
In this example template, the training job produces a model artifact.
The `save_model()` and `save_labels()` functions save that model artifact to the `model_output_directory`.

Once a training job is complete, Viam checks the output directory and creates a package with all of the contents of the directory, creating or updating a registry item for the ML model.
When the training job completes, Viam checks the output directory for the model artifact, packages the model, and uploads it to the registry.

You must fill in these functions.
You must fill in the `save_model()` and `save_labels()` functions.

As an example, you can refer to the logic from <file>model/training.py</file> from this [example classification training script](https://github.com/viam-modules/classification-tflite) that trains a classification model using TensorFlow and Keras.
As an example, refer to the logic from <file>model/training.py</file> from this [example classification training script](https://github.com/viam-modules/classification-tflite) that trains a classification model using TensorFlow and Keras.

{{% /tablestep %}}
{{% tablestep number=8 %}}
{{% tablestep number=7 %}}
**Update the main method**

Update the main to call the functions you have just created.

{{% /tablestep %}}
{{% tablestep number=9 %}}
**Using Viam APIs in a training script**
{{% tablestep number=8 %}}
**Use Viam APIs in a training script**

If you need to access any of the [Viam APIs](/dev/reference/apis/) within a custom training script, you can use the environment variables `API_KEY` and `API_KEY_ID` to establish a connection.
These environment variables will be available to training scripts.
To access [Viam APIs](/dev/reference/apis/) within a custom training script, use the environment variables `API_KEY` and `API_KEY_ID` to establish a connection.

```python
async def connect() -> ViamClient:
Expand Down Expand Up @@ -807,13 +801,13 @@ For example:
viam train submit custom from-registry --dataset-id=<INSERT DATASET ID> \
--org-id=<INSERT ORG ID> --model-name=MyRegistryModel \
--model-version=2 --version=1 \
--script-name=mycompany:MyCustomTrainingScript
--script-name=mycompany:MyCustomTrainingScript \
--args=custom_arg1=3,custom_arg2="'green_square blue_star'"
```

This command submits a training job to the previously uploaded `MyCustomTrainingScript` with another input dataset, which trains `MyRegistryModel` and publishes that to the registry.

You can get the dataset id from the dataset page or using the [`viam dataset list`](/dev/tools/cli/#dataset) command.
You can get the dataset id from the **DATASET** tab of the **DATA** page or by running the [`viam dataset list`](/dev/tools/cli/#dataset) command.

{{% /tab %}}
{{< /tabs >}}
Expand Down
2 changes: 1 addition & 1 deletion docs/data-ai/reference/mlmodel-design.md
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,7 @@ aliases:
- /operate/reference/advanced-modules/mlmodel-design/
---

The [Machine Learning (ML) model service](/data-ai/ai/deploy/) allow you to deploy machine learning models to your smart machine.
The [Machine Learning (ML) model service](/data-ai/ai/deploy/) allows you to deploy machine learning models to your smart machine.
Vision services, like [an `"mlmodel"` detector](/dev/reference/apis/services/vision/#detections) or [classifier](/dev/reference/apis/services/vision/#classifications), enable your machines to identify and classify objects in images with the deployed models' predictions.

The two services work closely together, with the vision service relying on the deployed ML model to make inferences.
Expand Down
8 changes: 0 additions & 8 deletions docs/dev/reference/apis/data-client.md
Original file line number Diff line number Diff line change
Expand Up @@ -128,11 +128,3 @@ Once you have instantiated a `DataClient`, you can run [API methods](#api) again
{{< readfile "/static/include/app/apis/generated/data.md" >}}

{{< readfile "/static/include/app/apis/generated/dataset.md" >}}

## Find part ID

To copy the ID of your machine part, select the part status dropdown to the right of your machine's location and name on the top of its page and click the copy icon next to **Part ID**.

For example:

{{<imgproc src="/build/program/data-client/grab-part-id.png" resize="1000x" class="shadow imgzoom" style="width: 500px" declaredimensions=true alt="Part ID displayed in the Viam app.">}}
2 changes: 1 addition & 1 deletion docs/dev/reference/glossary/pin-number.md
Original file line number Diff line number Diff line change
Expand Up @@ -11,4 +11,4 @@ This number is distinct from the GPIO number assigned to general purpose input/o
For example, pin number "11" on a NVIDIA Jetson Nano is GPIO "50", and pin number "11" on a Raspberry Pi 4 is GPIO "17".
When Viam documentation refers to pin number, it will always mean the pin's physical index and not GPIO number.

Pin numbers are found on a board's pinout diagram and data sheet.
Pin numbers are found on a board's pinout diagram and datasheet.
8 changes: 8 additions & 0 deletions docs/operate/reference/architecture/parts.md
Original file line number Diff line number Diff line change
Expand Up @@ -58,6 +58,14 @@ If it cannot connect to them, the part will still successfully start up.

![Example of a remote and a two part machine where the main (and only) part of machine 1 remotes into the main part of machine 2, and thus has access to all resources of machine 2.](/build/configure/parts/remotes-diagram.png)

### Find part ID

To copy the ID of your machine part, select the part status dropdown to the right of your machine's location and name on the top of its page and click the copy icon next to **Part ID**.

For example:

{{<imgproc src="/build/program/data-client/grab-part-id.png" resize="1000x" class="shadow imgzoom" style="width: 500px" declaredimensions=true alt="Part ID displayed in the Viam app.">}}

## Configuration

### Configure a sub-part
Expand Down
4 changes: 2 additions & 2 deletions docs/operate/reference/components/camera/webcam.md
Original file line number Diff line number Diff line change
Expand Up @@ -258,14 +258,14 @@ sudo reboot
{{% /expand%}}

{{% expand "Images are dim on start up" %}}
If you are capturing camera data, it can happen that the camera captures and syncs miscolored or dark images upon start up.
If you are capturing camera data, it can happen that the camera captures and syncs discolored or dark images upon start up.
{{% /expand%}}

{{% expand "CSI Camera not working on a Raspberry Pi" %}}

If you are using a CSI camera v1.3 or v2.0, or v3.0, use the [`viam:camera:csi` module](https://github.com/viamrobotics/csi-camera/) instead.

For Raspberry Pi AI cameras like the IMX500 AI camera, you can use a module such as [this `viam-pi-ai-camera` vision service](https://github.com/HipsterBrown/viam-pi-ai-camera).
For Raspberry Pi AI cameras like the IMX500 AI camera, use a module such as [this `viam-pi-ai-camera` vision service](https://github.com/HipsterBrown/viam-pi-ai-camera).
For more information about the vision service, see [run inference](https://docs.viam.com/data-ai/ai/run-inference/).
{{% /expand%}}

Expand Down
Loading
Loading