iTeachSkillsApp

The iTeachSkills App for HoloLens2 developed by Unity

Contents

• iTeachSkillsApp
  • Contents
  • Environment Setup
    • 1. Create Conda Environment
    • 2. Install ROS1 Melodic as instructed in RoboStack
    • 3. Compile ros_tcp_endpoint Package in ROS1 Melodic
  • How to Test on Linux
    • Terminal 1: Start ROS1 Melodic
    • Terminal 2: Launch ros_tcp_endpoint
    • Terminal 3: Publish Images from a Video or a Scene Directory
    • Terminal 4: Run RVIZ to Visualize the Images
  • How to Test on Windows
    • Terminal 1: Start ROS1 Melodic
    • Terminal 2: Launch ros_tcp_endpoint
    • Terminal 3: Publish test images

Environment Setup

1. Create Conda Environment

• Create conda environment

mamba create -n iteachskills python=3.11

• Activate conda environment

mamba activate iteachskills

• Install PyTorch v2.5.1 with CUDA 11.8

python -m pip install torch==2.5.1 torchvision==0.20.1 --index-url https://download.pytorch.org/whl/cu118 --no-cache-dir

• Install ultralytics

python -m pip install ultralytics --no-cache-dir

2. Install ROS1 Melodic as instructed in RoboStack

• Setup channels

conda config --env --add channels conda-forge
conda config --env --add channels robostack-staging
conda config --env --remove channels defaults

• Install ROS1 Melodic

mamba install ros-noetic-desktop

• Reactivate conda environment

mamba deactivate
mamba activate iteachskills

• Install tools for local development

mamba install compilers cmake pkg-config make ninja colcon-common-extensions catkin_tools rosdep

• Additional dependencies for developing on windows (optional)

# Install the Visual Studio command prompt - if you use Visual Studio 2019:
mamba install vs2019_win-64

# Install the Visual Studio command prompt - if you use Visual Studio 2022:
mamba install vs2022_win-64

3. Compile ros_tcp_endpoint Package in ROS1 Melodic

• Clone the ros_tcp_endpoint package

cd ~/catkin_ws/src
git clone 'https://github.com/Unity-Technologies/ROS-TCP-Endpoint.git'

• Build the ros_tcp_endpoint package

cd ~/catkin_ws
catkin_make

How to Test on Linux

Terminal 1: Start ROS1 Melodic

mamba activate iteachskills
roscore

Terminal 2: Launch ros_tcp_endpoint

mamba activate iteachskills
# Source Catwin Workspace
source ~/catkin_ws/devel/setup.bash
# Launch ros_tcp_endpoint
roslaunch ros_tcp_endpoint endpoint.launch

Terminal 3: Publish Images from a Video or a Scene Directory

1. Publish test images from the demo.mp4 video

mamba activate iteachskills
python Python/image_publisher.py

2. Publish images from a scene directory

mamba activate iteachskills
python Python/image_publisher.py <path_to_scene_dir>

• The label results will be saved under the output/prompts directory.
• The the prompts for SAM2 could be obtained by:

  import json
  W = 640 # Image width
  H = 480 # Image height
  with open('prompts.json', 'r') as f:
      prompts = json.load(f)
  sam2_point_prompts = []
  for prompt in prompts:
      sam2_points = [
        (int(pt["x"] * W), int((1 - pt["y"]) * H, label))
        for pt, label in zip(prompt["points"], prompt["labels"])
        ]
      sam2_point_prompts.append(sam2_points)

Terminal 4: Run RVIZ to Visualize the Images

mamba activate iteachskills
rviz -d Python/image_viewer.rviz

How to Test on Windows

Terminal 1: Start ROS1 Melodic

mamba activate iteachskills
roscore

Terminal 2: Launch ros_tcp_endpoint

mamba activate iteachskills
# Source Catwin Workspace
call C:/Users/JikaiWang/Catkin/devel/setup.bat
# Launch ros_tcp_endpoint
roslaunch ros_tcp_endpoint endpoint.launch

Terminal 3: Publish test images

mamba activate iteachskills
python Python/image_publisher.py

📚 BibTex

Please cite iTeach if it helps your research 🙌:

@misc{padalunkal2024iteach,
  title         = {iTeach: In the Wild Interactive Teaching for Failure-Driven Adaptation of Robot Perception},
  author        = {Jishnu Jaykumar P and Cole Salvato and Vinaya Bomnale and Jikai Wang and Yu Xiang},
  year          = {2026},
  eprint        = {2410.09072},
  archivePrefix = {arXiv},
  primaryClass  = {cs.RO},
  url           = {https://arxiv.org/abs/2410.09072}
}

📬 Contact

For any clarification, comments, or suggestions, you can choose from the following options:

• Join the discussion forum. 💬
• Report an issue. 🛠️
• Contact Jishnu. 📧

🙏 Acknowledgements

This work was supported by the DARPA Perceptually-enabled Task Guidance (PTG) Program under contract number HR00112220005, the Sony Research Award Program, and the National Science Foundation (NSF) under Grant No.2346528. We thank Sai Haneesh Allu for assistance with the real-world experiments. 🙌