[ROS] Update Official Docker image docs (#2578)

* Remove references to dedicated ROS distros
as final ROS 1 version, Noetic, is now EOL

* Redirect primary links to documentation landing page
as docs page is more up-to-date and pertinent for users

* Remove retired ROS blog
- https://www.ros.org/news/2020/01/ros-blog-retired.html

* Update OSRF URL redirect

* Update example image tag to rolling
to avoid docs from becoming stale

* Consolidate branding and dated references

* Update Q&A links to stackexchange given migration
- https://discourse.ros.org/t/planned-migration-of-ros-answers-and-gazebo-answers-to-robotics-stack-exchange/28068
- https://discourse.ros.org/t/new-static-archives-for-ros-and-gazebo-answers/41346

* Simplify cloning overlay source

* Simplify use of mixin setting

* Split runner and builder stages
as example to save on image size
as well as on build time using caching

* Simplify scripting for dependencies

* Lint use of ARGs
for readability of each stage

* Simplify example cache mount for apt
and update rosdep along with apt list
to ensure package index is fresh

* Update categories for ROS
to include additional relevant entries

* Shorten derivation script

* Use ros-core tag for minimal installer

* Stage example image sizes for demos

* Simplify naming

* Omit cacher from size table
to focus on primary targets

* Modify and mount custom apt config
accounting for included docker-clean default

* Simplify RUN directive for apt config

* Stage WIP edits

* Update package cache before cloning source
to avoid running rosdep update for each source change
as docker build --no-cache can still be used to force update

* Fix use of EOF in Dockerfile

* Simply RUN directives

* Expand bullet points about optimizations

* Expand upon runner vs installer comparison

* Linting

* Fix grammar and spelling

* Linting

* Rename stage in apt get example
while keeping with uniquely spelled noun
that is self descriptive yet unmistakable

* Comment on overwriting defaults to persist minimal cache

Author: ruffsl

ros/content.md

@@ -1,10 +1,10 @@
-# What is [ROS](https://www.ros.org/)?
+# What is [ROS](https://docs.ros.org/)?
 
 The Robot Operating System (ROS) is a set of software libraries and tools that help you build robot applications. From drivers to state-of-the-art algorithms, and with powerful developer tools, ROS has what you need for your next robotics project. And it's all open source.
 
 > [wikipedia.org/wiki/Robot_Operating_System](https://en.wikipedia.org/wiki/Robot_Operating_System)
 
-[%%LOGO%%](https://www.ros.org/)
+[%%LOGO%%](https://docs.ros.org/)
 
 # How to use this image
 
@@ -13,7 +13,7 @@ The Robot Operating System (ROS) is a set of software libraries and tools that h
 To create your own ROS docker images and install custom packages, here's a simple example of installing the C++, Python client library demos using the official released Debian packages via apt-get.
 
 ```dockerfile
-FROM %%IMAGE%%:foxy
+FROM %%IMAGE%%:rolling-ros-core as aptgetter
 
 # install ros package
 RUN apt-get update && apt-get install -y \
@@ -28,8 +28,8 @@ CMD ["ros2", "launch", "demo_nodes_cpp", "talker_listener_launch.py"]
 Note: all ROS images include a default entrypoint that sources the ROS environment setup before executing the configured command, in this case the demo packages launch file. You can then build and run the Docker image like so:
 
 ```console
-$ docker build -t my/ros:app .
-$ docker run -it --rm my/ros:app
+$ docker build -t my/ros:aptgetter .
+$ docker run -it --rm my/ros:aptgetter
 [INFO] [launch]: process[talker-1]: started with pid [813]
 [INFO] [launch]: process[listener-2]: started with pid [814]
 [INFO] [talker]: Publishing: 'Hello World: 1'
@@ -44,59 +44,85 @@ $ docker run -it --rm my/ros:app
 To create your own ROS docker images and build custom packages, here's a simple example of installing a package's build dependencies, compiling it from source, and installing the resulting build artifacts into a final multi-stage image layer.
 
 ```dockerfile
-ARG FROM_IMAGE=%%IMAGE%%:foxy
+ARG FROM_IMAGE=%%IMAGE%%:rolling
 ARG OVERLAY_WS=/opt/ros/overlay_ws
 
 # multi-stage for caching
 FROM $FROM_IMAGE AS cacher
+ARG OVERLAY_WS
+
+# overwrite defaults to persist minimal cache
+RUN rosdep update --rosdistro $ROS_DISTRO && \
+    cat <<EOF > /etc/apt/apt.conf.d/docker-clean && apt-get update
+APT::Install-Recommends "false";
+APT::Install-Suggests "false";
+EOF
 
 # clone overlay source
-ARG OVERLAY_WS
 WORKDIR $OVERLAY_WS/src
-RUN echo "\
-repositories: \n\
-  ros2/demos: \n\
-    type: git \n\
-    url: https://github.com/ros2/demos.git \n\
-    version: ${ROS_DISTRO} \n\
-" > ../overlay.repos
-RUN vcs import ./ < ../overlay.repos
-
-# copy manifests for caching
-WORKDIR /opt
-RUN mkdir -p /tmp/opt && \
-    find ./ -name "package.xml" | \
-      xargs cp --parents -t /tmp/opt && \
-    find ./ -name "COLCON_IGNORE" | \
-      xargs cp --parents -t /tmp/opt || true
+RUN cat <<EOF | vcs import .
+repositories:
+  ros2/demos:
+    type: git
+    url: https://github.com/ros2/demos.git
+    version: ${ROS_DISTRO}
+EOF
+
+# derive build/exec dependencies
+RUN bash -e <<'EOF'
+declare -A types=(
+  [exec]="--dependency-types=exec"
+  [build]="")
+for type in "${!types[@]}"; do
+  rosdep install -y \
+    --from-paths \
+      ros2/demos/demo_nodes_cpp \
+      ros2/demos/demo_nodes_py \
+    --ignore-src \
+    --reinstall \
+    --simulate \
+    ${types[$type]} \
+    | grep 'apt-get install' \
+    | awk '{gsub(/'\''/,"",$4); print $4}' \
+    | sort -u > /tmp/${type}_debs.txt
+done
+EOF
 
 # multi-stage for building
 FROM $FROM_IMAGE AS builder
-
-# install overlay dependencies
 ARG OVERLAY_WS
-WORKDIR $OVERLAY_WS
-COPY --from=cacher /tmp/$OVERLAY_WS/src ./src
-RUN . /opt/ros/$ROS_DISTRO/setup.sh && \
-    apt-get update && rosdep install -y \
-      --from-paths \
-        src/ros2/demos/demo_nodes_cpp \
-        src/ros2/demos/demo_nodes_py \
-      --ignore-src \
-    && rm -rf /var/lib/apt/lists/*
+
+# install build dependencies
+COPY --from=cacher /tmp/build_debs.txt /tmp/build_debs.txt
+RUN --mount=type=cache,target=/etc/apt/apt.conf.d,from=cacher,source=/etc/apt/apt.conf.d \
+    --mount=type=cache,target=/var/lib/apt/lists,from=cacher,source=/var/lib/apt/lists \
+    --mount=type=cache,target=/var/cache/apt,sharing=locked \
+    < /tmp/build_debs.txt xargs apt-get install -y
 
 # build overlay source
+WORKDIR $OVERLAY_WS
 COPY --from=cacher $OVERLAY_WS/src ./src
-ARG OVERLAY_MIXINS="release"
 RUN . /opt/ros/$ROS_DISTRO/setup.sh && \
     colcon build \
       --packages-select \
         demo_nodes_cpp \
         demo_nodes_py \
-      --mixin $OVERLAY_MIXINS
+      --mixin release
+
+# multi-stage for running
+FROM $FROM_IMAGE-ros-core AS runner
+ARG OVERLAY_WS
+
+# install exec dependencies
+COPY --from=cacher /tmp/exec_debs.txt /tmp/exec_debs.txt
+RUN --mount=type=cache,target=/etc/apt/apt.conf.d,from=cacher,source=/etc/apt/apt.conf.d \
+    --mount=type=cache,target=/var/lib/apt/lists,from=cacher,source=/var/lib/apt/lists \
+    --mount=type=cache,target=/var/cache/apt,sharing=locked \
+    < /tmp/exec_debs.txt xargs apt-get install -y
 
-# source entrypoint setup
-ENV OVERLAY_WS $OVERLAY_WS
+# setup overlay install
+ENV OVERLAY_WS=$OVERLAY_WS
+COPY --from=builder $OVERLAY_WS/install $OVERLAY_WS/install
 RUN sed --in-place --expression \
       '$isource "$OVERLAY_WS/install/setup.bash"' \
       /ros_entrypoint.sh
@@ -105,11 +131,36 @@ RUN sed --in-place --expression \
 CMD ["ros2", "launch", "demo_nodes_cpp", "talker_listener_launch.py"]
 ```
 
-The example above starts by using [`vcstool`](https://github.com/dirk-thomas/vcstool) to clone source repos of interest into the cacher stage. One could similarly `COPY` code from the local build context into the source directory as well. Package manifest files are then cached in a temporary directory where the following builder stage may copy from to install necessary dependencies with [`rosdep`](https://github.com/ros-infrastructure/rosdep). This is done prior to copying the rest of the source files to preserve the multi-stage build cache, given unaltered manifests do not alter declared dependencies, saving time and bandwidth. The overlay is then built using [`colcon`](https://colcon.readthedocs.io/en/released/), the entrypoint updated to source the workspace, and the default command set to launch the demo.
+The example above consists of three sequential stages. The `cacher` stage first updates the apt lists and ROS index, uses [`vcstool`](https://github.com/dirk-thomas/vcstool) to clone a demo repo into the workspace source directory, and derives build and runtime dependency sets using [`rosdep`](https://docs.ros.org/en/rolling/Tutorials/Intermediate/Rosdep.html). The `builder` stage installs the derived build dependencies, sources the ROS install underlay, and compiles the source in release mode using [`colcon`](https://docs.ros.org/en/rolling/Tutorials/Beginner-Client-Libraries/Colcon-Tutorial.html). Finally, the `runner` stage installs only runtime dependencies, copies the compiled workspace artifacts, and sets up the environment to launch the demo. Note the example consists of several subtle optimizations:
+
+-	Multi-Stage Build
+	-	Dependency derivation, compilation, and runtime setup are partitioned
+	-	Maximizes cache retention despite package source or build/runtime changes
+	-	Greater concurrency, e.g., colcon build while runtime apt installs
+-	Persistent Cache Propagation
+	-	Use of [`--mount`](https://docs.docker.com/engine/reference/builder/#run---mount) to cache temp data without bloating layers
+	-	Maintain temporally consistent apt lists between parallel stages
+	-	Avoid needless network I/O between stages or across Docker rebuilds
+-	Minimal Image Size
+	-	Final stage builds from `ros-core` for smallest runtime image
+	-	Builds and installs only a select few packages in the workspace
+	-	Only workspace install artifacts are copied into final layers
 
-Note: `--from-paths` and `--packages-select` are set here as so to only install the dependencies and build for the demo C++ and Python packages, among many in the demo git repo that was cloned. To install the dependencies and build all the packages in the source workspace, merely change the scope by setting `--from-paths src/` and dropping the `--packages-select` arguments.
+For comparison, the resulting `runner` image is similar in size to the earlier `aptgetter` example. This allows you to develop and distribute custom ROS packages without significantly increasing image size compared to pre-built Debian installations:
 
-For more advance examples such as daisy chaining multiple overlay workspaces to improve caching of docker image build layers, using tools such as ccache to accelerate compilation with colcon, or using buildkit to save build time and bandwidth even when dependencies change, the project `Dockerfile`s in the ROS 2 [Navigation2](https://github.com/ros-planning/navigation2) repo are excellent resources.
+```console
+$ docker image ls my/ros --format "table {{.Tag}}\t{{.Size}}"
+TAG                SIZE
+aptgetter          504MB
+runner             510MB
+builder            941MB
+$ docker image ls ros --format "table {{.Tag}}\t{{.Size}}"
+TAG                SIZE
+rolling-ros-core   489MB
+rolling            876MB
+```
+
+For more advance examples such as daisy chaining multiple overlay workspaces to improve caching of docker image build layers, using tools such as ccache to accelerate compilation with colcon, or using buildkit to save build time and bandwidth even when dependencies change, the project `Dockerfile`s in the [Navigation2](https://github.com/ros-planning/navigation2) repo are excellent resources.
 
 ## Deployment use cases
 
@@ -119,19 +170,18 @@ Developing such complex systems with cutting edge implementations of newly publi
 
 With the advancements and standardization of software containers, roboticists are primed to acquire a host of improved developer tooling for building and shipping software. To help alleviate the growing pains and technical challenges of adopting new practices, we have focused on providing an official resource for using ROS with these new technologies.
 
-For a complete listing of supported architectures and base images for each ROS Distribution Release, please read the official REP on target platforms for either [ROS 1](https://www.ros.org/reps/rep-0003.html) or for [ROS 2](https://www.ros.org/reps/rep-2000.html).
+For a complete listing of supported architectures and base images for each ROS Distribution Release, please read the official REP on target platforms [here](https://www.ros.org/reps/rep-2001.html).
 
 ## Deployment suggestions
 
 The available tags include supported distros along with a hierarchy tags based off the most common meta-package dependencies, designed to have a small footprint and simple configuration:
 
 -	`ros-core`: minimal ROS install
 -	`ros-base`: basic tools and libraries (also tagged with distro name with LTS version as `latest`)
--	`ros1-bridge`: tools and libraries to run hybrid ROS 1 - ROS 2 systems and bridge messages between them
 
 In the interest of keeping `ros-core` tag minimal in image size, developer tools such as `rosdep`, `colcon` and `vcstools` are not shipped in `ros_core`, but in `ros-base` instead.
 
-The rest of the common meta-packages such as `desktop` are hosted on repos under OSRF's Docker Hub profile [here](https://hub.docker.com/r/osrf/ros/). These meta-packages include graphical dependencies and hook a host of other large packages such as X11, X server, etc. So in the interest of keeping the official images lean and secure, the desktop packages are just being hosted with OSRF's profile. For an extensive list of available variants, please read the official REP on target platforms for either [ROS 1](https://ros.org/reps/rep-0150.html) or for [ROS 2](https://www.ros.org/reps/rep-2001.html).
+The rest of the common meta-packages such as `desktop` are hosted on repos under OSRF's Docker Hub profile [here](https://hub.docker.com/r/osrf/ros/). These meta-packages include graphical dependencies and hook a host of other large packages such as X11, X server, etc. So in the interest of keeping the official images lean and secure, the desktop packages are just being hosted with OSRF's profile.
 
 ### Volumes
 
@@ -201,59 +251,10 @@ $ docker compose rm
 
 > Note: the auto-generated network, `ros_demos_default`, will persist until you explicitly remove it using `docker compose down`.
 
-### ROS 1 Bridge
-
-To ease ROS 2 migration, [`ros1_bridge`](https://index.ros.org/p/ros1_bridge) is a ROS 2 package that provides bidirectional communication between ROS 1 and ROS 2. As a minimal example, given the ROS 2 Dockerfile above, we'll create the ROS 1 equivalent below, and name the Dockerfile appropriately.
-
-```dockerfile
-FROM %%IMAGE%%:noetic
-
-# install ros package
-RUN apt-get update && apt-get install -y \
-      ros-${ROS_DISTRO}-ros-tutorials \
-      ros-${ROS_DISTRO}-common-tutorials && \
-    rm -rf /var/lib/apt/lists/*
-
-# launch ros package
-CMD ["roslaunch", "roscpp_tutorials", "talker_listener_launch"]
-```
-
-The compose file bellow spawns services for both talker listener demos while connecting the two via a dynamic bridge. You may then view the log output from both pairs of talker and listener nodes cross talking over the `/chatter` topic.
-
-```yaml
-services:
-  ros1:
-    build:
-      context: ./
-      dockerfile: ros1.Dockerfile
-
-  ros2:
-    build:
-      context: ./
-      dockerfile: ros2.Dockerfile
-
-  bridge:
-    image: ros:foxy-ros1-bridge
-    environment:
-      - "ROS_HOSTNAME=bridge"
-      - "ROS_MASTER_URI=http://ros1:11311"
-    command: ros2 run ros1_bridge dynamic_bridge
-```
-
 # More Resources
 
-[ROS.org](http://www.ros.org/): Main ROS website  
-[Q&A](https://answers.ros.org/questions/): Ask questions. Get answers  
+[Docs](https://docs.ros.org/): ROS Developer Documentation  
+[Q&A](https://robotics.stackexchange.com/): Ask questions. Get answers  
 [Forums](https://discourse.ros.org/): Hear the latest discussions  
-[Blog](http://www.ros.org/news/): Stay up-to-date  
 [Packages](https://index.ros.org/?search_packages=true): Discover indexed packages  
-[OSRF](https://www.osrfoundation.org/): Open Source Robotics Foundation
-
-## ROS 2
-
-[Index](https://docs.ros.org): ROS 2 Documentation  
-[Design](https://design.ros2.org/): ROS 2 Design Articles
-
-## ROS 1
-
-[Wiki](http://wiki.ros.org/Documentation): ROS 1 Documentation
+[OSRF](https://www.openrobotics.org/): Open Source Robotics Foundation

ros/metadata.json

@@ -1,6 +1,8 @@
 {
   "hub": {
     "categories": [
+      "languages-and-frameworks",
+      "machine-learning-and-ai",
       "operating-systems"
     ]
   }