Development workflows

[stonier]

Support simpler and more flexible development workflows.

• Work on an arbitrary subset of our repos - e.g. delphyne/delphyne_gui only → simpler & flexibly handle different use cases
• Dependency management via one tool - simpler & shouldn't need to reinvent wheels (use or drive requirements for rosdep)
• Minimise non-system dependencies - too much endeavour to maintain
• Minimise bundling of packages - e.g. tinyxml \in odm \in malidrive → enable dependency sharing & avoid release incompatibilities

These are essentially the precepts of the ROS federated ecosystem. It's a set of rules that works and enables scaling and flexibility. Just a bit trickier to handle with closed source packages sans a build farm, but it's doable. I believe it is a good goal.

FYI @scpeters just to kick-start the thread.

[agalbachicar]

Prereqs installed packages to construct the workspace

• The workspace installs via prereqs:
  • bash-completion
  • build-essential
  • curl
  • gdb
  • git
  • mercurial
  • python3
  • python3-setuptools
  • python3-vcstool
  • python3-rosdep
  • python3-colcon-common-extensions
  • ros-dashing-ament-cmake
  • clan8 and its family of utilitlies.
  • tmux

Prereqs installed packages to use maliput

• libpython3-dev

Prereqs installed packages to use delphyne

• libignition-common3-dev
• libignition-math6-dev
• libignition-msgs2-dev
• libignition-transport5-dev

Prereqs installed packages to use delphyne-gui

• libignition-common3-dev
• libignition-math6-dev
• libignition-msgs2-dev
• libignition-tools-dev
• libignition-cmake2-dev
• libignition-cmake1-dev
• libignition-rendering2-dev
• libignition-gui0-dev

Prereqs installed packages to use drake-vendor

• Pulls drake and installs its own prereqs list.

Extra packages in the workspace needed but not maintained

• pybind11: drake's fork.
• proj4: used in malidrive to parse xord origin
• ament_cmake_doxygen: required to execute doxygen documentation

[agalbachicar]

Per meeting on 10/28/2020 we will try to get rid of proj4 by implementing the functionality in maliput (math).

[agalbachicar]

pybind11 provides support to all python bindings in maliput and in delphyne. We are using that version because of compatibility with drake.

Related issues:

• Unfreeze pybind11 #109 --> dropped the custom pybind11 version
• Pass-by-value vs Pass-by-reference/pointer maliput#101 --> explains problems with Boost Python and unique_ptrs
• [Infra] pybind11 library version maliput#284 --> in particular, see this comment that explains why we have a custom pybind11 version.

[agalbachicar]

About tinyxml in ODRM, here are some @stonier 's notes from the CMakeLists.txt:

#
# OpenDrive comes with a very old version of tinyxml.
#
# - OpenDrive Version: 2.4.3
# - Xenial Version: 2.6.2
# - Used in: OdrReaderXML.cc
#
# *Status*: The internal version has been reproduced faithfully here.
# *Caveat*: That will have conflicts if someone's application tries to
# link against this opendrive and a system version of tinyxml.
#
# ALTERNATIVE I: Since it's only used in opendrive sources,
# build it internally, hide visibility. The advantage of this is that the
# opendrive sources themselves require no modification.
#
# ALTERNATIVE II: Use the system version, but you'll need a small script
# to make modifications to OdrReaderXML.cc (mostly prefixing the various
# enums with TINYXML_).
#

[agalbachicar]

We also have libgtest code in malidrive, delphyne and delphyne-gui. We should be better off removing it and using system dependency as in maliput.

[stonier]

Great, I think you're moving in the right direction re minimising dependencies:

• TinyXML: is this a no-op if we drop ODRM?
• Libgtest: concur, system dependency
• Proj: concur, eliminate by adding functionality to maliput-math(?)
• Pybind11: I wonder how much we need the system specific bindings that need the unique ptr functionality? e.g. perhaps we can just expose our own c++ api (sans unique pointer requirements) to be python bound (either with a non-custom pybind11 or boost's python) -- Plugin architecture for maliput maliput#373
• Ignition dependencies: I haven't dug in to this one yet, but wondering what is needed here to get it to properly work with package.xml/rosdep -- Migrate to newer version of ignition-gui delphyne_gui#332

Aside from minimising dependencies, other objectives I'd like to hit here (please comment if you think they make sense or not):

• Eliminate use of prereqs-install. It's awkwardly competing with package.xml / rosdep for dependency management). One tool for one job.
• Binary underlay (like /opt/ros/eloquent) that serves our own packaging. This enables flexible creation of workspaces.
• Solve any residual pure cmake / python problems in the workspace for dependency management (e.g. PROJ is entirely broken today). It should hopefully be entirely resolved by other items above.

[agalbachicar]

Re proj4 --> maliput/maliput#356

[agalbachicar]

TinyXML: is this a no-op if we drop ODRM?

The new implementation of malidrive relies on TinyXML2 from system packages (installed via rosdep) for xodr map parsing.

Pybind11: I wonder how much we need the system specific bindings that need the unique ptr functionality? e.g. perhaps we can just expose our own c++ api (sans unique pointer requirements) to be python bound (either with a non-custom pybind11 or boost's python)

I believe we can proxy the access to unique_ptrs. I also know @andrewbest-tri and @liangfok wanted to include boost-python bindings. Would that make sense in favor of deprecating pybind11? (I still need to analyze the impact on delphyne and delphyne-gui)

Eliminate use of prereqs-install. It's awkwardly competing with package.xml / rosdep for dependency management). One tool for one job.

Makes sense to me.

Binary underlay (like /opt/ros/eloquent) that serves our own packaging. This enables flexible creation of workspaces.

We are using dashing now. Is it a requisite or can we push this migration to a separate effort? I haven't checked how much it would take to migrate though, note that combinatoric escalation is a problem for CI infrastructure maintenance and budget.

Solve any residual pure cmake / python problems in the workspace for dependency management (e.g. PROJ is entirely broken today). It should hopefully be entirely resolved by other items above.

Understood.

[scpeters]

Re proj4 --> ToyotaResearchInstitute/maliput#356

I didn't mention this during the meeting, but it looks like libproj-dev is in Ubuntu, though the version in 18.04 is only 4.9.3, though the version in 20.04 is 6.3.1

I assume at some point we will move to 20.04; perhaps we could at least test with 6.3.1 to see if it is suitable? it could save us from maintaining extra code in maliput

[agalbachicar]

All custom gtest and gmock versions were removed from the workspace.

[agalbachicar]

Dumping a few decisions taken in weekly meetings:

Pybind11: I wonder how much we need the system specific bindings that need the unique ptr functionality? e.g. perhaps we can just expose our own c++ api (sans unique pointer requirements) to be python bound (either with a non-custom pybind11 or boost's python)

For maliput, we will split maliput bindings into a separate package and from there develop a plugin architecture. Backends will require a new load interface so they can be seamlessly loaded in python or any other language (not C++).

For delphyne, we still require pybind due to drake's interaction. We could move all the bindings to another package (potential increased and unnecessary complexity) and release from pybind11 dependency delphyne. It also brings another question onto the table, is it necessary to have delphyne on its own? Do we foresee or is it there already another package that uses delphyne but not delphyne-gui / delphyne-demos?

Ignition dependencies: I haven't dug in to this one yet, but wondering what is needed here to get it to properly work with package.xml/rosdep

Started a migration of these libraries to newer versions.