Questions

[rhaschke]

Thanks a lot for providing this software. I'm feeling much more comfortable with the Python code than with the Lisp code.
The last days I worked myself through the docs and I've got a couple of open questions, fundamental and concrete ones:

1. What is the goal/roadmap for PyCRAM? Is it intended as a replacement for CRAM?
   How feature-complete is it compared to CRAM?
   Particularly, I was missing the connection to RoboKudo (How is the BulletWorld populated from perception?) and KnowRob's logical reasoning. Other KnowRob features, like the inner world model and OWL, are directly implemented in PyCRAM.

2. I'm missing the internal simulation of a whole plan. All given examples directly perform a plan's individual actions. Is there a way to simulate a plan before execution? Should one perform the plan first in a prospection world and then on the real robot?

3. CRAM seems to operate on a symbolic level only: plans just consider key frames and existing motion designators just teleport the robot to the desired configuration. Do you also consider trajectory planning, i.e. generating a collision-free trajectory? IMO, this should be part of a planning framework.

4. The difference between Motion and Action Designators became not very clear.
   Sure, motion designators shall represent atomic low-level motions, while the latter are more abstract.
   Considering navigation for example, there exist NavigateAction and MoveMotion, which seem to perform exactly the same task: navigating the robot to a target location. The only difference is that MoveMotion takes a single target, while NavigateAction takes a list, but only the first element is used:
   

   pycram/src/pycram/designators/action_designator.py

   Line 305 in f73867a

   return NavigateActionPerformable(self.target_locations[0])

   
   So, seemingly both designators serve exactly the same purpose.

   More generally, all action designators seem to return just the first element of their targets list.
   Is this intended? If so, what is the purpose of having a list?

5. Other designators also just resolve to the first item:
   

   pycram/src/pycram/designator.py

   Line 679 in f73867a

   return next(iter(self))

   How/Is it possible to iterate through all possible plan resolutions (and verify them via internal simulation) instead of greedily choosing the first parameter always?

6. The difference between Reachable and Accessing Location designators didn't become clear as well.
   What is their semantic difference? Both seem to evaluate the reachability of an object.

7. What is meant by ExecutionType SEMI_REAL?

8. I noticed regular crashes of the python kernel when running a jupyter notebook with a world running in GUI mode.
   These crashes didn't occur on particular actions, but during idle time. Hence, I assume that this is related to the visualization. Is that a known problem? Do you suggest using rviz visualization via VizMarkerPublisher instead?

[Tigul]

Hi, thanks for your interest and the questions

1. Yes, PyCRAM is intended to be a replacement for CRAM and, at least in our group, mostly does already.
   Giving an exact estimate on how feature-complete PyCRAM is compared to CRAM is difficult but I would say PyCRAM implements all core-feautures and most of the features present in CRAM. However, it is currently lacking in regards to handling failures and reacting to them as well as reasoning of Designators.
   Reasoning is currently in development while failure handling is planned for the end of the year.
   There already is a rough implementation of failure handling (https://github.com/cram2/pycram/blob/dev/src/pycram/failure_handling.py)
   The connection to RoboKudo is implemented in the DetecingAction which also populates the BeliefState with the results from perception.

2. Yes, where a designator is executed is determined by the environment ("with simulated_robot" or "with real_robot"). The prospection world is intented as a kind of sandbox where reasoning and prospection can be done.
   So you can nest a with Use_prospection_world and a with simulated_robot which will execute the plan on the simulated robot in the prospection world.

3. We don't consider the generation of trajectories directly. However, we can use Giskard as a full-body ik solver during generation of robot base pose which is able to generate collision-free trajectories. So we don't generate trajectories directly but delegate this task to Giskard.

4. Action designators represent high-level concepts which needs more sophisticated steps and can consist of other action, location and object designators. Motion designator are low-level motions that directly translate to motions the robot can execute.
   NavigateAction might not be a good example for this distinction since Navigate is indeed just an alias for MoveMotion. Looking at the PickUpAction (https://github.com/cram2/pycram/blob/dev/src/pycram/designators/action_designator.py#L689) might be better.

5. The general idea about designators is that there is a DesignatorDescription which symbolically describes a set of Designator that fit the description. The list input of a DesignatorDescription is meant to provide possible solutions that can be used. During the resolution of a DesignatorDescription PyCRAM reasons about it and returns a Designator which can be performed with specific parameters.
   The implementation where the first element of the list is used is just a placeholder while the reasoning and knowledge part is developed.

6. Reachable generates poses from where a Pose or an Object is reachable, therefore it tries to find a pose form where the robot can reach the given pose (or object) as well as a pre-grasp pose.
   Accessing Location generates poses to open drawers, it tries to find a pose from which the robot can reach the handle of the drawer when it is closed as well when it is opened.
   So both check some form of reachablility but with different constrains.

7. SEMI_REAL is a special case for the HSR and is intended for using Giskard in simulation. For example, if you are not working on the real robot but want to test how a real robot would behave. This mode is basically the same as REAL_ROBOT but emulates a text-to-speech engine and the does not use MoveBase but instead teleports the robot in Giskard. (https://github.com/cram2/pycram/blob/dev/src/pycram/process_modules/hsrb_process_modules.py#L158)

8. If you are referring to the Segmentation Faults then yes this is a know issue. This seems to be the combination of the IPython kernel and the OpenGL implementation in PyBullet.
   But these problems stopped occurring for me roughly a year ago, I guess an update of the IPython core packages fixed this (I have version 8.12.0 of ipython)
   Otherwise running the Simulation in DIRECT mode and using the VizMarkerPublisher is also a valid solution which should fix this and is also using less resources.

I hope this answers some of your questions, if something is still unclear please don't hesitate to ask :)

[rhaschke]

Thanks for your answers so far. I have a few more questions:

3. We don't generate trajectories directly but delegate this task to Giskard.

As Giskard is a local (gradient-based) planner, it may not be able to find a collision-free trajectory.
How do you handle this case?

5. The implementation where the first element of the list is used is just a placeholder while the reasoning and knowledge part is developed.

So these designators are not yet fully implemented?
One central question of mine was: How CRAM decides between multiple solution options?
Can you assign a cost function or (success) probability distribution, which is then maximized?
Can you consider interdependent subtasks and pick only solution combinations that fulfill the whole task?

8. If you are referring to the Segmentation Faults then yes this is a know issue.
   But these problems stopped occurring for me roughly a year ago, I guess an update of the IPython core packages fixed this (I have version 8.12.0 of ipython)

I'm using ipython 8.26.0 and python 3.10.12 and the problem definitely exists (again).

[Tigul]

As Giskard is a local (gradient-based) planner, it may not be able to find a collision-free trajectory.
How do you handle this case?

In the case that the trajectory is not collision-free we sample another seed state for the robot and try again until a limit is reached in which case the goal is deemed unreachable.

So these designators are not yet fully implemented?
One central question of mine was: How CRAM decides between multiple solution options?
Can you assign a cost function or (success) probability distribution, which is then maximized?
Can you consider interdependent subtasks and pick only solution combinations that fulfill the whole task?

Yes, they are not completely implemented yet. However, I'm positive that that will change soon.
In PyCRAM the designator itself decides how the best solution is found, since reasoning is directly tied to the designator. For a more concrete example how this would look like, please see this example: https://github.com/cram2/pycram/blob/dev/examples/improving_actions.md
So you can create a specialized version of a designator which does its own kind of choosing a solution between multiple solutions.

I'm using ipython 8.26.0 and python 3.10.12 and the problem definitely exists (again).

Then this is definitely a problem, I would suggest for the moment you use the viz_marker_publisher in combination with RViz. I'll ask around and try to pinpoint the combination of packages and versions that seems to fix this issue.
Unfortunately, debugging this problem is near impossible for me :(

[rhaschke]

In the case that the trajectory is not collision-free we sample another seed state for the robot and try again

Doesn't starting from another seed state invalidate the trajectory for execution? How do you connect the current robot state to the new seed state?

[Tigul]

The new seed state is for calculating the specific trajectory, therefore the rest of the plan is unaffected by that.
The current robot state is usually connected to the found pose by navigating it there.