Skip to content

heap-use-after-free thrown in tests #1588

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
fmauch opened this issue Mar 12, 2025 · 6 comments
Closed

heap-use-after-free thrown in tests #1588

fmauch opened this issue Mar 12, 2025 · 6 comments
Labels

Comments

@fmauch
Copy link
Contributor

fmauch commented Mar 12, 2025

Describe the bug
Running the JTC tests with ASAN enabled raises a heap-use-after-free error. I stumbled upon this while writing tests for #1191 where I get crashes on some configurations. Tracing things down I realized that there are memory issues reported also on the master branch and I would like to get them fixed, first.

=================================================================
==197664==ERROR: AddressSanitizer: heap-use-after-free on address 0x5160000bbed8 at pc 0x79ea5f6fb42e bp 0x7ffc58fb3210 sp 0x7ffc58fb29b8
READ of size 15 at 0x5160000bbed8 thread T0
    #0 0x79ea5f6fb42d in memcpy ../../../../src/libsanitizer/sanitizer_common/sanitizer_common_interceptors_memintrinsics.inc:115
    #1 0x79ea5eef3a85  (/opt/ros/jazzy/lib/librclcpp.so+0xf3a85) (BuildId: a6e4def86fc9f569833617ff077616d6d3344142)
    #2 0x79ea5ef2c0be in rclcpp::get_logger(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) (/opt/ros/jazzy/lib/librclcpp.so+0x12c0be) (BuildId: a6e4def86fc9f569833617ff077616d6d3344142)
    #3 0x5965fd023844 in hardware_interface::LoanedStateInterface::~LoanedStateInterface() /home/feex/checkout/ros2_control_dev/colcon_ws/src/ros2_control/hardware_interface/include/hardware_interface/loaned_state_interface.hpp:61
    #4 0x79ea5f5c2713 in void std::_Destroy<hardware_interface::LoanedStateInterface>(hardware_interface::LoanedStateInterface*) /usr/include/c++/13/bits/stl_construct.h:151
    #5 0x79ea5f5c2713 in void std::_Destroy_aux<false>::__destroy<hardware_interface::LoanedStateInterface*>(hardware_interface::LoanedStateInterface*, hardware_interface::LoanedStateInterface*) /usr/include/c++/13/bits/stl_construct.h:163
    #6 0x79ea5f5c2713 in void std::_Destroy<hardware_interface::LoanedStateInterface*>(hardware_interface::LoanedStateInterface*, hardware_interface::LoanedStateInterface*) /usr/include/c++/13/bits/stl_construct.h:196
    #7 0x79ea5f5c2713 in void std::_Destroy<hardware_interface::LoanedStateInterface*, hardware_interface::LoanedStateInterface>(hardware_interface::LoanedStateInterface*, hardware_interface::LoanedStateInterface*, std::allocator<hardware_interface::LoanedStateInterface>&) /usr/include/c++/13/bits/alloc_traits.h:948
    #8 0x79ea5f5c2713 in std::vector<hardware_interface::LoanedStateInterface, std::allocator<hardware_interface::LoanedStateInterface> >::~vector() /usr/include/c++/13/bits/stl_vector.h:735
    #9 0x79ea5f5c2713 in controller_interface::ControllerInterfaceBase::~ControllerInterfaceBase() /home/feex/checkout/ros2_control_dev/colcon_ws/src/ros2_control/controller_interface/src/controller_interface_base.cpp:40
    #10 0x5965fd0e94b5 in controller_interface::ControllerInterface::~ControllerInterface() /home/feex/checkout/ros2_control_dev/colcon_ws/src/ros2_control/controller_interface/include/controller_interface/controller_interface.hpp:31
    #11 0x5965fd0e94b5 in joint_trajectory_controller::JointTrajectoryController::~JointTrajectoryController() /home/feex/checkout/ros2_control_dev/colcon_ws/src/ros2_controllers/joint_trajectory_controller/include/joint_trajectory_controller/joint_trajectory_controller.hpp:52
    #12 0x5965fd01d845 in std::_Sp_counted_base<(__gnu_cxx::_Lock_policy)2>::_M_release() /usr/include/c++/13/bits/shared_ptr_base.h:346
    #13 0x5965fd01d845 in std::_Sp_counted_base<(__gnu_cxx::_Lock_policy)2>::_M_release() /usr/include/c++/13/bits/shared_ptr_base.h:317
    #14 0x5965fd01e1f4 in std::__shared_count<(__gnu_cxx::_Lock_policy)2>::~__shared_count() /usr/include/c++/13/bits/shared_ptr_base.h:1071
    #15 0x5965fd01e1f4 in std::__shared_ptr<test_trajectory_controllers::TestableJointTrajectoryController, (__gnu_cxx::_Lock_policy)2>::~__shared_ptr() /usr/include/c++/13/bits/shared_ptr_base.h:1524
    #16 0x5965fd01e1f4 in std::shared_ptr<test_trajectory_controllers::TestableJointTrajectoryController>::~shared_ptr() /usr/include/c++/13/bits/shared_ptr.h:175
    #17 0x5965fd01e1f4 in test_trajectory_controllers::TrajectoryControllerTest::~TrajectoryControllerTest() /home/feex/checkout/ros2_control_dev/colcon_ws/src/ros2_controllers/joint_trajectory_controller/test/test_trajectory_controller_utils.hpp:227
    #18 0x5965fd0f8e59 in TestTrajectoryActions::~TestTrajectoryActions() /home/feex/checkout/ros2_control_dev/colcon_ws/src/ros2_controllers/joint_trajectory_controller/test/test_trajectory_actions.cpp:50
    #19 0x5965fd0f8e59 in TestTrajectoryActions_test_goal_tolerances_single_point_success_Test::~TestTrajectoryActions_test_goal_tolerances_single_point_success_Test() /home/feex/checkout/ros2_control_dev/colcon_ws/src/ros2_controllers/joint_trajectory_controller/test/test_trajectory_actions.cpp:395
    #20 0x5965fd0f8e59 in TestTrajectoryActions_test_goal_tolerances_single_point_success_Test::~TestTrajectoryActions_test_goal_tolerances_single_point_success_Test() /home/feex/checkout/ros2_control_dev/colcon_ws/src/ros2_controllers/joint_trajectory_controller/test/test_trajectory_actions.cpp:395
    #21 0x5965fd1a8e8a in void testing::internal::HandleSehExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) /opt/ros/jazzy/src/gtest_vendor/src/gtest.cc:2612
    #22 0x5965fd1a8e8a in void testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) /opt/ros/jazzy/src/gtest_vendor/src/gtest.cc:2648
    #23 0x5965fd16c4e2 in testing::TestInfo::Run() /opt/ros/jazzy/src/gtest_vendor/src/gtest.cc:2842
    #24 0x5965fd16cf6e in testing::TestSuite::Run() /opt/ros/jazzy/src/gtest_vendor/src/gtest.cc:3015
    #25 0x5965fd1890f7 in testing::internal::UnitTestImpl::RunAllTests() /opt/ros/jazzy/src/gtest_vendor/src/gtest.cc:5920
    #26 0x5965fd1a9e3a in bool testing::internal::HandleSehExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*) /opt/ros/jazzy/src/gtest_vendor/src/gtest.cc:2612
    #27 0x5965fd1a9e3a in bool testing::internal::HandleExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*) /opt/ros/jazzy/src/gtest_vendor/src/gtest.cc:2648
    #28 0x5965fd16d7b8 in testing::UnitTest::Run() /opt/ros/jazzy/src/gtest_vendor/src/gtest.cc:5484
    #29 0x5965fcfbfe76 in RUN_ALL_TESTS() /opt/ros/jazzy/src/gtest_vendor/include/gtest/gtest.h:2317
    #30 0x5965fcfbfe76 in main /opt/ros/jazzy/src/gmock_vendor/src/gmock_main.cc:71
    #31 0x79ea5e62a1c9 in __libc_start_call_main ../sysdeps/nptl/libc_start_call_main.h:58
    #32 0x79ea5e62a28a in __libc_start_main_impl ../csu/libc-start.c:360
    #33 0x5965fcfc0d04 in _start (/home/feex/checkout/ros2_control_dev/colcon_ws/build/joint_trajectory_controller/test_trajectory_actions+0x9dd04) (BuildId: 8adb1213f87544afe13846fc3f8445af8a5677a3)

0x5160000bbed8 is located 88 bytes inside of 552-byte region [0x5160000bbe80,0x5160000bc0a8)
freed by thread T0 here:
    #0 0x79ea5f6ff5e8 in operator delete(void*, unsigned long) ../../../../src/libsanitizer/asan/asan_new_delete.cpp:164
    #1 0x5965fd01dbd9 in std::__new_allocator<hardware_interface::StateInterface>::deallocate(hardware_interface::StateInterface*, unsigned long) /usr/include/c++/13/bits/new_allocator.h:172
    #2 0x5965fd01dbd9 in std::allocator_traits<std::allocator<hardware_interface::StateInterface> >::deallocate(std::allocator<hardware_interface::StateInterface>&, hardware_interface::StateInterface*, unsigned long) /usr/include/c++/13/bits/alloc_traits.h:517
    #3 0x5965fd01dbd9 in std::_Vector_base<hardware_interface::StateInterface, std::allocator<hardware_interface::StateInterface> >::_M_deallocate(hardware_interface::StateInterface*, unsigned long) /usr/include/c++/13/bits/stl_vector.h:390
    #4 0x5965fd01dbd9 in std::_Vector_base<hardware_interface::StateInterface, std::allocator<hardware_interface::StateInterface> >::~_Vector_base() /usr/include/c++/13/bits/stl_vector.h:369
    #5 0x5965fd01dbd9 in std::vector<hardware_interface::StateInterface, std::allocator<hardware_interface::StateInterface> >::~vector() /usr/include/c++/13/bits/stl_vector.h:738
    #6 0x5965fd01dbd9 in test_trajectory_controllers::TrajectoryControllerTest::~TrajectoryControllerTest() /home/feex/checkout/ros2_control_dev/colcon_ws/src/ros2_controllers/joint_trajectory_controller/test/test_trajectory_controller_utils.hpp:227

previously allocated by thread T0 here:
    #0 0x79ea5f6fe548 in operator new(unsigned long) ../../../../src/libsanitizer/asan/asan_new_delete.cpp:95
    #1 0x5965fd048c36 in std::__new_allocator<hardware_interface::CommandInterface>::allocate(unsigned long, void const*) /usr/include/c++/13/bits/new_allocator.h:151
    #2 0x5965fd048c36 in std::allocator_traits<std::allocator<hardware_interface::CommandInterface> >::allocate(std::allocator<hardware_interface::CommandInterface>&, unsigned long) /usr/include/c++/13/bits/alloc_traits.h:482
    #3 0x5965fd048c36 in std::_Vector_base<hardware_interface::CommandInterface, std::allocator<hardware_interface::CommandInterface> >::_M_allocate(unsigned long) /usr/include/c++/13/bits/stl_vector.h:381
    #4 0x5965fd048c36 in std::_Vector_base<hardware_interface::StateInterface, std::allocator<hardware_interface::StateInterface> >::_M_allocate(unsigned long) /usr/include/c++/13/bits/stl_vector.h:378
    #5 0x5965fd048c36 in std::vector<hardware_interface::StateInterface, std::allocator<hardware_interface::StateInterface> >::reserve(unsigned long) /usr/include/c++/13/bits/vector.tcc:79

SUMMARY: AddressSanitizer: heap-use-after-free ../../../../src/libsanitizer/sanitizer_common/sanitizer_common_interceptors_memintrinsics.inc:115 in memcpy
Shadow bytes around the buggy address:
  0x5160000bbc00: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
  0x5160000bbc80: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
  0x5160000bbd00: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
  0x5160000bbd80: fd fd fd fd fd fa fa fa fa fa fa fa fa fa fa fa
  0x5160000bbe00: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
=>0x5160000bbe80: fd fd fd fd fd fd fd fd fd fd fd[fd]fd fd fd fd
  0x5160000bbf00: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
  0x5160000bbf80: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
  0x5160000bc000: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
  0x5160000bc080: fd fd fd fd fd fa fa fa fa fa fa fa fa fa fa fa
  0x5160000bc100: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
Shadow byte legend (one shadow byte represents 8 application bytes):
  Addressable:           00
  Partially addressable: 01 02 03 04 05 06 07
  Heap left redzone:       fa
  Freed heap region:       fd
  Stack left redzone:      f1
  Stack mid redzone:       f2
  Stack right redzone:     f3
  Stack after return:      f5
  Stack use after scope:   f8
  Global redzone:          f9
  Global init order:       f6
  Poisoned by user:        f7
  Container overflow:      fc
  Array cookie:            ac
  Intra object redzone:    bb
  ASan internal:           fe
  Left alloca redzone:     ca
  Right alloca redzone:    cb
==197664==ABORTING

It seems that the destructor of a loaned interface is called while its memory is freed already.

To Reproduce
Steps to reproduce the behavior:

  1. Build ros2_control and ros2_controllers with asan enabled (e.g. by --mixin asan-gcc)
  2. Disable new-delete-type-mismatch checks: export ASAN_OPTIONS=new_delete_type_mismatch=0 (they are thrown from rclcpp and make using asan quite useless otherwise)
  3. Run the jtc action tests colcon test --event-handlers=console_direct+ --packages-select joint_trajectory_controller --ctest-args -R test_trajectory_actions4. See error

Expected behavior
No crashes ;-)

Screenshots
Running the test in a debugger (without asan) and watching the first line (auto logger = rclcpp::get_logger(state_interface_.get_name());) in the destructor yielded for example:

Image

Environment (please complete the following information):

@fmauch fmauch added the bug label Mar 12, 2025
@fmauch
Copy link
Contributor Author

fmauch commented Mar 18, 2025

I digged a little deeper and using some debug output it is pretty clear what's going on:

5: [INFO 2025-03-18 07:30:17.991] [LoanedCommandInterface]: LoanedCommandInterface joint1/position is being created with command interface at address 0x604db0051cd0
5: [INFO 2025-03-18 07:30:17.991] [LoanedCommandInterface]: LoanedCommandInterface joint1/velocity is being created with command interface at address 0x604db0168db0
5: [INFO 2025-03-18 07:30:17.991] [LoanedCommandInterface]: LoanedCommandInterface joint1/acceleration is being created with command interface at address 0x604db0168fe0
5: [INFO 2025-03-18 07:30:17.991] [LoanedCommandInterface]: LoanedCommandInterface joint1/effort is being created with command interface at address 0x604db0169210
5: [INFO 2025-03-18 07:30:18.292] [test_joint_trajectory_controller]: Received new action goal
5: [INFO 2025-03-18 07:30:18.292] [test_joint_trajectory_controller]: Accepted new action goal
5: [INFO 2025-03-18 07:30:18.583] [test_joint_trajectory_controller]: Goal reached, success!
5: [INFO 2025-03-18 07:30:19.994] [CommandInterface]: Destroying CommandInterface: joint1/effort with address 0x604db0169210
5: [INFO 2025-03-18 07:30:19.994] [CommandInterface]: Destroying CommandInterface: joint1/acceleration with address 0x604db0168fe0
5: [INFO 2025-03-18 07:30:19.994] [CommandInterface]: Destroying CommandInterface: joint1/velocity with address 0x604db0168db0
5: [INFO 2025-03-18 07:30:19.994] [CommandInterface]: Destroying CommandInterface: joint1/position with address 0x604db0051cd0
5: [INFO 2025-03-18 07:30:19.995] [test_joint_trajectory_controller]: Destroying controller 'test_joint_trajectory_controller'.
5: [INFO 2025-03-18 07:30:19.995] [test_joint_trajectory_controller]: Calling shutdown transition of controller node 'test_joint_trajectory_controller' due to destruction.
5: [INFO 2025-03-18 07:30:19.996] [LoanedCommandInterface]: LoanedCommandInterface joint1/position is being destroyed
5: [INFO 2025-03-18 07:30:19.996] [LoanedCommandInterface]: LoanedCommandInterface joint1/velocity is being destroyed
5: [INFO 2025-03-18 07:30:19.996] [LoanedCommandInterface]: LoanedCommandInterface ��8�K` is being destroyed

This could potentially be mitigated by making sure to delete the controller before the command interfaces get deleted in that particular case, but that would still leave the potential for this to happen rather large.

If the LoanedCommandInterface would store the CommandInterface as a shared pointer, it wouldn't run away from it, but I have the feeling that would jeopardize the whole concept of the loaned interface?

@mamueluth @saikishor as you have touched the LoanedCommandInterface recently and there has been added a preferred signature of passing a shared pointer to a command interface to the constructor: What do you think?

@saikishor
Copy link
Member

@fmauch I've already tried to do this change months back. Unfortunately, this is a breaking change as most of the tests use it. We will surely do it in rolling after we branch off. However, I can think of another easy fix for these situations

@saikishor
Copy link
Member

@fmauch Can you give it a shot with ros-controls/ros2_control#2112? This should most likely fix it

@fmauch
Copy link
Contributor Author

fmauch commented Mar 18, 2025

@fmauch Can you give it a shot with ros-controls/ros2_control#2112? This should most likely fix it

I'll give it a shot, yes!


Edit: Nope, that seems to be too late:

5: ==42551==ERROR: AddressSanitizer: heap-use-after-free on address 0x5160000ae3d8 at pc 0x787b060fb42e bp 0x7ffd94f8e490 sp 0x7ffd94f8dc38
5: READ of size 15 at 0x5160000ae3d8 thread T0
5:     #0 0x787b060fb42d in memcpy ../../../../src/libsanitizer/sanitizer_common/sanitizer_common_interceptors_memintrinsics.inc:115
5:     #1 0x787b054f3a85  (/opt/ros/jazzy/lib/librclcpp.so+0xf3a85) (BuildId: a6e4def86fc9f569833617ff077616d6d3344142)
5:     #2 0x787b0552c0be in rclcpp::get_logger(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) (/opt/ros/jazzy/lib/librclcpp.so+0x12c0be) (BuildId: a6e4def86fc9f569833617ff077616d6d3344142)
5:     #3 0x5c81d344d65c in hardware_interface::LoanedCommandInterface::~LoanedCommandInterface() (/home/feex/checkout/ros2_control_dev/build/joint_trajectory_controller/test_trajectory_actions+0x25c65c) (BuildId: ee5a54b07445a43220f1c1cd6fde1f15f64565c8)
5:     #4 0x5c81d35079c2 in void std::_Destroy<hardware_interface::LoanedCommandInterface>(hardware_interface::LoanedCommandInterface*) (/home/feex/checkout/ros2_control_dev/build/joint_trajectory_controller/test_trajectory_actions+0x3169c2) (BuildId: ee5a54b07445a43220f1c1cd6fde1f15f64565c8)
5:     #5 0x5c81d34f4429 in void std::_Destroy_aux<false>::__destroy<hardware_interface::LoanedCommandInterface*>(hardware_interface::LoanedCommandInterface*, hardware_interface::LoanedCommandInterface*) (/home/feex/checkout/ros2_control_dev/build/joint_trajectory_controller/test_trajectory_actions+0x303429) (BuildId: ee5a54b07445a43220f1c1cd6fde1f15f64565c8)
5:     #6 0x5c81d34dc408 in void std::_Destroy<hardware_interface::LoanedCommandInterface*>(hardware_interface::LoanedCommandInterface*, hardware_interface::LoanedCommandInterface*) (/home/feex/checkout/ros2_control_dev/build/joint_trajectory_controller/test_trajectory_actions+0x2eb408) (BuildId: ee5a54b07445a43220f1c1cd6fde1f15f64565c8)
5:     #7 0x787b05fafe5e in std::vector<hardware_interface::LoanedCommandInterface, std::allocator<hardware_interface::LoanedCommandInterface> >::_M_erase_at_end(hardware_interface::LoanedCommandInterface*) (/home/feex/checkout/ros2_control_dev/install/controller_interface/lib/libcontroller_interface.so+0x80e5e) (BuildId: 09b3f9f83e1e68650916b9d2fc532e8fe2d5c1e1)
5:     #8 0x787b05fab36c in std::vector<hardware_interface::LoanedCommandInterface, std::allocator<hardware_interface::LoanedCommandInterface> >::clear() (/home/feex/checkout/ros2_control_dev/install/controller_interface/lib/libcontroller_interface.so+0x7c36c) (BuildId: 09b3f9f83e1e68650916b9d2fc532e8fe2d5c1e1)
5:     #9 0x787b05f9fcc7 in controller_interface::ControllerInterfaceBase::release_interfaces() (/home/feex/checkout/ros2_control_dev/install/controller_interface/lib/libcontroller_interface.so+0x70cc7) (BuildId: 09b3f9f83e1e68650916b9d2fc532e8fe2d5c1e1)
5:     #10 0x787b05f98883 in controller_interface::ControllerInterfaceBase::~ControllerInterfaceBase() (/home/feex/checkout/ros2_control_dev/install/controller_interface/lib/libcontroller_interface.so+0x69883) (BuildId: 09b3f9f83e1e68650916b9d2fc532e8fe2d5c1e1)
5:     #11 0x5c81d3522c7c in controller_interface::ControllerInterface::~ControllerInterface() (/home/feex/checkout/ros2_control_dev/build/joint_trajectory_controller/test_trajectory_actions+0x331c7c) (BuildId: ee5a54b07445a43220f1c1cd6fde1f15f64565c8)
5:     #12 0x5c81d3523080 in joint_trajectory_controller::JointTrajectoryController::~JointTrajectoryController() (/home/feex/checkout/ros2_control_dev/build/joint_trajectory_controller/test_trajectory_actions+0x332080) (BuildId: ee5a54b07445a43220f1c1cd6fde1f15f64565c8)
5:     #13 0x5c81d3558b08 in test_trajectory_controllers::TestableJointTrajectoryController::~TestableJointTrajectoryController() (/home/feex/checkout/ros2_control_dev/build/joint_trajectory_controller/test_trajectory_actions+0x367b08) (BuildId: ee5a54b07445a43220f1c1cd6fde1f15f64565c8)
5:     #14 0x5c81d356c0b9 in void std::_Destroy<test_trajectory_controllers::TestableJointTrajectoryController>(test_trajectory_controllers::TestableJointTrajectoryController*) (/home/feex/checkout/ros2_control_dev/build/joint_trajectory_controller/test_trajectory_actions+0x37b0b9) (BuildId: ee5a54b07445a43220f1c1cd6fde1f15f64565c8)
5:     #15 0x5c81d355de5b in std::_Sp_counted_ptr_inplace<test_trajectory_controllers::TestableJointTrajectoryController, std::allocator<void>, (__gnu_cxx::_Lock_policy)2>::_M_dispose() (/home/feex/checkout/ros2_control_dev/build/joint_trajectory_controller/test_trajectory_actions+0x36ce5b) (BuildId: ee5a54b07445a43220f1c1cd6fde1f15f64565c8)

A workaround would probably have to be done in the test itself.

@saikishor
Copy link
Member

A workaround would probably have to be done in the test itself.

Most likely yes :(

@christophfroehlich
Copy link
Contributor

JTC got fixed with #1596, follow-up issue for the other controllers is pending

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging a pull request may close this issue.

3 participants