RSDK-5986: Resource-level logging for the C++ SDK #383

lia-viam · 2025-03-14T16:18:30Z

This PR introduces resource-level logging to the C++ SDK. Building off the work in #374, the Instance class now initializes the logging machinery and handles bookkeeping for resource-level logging.

We are still using Boost.Log for the logging but there is a pretty significant overhaul which makes use of some more advanced features of the library. The distinction between Boost.Log's concept of source and sink is helpful. There is a single logging source in the SDK, namely the source of message generated by user code (or the SDK) using the VIAM_LOG or VIAM_RESOURCE_LOG macros. Precisely one of two sinks may be active.

Console logging, initialized by default on application startup
RDK logging via RobotClient.
Starting up a ModuleService will automatically switch from 1. to 2.

Currently there is a lot of header/ABI leakage of Boost.Log components. Unfortunately this may be nontrivial to work around, given that Boost.Log uses ostream-style streaming of log messages, but I do think it is doable. That said, we currently have other Boost components in headers/APIs, so I think this is lower priority for now.

Possible bikeshed: the class currently called Logger might more accurately be called something like LogManager, given that it manages the SDK's log source and handles setup of the logging core and the console sink.

The unit tests now feature tests of console logging at the SDK level and the resource level, with tests of global filtering and resource-level filtering as well. Note that these are only console logging tests, but reviewers please advise if it may be possible to connect a RobotClient to MockRobot or similar in the tests, and override the gRPC Log method so we can inspect logs getting sent via the non-console log sink.

On recommendation from @stuqdog here are some sample log outputs:

Client-side logging, general SDK:

2025--03--17 14:27:57: [Viam C++ SDK] <info> [tests/test_log.cpp:35] info1
2025--03--17 14:27:57: [Viam C++ SDK] <error> [tests/test_log.cpp:36] error1
2025--03--17 14:27:57: [Viam C++ SDK] <trace> [tests/test_log.cpp:45] trace2

Client-side, resource-level:

2025--03--17 14:27:57: [DefaultSensor] <info> [tests/test_log.cpp:76] sensor info
2025--03--17 14:27:57: [DefaultSensor] <error> [tests/test_log.cpp:77] sensor error
2025--03--17 14:27:57: [ErrorSensor] <error> [tests/test_log.cpp:77] sensor error

Sample RDK output:

2025-03-17T18:42:21.057Z	INFO	rdk.modmanager.my-module.my-module.StdOut	pexec/managed_process.go:294	
\_ 2025--03--17 14:42:21: [Viam C++ SDK] <info> [log/logger.cpp:134] Initialized console logging
2025-03-17T18:42:21.059Z	INFO	rdk.modmanager.my-module.my-module.StdOut	pexec/managed_process.go:294	
\_ 2025--03--17 14:42:21: [Viam C++ SDK] <info> [module/service.cpp:275] Module listening on /var/folders/ <snip>

...

2025-03-17T18:42:21.071Z	INFO	rdk.modmanager.my-module.my-module.StdOut	pexec/managed_process.go:294	
\_ 2025--03--17 14:42:21: [Viam C++ SDK] <info> [log/logger.cpp:138] Disabling console logging

...

2025-03-17T18:42:21.073Z	INFO	mysensor	######### CONFIGURING MY SENSOR ########	{"log_ts":"2025-03-17T14:42:21.073Z"}

edit: fixed a bug where RDK was not logging file/line, now looks like

2025-04-16T15:50:03.183Z	INFO	mysensor	[simple/main.cpp:71] Command entry hello	{"log_ts":"2025-04-16T11:50:03.182Z"}

…stry-member

Co-authored-by: Ethan <[email protected]>

benjirewis

Looks awesome! Any plans on adding to the simple or complex module examples to show how to use the new VIAM_LOG?

benjirewis · 2025-03-19T15:03:52Z

src/viam/sdk/robot/client.cpp

+    const auto response = impl_->stub_->Log(ctx, req, &resp);
+    if (is_error_response(response)) {
+        // Manually override to force this to get logged to console so we don't set off an infinite
+        // loop


A classic; had to do something like this in go, too, I think.

lia-viam · 2025-03-19T17:14:40Z

Per @benjirewis suggestion just added some logging to the simple and complex module examples, although now I'm wondering if I should just replace all the use of cout and cerr in the examples directory with logging? cc @stuqdog

acmorrow

I was interested so had a quick look through. Just some high level thoughts and comments. No need to wait on me to move forward with this.

acmorrow · 2025-03-20T13:45:03Z

src/viam/sdk/log/logging.hpp

+#include <boost/log/attributes/clock.hpp>
+#include <boost/log/expressions/keyword.hpp>
+#include <boost/log/sinks/sync_frontend.hpp>
+#include <boost/log/sinks/text_ostream_backend.hpp>
+#include <boost/log/sources/severity_channel_logger.hpp>
+#include <boost/log/utility/manipulators/add_value.hpp>
+#include <boost/utility/string_view.hpp>


Does the long term plan still include going boost free in the API? Should there be a level of indirection about what logging backend is in use?

Yes definitely. The situation with Boost.Log's stream operator-based approach makes this difficult but I suspect it is still doable.

which is to say hopefully in a subsequent boost-insulating PR

acmorrow · 2025-03-20T13:54:07Z

src/viam/sdk/log/logging.hpp

+#define VIAM_LOG_IMPL(lg, level)                                                     \
+    BOOST_LOG_SEV((lg), ::viam::sdk::log_level::level)                               \
+        << ::boost::log::add_value(::viam::sdk::attr_file_type{},                    \
+                                   ::viam::sdk::log_detail::trim_filename(__FILE__)) \
+        << ::boost::log::add_value(::viam::sdk::attr_line_type{}, __LINE__)
+
+/// @brief Log macro for general SDK logs.
+/// @ingroup Log
+///
+/// Use this macro to generate log messages pertaining to the SDK at large.
+#define VIAM_LOG(level) VIAM_LOG_IMPL(::viam::sdk::LogManager::get().global_logger(), level)
+
+/// @brief Log macro for resource-level logs.
+/// @ingroup Log
+///
+/// This macro can only be called from the definition of a member function of a class inheriting
+/// @ref Resource. It will log messages to the log source of that specific resource, allowing
+/// resource-level log filtering.
+#define VIAM_RESOURCE_LOG(level) VIAM_LOG_IMPL(this->logger_, level)


Do these macros form part of the library API? Or, if we had the machinery (prelude/postlude headers with push/pop, etc.) would they not be visible to consumers of the library? I think they probably are intended for user consumption (the former case), for instance in the case of writing a module.

I wonder then if they should be in a dedicated header, logging_macros.hpp or similar, so that the Viam C++ SDK can prevent them from leaking into client code by way of its own usage of the logging macros in headers, but consumers can opt in to visibility by consuming the header directly.

I would say they are part of the API yes. Pursuant to eventually insulating Boost entirely (touching on another of your comments) I'm inclined to present the macros to users as a black box and not have them call any Boost.Log functions on their own.

I'm aware of other logging libraries that are more amenable to non-macro use but unfortunately the operator<< approach of this one makes it fairly unusable for practical purposes without a logging macro.

acmorrow · 2025-03-20T13:55:23Z

src/viam/sdk/log/logging.hpp

+
+}  // namespace log_detail
+
+BOOST_LOG_ATTRIBUTE_KEYWORD_TYPE(attr_channel, "Channel", std::string);


I have some worries about what the implications are for when we want to do symbol visibility.

This was bugging me a bit as well. There is a slight subtlety that BOOST_LOG_ATTRIBUTE_KEYWORD_TYPE just defines something like struct attr_channel_type whereas BOOST_LOG_ATTRIBUTE_KEYWORD defines struct attr_channel_type as well as const attr_channel_type attr_channel. We are using the former--is that still an issue?

In any case I'm going to hack away and see if I can't figure out a way to hide those because I don't like exposing them in a header either.

acmorrow · 2025-03-20T13:57:06Z

src/viam/sdk/log/logging.hpp

+                                 "TimeStamp",
+                                 boost::log::attributes::local_clock::value_type);
+
+#define VIAM_LOG_IMPL(lg, level)                                                     \


Maybe VIAM_SDK_LOG... to keep it VIAM_SDK as the macro prefix and viam::sdk as the namespace prefix.

acmorrow · 2025-03-20T13:59:10Z

src/viam/sdk/log/logging.hpp

+/// @brief Severity levels for the logger.
+/// @ingroup Log
+enum class log_level : std::int8_t {
+    trace = -2,


I think elsewhere for enum constants we do k_x:

viam-cpp-sdk/src/viam/sdk/services/mlmodel.hpp

Line 125 in 1cdb1e9

k_int8 = 0,

That's true although I wonder about this from a usability perspective, it feels weird for users to have to type VIAM_LOG(k_warn) rather than VIAM_LOG(warn) ... maybe this should be an exception to the rule? I've seen other libraries do macros-per-severity (eg VIAM_LOG_INFO) but this starts to get combinatorial since we already have the general- and resource-flavored log macros

I'm willing for it to be an exception. Maybe throw a comment on there indicating that it is.

stuqdog

A couple small questions on the code itself! Also:

I think there was something that was cribbed from SO? It'd be nice to make sure still that do at least one of a) document where it came from, or b) unit test it
I think we should update the README to indicate how users should (or should not) be interacting with boost log directly when using the SDK.
The format of the server side logs looks slightly off to me insofar as the properly formatted server side log doesn't seem to include a file or line number? Unless I'm missing something.

Otherwise this looks good to me!

stuqdog · 2025-04-16T13:37:28Z

src/viam/examples/mlmodel/example_audio_classification_client.cpp

-            std::cout << argv[0] << ": The path `" << tflite_module_path.c_str()
-                      << "` provided for `--tflite-module-path` is not an existing regular file"
-                      << std::endl;
+            VIAM_SDK_LOG(error)


(q) just wondering why we no longer output argv[0] here and below?

so argv[0] is executable name, and every single log message in the old code was prepending exe name, which I get the desire for but isn't really a pattern that we'd used anywhere else. This was the inspiration for providing the option to change the general channel/module name printed with all log messages, which is used on line 96 to basically set the global channel name to argv[0]

stuqdog · 2025-04-16T13:39:30Z

src/viam/examples/mlmodel/example_audio_classification_client.cpp


        return EXIT_SUCCESS;
    }
 } catch (const std::exception& ex) {
-    std::cout << argv[0] << ": "
-              << "Failed: a std::exception was thrown: `" << ex.what() << "``" << std::endl;
+    std::cerr << argv[0] << ": "


(q) why are we using cerr here instead of VIAM_SDK_LOG(error)? I'd expect that we'd catch errors around the logger itself when we create it so we probably would still be able to use the logger even if we caught an exception here?

since the entire main is in a function try-catch block, the Instance is out of scope within this catch block. It's possible (though unlikely) that the exception could even have originated from the Instance, so we can't be assured that it exists and logging is set up

stuqdog · 2025-04-16T13:40:40Z

src/viam/examples/modules/tflite/main.cpp

@@ -757,10 +757,10 @@ int serve(const std::string& socket_path) try {

    return EXIT_SUCCESS;
 } catch (const std::exception& ex) {
-    std::cout << "ERROR: A std::exception was thrown from `serve`: " << ex.what() << std::endl;


same q as above, why are we not using the logger here?

same answer as above :)

lia-viam and others added 30 commits February 18, 2025 11:02

initial commit of instance object

31437c3

wip: initial commit to get sdk building with instance

932ebaa

add missing implementation

04d0158

update examples to use instance

a105cef

only instance can construct registry

403f316

make tests work with registry member approach

27eeb13

pimpl registry and update examples

ce2aa8c

add instance management infra

e5cf6d0

Merge branch 'main' of github.com:viamrobotics/viam-cpp-sdk into regi…

85194da

…stry-member

update tflite module

5024d3a

silence spurious const warning

ef8ec8e

make method static

16113fd

Merge branch 'main' of github.com:viamrobotics/viam-cpp-sdk into regi…

216e5e2

…stry-member

update cml boost links

dbf1647

add logger to resource

d1dffb8

use static current to populate registry member

4771390

revert and adapt examples

18d26c9

make instance more of a black box

6018323

Merge branch 'registry-member' into logger

d4aeb86

add impl instance to git

5535616

Merge branch 'registry-member' into logger

4d2582c

Add instance comment boilerplate

eeac195

Co-authored-by: Ethan <[email protected]>

Add instance comment boilerplate

7560f6d

Co-authored-by: Ethan <[email protected]>

connect robot client logging when parent of module service

af08403

remove old ctor arg

4e25d35

prohibit multi instance

55baa6c

remove initialized flag

10cd711

meyers singleton registry

c745422

remove registry member and fix member spacing/ordering

75ab450

remove unused include

fc21c35

lia-viam added 7 commits March 17, 2025 17:37

fix out of date comment

2de9f40

rename to log manager and global logger

d681f6f

rename files to logging

3b8d7dc

use has_log_configuration

82eb403

unit test trim filename

ad079c8

handle fallback console logging for grpc log failures

6b83aed

Merge branch 'main' of github.com:viamrobotics/viam-cpp-sdk into logger

f9f39a1

benjirewis reviewed Mar 19, 2025

View reviewed changes

lia-viam added 3 commits March 19, 2025 13:07

add logging to simple module example

4eb8643

add logging to complex module example

418f882

Merge branch 'main' of github.com:viamrobotics/viam-cpp-sdk into logger

ee85b28

acmorrow reviewed Mar 20, 2025

View reviewed changes

lia-viam added 9 commits March 20, 2025 11:48

use VIAM_SDK macro prefix

40b5400

disable fail-fast

22d29e3

Merge branch 'main' of github.com:viamrobotics/viam-cpp-sdk into logger

ba154d1

link boost log and log_setup in public libs

2e9d015

set boost log dyn link in test script

b8dcf7c

clean up example camera and use logging instead of cout

f6c21ff

update example code to use logging

ac25b4e

add test and use set_global_resource_name

04c92ce

silence const warnings

9297062

lia-viam requested a review from stuqdog April 14, 2025 15:36

stuqdog reviewed Apr 16, 2025

View reviewed changes

lia-viam added 6 commits April 16, 2025 11:09

add remark on logging enums

b1aa44b

add explanatory comment

6018b94

Merge branch 'main' of github.com:viamrobotics/viam-cpp-sdk into logger

743699d

format file/line into rdk log message

bc5f165

document use of boost log

82f7104

remove period

2fa0860

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

RSDK-5986: Resource-level logging for the C++ SDK #383

RSDK-5986: Resource-level logging for the C++ SDK #383

lia-viam commented Mar 14, 2025 •

edited

Loading

benjirewis left a comment

benjirewis Mar 19, 2025

lia-viam commented Mar 19, 2025

acmorrow left a comment

acmorrow Mar 20, 2025

lia-viam Mar 20, 2025

lia-viam Mar 20, 2025

acmorrow Mar 20, 2025

lia-viam Mar 20, 2025

acmorrow Mar 20, 2025

lia-viam Mar 20, 2025

acmorrow Mar 20, 2025

acmorrow Mar 20, 2025

lia-viam Mar 20, 2025

acmorrow Mar 20, 2025

stuqdog left a comment

stuqdog Apr 16, 2025

lia-viam Apr 16, 2025

stuqdog Apr 16, 2025

lia-viam Apr 16, 2025

stuqdog Apr 16, 2025

lia-viam Apr 16, 2025


		} // namespace log_detail

		BOOST_LOG_ATTRIBUTE_KEYWORD_TYPE(attr_channel, "Channel", std::string);

RSDK-5986: Resource-level logging for the C++ SDK #383

Are you sure you want to change the base?

RSDK-5986: Resource-level logging for the C++ SDK #383

Conversation

lia-viam commented Mar 14, 2025 • edited Loading

benjirewis left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

lia-viam commented Mar 19, 2025

acmorrow left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

stuqdog left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

lia-viam commented Mar 14, 2025 •

edited

Loading