Is there a way to include debug info?

[alexcasalboni]

When the .wasm file raises an error, the resulting backtrace does not contain any reference to the original source files (even using wasmtime::WasmBacktraceDetails::Enable).

With Rust and Go components it's possible to use compile-time flags such as --release or -no-debug to control whether debug info is included or not in the .wasm file. So I was wondering if the same would be possible with componentize-py?

Here's an example of the backtrace I get on error (where the only useful reference to my component's code is line 28):

Error: Failed to call component: error while executing at wasm backtrace:
    0: 0xa9b968 - wit-component:adapter:wasi_snapshot_preview1!<wasm function 34>
    1: 0x2230183 - wit-component:shim!adapt-wasi_snapshot_preview1-fd_write
    2: 0x21b2244 - libc.so!__wasi_fd_write
    3: 0x21b1a9f - libc.so!write
    4: 0x14934 - <unknown>!<wasm function 782>
    5: 0xeba24e - _Py_write_impl
                    at /home/runner/work/componentize-py/componentize-py/cpython/builddir/wasi/../../Python/fileutils.c:1962:17              - _Py_write
                    at /home/runner/work/componentize-py/componentize-py/cpython/builddir/wasi/../../Python/fileutils.c:2033:12
    6: 0xf9dffd - _io_FileIO_write_impl
                    at /home/runner/work/componentize-py/componentize-py/cpython/builddir/wasi/../../Modules/_io/fileio.c:874:9              - _io_FileIO_write
                    at /home/runner/work/componentize-py/componentize-py/cpython/builddir/wasi/../../Modules/_io/clinic/fileio.c.h:388:20
    7: 0xd16038 - method_vectorcall_FASTCALL_KEYWORDS_METHOD
                    at /home/runner/work/componentize-py/componentize-py/cpython/builddir/wasi/../../Objects/descrobject.c:387:24
    8: 0xd09540 - _PyObject_VectorcallTstate
                    at /home/runner/work/componentize-py/componentize-py/cpython/builddir/wasi/../../Include/internal/pycore_call.h:92:11              - PyObject_VectorcallMethod
                    at /home/runner/work/componentize-py/componentize-py/cpython/builddir/wasi/../../Objects/call.c:887:24
    9: 0xfab74b - PyObject_CallMethodOneArg
                    at /home/runner/work/componentize-py/componentize-py/cpython/builddir/wasi/../../Include/cpython/abstract.h:103:12              - _textiowrapper_writeflush
                    at /home/runner/work/componentize-py/componentize-py/cpython/builddir/wasi/../../Modules/_io/textio.c:1629:15
   10: 0xfacee3 - _io_TextIOWrapper_write_impl
                    at /home/runner/work/componentize-py/componentize-py/cpython/builddir/wasi/../../Modules/_io/textio.c:1759:13              - _io_TextIOWrapper_write
                    at /home/runner/work/componentize-py/componentize-py/cpython/builddir/wasi/../../Modules/_io/clinic/textio.c.h:680:20
   11: 0xd64d6e - cfunction_vectorcall_O
                    at /home/runner/work/componentize-py/componentize-py/cpython/builddir/wasi/../../Objects/methodobject.c:509:24
   12: 0xd084da - _PyObject_VectorcallTstate
                    at /home/runner/work/componentize-py/componentize-py/cpython/builddir/wasi/../../Include/internal/pycore_call.h:92:11              - PyObject_CallOneArg
                    at /home/runner/work/componentize-py/componentize-py/cpython/builddir/wasi/../../Objects/call.c:401:12
   13: 0xd291f6 - PyFile_WriteObject
                    at /home/runner/work/componentize-py/componentize-py/cpython/builddir/wasi/../../Objects/fileobject.c:132:14
   14: 0xd292e5 - PyFile_WriteString
                    at /home/runner/work/componentize-py/componentize-py/cpython/builddir/wasi/../../Objects/fileobject.c:156:15
   15: 0xeab96b - _Py_WriteIndentedMargin
                    at /home/runner/work/componentize-py/componentize-py/cpython/builddir/wasi/../../Python/traceback.c:430:13              - _PyTraceBack_Print_Indented
                    at /home/runner/work/componentize-py/componentize-py/cpython/builddir/wasi/../../Python/traceback.c:1078:9
   16: 0xe96e54 - print_exception_traceback
                    at /home/runner/work/componentize-py/componentize-py/cpython/builddir/wasi/../../Python/pythonrun.c:911:15              - print_exception
                    at /home/runner/work/componentize-py/componentize-py/cpython/builddir/wasi/../../Python/pythonrun.c:1180:9
   17: 0xe96465 - print_exception_recursive
                    at /home/runner/work/componentize-py/componentize-py/cpython/builddir/wasi/../../Python/pythonrun.c:1474:13
   18: 0xe96122 - _PyErr_Display
                    at /home/runner/work/componentize-py/componentize-py/cpython/builddir/wasi/../../Python/pythonrun.c:1524:9
   19: 0xe9600c - PyErr_Display
                    at /home/runner/work/componentize-py/componentize-py/cpython/builddir/wasi/../../Python/pythonrun.c:1556:5
   20: 0xea872d - sys_excepthook_impl
                    at /home/runner/work/componentize-py/componentize-py/cpython/builddir/wasi/../../Python/sysmodule.c:772:5              - sys_excepthook
                    at /home/runner/work/componentize-py/componentize-py/cpython/builddir/wasi/../../Python/clinic/sysmodule.c.h:102:20
   21: 0xd64ea4 - cfunction_vectorcall_FASTCALL
                    at /home/runner/work/componentize-py/componentize-py/cpython/builddir/wasi/../../Objects/methodobject.c:422:24
   22: 0xd08285 - _PyObject_VectorcallTstate
                    at /home/runner/work/componentize-py/componentize-py/cpython/builddir/wasi/../../Include/internal/pycore_call.h:92:11              - _PyObject_FastCallTstate
                    at /home/runner/work/componentize-py/componentize-py/cpython/builddir/wasi/../../Include/internal/pycore_call.h:116:12              - _PyObject_FastCall
                    at /home/runner/work/componentize-py/componentize-py/cpython/builddir/wasi/../../Objects/call.c:334:12
   23: 0xe94994 - _PyErr_PrintEx
                    at /home/runner/work/componentize-py/componentize-py/cpython/builddir/wasi/../../Python/pythonrun.c:807:28
   24: 0xe96049 - PyErr_PrintEx
                    at /home/runner/work/componentize-py/componentize-py/cpython/builddir/wasi/../../Python/pythonrun.c:845:5
   25: 0x12f52 - <unknown>!<wasm function 291>
   26: 0xaf3c63 - libcomponentize_py_runtime.so!pyo3::err::PyErr::print::hbb86b960398edf90
   27: 0xadda83 - libcomponentize_py_runtime.so!componentize-py#Dispatch
   28: 0x221c1db - libcomponentize_py_bindings.so!edgee:components/data-collection#page-export

[alexcasalboni]

ping @benbrandt @dicej 🙏

[dicej]

@alexcasalboni Thanks for opening the issue, and sorry I didn't notice it earlier.

There are several aspects to the question of debug info:

• Native Wasm code (e.g. compiled Rust/C/C++/Fortan/etc. and componentize-py-generated Wasm code)
  • In the stack trace you provided above, it looks like the CPython C code does have debug info (or at least line/file info)
  • The Rust code does not, but you could address that by removing

    componentize-py/build.rs

    Line 114 in 539472b

    .arg("--release")

    and changing release to debug in

    componentize-py/build.rs

    Line 135 in 539472b

    let path = out_dir.join("wasm32-wasip1/release/libcomponentize_py_runtime.a");

    . We could also update the code to control release vs. debug via e.g. an environment variable
  • The componentize-py-generated Wasm code doesn't have any "source code" in the conventional sense; it's derived from the WIT files you give it, but it's mostly Component Model ABI lifting and lowering code with no direct correspondence to a source file. I imagine we could generate some useful DWARF info for it, but it would be a non-trivial project
  • If you were using a package with native extensions packaged as .so files and wanted to debug them, those would also need to be compiled using e.g. a -g flag to LLVM
• Interpreted Python code
  • Note that componentize-py does not compile Python code -- it just bundles it into the component to be interpreted by CPython. Wasmtime-produced backtraces will not contain any information about the Python-level backtrace, regardless of the presence or absence of DWARF debug info in the Wasm code. Instead, you'll need to use e.g. the traceback module to retrieve or print the Python-level backtrace, either from within your Python code or via CPython or PyO3, depending on where in the code you need it. If you're looking to print that from the host somehow, you'll need to produce a component which exports a function which does that. Finally, note that you won't be able to call it after a trap since the instance will already have been invalidated; you'll need to somehow arrange for that code to run before the trap happens.

Hope that helps; let me know if you have any other questions or thoughts.

[alexcasalboni]

Thank you for the clear explanation @dicej 🙏

It could be useful to easily enable non-release builds to identify issues in the native Wasm code (for ourselves, as the platform/host).

Since we enable 3rd-party developers to build and run components on our platform, it would be incredibly useful to surface the Python-level backtrace with references to the source code - which is what the component developer controls and it's the most likely source of any error happening when running the Python component.

I would expect PyO3 (or wasmtime) to automatically intercept any interpreter error and surface it so we can give developers visibility over what's going on in the interpreter without forcing them to instrument their code in a special way.

Update: funny enough, I have just tried again (with a simple division by zero in the component code) and got this backtrace:

Traceback (most recent call last):
  File "/0/dc_component.py", line 41, in page
ZeroDivisionError: division by zero
thread '<unnamed>' panicked at runtime/src/lib.rs:499:25:
Python function threw an unexpected exception
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
Error: Failed to call component: error while executing at wasm backtrace:
    0: 0xb71cd8 - libcomponentize_py_runtime.so!__rust_start_panic
    1: 0xb71b49 - libcomponentize_py_runtime.so!rust_panic
    2: 0xb71b1c - libcomponentize_py_runtime.so!std::panicking::rust_panic_with_hook::h683e04a3e610de41
    3: 0xb69f7e - libcomponentize_py_runtime.so!std::panicking::begin_panic_handler::_$u7b$$u7b$closure$u7d$$u7d$::h55bf48e5a4dad3ba
    4: 0xb69ea6 - libcomponentize_py_runtime.so!std::sys::backtrace::__rust_end_short_backtrace::hc61383746b72f4f5
    5: 0xb713e2 - libcomponentize_py_runtime.so!rust_begin_unwind
    6: 0xb94f50 - libcomponentize_py_runtime.so!core::panicking::panic_fmt::h6c2ed8e979e576c4
    7: 0xaeb470 - libcomponentize_py_runtime.so!componentize-py#Dispatch
    8: 0x2229b8b - libcomponentize_py_bindings.so!edgee:components/data-collection#page-export

The first 3 lines look like a full Python backtrace, pointing to the correct source file and line (dc_component.py, line 41) and the error type/message (ZeroDivisionError: division by zero).

I am not 100% sure of why this is showing up now 🤔 I'm assuming because we updated to wasmtime = "31.0.0"?

Update 2: I figured out why this wasn't showing up before. It seems like the backtrace is automatically printed to stderr, but until a couple of weeks ago we were only exposing stdout to the component via WASI (for local development).

Now we're doing this:

wasmtime_wasi::WasiCtx::builder()
    .inherit_stdout()
    .inherit_stderr()
    .build()

so the Python backtrace automagically appears in your terminal 👌 This is pretty good for local development, and we could do something similar in production with some clever mapping.

[dicej]

Yeah, libcomponentize_py_runtime.so does already print a Python-level backtrace in certain circumstances, e.g.

componentize-py/runtime/src/lib.rs

Line 487 in 539472b

error.print(py);

, and per #13, we'd like to expand that to other cases as well.

[dicej]

BTW, the component model currently specifies that a component instance is invalidated upon trapping, but it would be nice if it had a feature whereby we could call one of its exports one last time (e.g. in "locked-down" mode, where it doesn't have access to any of its imports) for the purpose of generating an interpreter-level stack trace. For example, componentize-py could automatically add such an export for every component it creates. I'll open an issue about this on the component model repo when I get a chance.

[alexcasalboni]

Thank you again 🙏

Feel free to close this, as I've realized the "issue" was on our side because of stderr.
(unless you prefer to keep it open or rename it to keep track of the Native Wasm code env variable)