--memray causes built-in junit-xml results writer to fail

[petr-tik]

Bug Report

Current Behavior A clear and concise description of the behavior.

python3.8 -m pytest --memray test_memray_marker.py --junit-xml=results.xml
============================================================================ test session starts ============================================================================
platform linux -- Python 3.8.6, pytest-6.2.5, py-1.10.0, pluggy-1.0.0
rootdir: /tmp/tmp.RVJ6YYtLcB
plugins: memray-1.0.0
collected 1 item                                                                                                                                                            

test_memray_marker.py M
INTERNALERROR> Traceback (most recent call last):
INTERNALERROR>   File "/home/petr_tik/.local/lib/python3.8/site-packages/_pytest/main.py", line 269, in wrap_session
INTERNALERROR>     session.exitstatus = doit(config, session) or 0
INTERNALERROR>   File "/home/petr_tik/.local/lib/python3.8/site-packages/_pytest/main.py", line 323, in _main
INTERNALERROR>     config.hook.pytest_runtestloop(session=session)
INTERNALERROR>   File "/home/petr_tik/.local/lib/python3.8/site-packages/pluggy/_hooks.py", line 265, in __call__
INTERNALERROR>     return self._hookexec(self.name, self.get_hookimpls(), kwargs, firstresult)
INTERNALERROR>   File "/home/petr_tik/.local/lib/python3.8/site-packages/pluggy/_manager.py", line 80, in _hookexec
INTERNALERROR>     return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
INTERNALERROR>   File "/home/petr_tik/.local/lib/python3.8/site-packages/pluggy/_callers.py", line 60, in _multicall
INTERNALERROR>     return outcome.get_result()
INTERNALERROR>   File "/home/petr_tik/.local/lib/python3.8/site-packages/pluggy/_result.py", line 60, in get_result
INTERNALERROR>     raise ex[1].with_traceback(ex[2])
INTERNALERROR>   File "/home/petr_tik/.local/lib/python3.8/site-packages/pluggy/_callers.py", line 39, in _multicall
INTERNALERROR>     res = hook_impl.function(*args)
INTERNALERROR>   File "/home/petr_tik/.local/lib/python3.8/site-packages/_pytest/main.py", line 348, in pytest_runtestloop
INTERNALERROR>     item.config.hook.pytest_runtest_protocol(item=item, nextitem=nextitem)
INTERNALERROR>   File "/home/petr_tik/.local/lib/python3.8/site-packages/pluggy/_hooks.py", line 265, in __call__
INTERNALERROR>     return self._hookexec(self.name, self.get_hookimpls(), kwargs, firstresult)
INTERNALERROR>   File "/home/petr_tik/.local/lib/python3.8/site-packages/pluggy/_manager.py", line 80, in _hookexec
INTERNALERROR>     return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
INTERNALERROR>   File "/home/petr_tik/.local/lib/python3.8/site-packages/pluggy/_callers.py", line 60, in _multicall
INTERNALERROR>     return outcome.get_result()
INTERNALERROR>   File "/home/petr_tik/.local/lib/python3.8/site-packages/pluggy/_result.py", line 60, in get_result
INTERNALERROR>     raise ex[1].with_traceback(ex[2])
INTERNALERROR>   File "/home/petr_tik/.local/lib/python3.8/site-packages/pluggy/_callers.py", line 39, in _multicall
INTERNALERROR>     res = hook_impl.function(*args)
INTERNALERROR>   File "/home/petr_tik/.local/lib/python3.8/site-packages/_pytest/runner.py", line 109, in pytest_runtest_protocol
INTERNALERROR>     runtestprotocol(item, nextitem=nextitem)
INTERNALERROR>   File "/home/petr_tik/.local/lib/python3.8/site-packages/_pytest/runner.py", line 126, in runtestprotocol
INTERNALERROR>     reports.append(call_and_report(item, "call", log))
INTERNALERROR>   File "/home/petr_tik/.local/lib/python3.8/site-packages/_pytest/runner.py", line 219, in call_and_report
INTERNALERROR>     hook.pytest_runtest_logreport(report=report)
INTERNALERROR>   File "/home/petr_tik/.local/lib/python3.8/site-packages/pluggy/_hooks.py", line 265, in __call__
INTERNALERROR>     return self._hookexec(self.name, self.get_hookimpls(), kwargs, firstresult)
INTERNALERROR>   File "/home/petr_tik/.local/lib/python3.8/site-packages/pluggy/_manager.py", line 80, in _hookexec
INTERNALERROR>     return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
INTERNALERROR>   File "/home/petr_tik/.local/lib/python3.8/site-packages/pluggy/_callers.py", line 60, in _multicall
INTERNALERROR>     return outcome.get_result()
INTERNALERROR>   File "/home/petr_tik/.local/lib/python3.8/site-packages/pluggy/_result.py", line 60, in get_result
INTERNALERROR>     raise ex[1].with_traceback(ex[2])
INTERNALERROR>   File "/home/petr_tik/.local/lib/python3.8/site-packages/pluggy/_callers.py", line 39, in _multicall
INTERNALERROR>     res = hook_impl.function(*args)
INTERNALERROR>   File "/home/petr_tik/.local/lib/python3.8/site-packages/_pytest/junitxml.py", line 589, in pytest_runtest_logreport
INTERNALERROR>     reporter.append_failure(report)
INTERNALERROR>   File "/home/petr_tik/.local/lib/python3.8/site-packages/_pytest/junitxml.py", line 201, in append_failure
INTERNALERROR>     assert report.longrepr is not None
INTERNALERROR> AssertionError

Input Code

put this in a file called test_memray_marker.py

import pytest

@pytest.mark.limit_memory("10B")
def test_memray():
    _allocated_array = ["substring" * str_len for str_len in range(20)]
    print(_allocated_array)

    assert True

Expected behavior/code

Running this completes successfully without throwing InternalError - AssertionError

python3.8 -m pytest --memray test_memray_marker.py --junit-xml=results.xml

Environment

• Python(s): python3.8, pytest-memray-1.0.0, pytest-6.2.5

Additional context/Screenshots

Thanks for the awesome pytest plugin!

[pablogsal]

Thanks for the issue @petr-tik .

This has been fixed by #4

[petr-tik]

thanks for the swift turnaround! I was actually quite keen to try this myself, but didn't manage to make tox run the tests.

can you please include some getting started instructions for potential new contributors?

I was wondering if the longrepr error message can include more informative output along the lines of:

test {item.nodeid} failed because it was limited to {limit_from_the_memray_marker} and allocated {item.max_allocated_memory} was allocated

[pablogsal]

I was wondering if the longrepr error message can include more informative output along the lines of:

It could, but at the level, this is happening there could be more different markers, so we need some map from the marker to the error template. This can be a fantastic contribution if you are interested.

thanks for the swift turnaround! I was actually quite keen to try this myself, but didn't manage to make tox run the tests.

Oh :( What problems are you experiencing when you ran tox? Could you paste them here or maybe opening a new issue?

[petr-tik]

this is happening there could be more different markers, so we need some map from the marker to the error template

what other markers do you have in mind? I can think of xfail and its custom derivations. Either way, I would be happy to attempt to generate something more informative using the information collected by memray. You might know a certain financial services company that is partial to junit xml test output!

Oh :( What problems are you experiencing when you ran tox? Could you paste them here or maybe opening a new issue?

Am I supposed to run tox or is there any other way to run python tests? I only have python3.8 installed on my box, so I wouldn't want to run tox on all environments.

[pablogsal]

Am I supposed to run tox or is there any other way to run python tests? I only have python3.8 installed on my box, so I wouldn't want to run tox on all environments.

You can run tox -e py38 to only run the 3.8 version but if you want to avoid tox in any case you can create a virtual environment, run make test-install (or install the requirement files in the root of the project) then you can run the tests by running pytest tests or make check.

[pablogsal]

I would be happy to attempt to generate something more informative using the information collected by memray. You might know a certain financial services company that is partial to junit xml test output!

That would be fantastic. We created the plugin as an example of the tooling that can be built with memray (some of the APIs used here are not stable yet, but there will be) so we are super excited to see what the community can come up with :)

We don't have any specific new markers in mind at the time as we are focusing currently in fixing existing edges with the profiler directly but I think this is an area where we can greatly improve. Any contribution or idea here is welcomed!

[petr-tik]

I opened a prototype PR, would be great to get your thoughts.

Thanks again for open sourcing this

[pablogsal]

Will try to review this tomorrow. Thanks a lot for your work!