GPU master test failure: test_python_ops.py

[tengyifei]

This is confusing to contributors because they assume their PR broke things

.sss........sss.....malloc(): unsorted double linked list corrupted
*** Received signal 6 ***
*** BEGIN MANGLED STACK TRACE ***
/usr/local/lib/python3.10/site-packages/_XLAC.cpython-310-x86_64-linux-gnu.so(+0x6eb7026)[0x7f4c76d08026]
/lib/x86_64-linux-gnu/libc.so.6(+0x38dd0)[0x7f4eadbd6dd0]
/lib/x86_64-linux-gnu/libc.so.6(gsignal+0x141)[0x7f4eadbd6d51]
/lib/x86_64-linux-gnu/libc.so.6(abort+0x123)[0x7f4eadbc0537]
/lib/x86_64-linux-gnu/libc.so.6(+0x7a458)[0x7f4eadc18458]
/lib/x86_64-linux-gnu/libc.so.6(+0x8174a)[0x7f4eadc1f74a]
/lib/x86_64-linux-gnu/libc.so.6(+0x84a64)[0x7f4eadc22a64]
/lib/x86_64-linux-gnu/libc.so.6(__libc_malloc+0x1a9)[0x7f4eadc23f89]
/usr/lib/x86_64-linux-gnu/libstdc++.so.6(_Znwm+0x15)[0x7f4e9ebee0b5]
/usr/local/lib/python3.10/site-packages/torch_xla_cuda_plugin/lib/pjrt_c_api_gpu_plugin.so(+0x39b38ee)[0x7f4ba454e8ee]
/usr/local/lib/python3.10/site-packages/torch_xla_cuda_plugin/lib/pjrt_c_api_gpu_plugin.so(+0x39b4fc0)[0x7f4ba454ffc0]
/usr/local/lib/python3.10/site-packages/torch_xla_cuda_plugin/lib/pjrt_c_api_gpu_plugin.so(+0x39b5371)[0x7f4ba4550371]
/usr/local/lib/python3.10/site-packages/torch_xla_cuda_plugin/lib/pjrt_c_api_gpu_plugin.so(+0x764d8e5)[0x7f4ba81e88e5]
/usr/local/lib/python3.10/site-packages/torch_xla_cuda_plugin/lib/pjrt_c_api_gpu_plugin.so(+0x764db23)[0x7f4ba81e8b23]
/usr/local/lib/python3.10/site-packages/torch_xla_cuda_plugin/lib/pjrt_c_api_gpu_plugin.so(+0x7768593)[0x7f4ba8303593]
/usr/local/lib/python3.10/site-packages/torch_xla_cuda_plugin/lib/pjrt_c_api_gpu_plugin.so(+0x9df28b)[0x7f4ba157a28b]
/usr/local/lib/python3.10/site-packages/torch_xla_cuda_plugin/lib/pjrt_c_api_gpu_plugin.so(+0x764fc18)[0x7f4ba81eac18]
/usr/local/lib/python3.10/site-packages/torch_xla_cuda_plugin/lib/pjrt_c_api_gpu_plugin.so(+0x7650dcb)[0x7f4ba81ebdcb]
/usr/local/lib/python3.10/site-packages/torch_xla_cuda_plugin/lib/pjrt_c_api_gpu_plugin.so(+0x773de98)[0x7f4ba82d8e98]
/usr/local/lib/python3.10/site-packages/torch_xla_cuda_plugin/lib/pjrt_c_api_gpu_plugin.so(+0x764cfc8)[0x7f4ba81e7fc8]
/usr/local/lib/python3.10/site-packages/torch_xla_cuda_plugin/lib/pjrt_c_api_gpu_plugin.so(+0x3996f95)[0x7f4ba4531f95]
/usr/local/lib/python3.10/site-packages/torch_xla_cuda_plugin/lib/pjrt_c_api_gpu_plugin.so(+0x390c709)[0x7f4ba44a7709]
/usr/local/lib/python3.10/site-packages/torch_xla_cuda_plugin/lib/pjrt_c_api_gpu_plugin.so(+0x390db56)[0x7f4ba44a8b56]
/usr/local/lib/python3.10/site-packages/torch_xla_cuda_plugin/lib/pjrt_c_api_gpu_plugin.so(+0xd2f186)[0x7f4ba18ca186]
/usr/local/lib/python3.10/site-packages/torch_xla_cuda_plugin/lib/pjrt_c_api_gpu_plugin.so(+0xd4919a)[0x7f4ba18e419a]
/usr/local/lib/python3.10/site-packages/torch_xla_cuda_plugin/lib/pjrt_c_api_gpu_plugin.so(+0xd4baa5)[0x7f4ba18e6aa5]
/usr/local/lib/python3.10/site-packages/torch_xla_cuda_plugin/lib/pjrt_c_api_gpu_plugin.so(+0xd0c881)[0x7f4ba18a7881]
/usr/local/lib/python3.10/site-packages/torch_xla_cuda_plugin/lib/pjrt_c_api_gpu_plugin.so(+0xae0ffe)[0x7f4ba167bffe]
/usr/local/lib/python3.10/site-packages/torch_xla_cuda_plugin/lib/pjrt_c_api_gpu_plugin.so(+0xaddf57)[0x7f4ba1678f57]
/usr/local/lib/python3.10/site-packages/torch_xla_cuda_plugin/lib/pjrt_c_api_gpu_plugin.so(+0xaa8dcd)[0x7f4ba1643dcd]
/usr/local/lib/python3.10/site-packages/torch_xla_cuda_plugin/lib/pjrt_c_api_gpu_plugin.so(+0xaaa6f1)[0x7f4ba16456f1]
/usr/local/lib/python3.10/site-packages/torch_xla_cuda_plugin/lib/pjrt_c_api_gpu_plugin.so(+0xa66c8f)[0x7f4ba1601c8f]
/usr/local/lib/python3.10/site-packages/torch_xla_cuda_plugin/lib/pjrt_c_api_gpu_plugin.so(+0x9fc7ee)[0x7f4ba15977ee]
/usr/local/lib/python3.10/site-packages/torch_xla_cuda_plugin/lib/pjrt_c_api_gpu_plugin.so(+0x9fc84c)[0x7f4ba159784c]
/usr/local/lib/python3.10/site-packages/torch_xla_cuda_plugin/lib/pjrt_c_api_gpu_plugin.so(+0xa01a48)[0x7f4ba159ca48]
/usr/local/lib/python3.10/site-packages/_XLAC.cpython-310-x86_64-linux-gnu.so(+0xd28b45c)[0x7f4c7d0dc45c]
/usr/local/lib/python3.10/site-packages/_XLAC.cpython-310-x86_64-linux-gnu.so(_ZN3xla14PjRtCApiClient14CompileAndLoadERKNS_14XlaComputationENS_14CompileOptionsE+0x87)[0x7f4c7d0dc767]
/usr/local/lib/python3.10/site-packages/_XLAC.cpython-310-x86_64-linux-gnu.so(_ZN9torch_xla7runtime21PjRtComputationClient7CompileESt6vectorINS0_17ComputationClient15CompileInstanceESaIS4_EE+0x4ef)[0x7f4c76cfd18f]
/usr/local/lib/python3.10/site-packages/_XLAC.cpython-310-x86_64-linux-gnu.so(_ZN9torch_xla16XLAGraphExecutor7CompileERSt6vectorIN3c1013intrusive_ptrINS_9XLATensorENS2_6detail34intrusive_target_default_null_typeIS4_EEEESaIS8_EEN4absl12lts_202308024SpanIKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEEEERKN5torch4lazy17LazyGraphExecutor20SyncTensorCollectionEPNSP_13PostOrderDataERKS1_INSO_5ValueESaISV_EERKS1_ImSaImEE+0xd26)[0x7f4c76a27aa6]
/usr/local/lib/python3.10/site-packages/_XLAC.cpython-310-x86_64-linux-gnu.so(_ZN9torch_xla16XLAGraphExecutor24SyncTensorsGraphInternalEPSt6vectorIN3c1013intrusive_ptrINS_9XLATensorENS2_6detail34intrusive_target_default_null_typeIS4_EEEESaIS8_EEN4absl12lts_202308024SpanIKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEEEERKN5torch4lazy17LazyGraphExecutor17SyncTensorsConfigEb+0x639)[0x7f4c76a29a39]
/usr/local/lib/python3.10/site-packages/_XLAC.cpython-310-x86_64-linux-gnu.so(_ZN9torch_xla16XLAGraphExecutor16SyncTensorsGraphEPSt6vectorIN3c1013intrusive_ptrINS_9XLATensorENS2_6detail34intrusive_target_default_null_typeIS4_EEEESaIS8_EEN4absl12lts_202308024SpanIKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEEEEbbb+0xc5)[0x7f4c76a2a135]
/usr/local/lib/python3.10/site-packages/_XLAC.cpython-310-x86_64-linux-gnu.so(_ZN9torch_xla16XLAGraphExecutor20SyncLiveTensorsGraphEPKN5torch4lazy13BackendDeviceEN3c108ArrayRefINSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEEEEb+0xb9)[0x7f4c76a2a439]
/usr/local/lib/python3.10/site-packages/_XLAC.cpython-310-x86_64-linux-gnu.so(+0x68fa979)[0x7f4c7674b979]
/usr/local/lib/python3.10/site-packages/_XLAC.cpython-310-x86_64-linux-gnu.so(+0x68faf6c)[0x7f4c7674bf6c]
/usr/local/lib/python3.10/site-packages/_XLAC.cpython-310-x86_64-linux-gnu.so(+0x68c1728)[0x7f4c76712728]
/usr/local/bin/../lib/libpython3.10.so.1.0(+0x129055)[0x7f4eadea6055]
/usr/local/bin/../lib/libpython3.10.so.1.0(_PyObject_MakeTpCall+0x148)[0x7f4eadea3088]
/usr/local/bin/../lib/libpython3.10.so.1.0(_PyEval_EvalFrameDefault+0x5c8f)[0x7f4eade9ffdf]
/usr/local/bin/../lib/libpython3.10.so.1.0(+0x11c478)[0x7f4eade99478]
/usr/local/bin/../lib/libpython3.10.so.1.0(_PyEval_EvalFrameDefault+0x4be2)[0x7f4eade9ef32]
/usr/local/bin/../lib/libpython3.10.so.1.0(+0x11c478)[0x7f4eade99478]
/usr/local/bin/../lib/libpython3.10.so.1.0(PyVectorcall_Call+0xbc)[0x7f4eadeae31c]
/usr/local/bin/../lib/libpython3.10.so.1.0(_PyEval_EvalFrameDefault+0x2ce6)[0x7f4eade9d036]
/usr/local/bin/../lib/libpython3.10.so.1.0(+0x11c478)[0x7f4eade99478]
/usr/local/bin/../lib/libpython3.10.so.1.0(+0x130e5b)[0x7f4eadeade5b]
/usr/local/bin/../lib/libpython3.10.so.1.0(_PyEval_EvalFrameDefault+0x2ce6)[0x7f4eade9d036]
/usr/local/bin/../lib/libpython3.10.so.1.0(+0x11c478)[0x7f4eade99478]
/usr/local/bin/../lib/libpython3.10.so.1.0(+0x130d02)[0x7f4eadeadd02]
/usr/local/bin/../lib/libpython3.10.so.1.0(_PyEval_EvalFrameDefault+0x325)[0x7f4eade9a675]
/usr/local/bin/../lib/libpython3.10.so.1.0(+0x11c478)[0x7f4eade99478]
/usr/local/bin/../lib/libpython3.10.so.1.0(_PyEval_EvalFrameDefault+0x720)[0x7f4eade9aa70]
/usr/local/bin/../lib/libpython3.10.so.1.0(+0x11c478)[0x7f4eade99478]
/usr/local/bin/../lib/libpython3.10.so.1.0(+0x130d02)[0x7f4eadeadd02]
/usr/local/bin/../lib/libpython3.10.so.1.0(_PyEval_EvalFrameDefault+0x13d8)[0x7f4eade9b728]
/usr/local/bin/../lib/libpython3.10.so.1.0(+0x11c478)[0x7f4eade99478]
/usr/local/bin/../lib/libpython3.10.so.1.0(+0x130d02)[0x7f4eadeadd02]
/usr/local/bin/../lib/libpython3.10.so.1.0(_PyEval_EvalFrameDefault+0x13d8)[0x7f4eade9b728]
/usr/local/bin/../lib/libpython3.10.so.1.0(+0x11c478)[0x7f4eade99478]
/usr/local/bin/../lib/libpython3.10.so.1.0(+0x130d02)[0x7f4eadeadd02]
/usr/local/bin/../lib/libpython3.10.so.1.0(_PyEval_EvalFrameDefault+0x13d8)[0x7f4eade9b728]
/usr/local/bin/../lib/libpython3.10.so.1.0(+0x11c478)[0x7f4eade99478]
/usr/local/bin/../lib/libpython3.10.so.1.0(+0x130ddc)[0x7f4eadeadddc]
/usr/local/bin/../lib/libpython3.10.so.1.0(_PyEval_EvalFrameDefault+0x2ce6)[0x7f4eade9d036]
/usr/local/bin/../lib/libpython3.10.so.1.0(+0x11c478)[0x7f4eade99478]
/usr/local/bin/../lib/libpython3.10.so.1.0(_PyObject_FastCallDictTstate+0x9d)[0x7f4eadea26fd]
/usr/local/bin/../lib/libpython3.10.so.1.0(_PyObject_Call_Prepend+0x5c)[0x7f4eadeac30c]
/usr/local/bin/../lib/libpython3.10.so.1.0(+0x1d083a)[0x7f4eadf4d83a]
/usr/local/bin/../lib/libpython3.10.so.1.0(_PyObject_MakeTpCall+0x80)[0x7f4eadea2fc0]
/usr/local/bin/../lib/libpython3.10.so.1.0(_PyEval_EvalFrameDefault+0x50bd)[0x7f4eade9f40d]
/usr/local/bin/../lib/libpython3.10.so.1.0(+0x11c478)[0x7f4eade99478]
/usr/local/bin/../lib/libpython3.10.so.1.0(+0x130ddc)[0x7f4eadeadddc]
/usr/local/bin/../lib/libpython3.10.so.1.0(_PyEval_EvalFrameDefault+0x2ce6)[0x7f4eade9d036]
/usr/local/bin/../lib/libpython3.10.so.1.0(+0x11c478)[0x7f4eade99478]
/usr/local/bin/../lib/libpython3.10.so.1.0(_PyObject_FastCallDictTstate+0x9d)[0x7f4eadea26fd]
/usr/local/bin/../lib/libpython3.10.so.1.0(_PyObject_Call_Prepend+0x5c)[0x7f4eadeac30c]
/usr/local/bin/../lib/libpython3.10.so.1.0(+0x1d083a)[0x7f4eadf4d83a]
/usr/local/bin/../lib/libpython3.10.so.1.0(_PyObject_MakeTpCall+0x80)[0x7f4eadea2fc0]
/usr/local/bin/../lib/libpython3.10.so.1.0(_PyEval_EvalFrameDefault+0x50bd)[0x7f4eade9f40d]
/usr/local/bin/../lib/libpython3.10.so.1.0(+0x11c478)[0x7f4eade99478]
/usr/local/bin/../lib/libpython3.10.so.1.0(+0x130ddc)[0x7f4eadeadddc]
/usr/local/bin/../lib/libpython3.10.so.1.0(_PyEval_EvalFrameDefault+0x2ce6)[0x7f4eade9d036]
/usr/local/bin/../lib/libpython3.10.so.1.0(+0x11c478)[0x7f4eade99478]
/usr/local/bin/../lib/libpython3.10.so.1.0(_PyObject_FastCallDictTstate+0x9d)[0x7f4eadea26fd]
/usr/local/bin/../lib/libpython3.10.so.1.0(_PyObject_Call_Prepend+0x5c)[0x7f4eadeac30c]
/usr/local/bin/../lib/libpython3.10.so.1.0(+0x1d083a)[0x7f4eadf4d83a]
/usr/local/bin/../lib/libpython3.10.so.1.0(_PyObject_MakeTpCall+0x80)[0x7f4eadea2fc0]
/usr/local/bin/../lib/libpython3.10.so.1.0(_PyEval_EvalFrameDefault+0x50bd)[0x7f4eade9f40d]
/usr/local/bin/../lib/libpython3.10.so.1.0(+0x11c478)[0x7f4eade99478]
/usr/local/bin/../lib/libpython3.10.so.1.0(_PyEval_EvalFrameDefault+0x720)[0x7f4eade9aa70]
/usr/local/bin/../lib/libpython3.10.so.1.0(+0x11c478)[0x7f4eade99478]
/usr/local/bin/../lib/libpython3.10.so.1.0(_PyEval_EvalFrameDefault+0x720)[0x7f4eade9aa70]
/usr/local/bin/../lib/libpython3.10.so.1.0(+0x11c478)[0x7f4eade99478]
/usr/local/bin/../lib/libpython3.10.so.1.0(_PyObject_FastCallDictTstate+0x109)[0x7f4eadea2769]
/usr/local/bin/../lib/libpython3.10.so.1.0(+0x12efd4)[0x7f4eadeabfd4]
/usr/local/bin/../lib/libpython3.10.so.1.0(+0x1262bb)[0x7f4eadea32bb]
/usr/local/bin/../lib/libpython3.10.so.1.0(_PyObject_MakeTpCall+0x148)[0x7f4eadea3088]
/usr/local/bin/../lib/libpython3.10.so.1.0(_PyEval_EvalFrameDefault+0x5c8f)[0x7f4eade9ffdf]
/usr/local/bin/../lib/libpython3.10.so.1.0(+0x11c478)[0x7f4eade99478]
/usr/local/bin/../lib/libpython3.10.so.1.0(_PyEval_EvalFrameDefault+0x325)[0x7f4eade9a675]
/usr/local/bin/../lib/libpython3.10.so.1.0(+0x11c478)[0x7f4eade99478]
/usr/local/bin/../lib/libpython3.10.so.1.0(PyEval_EvalCode+0x90)[0x7f4eadf14410]
/usr/local/bin/../lib/libpython3.10.so.1.0(+0x1a7f6d)[0x7f4eadf24f6d]
/usr/local/bin/../lib/libpython3.10.so.1.0(+0x1a3b96)[0x7f4eadf20b96]
/usr/local/bin/../lib/libpython3.10.so.1.0(+0x8e091)[0x7f4eade0b091]
/usr/local/bin/../lib/libpython3.10.so.1.0(_PyRun_SimpleFileObject+0x37c)[0x7f4eade0a80f]
/usr/local/bin/../lib/libpython3.10.so.1.0(_PyRun_AnyFileObject+0x84)[0x7f4eade0c901]
/usr/local/bin/../lib/libpython3.10.so.1.0(Py_RunMain+0x3c6)[0x7f4eadf31db6]
/usr/local/bin/../lib/libpython3.10.so.1.0(Py_BytesMain+0x39)[0x7f4eadf062b9]
/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xea)[0x7f4eadbc1d7a]
python3(_start+0x2a)[0x559ad1ea007a]
*** END MANGLED STACK TRACE ***

./test/run_tests.sh: line 51: 70030 Alarm clock             python3 "$@"

[ysiraichi]

It looks like this went through in the last OpenXLA pin update #8908. In that PR, we didn't get to this error because we stopped at another (possibly flaky) error before that.