scripts/seastar-addr2line hangs on I/O #2609
Comments
|
A quick review of the code of
While the code in question has multiple calls like these: https://github.com/scylladb/seastar/blob/master/scripts/addr2line.py#L102 |
|
The problem is only with llvm-addr2line, right? |
No, the problem reproduces with both |
|
The problem isn't with the pipes. Starting But So after sending an address line for decoding, But in this case, unexpectedly, You can see it in your strace too: In other words, it's this specific Scylla executable that breaks the |
It doesn't ignore the warning. It just aren't applicable here. The python docs warn you against creating a pipe cycle, because you can end up with process A sleeping until some bytes are read by process B, while process B is waiting until some bytes are read by process A. So in general calling But the protocol we are following in this case avoids the issue, because we only send one address at a time (which is guaranteed to fit in the buffer of a Linux pipe), and then we fully drain the stdout of the child before adding more input. So there is never a situation when we are trying to send more input to a child which is trying to add more output. And |
Is it? I don't think you can universally guarantee it.
I understand that this is what you want. The question is why did we have to go down to hacks instead of not to? The hack used by BTW the output of the Which seems to be exactly what seastar-addr2line expects: However the script hangs exactly the same way: And when I interrupt it: So, @michoecho your assumption about it being either about specific Scylla binary or addr2line in general seems baseless. They behave exactly as On top of that a similar assumption (a.k.a. as "hack") at the line 57 in the scripts/addr2line.py about And for a different Scylla binary: |
The version that comes with U24.04 is newer than the one you have on your Fedora. |
|
I think I know what's the pickle:
The latter means that If I was willing to fix this I'd stop relying on undocumented empiric hacks.
A standard way of handling such situations is to use non-blocking read, @michoecho. |
|
I see that the the "??: 0“ at the end of the line is kinda documented in the addr2line man page. Maybe simply fixing the "dummy_pattern" and feeding an empty line instead of a "0x0" can be a simple fix. |
addr2line.py invokes "addr2line" tool in a "server" mode where addresses are fed via stdin. It intentionally makes a corresponding "addr2line" tool generate a known "invalid address" pattern at the end of each address decoding by feeding it a "0x0" address in order to denote the end of it because a single address can be decoded into multiple lines due to inlining. The above strategy assumes that 0x0 address is always going to be interpreted as an invalid one by both add2line and by llvm-addr2line. However, at least llvm-addr2line 18.1.3 doesn't always interpret it this way. On the other hand an empty line does always generate an expected (invalid address) output. On top of that it looks like a new llvm-addr2line changed the "invalid address" pattern format. This patch adds a new "invalid input" pattern to a dummy_pattern and changes the way we generate such an output by pushing an empty line instead of a 0x0 address. Ref scylladb#2609
addr2line.py invokes "addr2line" tool in a "server" mode where addresses are fed via stdin. It intentionally makes a corresponding "addr2line" tool generate a known "invalid address" pattern at the end of each address decoding by feeding it a "0x0" address in order to denote the end of it because a single address can be decoded into multiple lines due to inlining. The above strategy assumes that 0x0 address is always going to be interpreted as an invalid one by both add2line and by llvm-addr2line. However, at least llvm-addr2line 18.1.3 doesn't always interpret it this way. On the other hand an empty line does always generate an expected (invalid address) output. On top of that it looks like a new llvm-addr2line changed the "invalid address" pattern format. This patch adds a new "invalid input" pattern to a dummy_pattern and changes the way we generate such an output by pushing an empty line instead of a 0x0 address. Ref scylladb#2609
addr2line.py invokes "addr2line" tool in a "server" mode where addresses are fed via stdin. It intentionally makes a corresponding "addr2line" tool generate a known "invalid address" pattern at the end of each address decoding by feeding it a "0x0" address in order to denote the end of it because a single address can be decoded into multiple lines due to inlining. The above strategy assumes that 0x0 address is always going to be interpreted as an invalid one by both add2line and by llvm-addr2line. However, at least llvm-addr2line 18.1.3 doesn't always interpret it this way. On the other hand an empty line does always generate an expected (invalid address) output. On top of that it looks like a new llvm-addr2line changed the "invalid address" pattern format. This patch adds a new "invalid input" pattern to a dummy_pattern and changes the way we generate such an output by pushing an empty line instead of a 0x0 address. Ref scylladb#2609
Sorry how would that help? If the |
addr2line.py invokes "addr2line" tool in a "server" mode where addresses are fed via stdin. It intentionally makes a corresponding "addr2line" tool generate a known "invalid address" pattern at the end of each address decoding by feeding it a "0x0" address in order to denote the end of it because a single address can be decoded into multiple lines due to inlining. The above strategy assumes that 0x0 address is always going to be interpreted as an invalid one by both add2line and by llvm-addr2line. However, at least llvm-addr2line 18.1.3 doesn't always interpret it this way. On the other hand an empty line does always generate an expected (invalid address) output. On top of that it looks like a new llvm-addr2line changed the "invalid address" pattern format. This patch adds a new "invalid input" pattern to a dummy_pattern and changes the way we generate such an output by pushing line that only has a ',' charachter instead of a 0x0 address. Ref scylladb#2609
addr2line.py invokes "addr2line" tool in a "server" mode where addresses are fed via stdin. It intentionally makes a corresponding "addr2line" tool generate a known "invalid address" pattern at the end of each address decoding by feeding it a "0x0" address in order to denote the end of it because a single address can be decoded into multiple lines due to inlining. The above strategy assumes that 0x0 address is always going to be interpreted as an invalid one by both add2line and by llvm-addr2line. However, at least llvm-addr2line 18.1.3 doesn't always interpret it this way. On the other hand an empty line does always generate an expected (invalid address) output. On top of that it looks like a new llvm-addr2line changed the "invalid address" pattern format. This patch adds a new "invalid input" pattern to a dummy_pattern and changes the way we generate such an output by pushing line that only has a ',' charachter instead of a 0x0 address. Ref scylladb#2609
addr2line.py invokes "addr2line" tool in a "server" mode where addresses are fed via stdin. It intentionally makes a corresponding "addr2line" tool generate a known "invalid address" pattern at the end of each address decoding by feeding it a "0x0" address in order to denote the end of it because a single address can be decoded into multiple lines due to inlining. The above strategy assumes that 0x0 address is always going to be interpreted as an invalid one by both add2line and by llvm-addr2line. However, at least llvm-addr2line 18.1.3 doesn't always interpret it this way. On the other hand an empty line does always generate an expected (invalid address) output. On top of that it looks like a new llvm-addr2line changed the "invalid address" pattern format. This patch adds a new "invalid input" pattern to a dummy_pattern and changes the way we generate such an output by pushing line that only has a ',' charachter instead of a 0x0 address. Ref scylladb#2609
That's a valid point. I never encountered such a long delays with I did see that the binutils Anyway, on the PR above we seem to have progressed in working around the issue with The |
I only saw it that slow with During that part it would do some O(n^2) loop on the number of debug entries (of some type) which could be in the millions for highly inlined seastar application. Even when you don't hit that it's a low slower, which is why I added llvm-addr2line support (this also lets you decode binaries with newer DWARF versions). This one was at least partly fixed in later binutils versions.
I see. I have had success filing bugs against binutils for these issues, but in any case you still need to work around them in the meantime. |
addr2line.py invokes "addr2line" tool in a "server" mode where addresses are fed via stdin. It intentionally makes a corresponding "addr2line" tool generate a known "invalid address" pattern at the end of each address decoding by feeding it a "0x0" address in order to denote the end of it because a single address can be decoded into multiple lines due to inlining. The above strategy assumes that 0x0 address is always going to be interpreted as an invalid one by both add2line and by llvm-addr2line. However, at least llvm-addr2line 18.1.3 doesn't always interpret it this way. On the other hand an empty line does always generate an expected (invalid address) output. On top of that it looks like a new llvm-addr2line changed the "invalid address" pattern format. This patch adds a new "invalid input" pattern to a dummy_pattern and changes the way we generate such an output by pushing line that only has a ',' charachter instead of a 0x0 address. Ref #2609 Closes #2611
HEAD: ff24926
OS: Ubuntu 24.04
binutils version:
2.42-4ubuntu2.3LLVM version:
18.0-59~exp2Python version:
3.12.3-0ubuntu2Description
The most trivial invocation of
seastar-addr2linehangs on I/O (usingaddr2linefrombinutilsgives the same result):Interrupting it prints the following:
At the same time using
llvm-addr2linedirectly works just fine:The text was updated successfully, but these errors were encountered: