gRPC + Linkerd mirroring doesn't work #3002
Comments
|
Hi, {
"feature": {
"network": {
"outgoing": {
"ignore_localhost": true
}
}
}
}if that doesn't solve - feel free to book a debug session with me here: https://calendar.app.google/JihFZSDkbe8etmt37 |
|
My mirrord.json is extremely minimal, just tells bazel to resolve runfiles locally rather than remotely: Tried to add your suggestion but still not seeing traffic forwarded. Scheduled a slot at 1pm PST Wednesday, thanks for your help! |
|
Checking more closely with some c++ services as well, i believe our service mesh (linkerd) is potentially causing issues here, im noticing that traffic to unmeshed pods is being forwarded properly, but i dont see traffic from meshed pods |
aviramha
changed the title
|
Debugged with the user- stealing works, mirroring doesn't |
|
Related |
Bug Description
We have several languages and golang and c++ appear to be working basically out of the box. However while our python services start and seem to run fine, they don't see any grpc traffic forwarded.
I tried a couple different approaches, running with
python ...and compiling using cython (which we leverage to improve performance) to create an executable, but both hit this same root issue. Runningfileon the cythonized binary gives:Couldnt find any issues like this one, curious if this pattern-matches anything you've seen before. It's curious that it only happens to python. If it helps, we're compiling / running using bazel
Steps to Reproduce
I'm hoping this matches something you've seen before - our setup is quite complicated so a reproducible example might be tricky but can do this if it would be helpful
Backtrace
No response
mirrord layer logs
There are a lot, including what i feel is relevant:
On startup
After the service begins, I repeatedly see this log. I expect the service to be serving on :10010
�[2m2025-01-03T00:18:08.421680Z�[0m �[35mTRACE�[0m ThreadId(03) �[2mmirrord_layer::socket::ops�[0m�[2m:�[0m in connect LazyLock( Mutex { data: { 12: UserSocket { domain: 2, type_: 1, protocol: 0, state: Listening( Bound { requested_address: 0.0.0.0:10010, address: 0.0.0.0:10010, }, ), kind: Tcp( 1, ), }, 16: UserSocket { domain: 1, type_: 524289, protocol: 0, state: Initialized, kind: Tcp( 524289, ), }, 9: UserSocket { domain: 2, type_: 524289, protocol: 0, state: Listening( Bound { requested_address: 0.0.0.0:9000, address: 0.0.0.0:9000, }, ), kind: Tcp( 524289, ), }, 15: UserSocket { domain: 2, type_: 524289, protocol: 0, state: Connected( Connected { remote_address: Ip( 52.218.185.57:443, ), local_address: Ip( 10.10.39.115:42654, ), layer_address: Some( Ip( 127.0.0.1:43875, ), ), }, ), kind: Tcp( 524289, ), }, 13: UserSocket { domain: 2, type_: 524289, protocol: 0, state: Connected( Connected { remote_address: Ip( 169.254.169.254:80, ), local_address: Ip( 10.10.39.115:59038, ), layer_address: Some( Ip( 127.0.0.1:35321, ), ), }, ), kind: Tcp( 524289, ), }, 14: UserSocket { domain: 2, type_: 1, protocol: 0, state: Listening( Bound { requested_address: 0.0.0.0:10000, address: 0.0.0.0:10000, }, ), kind: Tcp( 1, ), }, }, poisoned: false, .. }, )mirrord intproxy logs
2025-01-03T00:16:50.400598Z INFO mirrord_intproxy::layer_initializer: new session, process_info: ProcessInfo { pid: 1177389, name: "..._ccbin", cmdline: ["..._ccbin", "--run_dir", "/rundir", "--resolved_workspace", "/root/workspace/workspace.pb.txt", "--log_level", "INFO"], loaded: true } at mirrord/intproxy/src/layer_initializer.rs:63 2025-01-03T00:17:13.243504Z INFO mirrord_intproxy::layer_initializer: new session, process_info: ProcessInfo { pid: 1177490, name: "uname", cmdline: ["uname", "-p"], loaded: true } at mirrord/intproxy/src/layer_initializer.rs:63mirrord agent logs
mirrord config
{ "agent": { "privileged": true }, "experimental": { "trust_any_certificate": true }, "feature": { "fs": { "local": [ ".*.runfiles\/.*", ".*.runfiles_manifest$" ] } } }mirrord CLI version
3.128.0
mirrord-agent version
3.128.0
mirrord-operator version (if relevant)
No response
plugin kind and version (if relevant)
No response
Your operating system and version
Ubuntu 22.04.2 LTS
Local process
ELF 64-bit LSB pie executable, x86-64, version 1 (SYSV), dynamically linked, interpreter /lib64/ld-linux-x86-64.so.2, for GNU/Linux 3.2.0, BuildID[sha1]=a83e70e25eebc70926938d868e78b474efaa369c, not stripped
Local process version
No response
Additional Info
No response
The text was updated successfully, but these errors were encountered: