Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

port-forwarding takes time to be stable with MTU > 49192 on kernel >= 4.20 #128

Open
AkihiroSuda opened this issue Aug 1, 2019 · 2 comments
Open

port-forwarding takes time to be stable with MTU > 49192 on kernel >= 4.20 #128

AkihiroSuda opened this issue Aug 1, 2019 · 2 comments
Labels
bug Something isn't working help wanted Extra attention is needed

Comments

@AkihiroSuda
Copy link
Member

AkihiroSuda commented Aug 1, 2019 •
edited
Loading 

$ MTU=65520 ./benchmarks/benchmark-iperf3-reverse.sh
...
[  5] local 127.0.0.1 port 35852 connected to 127.0.0.1 port 15201
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec  5.00 MBytes  41.9 Mbits/sec    3   1.31 MBytes
[  5]   1.00-2.00   sec  0.00 Bytes  0.00 bits/sec    0   1.31 MBytes
[  5]   2.00-3.00   sec  0.00 Bytes  0.00 bits/sec    0   1.31 MBytes
[  5]   3.00-4.00   sec  0.00 Bytes  0.00 bits/sec    0   1.31 MBytes
[  5]   4.00-5.00   sec  0.00 Bytes  0.00 bits/sec    0   1.31 MBytes
[  5]   5.00-6.00   sec   206 MBytes  1.73 Gbits/sec    0   1.81 MBytes
[  5]   6.00-7.00   sec   780 MBytes  6.55 Gbits/sec    0   1.81 MBytes
[  5]   7.00-8.00   sec   732 MBytes  6.14 Gbits/sec    0   1.81 MBytes
[  5]   8.00-9.00   sec   828 MBytes  6.94 Gbits/sec    0   1.81 MBytes
[  5]   9.00-10.00  sec   896 MBytes  7.52 Gbits/sec    0   1.81 MBytes
[  5]  10.00-11.00  sec  1024 MBytes  8.59 Gbits/sec    0   1.81 MBytes
[  5]  11.00-12.00  sec  1019 MBytes  8.55 Gbits/sec    0   1.81 MBytes
[  5]  12.00-13.00  sec  1.05 GBytes  8.99 Gbits/sec    0   1.81 MBytes
[  5]  13.00-14.00  sec   758 MBytes  6.35 Gbits/sec    0   1.81 MBytes
[  5]  14.00-15.00  sec   824 MBytes  6.91 Gbits/sec    0   1.81 MBytes
[  5]  15.00-16.00  sec   684 MBytes  5.73 Gbits/sec    1   1.81 MBytes
[  5]  16.00-17.00  sec   886 MBytes  7.44 Gbits/sec    0   1.81 MBytes
[  5]  17.00-18.00  sec   898 MBytes  7.53 Gbits/sec    0   1.81 MBytes
[  5]  18.00-19.00  sec   882 MBytes  7.40 Gbits/sec    0   1.81 MBytes
[  5]  19.00-20.00  sec   866 MBytes  7.27 Gbits/sec    0   1.81 MBytes
[  5]  20.00-21.00  sec   665 MBytes  5.58 Gbits/sec    1   1.81 MBytes
[  5]  21.00-22.00  sec   844 MBytes  7.08 Gbits/sec    0   1.81 MBytes
[  5]  22.00-23.00  sec   771 MBytes  6.47 Gbits/sec    0   1.81 MBytes
[  5]  23.00-24.00  sec   779 MBytes  6.54 Gbits/sec    0   1.81 MBytes
[  5]  24.00-25.00  sec   808 MBytes  6.77 Gbits/sec    0   1.81 MBytes
[  5]  25.00-26.00  sec   779 MBytes  6.53 Gbits/sec    0   1.81 MBytes
[  5]  26.00-27.00  sec   851 MBytes  7.14 Gbits/sec    0   1.87 MBytes
[  5]  27.00-28.00  sec   730 MBytes  6.12 Gbits/sec    0   1.87 MBytes
[  5]  28.00-29.00  sec   835 MBytes  7.00 Gbits/sec    0   1.87 MBytes
[  5]  29.00-30.00  sec   861 MBytes  7.23 Gbits/sec    0   2.06 MBytes
[  5]  30.00-31.00  sec   901 MBytes  7.56 Gbits/sec    0   2.06 MBytes
[  5]  31.00-32.00  sec   749 MBytes  6.28 Gbits/sec    0   2.06 MBytes
[  5]  32.00-33.00  sec   830 MBytes  6.96 Gbits/sec    0   2.06 MBytes
[  5]  33.00-34.00  sec   839 MBytes  7.04 Gbits/sec    0   2.06 MBytes
[  5]  34.00-35.00  sec   852 MBytes  7.15 Gbits/sec    0   2.06 MBytes
[  5]  35.00-36.00  sec   850 MBytes  7.13 Gbits/sec    0   2.06 MBytes
[  5]  36.00-37.00  sec   889 MBytes  7.46 Gbits/sec    0   2.19 MBytes
[  5]  37.00-38.00  sec   931 MBytes  7.81 Gbits/sec    0   2.19 MBytes
[  5]  38.00-39.00  sec   881 MBytes  7.39 Gbits/sec    0   2.19 MBytes
[  5]  39.00-40.00  sec   811 MBytes  6.81 Gbits/sec    0   2.19 MBytes
[  5]  40.00-41.00  sec   806 MBytes  6.76 Gbits/sec    0   2.19 MBytes
[  5]  41.00-42.00  sec   790 MBytes  6.63 Gbits/sec    0   2.19 MBytes
[  5]  42.00-43.00  sec   788 MBytes  6.61 Gbits/sec    0   2.19 MBytes
[  5]  43.00-44.00  sec   925 MBytes  7.76 Gbits/sec    0   2.19 MBytes
[  5]  44.00-45.00  sec   825 MBytes  6.92 Gbits/sec    0   2.19 MBytes
[  5]  45.00-46.03  sec   642 MBytes  5.22 Gbits/sec    1   2.19 MBytes
[  5]  46.03-47.00  sec   524 MBytes  4.54 Gbits/sec    0   2.19 MBytes
[  5]  47.00-48.00  sec   555 MBytes  4.66 Gbits/sec    0   2.19 MBytes
[  5]  48.00-49.00  sec   778 MBytes  6.52 Gbits/sec    0   2.19 MBytes
[  5]  49.00-50.00  sec   739 MBytes  6.20 Gbits/sec    0   2.19 MBytes
[  5]  50.00-51.00  sec   821 MBytes  6.89 Gbits/sec    0   2.25 MBytes
[  5]  51.00-52.00  sec   941 MBytes  7.90 Gbits/sec    0   2.25 MBytes
[  5]  52.00-53.00  sec   721 MBytes  6.05 Gbits/sec    0   2.25 MBytes
[  5]  53.00-54.00  sec   891 MBytes  7.48 Gbits/sec    0   2.25 MBytes
[  5]  54.00-55.00  sec   805 MBytes  6.75 Gbits/sec    0   2.25 MBytes
[  5]  55.00-56.00  sec   940 MBytes  7.89 Gbits/sec    0   2.25 MBytes
[  5]  56.00-57.00  sec   730 MBytes  6.12 Gbits/sec    0   2.25 MBytes
[  5]  57.00-58.00  sec   804 MBytes  6.74 Gbits/sec    0   2.25 MBytes
[  5]  58.00-59.00  sec   835 MBytes  7.00 Gbits/sec    0   2.25 MBytes
[  5]  59.00-60.00  sec   904 MBytes  7.58 Gbits/sec    0   2.25 MBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-60.00  sec  43.5 GBytes  6.23 Gbits/sec    6             sender
[  5]   0.00-60.00  sec  43.5 GBytes  6.23 Gbits/sec                  receive

Bad bandwidth in the first few seconds when MTU > 49192.

This seems related to #define TCP_MAXSEG_MAX 32768

slirp4netns/vendor/libslirp/src/tcp.h

Line 46 in ed51817

#define TCP_MAXSEG_MAX 32768

Modifying the TCP_MAXSEG_MAX value affects the threshold MTU to hit this issue.

Tested commit: ed51817

Reproducible envs:

  • Fedora 30, kernel 4.20.0-1.fc30.x86_64 #1 (VirtualBox)
  • Ubuntu 19.04, kernel 5.0.0-23-generic #24-Ubuntu (VMware Fusion)
  • Ubuntu 19.04, kernel 5.0.0-1011-gcp #11-Ubuntu (GCE n1-standard-2)
  • Fedora 30, kernel 5.0.9-301.fc30.x86_64 #1 (VirtualBox)
  • Fedora 30, kernel 5.1.20-300.fc30.x86_64 #1 (VirtualBox)

Unreproducible envs:

  • Ubuntu 14.04, kernel 4.4.0-101-generic (Travis)
  • Debian 9.8, kernel 4.9.0-8-amd64 #1 (GCE n1-standard-2)
  • Ubuntu 18.04, kernel 4.15.0-1037-gcp #39-Ubuntu (GCE n1-standard-2)
  • Fedora 29, kernel 4.18.16-300.fc29.x86_64 #1 (VirtualBox)
  • Debian 10, kernel 4.19.0-5-cloud-amd64 #1 (GCE n1-standard-2)

Seems related to the kernel version (>= 4.20)
@AkihiroSuda
Copy link
Member Author

AkihiroSuda commented Aug 1, 2019 •
edited
Loading

It is weird that the issue only happens in kernel >= 4.20, but as the TCP_MAXSEG_MAX value affects the threshold MTU to his the issue, probably this issue is on slirp4netns side, not on kernel side.

@AkihiroSuda AkihiroSuda added bug Something isn't working help wanted Extra attention is needed labels Aug 1, 2019
@AkihiroSuda AkihiroSuda changed the title Unstable port-forwarding with MTU > 49192 and recent kernel Unstable port-forwarding with MTU > 49192 on recent kernel Aug 1, 2019
@AkihiroSuda
Copy link
Member Author

AkihiroSuda commented Aug 1, 2019 •
edited
Loading

https://kernelnewbies.org/Linux_4.20#Networking

Suspicious commit (unconfirmed): tcp: up initial rmem to 128KB and SYN rwin to around 64KB torvalds/linux@a337531

@AkihiroSuda AkihiroSuda changed the title Unstable port-forwarding with MTU > 49192 on recent kernel Unstable port-forwarding with MTU > 49192 on kernel >= 4.20 Aug 1, 2019
@AkihiroSuda AkihiroSuda changed the title Unstable port-forwarding with MTU > 49192 on kernel >= 4.20 port-forwarding takes time to be stable with MTU > 49192 on kernel >= 4.20 Aug 1, 2019
AkihiroSuda added a commit to AkihiroSuda/slirp4netns that referenced this issue Aug 20, 2019
@AkihiroSuda
man: add workaround for the port-fowarding performance issue w/ kerne…
91bd8bc 
…l 4.20

The kernel 4.20 bumped up the default value of `/proc/sys/net/ipv4/tcp_rmem` from 87380 to 131072.
This is known to slow down slirp4netns port forwarding: rootless-containers#128

As a workaround, it is recommended to modify `/proc/sys/net/ipv4/tcp_rmem` inside the namespace manually.
(No real root privilege is needed)

Update rootless-containers#128

Signed-off-by: Akihiro Suda <[email protected]>
@AkihiroSuda AkihiroSuda mentioned this issue Aug 20, 2019
Closed
AkihiroSuda added a commit to AkihiroSuda/slirp4netns that referenced this issue Aug 20, 2019
@AkihiroSuda
man: add workaround for the port-fowarding performance issue w/ kerne…
0f59fc9 
…l 4.20

The kernel 4.20 bumped up the default value of `/proc/sys/net/ipv4/tcp_rmem` from 87380 to 131072.
This is known to slow down slirp4netns port forwarding: rootless-containers#128

As a workaround, it is recommended to modify `/proc/sys/net/ipv4/tcp_rmem` inside the namespace manually.
(No real root privilege is needed)

Update rootless-containers#128

Signed-off-by: Akihiro Suda <[email protected]>
AkihiroSuda added a commit to AkihiroSuda/slirp4netns that referenced this issue Aug 20, 2019
@AkihiroSuda
man: add workaround for the port-fowarding performance issue w/ kerne…
d29e6b0 
…l 4.20

The kernel 4.20 bumped up the default value of `/proc/sys/net/ipv4/tcp_rmem` from 87380 to 131072.
torvalds/linux@a337531

This is known to slow down slirp4netns port forwarding: rootless-containers#128

As a workaround, it is recommended to modify `/proc/sys/net/ipv4/tcp_rmem` inside the namespace manually.
(No real root privilege is needed)

Update rootless-containers#128

Signed-off-by: Akihiro Suda <[email protected]>
AkihiroSuda added a commit to AkihiroSuda/slirp4netns that referenced this issue Aug 20, 2019
@AkihiroSuda
add workaround for the port-fowarding performance issue w/ kernel 4.20
f014fd0 
The kernel 4.20 bumped up the default value of `/proc/sys/net/ipv4/tcp_rmem` from 87380 to 131072.
torvalds/linux@a337531

This is known to slow down slirp4netns port forwarding: rootless-containers#128

As a workaround, it is recommended to modify `/proc/sys/net/ipv4/tcp_rmem` inside the namespace.

The file is automatically modified when running with `--configure`.

Update rootless-containers#128

Signed-off-by: Akihiro Suda <[email protected]>
@giuseppe giuseppe mentioned this issue Oct 21, 2019
Closed
@AkihiroSuda AkihiroSuda mentioned this issue Feb 2, 2021
Merged
@rlpowell rlpowell mentioned this issue Feb 15, 2022
Open
@paulcostinean paulcostinean mentioned this issue Nov 12, 2022
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug Something isn't working help wanted Extra attention is needed
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@AkihiroSuda