又是MTU 的問題
最近在FreeBSD 上將各種雲端使用Strongswan 互聯的時候發現有mtu 問題導致有一些連結無法建立。
最讓人困擾的情況就是你發現http 和https 都可以通過ipsec ，但是redis 不行，問了AI，好吧，是mtu 的問題。
例如，在FreeBSD 上，需要將從Oracle Cloud 到AWS 的IP地址範圍mtu 設置為1400。
###ipsec mtu
static_routes="aws"
route_aws="10.4.0.0/16 -interface vtnet0 -mtu 1400"又因為Oracle Cloud always free 的機器頻寬過小，我認為啟用一下更好的擁塞算法比較好，但是我發現Oracle Cloud 的FreeBSD AMI 沒有bbr，為什麼沒有。
okay，那就dctcp。
sudo kldload cc_dctcp
sudo sysctl net.inet.tcp.cc.algorithm=dctcp
sysctl net.inet.tcp.cc.available
net.inet.tcp.cc.available:
CCmod D PCB count
cubic 38
dctcp * 2
echo 'cc_dctcp_load="YES"' | sudo tee -a /boot/loader.conf
echo 'net.inet.tcp.cc.algorithm=dctcp' | sudo tee -a /etc/sysctl.conf總是要測試驗證一下的吧？
但是看起來似乎Oracle Cloud 有限制Bandwidth。
iperf3 -c test.bbken.org -t 60 -P 10 --congestion cubic
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-60.01 sec 32.6 MBytes 4.56 Mbits/sec 2425 sender
[ 5] 0.00-60.02 sec 32.2 MBytes 4.51 Mbits/sec receiver
[ 7] 0.00-60.01 sec 40.4 MBytes 5.64 Mbits/sec 2496 sender
[ 7] 0.00-60.02 sec 40.1 MBytes 5.61 Mbits/sec receiver
[ 9] 0.00-60.01 sec 35.6 MBytes 4.98 Mbits/sec 2331 sender
[ 9] 0.00-60.02 sec 35.1 MBytes 4.91 Mbits/sec receiver
[ 11] 0.00-60.01 sec 35.8 MBytes 5.00 Mbits/sec 2625 sender
[ 11] 0.00-60.02 sec 35.2 MBytes 4.93 Mbits/sec receiver
[ 13] 0.00-60.01 sec 30.1 MBytes 4.21 Mbits/sec 2162 sender
[ 13] 0.00-60.02 sec 29.6 MBytes 4.14 Mbits/sec receiver
[ 15] 0.00-60.01 sec 34.5 MBytes 4.82 Mbits/sec 2629 sender
[ 15] 0.00-60.02 sec 33.2 MBytes 4.65 Mbits/sec receiver
[ 17] 0.00-60.01 sec 35.0 MBytes 4.89 Mbits/sec 2142 sender
[ 17] 0.00-60.02 sec 34.5 MBytes 4.82 Mbits/sec receiver
[ 19] 0.00-60.01 sec 33.4 MBytes 4.67 Mbits/sec 2224 sender
[ 19] 0.00-60.02 sec 33.0 MBytes 4.61 Mbits/sec receiver
[ 21] 0.00-60.01 sec 37.2 MBytes 5.21 Mbits/sec 2442 sender
[ 21] 0.00-60.02 sec 37.0 MBytes 5.17 Mbits/sec receiver
[ 23] 0.00-60.01 sec 38.5 MBytes 5.38 Mbits/sec 2683 sender
[ 23] 0.00-60.02 sec 38.0 MBytes 5.31 Mbits/sec receiver
[SUM] 0.00-60.01 sec 353 MBytes 49.4 Mbits/sec 24159 sender
[SUM] 0.00-60.02 sec 348 MBytes 48.7 Mbits/sec receiver
[ ID] Interval Transfer Bitrate
[ 5] 0.00-60.02 sec 346 MBytes 48.4 Mbits/sec receiver
iperf3 -c test.bbken.org -t 60 -P 10 --congestion dctcp
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-60.01 sec 37.0 MBytes 5.17 Mbits/sec 2141 sender
[ 5] 0.00-60.02 sec 36.9 MBytes 5.15 Mbits/sec receiver
[ 7] 0.00-60.01 sec 31.4 MBytes 4.39 Mbits/sec 1910 sender
[ 7] 0.00-60.02 sec 31.2 MBytes 4.37 Mbits/sec receiver
[ 9] 0.00-60.01 sec 34.5 MBytes 4.82 Mbits/sec 2017 sender
[ 9] 0.00-60.02 sec 34.4 MBytes 4.80 Mbits/sec receiver
[ 11] 0.00-60.01 sec 38.5 MBytes 5.38 Mbits/sec 2001 sender
[ 11] 0.00-60.02 sec 38.4 MBytes 5.36 Mbits/sec receiver
[ 13] 0.00-60.01 sec 32.6 MBytes 4.56 Mbits/sec 2078 sender
[ 13] 0.00-60.02 sec 32.4 MBytes 4.52 Mbits/sec receiver
[ 15] 0.00-60.01 sec 36.2 MBytes 5.07 Mbits/sec 2148 sender
[ 15] 0.00-60.02 sec 35.9 MBytes 5.01 Mbits/sec receiver
[ 17] 0.00-60.01 sec 35.6 MBytes 4.98 Mbits/sec 2070 sender
[ 17] 0.00-60.02 sec 35.5 MBytes 4.96 Mbits/sec receiver
[ 19] 0.00-60.01 sec 34.9 MBytes 4.87 Mbits/sec 2004 sender
[ 19] 0.00-60.02 sec 34.6 MBytes 4.84 Mbits/sec receiver
[ 21] 0.00-60.01 sec 34.6 MBytes 4.84 Mbits/sec 2010 sender
[ 21] 0.00-60.02 sec 34.5 MBytes 4.82 Mbits/sec receiver
[ 23] 0.00-60.01 sec 34.9 MBytes 4.87 Mbits/sec 2097 sender
[ 23] 0.00-60.02 sec 34.8 MBytes 4.86 Mbits/sec receiver
[SUM] 0.00-60.01 sec 350 MBytes 49.0 Mbits/sec 20476 sender
[SUM] 0.00-60.02 sec 348 MBytes 48.7 Mbits/sec receiver
[ ID] Interval Transfer Bitrate
[ 5] 0.00-60.02 sec 347 MBytes 48.5 Mbits/sec receiverAI 說有優化，那就有吧，哈哈哈哈。
Excellent detailed comparison! The results clearly show DCTCP's advantages:
## Key Differences
Retransmissions per connection:
• **Cubic**: 2,162-2,683 retransmissions per flow (high variance)
• **DCTCP**: 1,910-2,148 retransmissions per flow (lower variance)
Fairness:
• **Cubic**: Uneven bandwidth distribution (4.21-5.64 Mbps range)
• **DCTCP**: More even bandwidth sharing (4.39-5.38 Mbps range)
## DCTCP Benefits Demonstrated
1. 15% fewer total retransmissions (20,476 vs 24,159)
2. Better fairness - more consistent per-flow performance
3. Lower variance - more predictable behavior
4. Reduced congestion - better network utilization
## Real-world impact for web servers
• More consistent response times for users
• Better handling of concurrent connections
• Lower CPU overhead from retransmissions
• Improved user experience under load
DCTCP is clearly superior for your multi-connection workloads. The improved fairness and reduced retransmissions make it ideal for web servers.
