Troubleshooting large file transfers across wide area networks

Flow control troubleshooting

Solaris

http://www.princeton.edu/~unix/Solaris/troubleshoot/netstat.html

 # netstat -i
 Name  Mtu  Net/Dest      Address        Ipkts  Ierrs Opkts  Oerrs Collis Queue 
 lo0   8232 localhost     localhost      11982693 0     11982693 0     0      0     
 nxge1 1500 wdcdvuv10-adm.hmco.com wdcdvuv10-adm.hmco.com 129186999 0  258305840 0     0   0     
 nxge2 1500 wdcdvuv10-35.hmco.com wdcdvuv10-35.hmco.com 1116133305 0     1099063045 0  0  0     
 nxge130000 1500 wdcdvuv10-130 wdcdvuv10-130  247847548 0     180990116 0     0      0     
 nxge132000 1500 wdcdvuv10.hmco.com wdcdvuv10.hmco.com 247847548 0     180990116 0     0      0

Check only the following columns: <Input errors> <Output errors> <Collisions> <Queue length>\\ Run on wdcdvuv10 (global zone) to check interface for zones

 # netstat -i | grep nxge2 | awk '{ print $6 " " $8 " " $9 " " $10 }'
 0 0 0 0

Should all be zero if all is working well.

http://www.brendangregg.com/Perf/network.html

 # kstat -p | grep nxge2
 # kstat -p | grep "tcp:"

Flow Control monitoring

 # date; kstat -p 'tcp:0:tcpstat:tcp_flwctl_on'            
 tcp:0:tcpstat:tcp_flwctl_on     16330314

Improving flow control

http://www.psc.edu/networking/projects/tcptune/#Solaris

 # ndd /dev/tcp tcp_wscale_always  #(should be 1)
 # ndd /dev/tcp tcp_tstamp_if_wscale  #(should be 1)
 # ndd /dev/tcp tcp_sack_permitted  #(should be 2)

In Solaris 10: Use "ndd "set /dev/tcp tcp_wscale_always 1" This should be enabled by default.

Set/get the maximum (send or receive) TCP buffer size an application can request:

 # ndd -get /dev/tcp tcp_max_buf
 4000000
 # ndd -set /dev/tcp tcp_max_buf 4000000

Set/get the maximum congestion window:

 # ndd -get /dev/tcp tcp_cwnd_max
 4000000
 # ndd -set /dev/tcp tcp_cwnd_max 4000000

Set/get the default send and receive buffer sizes:

 # ndd -get /dev/tcp tcp_xmit_hiwat
 4000000
 # ndd -get /dev/tcp tcp_recv_hiwat
 4000000
 # ndd -set /dev/tcp tcp_xmit_hiwat 4000000
 # ndd -set /dev/tcp tcp_recv_hiwat 4000000

Update: 2010-11-10 7:45am ET:

I have now upped them to 1GB (temporarily) to see if this helps any:

 inscoek@wdcdvuw15:/home/inscoek> ./get_buffers.sh 
 Current TCP buffer settings on abcdvuw15 at Wednesday, November 10, 2010  7:48:26 AM EST
 
 tcp_max_buf:
 1000000000
 tcp_cwnd_max:
 1000000000
 tcp_xmit_hiwat:
 1000000000
 tcp_recv_hiwat
 1000000000

http://unix.derkeiler.com/Newsgroups/comp.unix.solaris/2006-05/msg01029.html

on the 6800...

 # route change <route-to-15K> -sendpipe 403456
 # route change <route-to-15K> -recvpipe 403456
 # route get <route-to-15K>

on the 15K...

 # route change <route-to-6800> -sendpipe 403456
 # route change <route-to-6800> -recvpipe 403456
 # route get <route-to-6800>

Calculate the window more or less as:

 (44*1024*1024*.070)/8 = 403456
 ------------ --- - ------
 | | | |
 | | | ------- window size in bytes
 | | ------------- bits/byte
 | ----------------- delay (rtt)
 --------------------------- ~bandwidth of ds3 (bits/sec)

(don't know if bandwidth should be expressed as powers of 2 or 10) but 44Mb/s is the right ball park for ds3. Perhaps someone else can clarify.

check the values for these using ndd.

 /dev/tcp tcp_max_buf
 /dev/tcp tcp_wscale_always
 /dev/tcp tcp_tstamp_if_wscale
 /dev/tcp tcp_deferred_acks_max
 /dev/tcp tcp_compression_enabled

Window scaling is negotiated only in the 3-way handshake so far as I know. The window size may change, but scaling should not once the connection is established. you might try increasing tcp_deferred_acks_max in case exceeding this causes the window to shrink. Also just an sysadmin. Good luck.

Tools

TCP Windows calculator - http://www.kehlet.cx/docs/tcpwin.php

Troubleshooting large file transfers across wide area networks

Contents

Flow control troubleshooting

Solaris

Flow Control monitoring

Improving flow control

Tools

Further reading

Navigation menu