Re: TCP network cost

On Mon, Feb 16, 2009 at 11:04 PM, Ross J. Reedstrom <reedstrm(at)rice(dot)edu> wrote:
> Recently I've been working on improving the performance of a system that
> delivers files stored in postgresql as bytea data. I was surprised at
> just how much a penalty I find moving from a domain socket connection to
> a TCP connection, even localhost. For one particular 40MB file (nothing
> outragous) I see ~ 2.5 sec. to download w/ the domain socket, but ~ 45 sec
> for a TCP connection (either localhost, name of localhost, or from
> another machine 5 hops away (on campus - gigabit LAN) Similar numbers
> for 8.2.3 or 8.3.6 (on Linux/Debian etch + backports)
>
> So, why the 20 fold penalty for using TCP? Any clues on how to trace
> what's up in the network IO stack?

TCP has additional overhead as well as going through the IP stack
which for non-tuned Linux kernels is pretty limiting.

long story short, there are things in /proc you can use to increase
buffers and window sizes which will help with large TCP streams (like
a 40MB file for example). There's a lot of documentation on the net
for how to tune the Linux IP stack so I won't repeat it here.

Now, having your DB box 5 hops away is going to add a lot of latency
and any packet loss is going to kill TCP throughput- especially if you
increase window sizes. I'd recommend something like "mtr" to map the
network traffic (make sure you run it both ways in case you have an
asymmetric routing situation) for a long period of time to look for
hiccups.

--
Aaron Turner
http://synfin.net/
http://tcpreplay.synfin.net/ - Pcap editing and replay tools for Unix & Windows
Those who would give up essential Liberty, to purchase a little
temporary Safety,
deserve neither Liberty nor Safety.
-- Benjamin Franklin