Re: Receive buffer size for the statistics socket

On 05/14/2017 09:54 PM, Tom Lane wrote:
> Also, it's clear that a session could easily shove much more than 8KB at
> a time out to the stats collector, because what we're doing in the stats
> test does not involve touching any very large number of tables. So I
> think this is not just a test failure but is telling us about a plausible
> mechanism for real-world statistics drops.
>
> I observe a default receive buffer size around 124K on my Linux box,
> which seems like it'd be far less prone to overflow than 8K.
>
> I propose that it'd be a good idea to try to set the stats socket's
> receive buffer size to be a minimum of, say, 100K on all platforms.
> Code for this would be analogous to what we already have in pqcomm.c
> (circa line 760) for forcing up the send buffer size, but SO_RCVBUF
> not SO_SNDBUF.

Seems reasonable.

> A further idea is that maybe backends should be tweaked to avoid
> blasting large amounts of data at the stats collector in one go.
> That would require more thought to design, though.

The data is already sent in small < 1 kB messages, I don't see what more
we can do in the sender side to avoid overwhelming the receiver. Except
reduce the amount of data sent overall. But that only goes so far, we
cannot eliminate the problem altogether, unless we also lose some detail.

It might nevertheless be worthwhile to reduce the overall volume. It
would avoid some overhead, even if the buffer is large enough, although
I don't remember pgstat being significant in any profiling I've done.
One thing that caught my eye at a quick glance is that we are sending
the # of tuples inserted/deleted/updated counters, even for read-only
cases. It seems straightforward to detect that special case, and use an
abbreviated version of PgStat_MsgTabstat without those counters, when we
haven't updated anything. But again, might not be worth the trouble.

- Heikki