Re: performance for high-volume log insertion

On Mon, 20 Apr 2009, Greg Smith wrote:

> On Mon, 20 Apr 2009, david(at)lang(dot)hm wrote:
>
>> any idea what sort of difference binary mode would result in?
>
> The win from switching from INSERT to COPY can be pretty big, further
> optimizing to BINARY you'd really need to profile to justify. I haven't
> found any significant difference in binary mode compared to overhead of the
> commit itself in most cases. The only thing I consistently run into is that
> timestamps can bog things down considerably in text mode, but you have to be
> pretty efficient in your app to do any better generating those those in the
> PostgreSQL binary format yourself. If you had a lot of difficult to parse
> data types like that, binary might be a plus, but it doesn't sound like that
> will be the case for what you're doing.
>
> But you don't have to believe me, it's easy to generate a test case here
> yourself. Copy some typical data into the database, export it both ways:
>
> COPY t to 'f';
> COPY t to 'f' WITH BINARY;
>
> And then compare copying them both in again with "\timing". That should let
> you definitively answer whether it's really worth the trouble.

while I fully understand the 'benchmark your situation' need, this isn't
that simple.

in this case we are trying to decide what API/interface to use in a
infrastructure tool that will be distributed in common distros (it's now
the default syslog package of debian and fedora), there are so many
variables in hardware, software, and load that trying to benchmark it
becomes effectivly impossible.

based on Stephan's explination of where binary could help, I think the
easy answer is to decide not to bother with it (the postgres text to X
converters get far more optimization attention than anything rsyslog could
deploy)

David Lang