Re: Processing very large TEXT columns (300MB+) using C/libpq

As an aside any halfway decent optimizer would realize that the results of
strlen() are unchanging as long as the contents of what it's passed isn't
modified. That's a common enough pattern that it should be checked.

What about buffer size? Are you using a smaller fetch size that results in
lots of little packets?

Is there a way to specify a connection that uses compression?

On Fri, Oct 20, 2017 at 10:55 AM, Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us> wrote:

> Cory Nemelka <cnemelka(at)gmail(dot)com> writes:
> > Yes, but I should be able to read them much faster. The psql client can
> > display an 11MB column in a little over a minute, while in C using libpg
> > library, it takes over an hour.
>
> Well, of course psql relies on libpq, so it seems unlikely that libpq
> itself is where the time is going. Have you tried applying a profiler?
> "perf" or "oprofile" or similar tool ought to pinpoint the culprit
> pretty easily.
>
> regards, tom lane
>
>
> --
> Sent via pgsql-admin mailing list (pgsql-admin(at)postgresql(dot)org)
> To make changes to your subscription:
> http://www.postgresql.org/mailpref/pgsql-admin
>