Re: libpq: PQgetCopyData() and allocation overhead

Thank you.

I meant to try PQExpBuffer — as you say, it fits much better with existing
libpq style. But then I hit on the callback idea which was even more
efficient, by a fair margin. It was also considerably simpler both inside
libpq and in the client code, eliminating all sorts of awkward questions
about who deallocates the buffer in which situations. So I ditched the
"dynamic buffer" concept and went with the callback.

One other possible alternative suggests itself: instead of taking a
callback and a context pointer, the function could probably just return a
struct: status/size, and buffer. Then the caller would have to figure out
whether there's a line in the buffer, and if so, process it. It seems like
more work for the client code, but it may make the compiler's optimisation
work easier.

Jeroen

On Fri, 3 Mar 2023 at 16:52, Jelte Fennema <postgres(at)jeltef(dot)nl> wrote:

> On Thu, 2 Mar 2023 at 20:45, Jeroen Vermeulen <jtvjtv(at)gmail(dot)com> wrote:
> > I'm attaching a diff now. This is not a patch, it's just a discussion
> piece.
>
> Did you try with PQExpBuffer? I still think that probably fits better
> in the API design of libpq. Obviously if it's significantly slower
> than the callback approach in this patch then it's worth considering
> using the callback approach. Overall it definitely seems reasonable to
> me to have an API that doesn't do an allocation per row.
>