Re: Speed dblink using alternate libpq tuple storage

Thank you for the comment,

> First, my priority is one-the-fly result processing,
> not the allocation optimizing. And this patch seems to make
> it possible, I can process results row-by-row, without the
> need to buffer all of them in PQresult. Which is great!
>
> But the current API seems clumsy, I guess its because the
> patch grew from trying to replace the low-level allocator.

Exactly.

> I would like to propose better one-shot API with:
>
> void *(*RowStoreHandler)(PGresult *res, PGresAttValue *columns);
>
> where the PGresAttValue * is allocated once, inside PQresult.
> And the pointers inside point directly to network buffer.

Good catch, thank you. The patch is dragging too much from the
old implementation. It is no need to copy the data inside
getAnotherTuple to do it, as you say.

> Ofcourse this requires replacing the current per-column malloc+copy
> pattern with per-row parse+handle pattern, but I think resulting
> API will be better:
>
> 1) Pass-through processing do not need to care about unnecessary
> per-row allocations.
>
> 2) Handlers that want to copy of the row (like regular libpq),
> can optimize allocations by having "global" view of the row.
> (Eg. One allocation for row header + data).
>
> This also optimizes call patterns - first libpq parses packet,
> then row handler processes row, no unnecessary back-and-forth.
>
>
> Summary - current API has various assumptions how the row is
> processed, let's remove those.

Thank you, I rewrite the patch to make it realize.

regards,

--
Kyotaro Horiguchi
NTT Open Source Software Center