Re: postgres_fdw batching vs. (re)creating the tuple slots

Hi,

On 2021-05-30 22:22:10 +0200, Tomas Vondra wrote:
> There are two problems at play, here. Firstly, the way it's coded now
> the slots are pretty much re-created for each batch. So with 1M rows and
> batches of 32k rows, that's ~30x drop/create. That seems a bit wasteful,
> and it shouldn't be too difficult to keep the slots across batches. (We
> can't initialize all the slots in advance, because we don't know how
> many will be needed, but we don't have to release them between batches.)

Yea, that sounds like an obvious improvement.

> I'll take a look at fixing the first point, but I'm not entirely sure
> how much will that improve the situation.

Hm - I'd not expect this to still show up in the profile afterwards,
when you insert >> 32k rows. Still annoying when a smaller number is
inserted, of course.

> The other problem is that ResourceArrayAdd/Remove seem to behave a bit
> poorly with very many elements - I'm not sure if it's O(N^2) or worse,
> but growing the array and linear searches seem to be a bit expensive.

Hm. I assume this is using the hashed representation of a resowner array
most of the time, not the array one? I suspect the problem is that
pretty quickly the ResourceArrayRemove() degrades to a linear search,
because all of the resowner entries are the same, so the hashing doesn't
help us at all. The peril of a simplistic open-coded hash table :(

I think in this specific situation the easiest workaround is to use a
copy of the tuple desc, instead of the one in the relcache - the copy
won't be refcounted.

The whole tupledesc refcount / resowner stuff is a mess. We don't really
utilize it much, and pay a pretty steep price for maintaining it.

This'd be less of an issue if we didn't store one resowner item for each
reference, but kept track of the refcount one tupdesc resowner item
has. But there's no space to store that right now, nor is it easy to
make space, due to the way comparisons work for resowner.

Greetings,

Andres Freund