Re: Fdw batch insert error out when set batch_size > 65535

On 6/9/21 8:28 AM, Tom Lane wrote:
> I wrote:
>> Bharath Rupireddy <bharath(dot)rupireddyforpostgres(at)gmail(dot)com> writes:
>>>> I've added a simple regression test to postgres_fdw, testing that batch
>>>> sizes > 65535 work fine, and pushed the fix.
>
>>> I was earlier thinking of adding one, but stopped because it might
>>> increase the regression test execution time. It looks like that's true
>>> - with and without the test case it takes 17 sec and 4 sec
>>> respectively on my dev system which is 4X slower. I'm not sure if this
>>> is okay.
>
>> The cost, versus the odds of ever detecting a problem, doesn't
>> seem like a good tradeoff.
>
> I took a quick look and noted that on buildfarm member longfin
> (to take a random example that's sitting a few feet from me),
> the time for contrib-install-check went from 34 seconds before
> this patch to 40 seconds after. I find that completely
> unacceptable compared to the likely value of this test case.
>

Note that the problem here is [1] - we're creating a lot of slots
referencing the same tuple descriptor, which inflates the duration.
There's a fix in the other thread, which eliminates ~99% of the
overhead. I plan to push that fix soon (a day or two).

regards

--
Tomas Vondra
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company