Re: libpq contention due to gss even when not using gss

> On 14 Jun 2024, at 10:46, Dmitry Dolgov <9erthalion6(at)gmail(dot)com> wrote:
>
>> On Thu, Jun 13, 2024 at 10:30:24AM GMT, Andres Freund wrote:
>>>> To investigate a report of both postgres and pgbouncer having issues when a
>>>> lot of new connections aree established, I used pgbench -C. Oddly, on an
>>>> early attempt, the bottleneck wasn't postgres+pgbouncer, it was pgbench. But
>>>> only when using TCP, not with unix sockets.
>>>>
>>>> c=40;pgbench -C -n -c$c -j$c -T5 -f <(echo 'select 1') 'port=6432 host=127.0.0.1 user=test dbname=postgres password=fake'
>>>>
>>>> host=127.0.0.1: 16465
>>>> host=127.0.0.1,gssencmode=disable 20860
>>>> host=/tmp: 49286
>>>>
>>>> Note that the server does *not* support gss, yet gss has a substantial
>>>> performance impact.
>>>>
>>>> Obviously the connection rates here absurdly high and outside of badly written
>>>> applications likely never practically relevant. However, the number of cores
>>>> in systems are going up, and this quite possibly will become relevant in more
>>>> realistic scenarios (lock contention kicks in earlier the more cores you
>>>> have).
>>>
>>> By not supporting gss I assume you mean having built with --with-gssapi,
>>> but only host (not hostgssenc) records in pg_hba, right?
>>
>> Yes, the latter. Or not having kerberos set up on the client side.
>
> I've been experimenting with both:
>
> * The server is built without gssapi, but the client does support it.
> This produces exactly the contention you're talking about.
>
> * The server is built with gssapi, but do not use it in pg_hba, the
> client does support gssapi. In this case the difference between
> gssencmode=disable/prefer is even more dramatic in my test case
> (milliseconds vs seconds) due to the environment with configured
> kerberos (for other purposes, thus gss_init_sec_context spends huge
> amount of time to still return nothing).
>
> At the same time after quick look I don't see an easy way to avoid that.
> Current implementation tries to initialize gss before getting any
> confirmation from the server whether it's supported. Doing this other
> way around would probably just shift overhead to the server side.

The main problem seems to be that we check whether or not there is a credential
cache when we try to select encryption but not yet authentication, as a way to
figure out if gssenc it as all worth trying? I experimented with deferring it
with potentially cheaper heuristics in encryption selection, but it seems hard
to get around since other methods were even more expensive.

--
Daniel Gustafsson