Re: BUG #17326: Postgres crashed when pg_reload_conf() with ssl certificate parameters

From: Dmitry Dolgov <9erthalion6(at)gmail(dot)com>
To: Michael Paquier <michael(at)paquier(dot)xyz>
Cc: "James Pang (chaolpan)" <chaolpan(at)cisco(dot)com>, "pgsql-bugs(at)lists(dot)postgresql(dot)org" <pgsql-bugs(at)lists(dot)postgresql(dot)org>
Subject: Re: BUG #17326: Postgres crashed when pg_reload_conf() with ssl certificate parameters
Date: 2021-12-14 17:36:54
Message-ID: 20211214173654.i54n54e7v73dvflj@localhost
Views: Raw Message | Whole Thread | Download mbox | Resend email
Thread:
Lists: pgsql-bugs

> On Tue, Dec 14, 2021 at 04:46:04PM +0100, Dmitry Dolgov wrote:
> > On Mon, Dec 13, 2021 at 08:10:57PM +0900, Michael Paquier wrote:
> > On Mon, Dec 13, 2021 at 07:06:16AM +0000, James Pang (chaolpan) wrote:
> > > Edit postgresql.conf to change ssl_certificate parameter ,
> >
> > Do you mean ssl_cert_file here? Also, something that's not completely
> > clear to me is if this is a problem with a vanilla PostgreSQL
> > instance or if this is related to the pgaudit extension set_user, as
> > it has been mentioned as one potential origin of the problem upthread,
> > but you are not telling if this is the case here. So what do you have
> > for shared_preload_libraries in this crash?
> >
> > > #9 0x00007ff49a78059c in ssl_cert_clear_certs () from /lib64/libssl.so.1.1
> > > #10 0x00007ff49a780645 in ssl_cert_free () from /lib64/libssl.so.1.1
> > > #11 0x00007ff49a78a25c in SSL_CTX_free () from /lib64/libssl.so.1.1
> > > #12 0x000000000068b6b8 in be_tls_init ()
> > > #13 0x00000000007271e1 in SIGHUP_handler ()
> >
> > Why is secure_initialize() not showing up in this stack? That would
> > be the caller of be_tls_init() in the SIGHUP handler. The version of
> > OpenSSL you are linking your binaries to would be useful here. That
> > would be a 1.1.0 or a 1.1.1, no? Any specific minor version letter?
>
> I think I can actually reproduce the issue. In my case the stack is
> fine, it contains secure_initialize, and overall it looks like some sort
> of memory corruption -- at least openssl gets segfault because it can't
> access some memory address it tries to verify in asn1_primitive_free.
> Not sure yet why, investigating.

After a short investigation looks like it's set_user problem. The
extension has duplicating set of parameters, where one is the actual set
and another one is "deprecated options". If I have both sets set
simultaneously in configuration (e.g. set_user.superuser_whitelist and
set_user.superuser_allowlist), on sighup in

set_config_option / PGC_STRING branch / makeDefault condition

something weird happens after set_extra_field, and after this point ssl
context memory seems to be corrupted. Right before that an assign_hook
from set_user is invoked to do something around "deprecated" options,
that's why it looks suspicious. As soon as no "deprecated" options left
in the config the issue disappears.

In response to

Responses

Browse pgsql-bugs by date

  From Date Subject
Next Message Vincent Veyron 2021-12-14 18:41:45 Re: When Update balloons memory
Previous Message Tom Lane 2021-12-14 16:28:07 Re: BUG #17336: logtape sort performance and overflow