RE: BUG #17326: Postgres crashed when pg_reload_conf() with ssl certificate parameters

It's a new project that need security compliance , SSL is a MUST here , and pgaudit,set_user is installed here too to meeting the compliance request. We test renew SSL certificate, and change the ssl_cert_file and ssl_key_file parameter to renewed ssl certificates.
ssl = on
ssl_ciphers = 'HIGH:MEDIUM:+3DES:!aNULL'

ssl_crl_file = ''
#ssl_min_protocol_version = 'TLSv1.2'
ssl_ca_file = '/var/lib/pgsql/sslrenew/idtrca.cer'
#ssl_cert_file = '/var/lib/pgsql/sslrenew/postgres-109798.crt'
#ssl_key_file = '/var/lib/pgsql/sslrenew/postgres-109798.key'

ssl_cert_file = '/var/lib/pgsql/sslrenew/postgres014-110388.crt'
ssl_key_file = '/var/lib/pgsql/sslrenew/postgres014-11038.key'

--
shared_preload_libraries = 'orafce,pgaudit,pg_cron,pg_stat_statements,pg_prewarm,set_user'
pgaudit.log_catalog='on'
pgaudit.log_level='log'
pgaudit.log_parameter=on
pgaudit.log_statement_once=off
pgaudit.log='all, -misc'
pgaudit.log='ddl,role'
pgaudit.role='postgres,jamet'

#set_user
set_user.superuser_whitelist = '+dba'
#set_user.superuser_allowlist = '+dba'
set_user.block_log_statement=on
#set_user.nosuperuser_target_whitelist = ''
set_user.nosuperuser_target_allowlist = ''

#pre_warm
pg_prewarm.autoprewarm = true
pg_prewarm.autoprewarm_interval = 600

the Operating system got some security hardening too, too meet compliance requirement. The OpenSSL 1.1.1g with FIPS enabled.
$ openssl version
OpenSSL 1.1.1g FIPS 21 Apr 2020

Yes, interesting thing is when I remove all extensions and try the test again, then install orafce, pg_background, pgaudit, looks like not reproduced the issue, until install set_user rpm it's ok, but when create extension again, reproduced the issue.

=# \dx
List of installed extensions
Name | Version | Schema | Description
--------------------+---------+------------+----------------------------
--------------------+---------+------------+----------------------------
--------------------+---------+------------+----------------------------
--------------------+---------+------------+-----------
amcheck | 1.2 | public | functions for verifying relation integrity
orafce | 3.15 | public | Functions and operators that emulate a subset of functions and packages from the Oracle RDBMS
pageinspect | 1.8 | public | inspect the contents of database pages at a low level
pg_background | 1.0 | public | Run SQL queries in the background
pg_buffercache | 1.3 | public | examine the shared buffer cache
pg_cron | 1.4 | public | Job scheduler for PostgreSQL
pg_freespacemap | 1.2 | public | examine the free space map (FSM)
pg_permissions | 1.1 | public | view object permissions and compare them with the desired state
pg_stat_statements | 1.8 | public | track planning and execution statistics of all SQL statements executed
pgaudit | 1.5 | public | provides auditing functionality
pgstattuple | 1.5 | public | show tuple-level statistics
plpgsql | 1.0 | pg_catalog | PL/pgSQL procedural language
postgres_fdw | 1.0 | public | foreign-data wrapper for remote PostgreSQL servers
set_user | 3.0 | public | similar to SET ROLE but with added logging
(14 rows)

Thanks,

James

-----Original Message-----
From: Dmitry Dolgov <9erthalion6(at)gmail(dot)com>
Sent: Tuesday, December 14, 2021 11:46 PM
To: Michael Paquier <michael(at)paquier(dot)xyz>
Cc: James Pang (chaolpan) <chaolpan(at)cisco(dot)com>; pgsql-bugs(at)lists(dot)postgresql(dot)org
Subject: Re: BUG #17326: Postgres crashed when pg_reload_conf() with ssl certificate parameters

> On Mon, Dec 13, 2021 at 08:10:57PM +0900, Michael Paquier wrote:
> On Mon, Dec 13, 2021 at 07:06:16AM +0000, James Pang (chaolpan) wrote:
> > Edit postgresql.conf to change ssl_certificate parameter ,
>
> Do you mean ssl_cert_file here? Also, something that's not completely
> clear to me is if this is a problem with a vanilla PostgreSQL instance
> or if this is related to the pgaudit extension set_user, as it has
> been mentioned as one potential origin of the problem upthread, but
> you are not telling if this is the case here. So what do you have for
> shared_preload_libraries in this crash?
>
> > #9 0x00007ff49a78059c in ssl_cert_clear_certs () from
> > /lib64/libssl.so.1.1
> > #10 0x00007ff49a780645 in ssl_cert_free () from /lib64/libssl.so.1.1
> > #11 0x00007ff49a78a25c in SSL_CTX_free () from /lib64/libssl.so.1.1
> > #12 0x000000000068b6b8 in be_tls_init ()
> > #13 0x00000000007271e1 in SIGHUP_handler ()
>
> Why is secure_initialize() not showing up in this stack? That would
> be the caller of be_tls_init() in the SIGHUP handler. The version of
> OpenSSL you are linking your binaries to would be useful here. That
> would be a 1.1.0 or a 1.1.1, no? Any specific minor version letter?

I think I can actually reproduce the issue. In my case the stack is fine, it contains secure_initialize, and overall it looks like some sort of memory corruption -- at least openssl gets segfault because it can't access some memory address it tries to verify in asn1_primitive_free.
Not sure yet why, investigating.