Re: BUG #14683: *** glibc detected *** SELECT: double free or corruption

Hi Tom,

Not sure how to install debug symbols and whether it would be a good idea on a Production system.

The information you asked for is below. Note that we have had increased logging configured for a while now so that we can run pgbadger. These issues seem to have happened when the load was increased by adding new functionality. We have switched this extra functionality off for the time being so that the database is stable again.

NON DEFAUL PARAMETERS
work_mem = '32MB'
listen_addresses = '*'
max_connections = '300'
superuser_reserved_connections = '13'
shared_buffers = '9GB'
effective_cache_size = '18GB'
maintenance_work_mem = '256MB'
wal_level = 'hot_standby'
archive_mode = 'on'
archive_command = 'cp /var/lib/pgsql/9.5/data/%p /var/lib/pgsql/9.5/wal_archive/%f'
max_wal_senders = '10'
wal_keep_segments = '5000'
hot_standby = 'on'
max_stack_depth = '4MB'
search_path = '"$user", public, repmgr_dbp01_cluster'
shared_preload_libraries = 'pg_stat_statements'
pg_stat_statements.max = '5000'
pg_stat_statements.track = 'all'
log_min_duration_statement = '10'
log_line_prefix = '%t [%p]: [%l-1] user=%u,db=%d,app=%a,client=%h '
log_checkpoints = 'on'
log_connections = 'on'
log_disconnections = 'on'
log_lock_waits = 'on'
log_temp_files = '0'
log_statement = 'none'
log_autovacuum_min_duration = '0'
log_directory = '/mnt/pg_log'
timezone = 'UTC'
synchronous_commit = 'off'
checkpoint_timeout = '30min'
checkpoint_completion_target = '0.9'
max_wal_size = '18GB'

EXTENSIONS
name
--------------------
plpgsql
pg_stat_statements
pg_buffercache
uuid-ossp

Thanks for helping.

Kind regards ... Ray

-----Original Message-----
From: Tom Lane [mailto:tgl(at)sss(dot)pgh(dot)pa(dot)us]
Sent: 02 June 2017 16:10
To: Ray Warren
Cc: Michael Paquier; PostgreSQL mailing lists
Subject: Re: [BUGS] BUG #14683: *** glibc detected *** SELECT: double free or corruption

Ray Warren <Ray(dot)Warren(at)artsalliancemedia(dot)com> writes:
> ======= Backtrace: =========
> /lib64/libc.so.6(+0x75dee)[0x7f1d6ba7edee]
> /lib64/libc.so.6(+0x78ff6)[0x7f1d6ba81ff6]
> /lib64/libc.so.6(+0x7babe)[0x7f1d6ba84abe]
> /lib64/libc.so.6(realloc+0x158)[0x7f1d6ba84f08]
> postgres: postgres etms 172.30.8.63(53338) SELECT[0x7c8d72]
> postgres: postgres etms 172.30.8.63(53338)
> SELECT(repalloc+0x25)[0x7c9295]
> postgres: postgres etms 172.30.8.63(53338)
> SELECT(enlargeStringInfo+0x4e)[0x5e505e]
> postgres: postgres etms 172.30.8.63(53338)
> SELECT(appendStringInfoChar+0x28)[0x5e5178]
> postgres: postgres etms 172.30.8.63(53338) SELECT[0x7a6724]
> postgres: postgres etms 172.30.8.63(53338)
> SELECT(EmitErrorReport+0xa2f)[0x7a963f]
> postgres: postgres etms 172.30.8.63(53338)
> SELECT(errfinish+0x8d)[0x7a97dd]
> postgres: postgres etms 172.30.8.63(53338)
> SELECT(FileClose+0x1c5)[0x6966d5]
> postgres: postgres etms 172.30.8.63(53338)
> SELECT(BufFileClose+0x32)[0x698652]
> postgres: postgres etms 172.30.8.63(53338)
> SELECT(ExecHashJoin+0x31b)[0x5ce60b]
> postgres: postgres etms 172.30.8.63(53338)
> SELECT(ExecProcNode+0x168)[0x5bc9f8]
> postgres: postgres etms 172.30.8.63(53338)
> SELECT(ExecSort+0xf9)[0x5d86c9]
> postgres: postgres etms 172.30.8.63(53338)
> SELECT(ExecProcNode+0x148)[0x5bc9d8]

These stack traces would be a lot more useful if you had debug symbols installed, I think.

This trace is rather unusual because it appears to show FileClose trying to log something. That would be expected if you have log_temp_files enabled, but otherwise it would indicate an unexpected failure. More generally, what nondefault configuration settings are you using, and do you have any extensions installed?

regards, tom lane