Re: pg_upgrade of 11 -> 13: free(): invalid pointer

On 11/17/20 8:59 AM, Jeremy Wilson wrote:
> I’m continuing my upgrade journey, this time from 11 to 13, and the process is dying in the copy phase, always on the same DB:
>
> —
> Performing Upgrade
> ------------------
> Analyzing all rows in the new cluster ok
> Freezing all rows in the new cluster ok
> Deleting files from new pg_xact ok
> Copying old pg_xact to new server ok
> Setting next transaction ID and epoch for new cluster ok
> Deleting files from new pg_multixact/offsets ok
> Copying old pg_multixact/offsets to new server ok
> Deleting files from new pg_multixact/members ok
> Copying old pg_multixact/members to new server ok
> Setting next multixact ID and offset for new cluster ok
> Resetting WAL archives ok
> Setting frozenxid and minmxid counters in new cluster ok
> Restoring global objects in the new cluster ok
> Restoring database schemas in the new cluster
> messages
> *failure*
>
> Consult the last few lines of "pg_upgrade_dump_16387.log" for
> the probable cause of the failure.
> Failure, exiting
> —
>
> The log contains (which is different each time):
>
> —
> pg_restore: WARNING: terminating connection because of crash of another server process
> DETAIL: The postmaster has commanded this server process to roll back the current transaction and exit, because another server process exited abnormally and possibly corrupted shared memory.
> HINT: In a moment you should be able to reconnect to the database and repeat your command.
> pg_restore: creating COMMENT "public.FUNCTION "st_isempty"("rast" "public"."raster")"
> pg_restore: while PROCESSING TOC:
> pg_restore: from TOC entry 5338; 0 0 COMMENT FUNCTION "st_isempty"("rast" "public"."raster") postgres
> pg_restore: error: could not execute query: server closed the connection unexpectedly
> This probably means the server terminated abnormally
> before or while processing the request.
> Command was: COMMENT ON FUNCTION "public"."st_isempty"("rast" "public"."raster") IS 'args: rast - Returns true if the raster is empty (width = 0 and height = 0). Otherwise, returns false.’;
> —
>
> And the pgsql13 server log contains:
>
> —
> 2020-11-17 11:51:40.953 EST [96545] LOG: database system is ready to accept connections
> free(): invalid pointer
> 2020-11-17 11:51:42.880 EST [96545] LOG: server process (PID 96575) was terminated by signal 6: Aborted
> 2020-11-17 11:51:42.880 EST [96545] LOG: terminating any other active server processes
> 2020-11-17 11:51:42.880 EST [96582] WARNING: terminating connection because of crash of another server process
> 2020-11-17 11:51:42.880 EST [96582] DETAIL: The postmaster has commanded this server process to roll back the current transaction and exit, because another server process exited abnormally and possibly corrupted shared memory.
> 2020-11-17 11:51:42.880 EST [96582] HINT: In a moment you should be able to reconnect to the database and repeat your command.
> 2020-11-17 11:51:42.884 EST [96545] LOG: all server processes terminated; reinitializing
> 2020-11-17 11:51:42.904 EST [96545] LOG: received fast shutdown request
> 2020-11-17 11:51:42.905 EST [96585] LOG: database system was interrupted; last known up at 2020-11-17 11:51:42 EST
> 2020-11-17 11:51:42.906 EST [96585] LOG: database system was not properly shut down; automatic recovery in progress
> 2020-11-17 11:51:42.906 EST [96585] LOG: redo starts at E0/DB6B2960
> 2020-11-17 11:51:42.907 EST [96545] LOG: abnormal database system shutdown
> 2020-11-17 11:51:42.909 EST [96545] LOG: database system is shut down
> —
>
> So I’m assuming it’s that free() call. Servers have PostGIS 3.0 on them, all installed from repo, and running CentOS 8.

Was this after a clean install of the corrected RPM's?

--
Adrian Klaver
adrian(dot)klaver(at)aklaver(dot)com