Re: pg_dump crashes

On 5/22/20 5:37 AM, Nico De Ranter wrote:
> Hi all,
>
> Postgres version: 9.5
> OS: Ubuntu 18.04.4
>
> I have a 144GB Bacula database that crashes the postgres daemon when I
> try to do a pg_dump.
> At some point the server ran out of diskspace for the database storage.
> I expanded the lvm and rebooted the server. It seemed to work fine,
> however when I try to dump the bacula database the postgres daemon dies
> after about 37GB.
>
> I tried copying the database to another machine and upgrading postgres
> to 11 using pg_upgrade.  The upgrade seems to work but I still get
> exactly the same problem when trying to dump the database.
>
> postgres(at)core4:~$ pg_dumpall --cluster 11/main --file=dump.sql
> pg_dump: Dumping the contents of table "file" failed: PQgetCopyData()
> failed.
> pg_dump: Error message from server: server closed the connection
> unexpectedly
> This probably means the server terminated abnormally
> before or while processing the request.
> pg_dump: The command was: COPY public.file (fileid, fileindex, jobid,
> pathid, filenameid, deltaseq, markid, lstat, md5) TO stdout;
> pg_dumpall: pg_dump failed on database "bacula", exiting

What happens if you try to dump just this table?

Something along lines of:

pg_dump -t file -d some_db -U some_user

Have you looked at the system logs to see if it is the OS killing the
process?

>
> In the logs I see:
>
> 2020-05-22 14:23:30.649 CEST [12768] LOG:  server process (PID 534) was
> terminated by signal 11: Segmentation fault
> 2020-05-22 14:23:30.649 CEST [12768] DETAIL:  Failed process was
> running: COPY public.file (fileid, fileindex, jobid, pathid, filenameid,
> deltaseq, markid, lstat, md5) TO stdout;
> 2020-05-22 14:23:30.651 CEST [12768] LOG:  terminating any other active
> server processes
> 2020-05-22 14:23:30.651 CEST [482] WARNING:  terminating connection
> because of crash of another server process
> 2020-05-22 14:23:30.651 CEST [482] DETAIL:  The postmaster has commanded
> this server process to roll back the current transaction and exit,
> because another server process exited abnormally and possibly corrupted
> shared memory.
> 2020-05-22 14:23:30.651 CEST [482] HINT:  In a moment you should be able
> to reconnect to the database and repeat your command.
> 2020-05-22 14:23:30.652 CEST [12768] LOG:  all server processes
> terminated; reinitializing
> 2020-05-22 14:23:30.671 CEST [578] LOG:  database system was
> interrupted; last known up at 2020-05-22 14:15:19 CEST
> 2020-05-22 14:23:30.809 CEST [578] LOG:  database system was not
> properly shut down; automatic recovery in progress
> 2020-05-22 14:23:30.819 CEST [578] LOG:  redo starts at 197/D605EA18
> 2020-05-22 14:23:30.819 CEST [578] LOG:  invalid record length at
> 197/D605EA50: wanted 24, got 0
> 2020-05-22 14:23:30.819 CEST [578] LOG:  redo done at 197/D605EA18
> 2020-05-22 14:23:30.876 CEST [12768] LOG:  database system is ready to
> accept connections
> 2020-05-22 14:29:07.511 CEST [12768] LOG:  received fast shutdown request
>
>
> Any ideas how to fix or debug this?
>
> Nico
>
> --
>
> Nico De Ranter
>
> Operations Engineer
>
> T. +32 16 38 72 10
>
>
> <http://www.esaturnus.com>
>
> <http://www.esaturnus.com>
>
>
> eSATURNUS
> Philipssite 5, D, box 28
> 3001 Leuven – Belgium
>
>
>
> T. +32 16 40 12 82
> F. +32 16 40 84 77
> www.esaturnus.com <http://www.esaturnus.com>
>
> ** <http://www.esaturnus.com/>
>
> *For Service & Support :*
>
> Support Line Belgium: +32 2 2009897
>
> Support Line International: +44 12 56 68 38 78
>
> Or via email : medical(dot)services(dot)eu(at)sony(dot)com
> <mailto:medical(dot)services(dot)eu(at)sony(dot)com>
>
>

--
Adrian Klaver
adrian(dot)klaver(at)aklaver(dot)com