Starting Postgres when there is no disk space

From: Igal Sapir <igal(at)lucee(dot)org>
To: "Psql_General (E-mail)" <pgsql-general(at)postgresql(dot)org>
Subject: Starting Postgres when there is no disk space
Date: 2019-05-02 00:07:01
Message-ID: CA+zig08KvQGq3O=QAtXO18=-13nV4nYJpKerSOKGs7GWrYDGeA@mail.gmail.com
Views: Raw Message | Whole Thread | Download mbox | Resend email
Thread:
Lists: pgsql-general

I have Postgres running in a Docker container with PGDATA mounted from the
host. Postgres consume all of the disk space, 130GB [1], and can not be
started [2]. The database has a lot of bloat due to much many deletions.
The problem is that now I can not start Postgres at all.

I mounted an additional partition with 100GB, hoping to fix the bloat with
a TABLESPACE in the new mount, but how can I do anything if Postgres will
not start in the first place?

I expected there to be a tool that can defrag the database files, e.g. a
"vacuumdb" utility that can run without Postgres. Or maybe run Postgres
and disable the WAL so that no new disk space will be required.

Surely, I'm not the first one to experience this issue. How can I fix this?

Thank you,

Igal

[1]
root(at)ff818ff7550a:/# du -h --max-depth=1 /pgdata
625M /pgdata/pg_wal
608K /pgdata/global
0 /pgdata/pg_commit_ts
0 /pgdata/pg_dynshmem
8.0K /pgdata/pg_notify
0 /pgdata/pg_serial
0 /pgdata/pg_snapshots
16K /pgdata/pg_subtrans
0 /pgdata/pg_twophase
16K /pgdata/pg_multixact
130G /pgdata/base
0 /pgdata/pg_replslot
0 /pgdata/pg_tblspc
0 /pgdata/pg_stat
0 /pgdata/pg_stat_tmp
7.9M /pgdata/pg_xact
4.0K /pgdata/pg_logical
0 /pgdata/tmp
130G /pgdata

[2]
postgres(at)1efd26b999ca:/$ /usr/lib/postgresql/11/bin/pg_ctl start
waiting for server to start....2019-05-01 20:43:59.301 UTC [34] LOG:
listening on IPv4 address "0.0.0.0", port 5432
2019-05-01 20:43:59.301 UTC [34] LOG: listening on IPv6 address "::", port
5432
2019-05-01 20:43:59.303 UTC [34] LOG: listening on Unix socket
"/var/run/postgresql/.s.PGSQL.5432"
2019-05-01 20:43:59.322 UTC [35] LOG: database system shutdown was
interrupted; last known up at 2019-05-01 19:37:32 UTC
2019-05-01 20:43:59.863 UTC [35] LOG: database system was not properly
shut down; automatic recovery in progress
2019-05-01 20:43:59.865 UTC [35] LOG: redo starts at 144/4EFFFC18
...2019-05-01 20:44:02.389 UTC [35] LOG: redo done at 144/74FFE060
2019-05-01 20:44:02.389 UTC [35] LOG: last completed transaction was at
log time 2019-04-28 05:05:24.687581+00
.2019-05-01 20:44:03.474 UTC [35] PANIC: could not write to file
"pg_logical/replorigin_checkpoint.tmp": No space left on device
2019-05-01 20:44:03.480 UTC [34] LOG: startup process (PID 35) was
terminated by signal 6: Aborted
2019-05-01 20:44:03.480 UTC [34] LOG: aborting startup due to startup
process failure
2019-05-01 20:44:03.493 UTC [34] LOG: database system is shut down
stopped waiting
pg_ctl: could not start server
Examine the log output.

Responses

Browse pgsql-general by date

  From Date Subject
Next Message Sergey Koposov 2019-05-02 00:59:52 strange nested loop row count estimates
Previous Message Thomas Munro 2019-05-01 22:09:05 Re: Upgrading locale issues