Re: Enhance pg_dump multi-threaded streaming (WAS: Re: filesystem full during vacuum - space recovery issues)

From: Scott Ribe <scott_ribe(at)elevated-dev(dot)com>
To: Thomas Simpson <ts(at)talentstack(dot)to>
Cc: Pgsql-admin <pgsql-admin(at)lists(dot)postgresql(dot)org>, pgsql-hackers(at)lists(dot)postgresql(dot)org
Subject: Re: Enhance pg_dump multi-threaded streaming (WAS: Re: filesystem full during vacuum - space recovery issues)
Date: 2024-07-19 13:26:47
Message-ID: 6F46CFEB-3C3A-4CEC-89DF-57D8225A5863@elevated-dev.com
Views: Raw Message | Whole Thread | Download mbox | Resend email
Thread:
Lists: pgsql-admin pgsql-hackers

Do you actually have 100G networking between the nodes? Because if not, a single CPU should be able to saturate 10G.

Likewise the receiving end would need disk capable of keeping up. Which brings up the question, why not write to disk, but directly to the destination rather than write locally then copy?

Do you require dump-reload because of suspected corruption? That's a tough one. But if not, if the goal is just to get up and running on a new server, why not pg_basebackup, streaming replica, promote? That depends on the level of data modification activity being low enough that pg_basebackup can keep up with WAL as it's generated and apply it faster than new WAL comes in, but given that your server is currently keeping up with writing that much WAL and flushing that many changes, seems likely it would keep up as long as the network connection is fast enough. Anyway, in that scenario, you don't need to care how long pg_basebackup takes.

If you do need a dump/reload because of suspected corruption, the only thing I can think of is something like doing it a table at a time--partitioning would help here, if practical.

In response to

Responses

Browse pgsql-admin by date

  From Date Subject
Next Message Thomas Simpson 2024-07-19 13:46:14 Re: Enhance pg_dump multi-threaded streaming (WAS: Re: filesystem full during vacuum - space recovery issues)
Previous Message Ron Johnson 2024-07-19 12:21:36 Re: Enhance pg_dump multi-threaded streaming (WAS: Re: filesystem full during vacuum - space recovery issues)

Browse pgsql-hackers by date

  From Date Subject
Next Message Tomas Vondra 2024-07-19 13:28:44 Re: [PATCH] Add additional extended protocol commands to psql: \parse and \bindx
Previous Message Robert Haas 2024-07-19 13:26:39 Re: Things I don't like about \du's "Attributes" column