From: | "Bossart, Nathan" <bossartn(at)amazon(dot)com> |
---|---|
To: | Bharath Rupireddy <bharath(dot)rupireddyforpostgres(at)gmail(dot)com> |
Cc: | "pgsql-hackers(at)postgresql(dot)org" <pgsql-hackers(at)postgresql(dot)org> |
Subject: | Re: O(n) tasks cause lengthy startups and checkpoints |
Date: | 2021-12-10 19:03:17 |
Message-ID: | 6DFA78EC-EF4D-4CB0-82C8-AD5DA1B8E0AC@amazon.com |
Views: | Raw Message | Whole Thread | Download mbox | Resend email |
Thread: | |
Lists: | pgsql-hackers |
On 12/6/21, 11:23 AM, "Bossart, Nathan" <bossartn(at)amazon(dot)com> wrote:
> On 12/6/21, 3:44 AM, "Bharath Rupireddy" <bharath(dot)rupireddyforpostgres(at)gmail(dot)com> wrote:
>> Thanks. As I said upthread we've been discussing the approach of
>> offloading some of the checkpoint tasks like (deleting snapshot files)
>> internally for quite some time and I would like to share a patch that
>> adds a new background cleaner process (currently able to delete the
>> logical replication snapshot files, if required can be extended to do
>> other tasks as well). I don't mind if it gets rejected. Please have a
>> look.
>
> Thanks for sharing! I've also spent some time on a patch set, which I
> intend to share once I have handling for all four tasks (so far I have
> handling for CheckPointSnapBuild() and RemovePgTempFiles()). I'll
> take a look at your patch as well.
Well, I haven't had a chance to look at your patch, and my patch set
still only has handling for CheckPointSnapBuild() and
RemovePgTempFiles(), but I thought I'd share what I have anyway. I
split it into 5 patches:
0001 - Adds a new "custodian" auxiliary process that does nothing.
0002 - During startup, remove the pgsql_tmp directories instead of
only clearing the contents.
0003 - Split temporary file cleanup during startup into two stages.
The first renames the directories, and the second clears them.
0004 - Moves the second stage from 0003 to the custodian process.
0005 - Moves CheckPointSnapBuild() to the custodian process.
This is still very much a work in progress, and I've done minimal
testing so far.
Nathan
Attachment | Content-Type | Size |
---|---|---|
v1-0005-Move-removal-of-old-serialized-snapshots-to-custo.patch | application/octet-stream | 3.8 KB |
v1-0004-Move-pgsql_tmp-file-removal-to-custodian-process.patch | application/octet-stream | 5.1 KB |
v1-0003-Split-pgsql_tmp-cleanup-into-two-stages.patch | application/octet-stream | 9.8 KB |
v1-0002-Also-remove-pgsql_tmp-directories-during-startup.patch | application/octet-stream | 4.8 KB |
v1-0001-Introduce-custodian.patch | application/octet-stream | 18.6 KB |
From | Date | Subject | |
---|---|---|---|
Next Message | Alvaro Herrera | 2021-12-10 20:02:08 | Re: Column Filtering in Logical Replication |
Previous Message | Godfrin, Philippe E | 2021-12-10 18:49:05 | RE: [EXTERNAL] Re: speed up verifying UTF-8 |