Re: O(n) tasks cause lengthy startups and checkpoints

From: Nathan Bossart <nathandbossart(at)gmail(dot)com>
To: Andres Freund <andres(at)anarazel(dot)de>
Cc: Simon Riggs <simon(dot)riggs(at)enterprisedb(dot)com>, Robert Haas <robertmhaas(at)gmail(dot)com>, "Bossart, Nathan" <bossartn(at)amazon(dot)com>, Bharath Rupireddy <bharath(dot)rupireddyforpostgres(at)gmail(dot)com>, Maxim Orlov <orlovmg(at)gmail(dot)com>, Amul Sul <sulamul(at)gmail(dot)com>, Bruce Momjian <bruce(at)momjian(dot)us>, "pgsql-hackers(at)postgresql(dot)org" <pgsql-hackers(at)postgresql(dot)org>
Subject: Re: O(n) tasks cause lengthy startups and checkpoints
Date: 2022-07-06 16:51:10
Message-ID: 20220706165110.GB2135371@nathanxps13
Views: Raw Message | Whole Thread | Download mbox | Resend email
Thread:
Lists: pgsql-hackers

Here's a new revision where I've attempted to address all the feedback I've
received thus far. Notably, the custodian now uses a queue for registering
tasks and determining which tasks to execute. Other changes include
splitting the temporary file functions apart to avoid consecutive boolean
flags, using a timestamp instead of an integer for the staging name for
temporary directories, moving temporary directories to a dedicated
directory so that the custodian doesn't need to scan relation files,
ERROR-ing when something goes wrong when cleaning up temporary files,
executing requested tasks immediately in single-user mode, and more.

--
Nathan Bossart
Amazon Web Services: https://aws.amazon.com

Attachment Content-Type Size
v7-0001-Introduce-custodian.patch text/x-diff 24.1 KB
v7-0002-Also-remove-pgsql_tmp-directories-during-startup.patch text/x-diff 4.8 KB
v7-0003-Split-pgsql_tmp-cleanup-into-two-stages.patch text/x-diff 14.2 KB
v7-0004-Move-pgsql_tmp-file-removal-to-custodian-process.patch text/x-diff 5.4 KB
v7-0005-Move-removal-of-old-serialized-snapshots-to-custo.patch text/x-diff 4.5 KB
v7-0006-Move-removal-of-old-logical-rewrite-mapping-files.patch text/x-diff 9.2 KB

In response to

Responses

Browse pgsql-hackers by date

  From Date Subject
Next Message Robert Haas 2022-07-06 17:11:07 Re: Possible fails in pg_stat_statements test
Previous Message Nathan Bossart 2022-07-06 16:41:05 Re: archive modules