After Startup Processing (was Re: Remaining beta blockers)

On 29 April 2013 01:40, Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us> wrote:
> Robert Haas <robertmhaas(at)gmail(dot)com> writes:
>> On Sun, Apr 28, 2013 at 11:41 AM, Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us> wrote:
>>> Well, it's fairly clear *how* to do it: add some more processing that
>>> occurs after we've completed crash replay. We already have some of
>>> that, eg completion of partial splits in btrees, so it's not that much
>>> of a stretch; it's just a lot of code that's not been written yet.
>
>> As far as I can see, that would require starting a separate backend
>> process for every database, and keeping track of which of those
>> completed their post-recovery work, and disallowing connections to any
>> given database until the post-recovery work for that database had been
>> completed. That seems to add quite a few failure modes that we don't
>> have today, which is why I haven't been much interested in going that
>> route.
>
> Well, I didn't say it would be easy or quick ;-). But you're presuming
> quite a number of design elements that don't seem essential to me.
>
> What I'd be inclined to think about as prerequisite work is fixing
> things so that a process could reattach to a new database, after
> flushing all its caches. That's a feature that's been requested plenty,
> and could have applications in autovacuum or other places besides this,
> and would certainly get lots of testing outside of crash recovery.
>
> Having done that, we could imagine that the startup process itself
> cycles through all the databases and does the fixup work, rather than
> complicating the postmaster logic as suggested above. Potentially this
> could replace the filesystem-scan-driven fixup logic that exists in it
> now, if it turns out to be faster to search for unlogged tables via a
> catalog scan rather than directory scans.

I'd rather leave the Startup process alone, it does too much already.

It's easy enough to start a new process that can run SQL properly
after the Startup process has gone, maybe we can have more than one.
Hopefully now you can see why I'd like to change the meaning of
database so we can have one process talk to multiple databases at same
time. It helps for a variety of system tasks, like autovacuum,
autoreindex, logical replication etc..

The AfterStartup processing can take normal session locks on the
objects it needs to work on and keep them til its done.

The general background to this is exactly what we need to rebuild
damaged indexes, or allow unlogged indexes on logged tables which can
be rebuilt on crash.

--
Simon Riggs http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Training & Services