Re: Proposal: Another attempt at vacuum improvements

On Wed, May 25, 2011 at 10:36 PM, Robert Haas <robertmhaas(at)gmail(dot)com> wrote:

>
>> I don't see how you'd store
>> anything in the hole without it being in a fixed place, where it would
>> eventually be hit by either the line pointer array or tuple data.
>
> The point is that it doesn't matter.  Suppose we put it just after the
> line pointer array.

I think the point is you can not *always* put it just after the line
pointer array without possibly shuffling the tuples. Remember we need
to put the LSN when the dead line pointer is generated because we
decided to prune away the dead tuple. Say, for example, the page is
completely full and there are no dead line pointers and hence no LSN
on the page. Also there is no free space after the line pointer array.
Now say we prune dead tuples and generate dead line pointers, but the
last line pointer in the array is still in-use and the first tuple
immediately after the line pointer array is live. Since you generated
dead line pointers you want to store the LSN on the page. Now, there
is no way you can store is after the line pointer array without moving
the live tuple somewhere else.

Thats the point me and Alvaro are making. Do you agree with that logic
? Now that does not matter because we would always generate dead line
pointers holding a buffer cleanup lock and hence we are free to
shuffle tuples around. May be we are digressing on a trivial detail
here, but I hope I got it correct.

> Any time we're thinking about extending the line
> pointer array, we already have an exclusive lock on the buffer.  And
> if we already have a exclusive lock on the buffer, then we can reclaim
> the dead line pointers and now we no longer need the saved LSN, so
> writing over it is perfectly fine.
>

The trouble is you may not be able to shrink the line pointer array.
But of course, you can reuse the reclaimed dead line pointers. I would
still advocate doing that during the pruning operation because we want
to emit WAL records for the operation.

> OK, I lied: if we have an exclusive buffer lock, but the last vacuum
> either failed, or is still in progress, then the LSN might not be old
> enough for us to reclaim the dead line pointers yet.  So ideally we'd
> like to hold onto it.  We can do that by either (a) moving the LSN out
> another 6 bytes, if there's enough room; or (b) deciding not to put
> the new tuple on this page, after all.  There's no situation in which
> we absolutely HAVE to get another tuple onto this particular page.  We
> can just decide that the effective size of a page that contains dead
> line pointers is effectively 8 bytes less.  The alternative is to eat
> up 8 bytes of space on ALL pages, whether they contain dead line
> pointers or not.
>

I think we are on the same page as far as storing LSN if and only if
its required. But what was not convincing is the argument that you can
*always* find free space for the LSN without moving things around.

Let me summarize the sequence of operations and let me know if you
still disagree with the general principle:

1. There are no dead line pointers in the page - we are good.
2. Few tuples become dead, HOT pruning is invoked either during normal
operation or heap vacuum. The dead tuples are pruned away and
truncated to dead line pointers. We already hold cleanup lock on the
buffer. We set the flag in the page header and store the LSN (either
at the end of line pointer array or at the end of the page)
3. Someday index vacuum is run and it removes the index pointers to
the dead line pointers. We remember the start LSN of the index vacuum
somewhere, may be as a pg_class attribute (how does index vacuum get
the list of dead line pointers is not material in the general scheme
of things)
4. When the page is again chosen for pruning, we check if the flag is
set in the header. If so, get the LSN stored in the page, check it
against the last successful index vacuum LSN and if its precedes the
index vacuum LSN, we turn the LP_DEAD line pointers to LP_UNUSED. The
special LSN can be removed unless new LP_DEAD line pointers get
generated during the pruning, otherwise its overwritten with the
current LSN. Since we hold the buffer cleanup lock, the special LSN
storage can be reclaimed by shuffling things around.

Thanks,
Pavan

--
Pavan Deolasee
EnterpriseDB     http://www.enterprisedb.com