Re: SLRU limits

On 09.06.2011 15:50, Robert Haas wrote:
> On Thu, Jun 9, 2011 at 7:46 AM, Heikki Linnakangas
> <heikki(dot)linnakangas(at)enterprisedb(dot)com> wrote:
>> While reviewing the SLRU code in predicate.c again, I remembered this old
>> thread:
>>
>> http://archives.postgresql.org/pgsql-hackers/2010-12/msg02374.php
>>
>> SLRU has a limit of 64k segment files, because the files are named using
>> four hex digits like "00CE". Kevin's math shows that that's just enough to
>> store 2^32 four-byte integers, which wasn't enough for predicate.c, which
>> needs to store uint64s. Kevin worked around that by simply limiting the max
>> range of open xids to fit the SLRU limit, ie. 2^31. However, that math was
>> based on 8k block size, and the situation is worse for smaller block sizes.
>> If you set BLCKSZ to 2048 or less, pg_subtrans can only hold 1 billion
>> transactions. With 1024 block size, only half a billion.
>
> I'm pretty unexcited about this. It's not terribly sane to keep a
> transaction open for half a billion XIDs anyway, because of VACUUM.

I agree, but if you or your application is insane enough to do that
anyway, it's not appropriate response for the system to give warnings
about apparent XID wrap-around, and in the worst case mix up
subtransactions belonging to different transactions, possibly leading to
data loss. I have not actually tested that, but I don't see any
safeguards to stop it from happening. Of course, another alternative is
to add such safeguards.

> And I would guess that there's a lot more interest in raising BLCKSZ
> than lowering it. It might not be a bad idea to adopt the fix you
> propose anyway, but it doesn't seem urgent.

I guess we could fix pg_subtrans by not allowing BLCKSZ < 8k. That
leaves the problem with pg_serial. Kevin has already worked around, but
I'm not very happy with that workaround.

If we don't want to change it wholesale, one option would be to support
different lengths of filenames in slru.c for different slrus. At a quick
glance, it seems pretty easy. That would allow keeping clog unchanged -
that's the one that's most likely to have unforeseen consequences if
changed. pg_subtrans and pg_serial are more ephemeral, they don't need
to be retained over shutdown, so they seem less likely to cause trouble.
That seems like the best option to me.

--
Heikki Linnakangas
EnterpriseDB http://www.enterprisedb.com