Re: 9.3: summary of corruption detection / checksums / CRCs discussion

On Wed, Apr 25, 2012 at 9:28 AM, Merlin Moncure <mmoncure(at)gmail(dot)com> wrote:
>> That patch actually has more than one optimization in it, I think, but
>> the basic idea is that if we could figure out a way to set
>> HEAP_XMIN_COMMITTED when loading data into a table created or
>> truncated within the same transaction, the need to set hint bits on
>> first scan of the table would be eliminated.  Writing the xmin as
>> FrozenTransactionId would save even more, though it introduces some
>> additional complexity.
>
> This would be great but it's only a corner case.  A pretty common
> application flow is to write a large number of records, scan them,
> update them, scan them again, delete them, etc. in a table that's
> already established and possibly pretty large.  Unfortunately this
> type of work doesn't get a lot of coverage with the common benchmarks.
>
> Also, wouldn't the extra out of band wal traffic from hint bits
> exacerbate contention issues on the wal insert lock?

Hm, You probably remember the process local hint bit cache. While my
implementation had a lot of issues and it's arguable if I'm a patient
enough C programmer to come up with one to pass muster, it did work as
advertised: it greatly reduced to the point of elimination hint bit
traffic in cases like this, including the bulk load case. Some
variant of the approach would probably take at least some of the sting
out of the logged hint bits if that's the direction things have to go.

That doesn't help high concurrency cases where the #tuples/xid is low
though. And even a small increase in contention on the wal insert
lock could mean a measurable reduction in scaling unless that problem
is solved independently.

merlin