Re: Why our counters need to be time-based WAS: WIP: cross column correlation ...

On Mon, Feb 28, 2011 at 2:31 PM, Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us> wrote:
> Robert Haas <robertmhaas(at)gmail(dot)com> writes:
>> On Mon, Feb 28, 2011 at 1:50 PM, Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us> wrote:
>>> Ultimately we need to think of a reporting mechanism that's a bit
>>> smarter than "rewrite the whole file for any update" ...
>
>> Well, we have these things called "tables".  Any chance of using those?
>
> Having the stats collector write tables would violate the classical form
> of the heisenberg principle (thou shalt avoid having thy measurement
> tools affect that which is measured), not to mention assorted practical
> problems like not wanting the stats collector to take locks or run
> transactions.
>
> The ideal solution would likely be for the stats collector to expose its
> data structures as shared memory, but I don't think we get to do that
> under SysV shmem --- it doesn't like variable-size shmem much.  Maybe
> that's another argument for looking harder into mmap or POSIX shmem,
> although it's not clear to me how well either of those fixes that.

Well, certainly, you could make it work with mmap() - you could
arrange a mechanism whereby anyone who tries to reference off the end
of the portion they've mapped calls stat() on the file and remaps it
at its now-increased size. But you'd need to think carefully about
locking and free-space management, which is where it starts to sound
an awful lot like you're reinventing the idea of a heap. Maybe
there's a way to design some kind of lighter weight mechanism, but the
complexity of the problem is not obviously a lot less than the general
problem of storing frequently updated tabular data.

--
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company