Re: serverlog rotation/functions

Andreas Pflug wrote:
> Bruce Momjian wrote:
> >>Ok, no limit (but a default maximum of 50k remains). And since it's
> >>superuser only, he hopefully knows what he does.
> >
> >
> > Huh? Why have a default maximum?
>
> Just for convenience. Both start and size are optional parameters, but
> with start=0 and size=50000. Well, it's a very special function anyway,
> so we could require the user to supply all parameters. I'll remove it.

Agreed, and maybe a zero value gets the entire file.

> > Ah, I think it needs to close that as soon as it starts. Don't other
> > subprocesses do that? That shared memory is very fragile and we don't
> > want an errant pointer poking in there.
>
> The result of an errant pointer writing to that shred mem would be
>
> 1) wrong pid for SysLogger, so it can't be signalled to rotate from backends
> 2) wrong timestamp, so backends don't know the latest logfile.
>
> Nothing particularly crash prone really.

No, I am thinking the program goes crazy and writes everywhere.

> > This seems clean and fast enough to me:
> >
> > SELECT filename
> > FROM pg_dir_ls('/var/log')
> > ORDER BY 1 DESC
> > LIMIT 1
>
> For a logfile listing function, this would look
>
> SELECT MAX(startdate)
> FROM pg_logfile_ls()
>
>
> >
> > Considering that any query from a client is going to have to go through
> > the parser and be executed, an 'ls' in a directory just isn't a
> > measurable performance hit.
> >
> > If you want run a test that does an 'ls' and one that doesn't to see
> > that there is no measurable performance difference.
> >
> So while a simple PG_RETURN_TIMESTAMP(logfiletimestamp) compared to a
> lengthy setof returning function is drastically faster, this appears
> much less drastic with parser overhead.
>
> > I would not worry about the clock going backward. PostgreSQL would have
> > enough problems with timestamp columns moving backward that the file log
> > times are the least of our problems.
>
> I see, so the admin is in trouble anyway (what about PITR? Data column
> deviations appear harmless compared to restoration based on timestamps).

PITR uses WAL numbering so it would be fine, but the timestamps on the
commit records would have problems.

> > You can't know that you aren't reading corrupt data if you read shared
> > memory without a lock. What if the write is happening as you read?
>
> I thought about this quite a while.
>
> If the shmem fields aren't written atomically (one is 32bit, one 64 bit,
> probably on dword boundaries so writing will happen at least processor
> bus wide, do we support any 16 bit processor?) the corruption
> consequences as above apply. In the case of the timestamp, the high word
> will rarely change anyway, only every 2^32 seconds...
>
> Concurrent access on the logger pid would mean to call
> pg_logfile_rotate() while a killed logger is being restarted, which is
> creating a new logfile then anyway. This would send a SIGINT into outer
> space, maybe to the bgwriter triggering a checkpoint, or the postmaster
> shutting it down (gracefully, still unwanted).
>
> BTW, the consequences of a trigger flag in shmem would be less because
> all that could happen was a log rotation (which appends to existing
> files, just in case syslogger died in the milliseconds after a rotation).
>
> >
> > The only clean solution I can think of is to write an operating system
> > file that contains the current log filename and read from that. I
> > believe such writes are atomic. But again, this seems like overkill to
> > me.
>
> Ah wait.
> Digging further behind SIGUSR1 I now *do* see a solution without pid in
> shmem, using SendPostmasterSignal. Well, a little hint from gurus would
> have helped...
>
> I'll convert to this, *dropping* all shmem.

Yes, that is the usual method. We signal the postmaster and it then
does the signalling to the logger. I thought you had looked at other
backend signalling examples so I didn't explain it.

Now, one really good efficiency would be to use LISTEN/NOTIFY so clients
could know when new data has appeared in the log, or the log file is
rotated. Now that's an efficiency! However, let's get this
infrastructure completed first. One wacky idea would be for the
clients to LISTEN on 'pg_new_logfile' and have the logger do
system('psql -c "NOTIFY pg_new_logfile" template1') or something like
that.

--
Bruce Momjian | http://candle.pha.pa.us
pgman(at)candle(dot)pha(dot)pa(dot)us | (610) 359-1001
+ If your life is a hard drive, | 13 Roberts Road
+ Christ can be your backup. | Newtown Square, Pennsylvania 19073