Re: How could we make it simple to access the log as a table?

On 5/28/12 2:55 PM, Robert Haas wrote:
> As far as CSV goes, I think the biggest deficiency is that there's a
> mismatch between the way that log files are typically named (e.g. one
> per day, or one per hour) and the way that a CSV foreign table is
> created (you've got to point it at one particular file). Maybe we
> could have a CSV reader that understands PostgreSQL-format CSV logs,
> but you point it at a directory, rather than a single file, and it
> reads all the CSV files in the directory. And maybe it could also be
> smart enough that if you've got a WHERE clause that filter by date, it
> uses that to skip any files that can be proven irrelevant. So the
> user can just turn on CSV logging, point the FDW at the log directory,
> and away they go.

The idea is a really good one, except for one thing: this pattern is undoubtedly not unique to PG CSV logs.

ISTM it would be extremely useful to have a generic ability to define a filename pattern and have all files matching that pattern be pulled in via FDW; possibly be treating each file as a partition, possibly complete with exclusion constraints.

ISTM that what's really needed here are a bunch of separate improvements to our logging and/or FDW, each of which will add usefulness.
--
Jim C. Nasby, Database Architect jim(at)nasby(dot)net
512.569.9461 (cell) http://jim.nasby.net