Re: proposal: possibility to read dumped table's name from file

On 7/14/21 2:18 AM, Stephen Frost wrote:
> Greetings,
>
> * Alvaro Herrera (alvherre(at)2ndquadrant(dot)com) wrote:
>> On 2021-Jul-13, Stephen Frost wrote:
>>> The simplest possible format isn't going to work with all the different
>>> pg_dump options and it still isn't going to be 'simple' since it needs
>>> to work with the flexibility that we have in what we support for object
>>> names,
>>
>> That's fine. If people want a mechanism that allows changing the other
>> pg_dump options that are not related to object filtering, they can
>> implement a configuration file for that.
>
> It's been said multiple times that people *do* want that and that they
> want it to all be part of this one file, and specifically that they
> don't want to end up with a file structure that actively works against
> allowing other options to be added to it.
>

I have no problem believing some people want to be able to specify
pg_dump parameters in a file, similarly to IMPDP/EXPDP parameter files
etc. That seems useful, but I doubt they considered the case with many
filter rules ... which is what "my people" want.

Not sure how keeping the filter rules in a separate file (which I assume
is what you mean by "file structure"), with a format tailored for filter
rules, works *actively* against adding options to the "main" config.

I'm not buying the argument that keeping some of the stuff in a separate
file is an issue - plenty of established tools do that, the concept of
"including" a config is not a radical new thing, and I don't expect we'd
have many options supported by a file.

In any case, I think user input is important, but ultimately it's up to
us to reconcile the conflicting requirements coming from various users
and come up with a reasonable compromise design.

>>> I don't know that the options that I suggested previously would
>>> definitely work or not but they at least would allow other projects like
>>> pgAdmin to leverage existing code for parsing and generating these
>>> config files.
>>
>> Keep in mind that this patch is not intended to help pgAdmin
>> specifically. It would be great if pgAdmin uses the functionality
>> implemented here, but if they decide not to, that's not terrible. They
>> have survived decades without a pg_dump configuration file; they still
>> can.
>
> The adding of a config file for pg_dump should specifically be looking
> at pgAdmin as the exact use-case for having such a capability.
>
>> There are several votes in this thread for pg_dump to gain functionality
>> to filter objects based on a simple specification -- particularly one
>> that can be written using shell pipelines. This patch gives it.
>
> And several votes for having a config file that supports, or at least
> can support in the future, the various options which pg_dump supports-
> and active voices against having a new file format that doesn't allow
> for that.
>

IMHO the whole "problem" here stems from the question whether there
should be a single universal pg_dump config file, containing everything
including the filter rules. I'm of the opinion it's better to keep the
filter rules separate, mainly because:

1) simplicity - Options (key/value) and filter rules (with more internal
structure) seem quite different, and mixing them in the same file will
just make the format more complex.

2) flexibility - Keeping the filter rules in a separate file makes it
easier to reuse the same set of rules with different pg_dump configs,
specified in (much smaller) config files.

So in principle, the "main" config could use e.g. TOML or whatever we
find most suitable for this type of key/value config file (or we could
just use the same format as for postgresql.conf et al). And the filter
rules could use something as simple as CSV (yes, I know it's not great,
but there's plenty of parsers, it handles multi-line strings etc.).

>>> I'm not completely against inventing something new, but I'd really
>>> prefer that we at least try to make something existing work first
>>> before inventing something new that everyone is going to have to deal
>>> with.
>>
>> That was discussed upthread and led nowhere.
>
> You're right- no one followed up on that. Instead, one group continues
> to push for 'simple' and to just accept what's been proposed, while
> another group counters that we should be looking at the broader design
> question and work towards a solution which will work for us down the
> road, and not just right now.
>

I have quite thick skin, but I have to admit I rather dislike how this
paints the people arguing for simplicity.

IMO simplicity is a perfectly legitimate (and desirable) design feature,
and simpler solutions often fare better in the long run. Yes, we need to
look at the broader design, no doubt about that.

> One thing remains clear- there's no consensus here.
>

True.

regards

--
Tomas Vondra
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company