Re: proposal: possibility to read dumped table's name from file

Greetings,

* Justin Pryzby (pryzby(at)telsasoft(dot)com) wrote:
> On Wed, Nov 11, 2020 at 06:49:43AM +0100, Pavel Stehule wrote:
> > >> Perhaps this feature could co-exist with a full blown configuration for
> > >> pg_dump, but even then there's certainly issues with what's proposed-
> > >> how would you handle explicitly asking for a table which is named
> > >> " mytable" to be included or excluded? Or a table which has a newline
> > >> in it? Using a standardized format which supports the full range of
> > >> what we do in a table name, explicitly and clearly, would address these
> > >> issues and also give us the flexibility to extend the options which
> > >> could be used through the configuration file beyond just the filters in
> > >> the future.
>
> I think it's a reasonable question - why would a new configuration file option
> include support for only a handful of existing arguments but not the rest.

Even if the first version of having a config file for pg_dump only
supported some options, that would be reasonable imv, but I dislike the
idea of building it in such a way that it'll be awkward to add more
options to it in the future, something that I definitely think people
would like to see (I know I would...).

> > > This is the correct argument - I will check a possibility to use strange
> > > names, but there is the same possibility and functionality like we allow
> > > from the command line. So you can use double quoted names. I'll check it.
> >
> > I checked
> > echo "+t \"bad Name\"" | /usr/local/pgsql/master/bin/pg_dump --filter=/dev/stdin
> > It is working without any problem
>
> I think it couldn't possibly work with newlines, since you call pg_get_line().

Yeah, I didn't really believe that it actually worked but hadn't had a
chance to demonstrate that it didn't yet.

> I realize that entering a newline into the shell would also be a PITA, but that
> could be one *more* reason to support a config file - to allow terrible table
> names to be in a file and avoid writing dash tee quote something enter else
> quote in a pg_dump command, or shell script.

Agreed.

> I fooled with argument parsing to handle reading from a file in the quickest
> way. As written, this fails to handle multiple config files, and special table
> names, which need to support arbitrary, logical lines, with quotes surrounding
> newlines or other special chars. As written, the --config file is parsed
> *after* all other arguments, so it could override previous args (like
> --no-blobs --no-blogs, --file, --format, --compress, --lock-wait), which I
> guess is bad, so the config file should be processed *during* argument parsing.
> Unfortunately, I think that suggests duplicating parsing of all/most the
> argument parsing for config file support - I'd be happy if someone suggested a
> better way.

This still feels like we're trying to quickly hack-and-slash at adding a
config file option rather than thinking through what a sensible design
for a pg_dump config file would look like. Having a way to avoid having
multiple places in the code that has to handle all the possible options
is a nice idea but, as I tried to allude to up-thread, I fully expect
that once we've got this config file capability that we're going to want
to add things to it that would be difficult to utilize through the
command-line and so I expect these code paths to diverge anyway.

I would imagine something like:

[connection]
db-host=whatever
db-port=5433
...

[output]
owners = true
privileges = false
format = "custom"
file = "myoutput.dump"

# This is a comment
[include]
tables = [ "sometable", "table with spaces",
"table with quoted \"",
"""this is my table
with a carriage return""", "anothertable" ]
table-patterns = [ "table*" ]
schemas = [ "myschema" ]

[exclude]
tables = [ "similar to include" ]
functions = [ "somefunction(int)" ]

etc, etc ...

Thanks,

Stephen