Re: Why conf.d should be default, and auto.conf and recovery.conf should be in it

Tom Lane wrote:
> Stephen Frost <sfrost(at)snowman(dot)net> writes:
> > * Alvaro Herrera (alvherre(at)2ndquadrant(dot)com) wrote:
> >> 1. it is to be read automatically by the server without need for an
> >> "include_dir conf.d" option in the main postgresql.conf.
>
> > I am not thrilled with the idea that we're claiming ownership of the
> > directory which postgresql.conf happens to reside in and you had better
> > not have any 'conf.d' directory in there unless it's the one for PG.
>
> If we were to do this, I'd argue that the location of this hard-wired
> config directory ought to be selected by a configure option. And
> in fact, I'd argue that the default behavior with no such option be
> that there's no such hard-wired directory. That puts it entirely
> on the packager to decide what makes sense as a location.

I fail to see the sense in this. Surely the only useful option is to
use a relative path: an absolute path is completely useless because then
there's no way to have multiple clusters using the same executables, as
you say. Since it has to be relative to *something*, there aren't that
many options: we could make it relative to the directory where the
executables reside (which would be absolutely pointless), or relative to
the data directory (which we already ruled out for reasons well
understood), or relative to the directory containing the other
configuration files.

> On the whole though, I find the argument that we can't configure this
> in postgresql.conf to be exceedingly weak. In particular, the idea
> that you can puppet-ify a configuration without any knowledge of the
> distro you're targeting is ridiculous on its face,

Surely puppetification is not the only use case for conf.d. For
example, suppose for a minute that a module providing an extension wants
to offer a program to initialize its configuration file into some
configuration directory (say, an hypothetical PostGIS package drops a
"postgis.conf" into conf.d with initial values for all the config
options it provides). If you're not administering postgresql.conf with
puppet, you'd like to be able to do this to the clusters you have
without having to figure out *where* to put the files (that is, without
having to parse and possibly modify postgresql.conf). Just dump it in
the cluster's conf.d.

> as is the idea that we/postgres can dictate configuration file
> location conventions to packagers who have distro rules to follow.

I don't see that this is ridiculous at all. Configuration files always
go together; a distro rule that some config files go in some place and
other configuration files go somewhere else, sounds more ridiculous to
me, actually.

> There isn't going to be a "one location to rule them all", and that
> means the argument that a location determined by postgresql.conf is
> too unstable does not really hold water.

postgresql.conf's location already rules them all. Remember, my
proposal upthread was to have conf.d be located in the same directory
that holds postgresql.conf.

> Another point here is that hard-wiring a config directory location
> into the executables completely breaks many scenarios for running
> multiple clusters with the same executables.

Therefore the proposal is not to hardwire the location in the
executables.

A server with multiple clusters using the same executables needs a way
to specify different ports for each (at the very least); therefore they
already have separate postgresql.conf. Each directory containing such
file can contain a conf.d subdirectory with extra stuff perfectly well.

Stephen Frost wrote:

> * Alvaro Herrera (alvherre(at)2ndquadrant(dot)com) wrote:

> > In /etc you have /etc/postfix/ where you have Postfix files, and
> > /etc/exim4 where you have Exim files, and /etc/apache2 or whatever. And
> > if each of those software packages uses a "conf.d" or "conf-enabled"
> > directory, they use a hardcoded name for a place inside their own
> > subdirectory, not one directly in /etc.
>
> I missed the part of this where you point out that Apache on Debian has
> some kind of problem because it's possible for an admin to remove the
> 'IncludeDir sites-enabled' line from apache2.conf.

I don't see that this is parallel. For one thing, I don't think you
would normally have multiple apache2.conf files. In the other hand,
even if you had more than one, surely the one that's managed by the
packaging system, if there is one, is not going to want to have that
line changed, because that would immediately break all modules installed
by packages.

> > What you don't have is /etc/exim4.conf and /etc/postfix.conf and
> > /etc/apache.conf, where each of those files specifies that I would
> > please like to have extra files loaded from /etc/exim.conf.d and
> > /etc/apache.conf.d, and that packages please figure out by parsing my
> > config file in whatever random format I have invented to figure out
> > where snippets are to be stored.
>
> Right, there's absolutely zero cases of /etc/*.conf files these days.
> Nor are there any /etc/*.conf.d directories.
>
> </sarcasm>

I just checked my system, and it seems to me that most cases that have
.conf files and something.conf.d directly in /etc are more because of
historical reasons than because they have made a conscious, contemporary
decision to do that. Do we want to replicate choices made in the
1990's?

> Sure, maybe we all feel that there should be an /etc/pam directory
> instead of /etc/pam.conf and /etc/pam.d, but that doesn't make it so,
> and we're only talking about a subset (perhaps a majority, but it's
> certainly not a totality) of PG users- not everyone is using Debian or
> RH or even *Linux* or GNU tools to run PG with.

Yeah, so instead of making a rational choice for people using all
platforms, mirroring the rational choice made by Linux/Debian/GNU, lets
make an irrational choice so that everybody will be constantly bitten by
it, regardless of platform.

</sarcasm>

> > > > 2. it is to be placed alongside postgresql.conf, not necessarily in
> > > > PGDATA.
> > >
> > > I certainly agree with this idea- but I feel it should be configurable.
> > > The path which is passed in should be relative to the location of
> > > postgresql.conf.
> >
> > What would be the point of having it be configurable?
>
> To allow admins to configure it? Perhaps their policy is that it must
> be postgresql.conf.d

This example sounds rather weak. I mean, if they *must* have
/etc/postgresql/9.2/main/postgresql.conf.d instead of
/etc/postgresql/9.2/main/conf.d, why aren't they forced to have

Now, if we want to hardcode the name postgresql.conf.d instead of just
conf.d, that's fine too. Just have *one* name and stick with it, not a
configurable choice which subsequently becomes a PITA.

> or that they want it served off of an NFS mount instead of being
> pulled from /etc,

So conf.d would be in an NFS mount but postgresql.conf would not?

> or maybe they only back up /srv but they don't want to change where
> the package manager dumps the postgresql.conf that's used for starting
> up the cluster?

> > > > 4. there is no point in "disabling" it, and thus we offer no mechanism
> > > > to do that.
> > >
> > > No. There should be a way to disable it.
> >
> > What's you rationale for this?
>
> I don't want PG randomly guessing at directories to go poking around in
> for config files. It's also documentation.

Documentation is fine. We can have a comment at the top of
postgresql.conf saying "also see files in postgresql.conf.d/" or some
such. I can hardly thing that opening and reading a subdirectory named
conf.d or postgresql.conf.d, *in the same directory as postgresql.conf*
is "randomly poking".

> > > > 5. its entries override what is set in postgresql.conf, and are in turn
> > > > overridden by what is set in postgresql.auto.conf.
> > >
> > > For my part, I'd rather my configurations be deterministic and would
> > > prefer that we error on obvious misconfigurations where values are set
> > > and then reset.
> >
> > Any convention we establish will make configurations deterministic.
> > Random variations such as making the conf.d directory be configurable
> > would make them non-deterministic. ISTM you're contradicting yourself
> > here.
>
> Perhaps you see it that way, but I certainly don't. Having conf.d be
> configurable doesn't mean that we have one file with "work_mem = 1GB"
> and another with "work_mem = 5GB" where it's not at all clear which one
> is actually what gets used.

There must be a way to jump over the chasm of logic like you did in the
above paragraph, but I don't see it.

If you want to argue that multiple lines setting the same variable
should raise an error instead of "last definition wins", please do
propose that separately.

> > > It will be a useful mechanism for puppet even with the ability to
> > > disable it in postgresql.conf (you're probably managing that with puppet
> > > also, or keeping it at the system default which will likely include the
> > > include_dir option by default- but, yes, you'd need to check that) and
> > > without the "override previous definition" option.
> >
> > Meh. External packages (say, postgis) will be able to add their own
> > files into conf.d without having to check what its actual location is,
> > and without having to check whether the thing is enabled in the first
> > place. If we make it disable-able, packages will not be able to do
> > that; or rather, they will be able to add files, but they will have no
> > effect.
>
> Which cluster directory is the packager of PostGIS going to put his
> config snippets into, exactly? Even if you force the hard-code conf.d
> idea, the packager will still need to figure out what clusters exist.

This was a thinko on my part, I admit, which doesn't invalidate the rest
of my argumentation. I think the way to think about this would be
something like having mode in pg_ctlcluster (for example) that enables
postgis for a certain cluster, which means it drops a certain file in
the cluster's conf.d directory, as I described above.

> Although, if you could make the conf.d directory configurable, and have
> more than one, then we actually could have a "system-wide" directory and
> then a per-cluster directory, which *would* make things easier for
> packagers. But, oh wait, don't forget that the "system-wide" one would
> still need to be major-version-specific...

Hmm. I guess you could have /etc/postgresql/9.2/conf.d which is used
for all clusters, yeah. The need for this is not as clear as a
per-cluster conf.d though, and IMO would be a separate proposal.

--
Álvaro Herrera http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Training & Services