[RFC] Removing "magic" oids

Hi,

In my opinion the current WITH OIDs system has numerous weaknesses:

1) The fact that oids are so magic means that if we get pluggable
storage, the design of the potential pluggable systems is constrained
and similar magic has to be present everywhere.

2) The fact that the oids in each table have the same counter to be
based on means that oid wraparounds have much worse consequences
performance wise than necessary. E.g. once the global counter has
wrapped, all toast tables start to be significantly slower.

It would be much better if most database objects had their own
counters.

3) For some oid using objects (toast, large objects at the very least)
it'd be quite worthwhile to switch to 8 byte ids. Currently that's
hard to do, because it'd break on-disk compatibility.

4) There's a lot of special case code around for dealing with oids.

5a) The fact that system table oids don't show up in selects by default
makes it more work than necessary to look at catalogs.

5b) Similarly, it's fairly annoying when debugging not to trivially see
oids for catalog structs.

I think we should drop WITH OIDs support. pg_dump should convert WITH
OIDs tables into tables that have an explicit oid column (with an
appropriate default function), pg_upgrade should refuse to upgrade them.
We've defaulted WITH OIDs to off for quite a while now, so that's
hopefully not going to be too painful.

For catalog tables, I think we should just add explicit oid columns.
That obviously requires a fair amount of change, but it's not too bad.
One issue here is that we we want to support "manual" inserts both for
initdb, and for emergency work.

The attached *PROTOTYPE* *WIP* patch removes WITH OIDs support, converts
catalog table oids to explicit oid columns, and makes enough
infrastructure changes to make plain pg_regress pass (after the
necessary changes of course). Only superficial psql and no pg_dump
changes.

There's plenty of issues with the prototype, but overall I'm pretty
pleased.

There's three major areas I'm not so sure about:

1) There's a few places dealing with system tables that don't deal with
a hardcoded system table. Since there's no notion of "table has oid"
and "which column is the oid column) anymore, we need some way to
deal with that. So far I've just extended existing objectaddress.c
code to deal with that, but it's not that pretty.

2) We need to be able to manually insert into catalog tables. Both
initdb and emergency surgery. My current hack is doing so directly
in nodeModifyTable.c but that's beyond ugly. I think we should add
an explicit DEFAULT clause to those columns with something like
nextoid('tablename', 'name_of_index') or such.

3) The quickest way to deal with the bootstrap code was to just assign
all oids for oid carrying tables that don't yet have any assigned.
That doesn't generally seem terrible, although it's certainly badly
implemented right now. That'd mean we'd have three ranges of oids
probably, unless we somehow have the bootstrap code advance the
in-database oid counter to a right state before we start with
post-bootstrap work. I like the idea of all bootstrap time oids
being determined by genbki.pl (I think that'll allow to remove some
code too).

I do wonder about ripping the oid counter out entirely, and replacing it
with sequences. But that seems like a separate project.

I'll polish this up further in the not too far away future. But I'd
really like to get some feedback before I sink more time into this.

Greetings,

Andres Freund