Re: backup manifests

Greetings,

* Tom Lane (tgl(at)sss(dot)pgh(dot)pa(dot)us) wrote:
> Robert Haas <robertmhaas(at)gmail(dot)com> writes:
> > AFAICS, the only options to make that work with JSON are (1) introduce
> > a new hand-coded JSON parser designed for frontend operation, (2) add
> > a dependency on an external JSON parser that we can use from frontend
> > code, or (3) adapt the existing JSON parser used in the backend so
> > that it can also be used in the frontend.
> > ... I'd
> > be willing to do (3) if somebody could explain to me how to solve the
> > problems with porting that code to work on the frontend side, but the
> > only suggestion so far as to how to do that is to port memory
> > contexts, elog/report, and presumably encoding handling to work on the
> > frontend side. That seems to me to be an unreasonably large lift,
>
> Yeah, agreed. The only consideration that'd make that a remotely
> sane idea is that if somebody did the work, there would be other
> uses for it. (One that comes to mind immediately is cleaning up
> ecpg's miserably-maintained fork of the backend datetime code.)
>
> But there's no denying that it would be a large amount of work
> (if it's even feasible), and nobody has stepped up to volunteer.
> It's not reasonable to hold up this particular feature waiting
> for that to happen.

Sure, it'd be work, and for "adding a simple backup manifest", maybe too
much to be worth considering ... but that's not what is going on here,
is it? Are we really *just* going to add a backup manifest to
pg_basebackup and call it done? That's not what I understood the goal
here to be but rather to start doing a lot of other things with
pg_basebackup beyond just having a manifest and if you think just a bit
farther down the path, I think you start to realize that you're going to
need this base set of capabilities to get to a point where pg_basebackup
(or whatever it ends up being called) is able to have the kind of
capabilities that exist in other PG backup software already.

I'm sure I don't need to say where to find it, but I can point you to a
pretty good example of a similar effort, and we didn't start with "build
a manifest into a custom format" as the first thing implemented, but
rather a great deal of work was first put into building out things like
logging, memory management/contexts, error handling/try-catch, having a
string type, a variant type, etc.

In some ways, it's kind of impressive what we've got in our front-ends
tools even though we don't have these things, really, and certainly not
all in one nice library that they all use... but at the same time, I
think that lack has also held those tools back, pg_basebackup among
them.

Anyway, off my high horse, I'll just say I agree w/ David and David wrt
using JSON for this over hacking together yet another format. We didn't
do that as thoroughly as we should have (we've got a JSON parser and all
that, and use JSON quite a bit, but the actual manifest format is a mix
of ini-style and JSON, because it's got more in it than just a list of
files, something that I suspect will also end up being true of this down
the road and for good reasons, and we started with the ini format and
discovered it sucked and then started embedding JSON in it...), and
we've come to realize that was a bad idea, and intend to fix it in our
next manifest major version bump. Would be unfortunate to see PG making
that same mistake.

Thanks,

Stephen