Re: Support for pg_receivexlog --post-segment command

On Fri, Jan 6, 2017 at 1:45 PM, Feike Steenbergen <
feikesteenbergen(at)gmail(dot)com> wrote:

> Hi all,
>
> When reading through "Support for pg_receivexlog --format=plain|tar"[1], I
> came across a notion from Magnus Hagander that has crossed my mind a few
> times as well in the past years. As the feature proposed here is not
> directly related to that thread, I thought it best to start a new thread to
> discuss.
>
> > I have been talking to David about it a couple of times, and he agreed
> that it'd be useful to have a post-segment command. We haven't discussed it
> in much detail though. I'll add him to direct-cc here to see if he has any
> further input :)
>
> I'm coming across a few usecases where this would seem very useful. I'm
> looking to push finished segments to some api as soon as possible. Having
> the post-segment command would allow me to get there. I've tried the
> following approaches, none of them are very satisfying however:
>
> - periodic checking of new files (cron)
> - using inotify
> - tailing verbose pg_receivexlog output to see when a segment was switched
>

Yeah, these are all kind of ugly. Or platform specific. Or both.

You can make them work, but they feel hackish.

> > It could be that the best idea is to just notify some other process of
> what's happening. But making it an external command would give that a lot
> of flexibility. Of course, we need to be careful not to put ourselves back
> in the position we are in with archive_command, in that it's very difficult
> to write a good one.
>
> A signal for would be good enough for my use case, I don't necessarily
> need all the bookkeeping to ensure the post-segment command was finished
> successfully. However I can see people expecting similar behaviour for the
> post-segment command as for the archive_command. If we would use an
> external command, does this also imply that we need some bookkeeping around
> which segments are ready and done, similar to what is done on the server in
> the archive_status directory?
>

I'd really like to avoid having to re-implement that functionality in
pg_receivexlog. Given the experiences we have with how difficult it is to
get right in archive_command...

I think we're better off clearly documenting that we don't care about it.
And basically let the external command be responsible for that part.

So for example, your typical backup manager would listen to this signal or
whatever to react quickly. But it would *also* have some sort of fallback.
For example, whenever it's triggered it also checks if there are any
previous segments it missed (this would also cover the startup sequence).

Since we never actually remove anything (unlike archive_command which has
the integration with wal segment rotation), I think this can be done
perfectly safe.

Looking at the usecases where you have been doing it, are there any where
this would not work?

--
Magnus Hagander
Me: http://www.hagander.net/
Work: http://www.redpill-linpro.com/