Re: How to redirect output from PostgreSQL pg_recvlogical to a file or a pipe?

On 1/12/24 14:03, David Ventimiglia wrote:

Reply to list also.
Ccing list
> Hi Adrian,
>
> I left out the creation of the slot both from the SO question and from
> this mailing list question, because I believe it's a red herring.  I run
> into the same problem with the default output plugin as I do with the
> wal2json plugin.  However, the problem is a little different from what I
> described.  It turns out it's not output redirection from pg_recvlogical
> sending to stdout that fails.  Rather, it's output redirection from
> pg_recvlogical sending to stdout and then passed through a pipeline that
> fails.  Or something like that.
>
> This works.  The sample.txt file is non-empty.
>
> pg_recvlogical -d postgres --slot=test --create-slot
> pg_recvlogical -d postgres -n --slot=test --start -f - > sample.txt
>
>
> This does not work.  The sample.txt file is empty.
>
> pg_recvlogical -d postgres --slot=test --create-slot
> pg_recvlogical -d postgres -n --slot=test --start -f - | awk
> '{print}' > sample.txt
>
>
> Weirdly, this works.  The sample.txt file is non-empty.
>
> pg_recvlogical -d postgres --slot=test --create-slot
> pg_recvlogical -d postgres -n --slot=test --start -f - | cat >
> sample.txt
>
>
> FWIW, this is demonstrated in this screen-cast
> <https://asciinema.org/a/631166>.  I suspect it has something to do with
> the flushing of buffers, as other people have suggested.
>
> Thanks!
> David
>
>
>
>
> On Fri, Jan 12, 2024 at 1:48 PM Adrian Klaver <adrian(dot)klaver(at)aklaver(dot)com
> <mailto:adrian(dot)klaver(at)aklaver(dot)com>> wrote:
>
> On 1/12/24 11:34, David Ventimiglia wrote:
> > Hello! How do I redirect logical decoding output from the
> PostgreSQL CLI
> > tool |pg_recvlogical| either to a file or to another command via
> a pipe?
> > I ask because when I try the obvious, no output is recorded or sent:
> >
> > |pg_recvlogical -d postgres --slot test --start -f - >>
> sample.jsonl |
> >
> > Lest there be any confusion, I already created the slot in an
> earlier
> > step. Moreover, I can verify that if I omit the output
> redirection |>>
> > sample| then it does work, insofar as it emits the expected change
> > events when I perform DML in another terminal window. When I
> include the
> > redirection (or alternatively, set up a pipeline), then nothing
> happens.
>
> You left out the important part of the SO question:
>
> pg_recvlogical -d postgres --slot test --create-slot -P wal2json
>
> I can get:
>
> pg_recvlogical -d postgres --slot test --start -f - >> sample.jsonl
>
> to work when I do:
>
> pg_recvlogical -d postgres --slot test --create-slot
>
> wal2json seems to be the issue.
>
> >
> > Note that I am aware of the option to pass a filename to the -f
> switch
> > to write to a file.  That works, but it's not what I'm after
> because it
> > doesn't help update my mental model of how this is supposed to work.
> > Based on my current (flawed) mental model built up from command line
> > experience with other tools, this /should/ work.  I should be
> able to
> > send the output to stdout and then redirect it to a file.  It
> surprises
> > me that I cannot.
> >
> > Anyway, thanks!
> >
> > Best,
> >
> > David
> >
>
> --
> Adrian Klaver
> adrian(dot)klaver(at)aklaver(dot)com <mailto:adrian(dot)klaver(at)aklaver(dot)com>
>

--
Adrian Klaver
adrian(dot)klaver(at)aklaver(dot)com