Re: Add Pipelining support in psql

From: Anthonin Bonnefoy <anthonin(dot)bonnefoy(at)datadoghq(dot)com>
To: Jelte Fennema-Nio <postgres(at)jeltef(dot)nl>
Cc: Michael Paquier <michael(at)paquier(dot)xyz>, PostgreSQL Hackers <pgsql-hackers(at)postgresql(dot)org>
Subject: Re: Add Pipelining support in psql
Date: 2024-12-04 12:52:05
Message-ID: CAO6_Xqr10ncZPmVzn1gUT2nHu8Bd+u3SkbgF4Uw9TL8FUA5fbQ@mail.gmail.com
Views: Raw Message | Whole Thread | Download mbox | Resend email
Thread:
Lists: pgsql-hackers

An updated version of the patch, still a bit rough around the edges.

I've added \flushrequest, \flush and \getrequests meta-commands.
\flushrequest and \getresults look interesting as they add an
additional protocol message to test, but it also introduces the
possibility to be in the middle of an aborted pipeline which adds some
complexity to the patch. I'm a bit skeptical of \flush as \getresults
already automatically flush through PQgetResult and I haven't thought
of interesting tests that could rely on \flush.

I've added a new prompt interpolation instead of trying to hijack the
existing %x: %P reports the pipeline status with either
'|num_syncs,num_queries,pending_results|' for an ongoing pipeline or
'+num_syncs,num_queries,pending_results+' for an aborted pipeline.
Those are definitely open to debate for those but I found those
informations useful while debugging.

With the new \flushrequest and \getresults, it's visible that %x
doesn't reflect the transaction status when pipelines are involved. If
I do:

postgres=# \startpipeline
postgres=|0,0,0|# BEGIN \bind \g
postgres=|0,1,0|*# \flushrequest
postgres=|0,1,1|*# \getresults
BEGIN
postgres=|0,0,0|#

There's an ongoing transaction block running on the server but %x will
be empty as there's no queued commands and the server results are
consumed. I'm tempted to change %x to not display anything when within
a pipeline.

Calling PQgetResult while a command is piped and not followed by a
flushrequest or a sync would block since the asyncstatus is busy. So
\getresults has to keep track of the number of pending results to
avoid calling PQgetResult if there's no pending results. This is to
cover cases like:

postgres=# \startpipeline
postgres=|0,0,0|# select 1 \bind \g
postgres=|0,1,0|*# \getresults
No pending results to get
postgres=|0,1,0|*#

There are some interesting behaviors with pipeline + copy. Running the
following:

\startpipeline
SELECT $1 \bind 'val1' \g
COPY psql_pipeline FROM STDIN \bind \g
\flushrequest
\getresults
3 test3
\.
\endpipeline

Will complain with a "message type 0x5a arrived from server while
idle" (but still work). It's also possible to trigger a protocol error
with a query error after a piped copy:

truncate pgbench_accounts;
\startpipeline
copy pgbench_accounts FROM stdin \bind \g
SELECT 1 \bind \g
\endpipeline
1 2 3 asd
\.

Copy will fail with "ERROR: unexpected message type 0x50 during COPY
from stdin" and connection will be terminated.

Attachment Content-Type Size
v03-0001-Add-pipelining-support-in-psql.patch application/octet-stream 47.8 KB

In response to

Responses

Browse pgsql-hackers by date

  From Date Subject
Next Message Amul Sul 2024-12-04 13:02:33 Re: NOT ENFORCED constraint feature
Previous Message David Rowley 2024-12-04 12:42:36 Re: Make tuple deformation faster