libpq: Process buffered SSL read bytes to support records >8kB on async API

From: Lars Kanis <lars(at)greiz-reinsdorf(dot)de>
To: pgsql-hackers(at)lists(dot)postgresql(dot)org
Subject: libpq: Process buffered SSL read bytes to support records >8kB on async API
Date: 2024-09-08 20:07:53
Message-ID: 2039ac58-d3e0-434b-ac1a-2a987f3b4cb1@greiz-reinsdorf.de
Views: Raw Message | Whole Thread | Download mbox | Resend email
Thread:
Lists: pgsql-hackers

Dear hackers,

I'm the maintainer of ruby-pg the ruby interface to the PostgreSQL
database. This binding uses the asynchronous API of libpq by default to
facilitate the ruby IO wait and scheduling mechanisms.

This works well with the vanilla postgresql server, but it leads to
starvation with other types of servers using the postgresql wire
protocol 3. This is because the current functioning of the libpq async
interface depends on a maximum size of SSL records of 8kB.

The following servers were reported to starve with ruby-pg:

* AWS RDS Aurora Serverless [1]
* YugabyteDb [2]
* CockroachDB [3]

They block infinitely on certain message sizes sent from the backend to
the libpq frontend. It is best described in [4]. A repro docker
composition is provided by YugabyteDB at [2].

To fix this issue the attached patch calls pqReadData() repeatedly in
PQconsumeInput() until there is no buffered SSL data left to be read.
Another solution could be to process buffered SSL read bytes in
PQisBusy() instead of PQconsumeInput() .

The synchronous libpq API isn't affected, since it supports arbitrary
SSL record sizes already. That's why I think that the asynchronous API
should also support bigger SSL record sizes.

Regards, Lars

[1] https://github.com/ged/ruby-pg/issues/325
[2] https://github.com/ged/ruby-pg/issues/588
[3] https://github.com/ged/ruby-pg/issues/583
[4] https://github.com/ged/ruby-pg/issues/325#issuecomment-737561270

Attachment Content-Type Size
0001-libpq-Process-buffered-SSL-read-bytes-to-support-rec.patch text/x-patch 1.7 KB

Responses

Browse pgsql-hackers by date

  From Date Subject
Next Message Thomas Munro 2024-09-08 20:55:31 Re: CI, macports, darwin version problems
Previous Message Noah Misch 2024-09-08 16:53:55 Re: Yet another way for pg_ctl stop to fail on Windows