November 14, 2024: PostgreSQL 17.1, 16.5, 15.9, 14.14, 13.17, and 12.21 Released!

PostgreSQL Weekly News - September 12, 2021

Posted on 2021-09-13 by PWN
PWN

PostgreSQL Weekly News - September 12, 2021

PostgreSQL Product News

pg_dumpbinary 2.5, a program used to dump a PostgreSQL database in binary format, released.

pgBadger v11.6, a PostgreSQL log analyzer and graph tool written in Perl, released.

[pgagroal 1.3.0, a high-performance protocol-native connection pool for PostgreSQL, released

PostgreSQL Jobs for September

https://archives.postgresql.org/pgsql-jobs/2021-09/

PostgreSQL in the News

Planet PostgreSQL: https://planet.postgresql.org/

PostgreSQL Weekly News is brought to you this week by David Fetter

Submit news and announcements by Sunday at 3:00pm PST8PDT to david@fetter.org.

Applied Patches

Michaël Paquier pushed:

Peter Eisentraut pushed:

Fujii Masao pushed:

  • Fix typo in comments. Author: Hou Zhijie Discussion: https://postgr.es/m/OS0PR01MB5716E6A6535FDFDC5A1B004194CE9@OS0PR01MB5716.jpnprd01.prod.outlook.com https://git.postgresql.org/pg/commitdiff/78aa616be74a13156f4fc8db3a131f1fdc2cce47

  • postgres_fdw: Allow application_name of remote connection to be set via GUC. This commit adds postgres_fdw.application_name GUC which specifies a value for application_name configuration parameter used when postgres_fdw establishes a connection to a foreign server. This GUC setting always overrides application_name option of the foreign server object. This GUC is useful when we want to specify our own application_name per remote connection. Previously application_name of a remote connection could be set basically only via options of a server object. But which meant that every session connecting to the same foreign server basically should use the same application_name. Also if we want to change the setting, we had to execute "ALTER SERVER ... OPTIONS ..." command. It was inconvenient. Author: Hayato Kuroda Reviewed-by: Masahiro Ikeda, Fujii Masao Discussion: https://postgr.es/m/TYCPR01MB5870D1E8B949DAF6D3B84E02F5F29@TYCPR01MB5870.jpnprd01.prod.outlook.com https://git.postgresql.org/pg/commitdiff/449ab6350526e99d33363706b759951ebad7928e

  • postgres_fdw: Revert unstable tests for postgres_fdw.application_name. Commit 449ab63505 added the tests that check that postgres_fdw.application_name GUC works as expected. But they were unstable and caused some buildfarm members to report the failure. This commit reverts those unstable tests. Reported-by: Tom Lane as per buildfarm Discussion: https://postgr.es/m/3220909.1631054766@sss.pgh.pa.us https://git.postgresql.org/pg/commitdiff/98dbef90eb29b13079ba3bd260b3c5818904ee86

  • Fix issue with WAL archiving in standby. Previously, walreceiver always closed the currently-opened WAL segment and created its archive notification file, after it finished writing the current segment up and received any WAL data that should be written into the next segment. If walreceiver exited just before any WAL data in the next segment arrived at standby, it did not create the archive notification file of the current segment even though that's known completed. This behavior could cause WAL archiving of the segment to be delayed until subsequent restartpoints or checkpoints created its notification file. To fix the issue, this commit changes walreceiver so that it creates an archive notification file of a current WAL segment immediately if that's known completed before receiving next WAL data. Back-patch to all supported branches. Reported-by: Kyotaro Horiguchi Author: Fujii Masao Reviewed-by: Kyotaro Horiguchi Discussion: https://postgr.es/m/20200630.165503.1465894182551545886.horikyota.ntt@gmail.com https://git.postgresql.org/pg/commitdiff/596ba75cb11173a528c6b6ec0142a282e42b69ec

  • pgbench: Stop counting skipped transactions as soon as timer is exceeded. When throttling is used, transactions that lag behind schedule by more than the latency limit are counted and reported as skipped. Previously, there was the case where pgbench counted all skipped transactions even if the timer specified in -T option was exceeded. This could take a very long time to do that especially when unrealistically high rate setting in -R option caused quite a lot of transactions that lagged behind schedule. This could prevent pgbench from ending immediately, and so pgbench could look like it got stuck to users. To fix the issue, this commit changes pgbench so that it stops counting skipped transactions as soon as the timer is exceeded. The timer can make pgbench end soon even when there are lots of skipped transactions that have not been counted yet. Note that there is no guarantee that all skipped transactions are counted under -T though there is under -t. This is OK in practice because it's very unlikely to happen with realistic setting. Also this is not the issue that this commit newly introdues. There used to be the case where pgbench ended without counting all skipped transactions since before. Back-patch to v14. Per discussion, we decided not to bother back-patch to the stable branches because it's hard to imagine the issue happens in practice (with realistic setting). Author: Yugo Nagata, Fabien COELHO Reviewed-by: Greg Sabino Mullane, Fujii Masao Discussion: https://postgr.es/m/20210613040151.265ff59d832f835bbcf8b3ba@sraoss.co.jp https://git.postgresql.org/pg/commitdiff/9bcbd7c581a5de3b915ef8fe0262e4abd9cb6e59

Tom Lane pushed:

  • Make timetz_zone() stable, and correct a bug for DYNTZ abbreviations. Historically, timetz_zone() has used time(NULL) as the reference point for deciding whether DST is active. That means its result can change intra-statement, requiring it to be marked VOLATILE (cf. 35979e6c3). But that definition is pretty inconsistent with the way we deal with timestamps elsewhere. Let's make it use the transaction start time ("now()") as the reference point instead. That lets it be marked STABLE, and also saves a kernel call per invocation. While at it, remove the function's use of pg_time_t and pg_localtime. Those are inconsistent with the other code in this area, which indeed created a bug: timetz_zone() delivered completely wrong answers if the zone was specified by a dynamic TZ abbreviation. (We need to do something about that in the back branches, but the fix will look different from this.) Aleksander Alekseev and Tom Lane Discussion: https://postgr.es/m/CAJ7c6TOMG8zSNEZtCn5SPe+cCk3Lfxb71ZaQwT2F4T7PJ_t=KA@mail.gmail.com https://git.postgresql.org/pg/commitdiff/388e71af880d579212c2489686903c2cfdea9032

  • Fix actively-misleading comments about the contents of struct pg_tm. pgtime.h documented the PG interpretation of tm_mon right alongside the POSIX interpretation of tm_year, with no hint that neither comment was correct throughout our code. Perhaps someday we ought to switch to using two separate struct definitions to provide a clearer indication of which semantics are in use where. But I fear the tedium-versus-safety-gain tradeoff would not be very good. Discussion: https://postgr.es/m/CAJ7c6TOMG8zSNEZtCn5SPe+cCk3Lfxb71ZaQwT2F4T7PJ_t=KA@mail.gmail.com https://git.postgresql.org/pg/commitdiff/89dba59590fdd03799a47daf8019890d4324fbcf

  • Further fix psql query-cancel test. The query to wait for pg_sleep to be running did no such thing, because the regex pattern it used could match itself. Report: https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=conchuela&dt=2021-09-06%2018%3A00%3A20 https://git.postgresql.org/pg/commitdiff/bd5846e4a9c1338ded5efcef53511f0d71f53f0e

  • Fix rewriter to set hasModifyingCTE correctly on rewritten queries. If we copy data-modifying CTEs from the original query to a replacement query (from a DO INSTEAD rule), we must set hasModifyingCTE properly in the replacement query. Failure to do this can cause various unpleasantness, such as unsafe usage of parallel plans. The code also neglected to propagate hasRecursive, though that's only cosmetic at the moment. A difficulty arises if the rule action is an INSERT...SELECT. We attach the original query's RTEs and CTEs to the sub-SELECT Query, but data-modifying CTEs are only allowed to appear in the topmost Query. For the moment, throw an error in such cases. It would probably be possible to avoid this error by attaching the CTEs to the top INSERT Query instead; but that would require a bunch of new code to adjust ctelevelsup references. Given the narrowness of the use-case, and the need to back-patch this fix, it does not seem worth the trouble for now. We can revisit this if we get field complaints. Per report from Greg Nancarrow. Back-patch to all supported branches. (The test case added here does not fail before v10, but there are plenty of places checking top-level hasModifyingCTE in 9.6, so I have no doubt that this code change is necessary there too.) Greg Nancarrow and Tom Lane Discussion: https://postgr.es/m/CAJcOf-f68DT=26YAMz_i0+Au3TcLO5oiHY5=fL6Sfuits6r+_w@mail.gmail.com Discussion: https://postgr.es/m/CAJcOf-fAdj=nDKMsRhQzndm-O13NY4dL6xGcEvdX5Xvbbi0V7g@mail.gmail.com https://git.postgresql.org/pg/commitdiff/362e2dcc46195faadd3fa0ba011dd9a8e3829e7a

  • In psql tab completion, offer spelled-out commands not abbreviations. Various psql backslash commands have both single-letter and long forms, for example \e and \edit. Previously, tab completion generally offered the single-letter form but not the long form. It seems more sensible to offer the long form, because (a) no useful completion can happen when you've already typed the single letter, and (b) if you're not so familiar with the command set as to know that, the long form is likely to be less confusing. Haiying Tang, reviewed by Dagfinn Ilmari Mannsåker and myself Discussion: https://postgr.es/m/OS0PR01MB61136018064660F095CB57A8FB129@OS0PR01MB6113.jpnprd01.prod.outlook.com https://git.postgresql.org/pg/commitdiff/7cffa2ed0c9f7f4d96bac7af5284c47e82af5ffa

  • Fix misleading comments about TOAST access macros. Seems to have been my error in commit aeb1631ed. Noted by Christoph Berg. Discussion: https://postgr.es/m/YTeLipdnSOg4NNcI@msg.df7cb.de https://git.postgresql.org/pg/commitdiff/490798451a3adc32b71b30e285bd99875d67fa2b

  • Avoid useless malloc/free traffic around getFormattedTypeName(). Coverity complained that one caller of getFormattedTypeName() failed to free the returned string. Which is true, but rather than fixing that one, let's get rid of this tedious and error-prone requirement. Now that getFormattedTypeName() caches its result, strdup'ing that result and expecting the caller to free it accomplishes little except to waste cycles. We do create a leak in the case where getTypes didn't make a TypeInfo for the type, but that basically shouldn't ever happen. Back-patch, as commit 6c450a861 was. This isn't a particularly interesting bug fix, but the API change seems like a hazard for future back-patching activity if we don't back-patch it. https://git.postgresql.org/pg/commitdiff/072e2f8a62002cb01ed6c4e161442e133509349e

  • Check for relation length overrun soon enough. We don't allow relations to exceed 2^32-1 blocks, because block numbers are 32 bits and the last possible block number is reserved to mean InvalidBlockNumber. There is a check for this in mdextend, but that's really way too late, because the smgr API requires us to create a buffer for the block-to-be-added, and we do not want to have any buffer with blocknum InvalidBlockNumber. (Such a case can trigger assertions in bufmgr.c, plus I think it might confuse ReadBuffer's logic for data-past-EOF later on.) So put the check into ReadBuffer. Per report from Christoph Berg. It's been like this forever, so back-patch to all supported branches. Discussion: https://postgr.es/m/YTn1iTkUYBZfcODk@msg.credativ.de https://git.postgresql.org/pg/commitdiff/8481f99896a192e9fd57f5e1a99e255e27680a10

  • Avoid fetching from an already-terminated plan. Some plan node types don't react well to being called again after they've already returned NULL. PortalRunSelect() has long dealt with this by calling the executor with NoMovementScanDirection if it sees that we've already run the portal to the end. However, commit ba2c6d6ce overlooked this point, so that persisting an already-fully-fetched cursor would fail if it had such a plan. Per report from Tomas Barton. Back-patch to v11, as the faulty commit was. (I've omitted a test case because the type of plan that causes a problem isn't all that stable.) Discussion: https://postgr.es/m/CAPV2KRjd=ErgVGbvO2Ty20tKTEZZr6cYsYLxgN_W3eAo9pf5sw@mail.gmail.com https://git.postgresql.org/pg/commitdiff/cba79a163267a44205e391137deb543f4f89bc8b

  • Fix some anomalies with NO SCROLL cursors. We have long forbidden fetching backwards from a NO SCROLL cursor, but the prohibition didn't extend to cases in which we rewind the query altogether and then re-fetch forwards. I think the reason is that this logic was mainly meant to protect plan nodes that can't be run in the reverse direction. However, re-reading the query output is problematic if the query is volatile (which includes SELECT FOR UPDATE, not just queries with volatile functions): the re-read can produce different results, which confuses the cursor navigation logic completely. Another reason for disliking this approach is that some code paths will either fetch backwards or rewind-and-fetch-forwards depending on the distance to the target row; so that seemingly identical use-cases may or may not draw the "cursor can only scan forward" error. Hence, let's clean things up by disallowing rewind as well as fetch-backwards in a NO SCROLL cursor. Ordinarily we'd only make such a definitional change in HEAD, but there is a third reason to consider this change now. Commit ba2c6d6ce created some new user-visible anomalies for non-scrollable cursors WITH HOLD, in that navigation in the cursor result got confused if the cursor had been partially read before committing. The only good way to resolve those anomalies is to forbid rewinding such a cursor, which allows removal of the incorrect cursor state manipulations that ba2c6d6ce added to PersistHoldablePortal. To minimize the behavioral change in the back branches (including v14), refuse to rewind a NO SCROLL cursor only when it has a holdStore, ie has been held over from a previous transaction due to WITH HOLD. This should avoid breaking most applications that have been sloppy about whether to declare cursors as scrollable. We'll enforce the prohibition across-the-board beginning in v15. Back-patch to v11, as ba2c6d6ce was. Discussion: https://postgr.es/m/3712911.1631207435@sss.pgh.pa.us https://git.postgresql.org/pg/commitdiff/c1b7a6c2731241cf5af4c08de54a64fc8999d727

  • Make pg_regexec() robust against out-of-range search_start. If search_start is greater than the length of the string, we should just return REG_NOMATCH immediately. (Note that the equality case should not be rejected, since the pattern might be able to match zero characters.) This guards various internal assumptions that the min of a range of string positions is not more than the max. Violation of those assumptions could allow an attempt to fetch string[search_start-1], possibly causing a crash. Jaime Casanova pointed out that this situation is reachable with the new regexp_xxx functions that accept a user-specified start position. I don't believe it's reachable via any in-core call site in v14 and below. However, extensions could possibly call pg_regexec with an out-of-range search_start, so let's back-patch the fix anyway. Discussion: https://postgr.es/m/20210911180357.GA6870@ahch-to https://git.postgresql.org/pg/commitdiff/e757080e041214cf6983e3e77ef01e83f1371d72

Álvaro Herrera pushed:

Noah Misch pushed:

Amit Kapila pushed:

Heikki Linnakangas pushed:

Andres Freund pushed:

  • windows: Only consider us to be running as service if stderr is invalid. Previously pgwin32_is_service() would falsely return true when postgres is started from somewhere within a service, but not as a service. That is e.g. always the case with windows docker containers, which some CI services use to run windows tests in. When postgres falsely thinks its running as a service, no messages are writting to stdout / stderr. That can be very confusing and causes a few tests to fail. To fix additionally check if stderr is invalid in pgwin32_is_service(). For that to work in backend processes, pg_ctl is changed to pass down handles so that postgres can do the same check (otherwise "default" handles are created). While this problem exists in all branches, there have been no reports by users, the prospective CI usage currently is only for master, and I am not a windows expert. So doing the change in only master for now seems the sanest approach. Author: Andres Freund andres@anarazel.de Reviewed-By: Magnus Hagander magnus@hagander.net Discussion: https://postgr.es/m/20210305185752.3up5eq2eanb7ofmb@alap3.anarazel.de https://git.postgresql.org/pg/commitdiff/76e38b37a5f179d4c9d2865ff31b79130407530b

Magnus Hagander pushed:

Daniel Gustafsson pushed: