== PostgreSQL Weekly News - April 19, 2020 ==

== PostgreSQL Weekly News - April 19, 2020 ==

== PostgreSQL Product News ==

pg_normalize_query 1.0, an extension to expose to SQL level a way to normalize
query similar that used by to pg_stat_statements, released.
https://github.com/fabriziomello/pg_normalize_query

pspg 3.0.0 a pager designed for PostgreSQL, released.
https://github.com/okbob/pspg/releases/tag/3.0.0

pgtt 2.0, an extension to implement global temporary tables, released.
https://github.com/darold/pgtt/releases/tag/v2.0

repmgr 5.1 a replication manager for PostgreSQL, released.
https://repmgr.org/docs/current/release-5.1.0.html

pgmetrics 1.9, a command-line tool for PostgreSQL metrics, released.
https://pgmetrics.io/

Pyrseas 0.9.1, a set of utilities for comparing and synchronizing Postgres
database schemas, released.
https://github.com/pyrseas/Pyrseas

pg_readonly 0.0.1, an extension that makes it possible to set all cluster
databases read-only, released.
https://github.com/pierreforstmann/pg_readonly

== PostgreSQL Jobs for April ==

http://archives.postgresql.org/pgsql-jobs/2020-04/

== PostgreSQL Local ==

PGCon 2020 will take place online on May 26-29, 2020.
https://www.pgcon.org/2020/

PostgresLondon 2020 will be July 7-8, 2020 with an optional training day on
July 6.
http://postgreslondon.org

PG Day Russia will be in Saint Petersburg on July 10, 2020.
https://pgday.ru/en/2020/

FOSS4G 2020, will take place in Calgary, Alberta, Canada August 24-29 2020.
the Call for Papers is currently open at https://2020.foss4g.org/speakers/
https://2020.foss4g.org/

PGDay Ukraine will take place September 5th, 2020 in Lviv at the Bank Hotel.
https://pgday.org.ua/

pgDay Israel 2020 will take place on September 10, 2020 in Tel Aviv.
http://pgday.org.il/

PGDay Austria will take place September 18, 2020 at Schloss Schoenbrunn
(Apothekertrakt) in Vienna.
https://pgday.at/en/

== PostgreSQL in the News ==

Planet PostgreSQL: http://planet.postgresql.org/

PostgreSQL Weekly News is brought to you this week by David Fetter

Submit news and announcements by Sunday at 3:00pm PST8PDT to david(at)fetter(dot)org(dot)

== Applied Patches ==

Tom Lane pushed:

- Doc: introduce and document "&zwsp;" for allowing optional line breaks. We
already had a couple of places using zero-width spaces for formatting hackery,
and we're going to need more if we ever want the PDF manuals to look decent.
But please let's not write hard-coded Unicode escapes. We can avoid that by
using a custom entity, which also provides a place to put a teeny bit of
documentation about what it is and how to use it. I'd previously posted a
patch using "&break;" for this, but on reflection that would be horrible to
grep for. Instead let's use "&zwsp;", based on the name of the Unicode symbol
("zero width space"). Discussion:
https://postgr.es/m/9326.1581457869@sss.pgh.pa.us
https://git.postgresql.org/pg/commitdiff/88d934f0387a66ba372643913f99e845d0ea144a

- Doc: introduce new layout for tables of functions and operators. We've long
fought with the draconian space limitations of our traditional table layout
for describing SQL functions and operators. This commit introduces a new
approach, though so far I've only applied it to a few of those tables. The
new way makes use of DocBook's support for different layouts in different rows
of a table, and allows the descriptions and examples for a function or
operator to run to several lines without as much ugliness and wasted space as
before. The core layout concept is now Name Signature
Description Example Example Result so that a function or
operator really has three table rows not one, but we group them to look like
one row by having the name column have only one entry for all three rows.
(Actually, there could be four or more rows if you wanted to have more than
one example, which is another thing that was painful before but works easily
now.) This is handled by a "morerows" annotation on the name entry, which
isn't perfect (notably, the toolchain is not smart enough to avoid breaking
these row groups across PDF pages) but there seems no better solution in
DocBook. The name column is normally fairly narrow, allowing plenty of space
for the other column(s), and not wasting too much space when one of the other
components runs to multiple lines. The varying row layout is managed by
defining named "spans" and then tagging entries with a "spanname" of "name",
"sig", "desc", "example", or "exresult". This provides a bit of semantic
annotation to go with the formatting improvement, which seems like a good
thing. (It seems that we have to re-define these spans afresh for each table,
which is annoying, but it's not any worse than the duplication involved in the
table headers. At least that gives us an opportunity to vary the relative
column widths per-table, which is handy since function tables sometimes need
much wider name columns than operator tables.) Signature entries should be
written in the style <function>fname</function>(<type>typename</type> ...)
<returnvalue>typename</returnvalue> The <returnvalue> tag produces a right
arrow before the result type name. (I'll document that convention in a
user-visible place later.) While this provides significantly more horizontal
space than before for examples, it's still true that PDF output is a lot
narrower than typical webpage viewing windows, so some examples need to be
broken in places where there is no whitespace. I've added &zwsp; markers in
suitable places to allow the tables to render warning-free in PDF. I've so
far converted only the date/time operator, date/time function, and enum
function tables in sections 9.9 and 9.10; these were chosen to provide a
reasonable sample of the formatting problems that need to be solved. Assuming
that this looks good on the website and doesn't provoke howls of anguish, I'll
work on the other similar tables in the near future. There's a moderate
amount of new editorial content in this patch along with the raw formatting
changes; for instance I had to write text descriptions for operators that
lacked them. I failed to resist the temptation to improve some other
descriptions and examples, too. Patch by me, with thanks to Alexander Lakhin
for assistance with figuring out some formatting issues. Discussion:
https://postgr.es/m/9326.1581457869@sss.pgh.pa.us
https://git.postgresql.org/pg/commitdiff/e894c61836e4b967f7ec65358fdaed2ba86ed238

- Account for collation when coercing the output of a SQL function. Commit
913bbd88d overlooked that the result of coerce_to_target_type might need
collation fixups. Per report from Andreas Joseph Krogh. Discussion:
https://postgr.es/m/VisenaEmail.72.37d08ec2b8cb8fb5.17179940cd3@tc7-visena
https://git.postgresql.org/pg/commitdiff/2d59643dbccaf73bc1f90875ea02dcad641379cd

- Fix cache reference leak in contrib/sepgsql. fixup_whole_row_references() did
the wrong thing with a dropped column, resulting in a commit-time warning
about a cache reference leak. I (tgl) added a test case exercising this, but
back-patched the test only as far as v10; the patch didn't apply cleanly to
9.6 and it didn't seem worth the trouble to adapt it. The bug is pretty old
though, so apply the code change all the way back. Michael Luo, with cosmetic
improvements by me Discussion:
https://postgr.es/m/BYAPR08MB5606D1453D7F50E2AF4D2FD29AD80@BYAPR08MB5606.namprd08.prod.outlook.com
https://git.postgresql.org/pg/commitdiff/fc576b7c4f3a5f045d443fc771d9e0f54b33e72b

- Fix possible future cache reference leak in ALTER EXTENSION ADD/DROP.
recordExtObjInitPriv and removeExtObjInitPriv were sloppy about calling
ReleaseSysCache. The cases cannot occur given current usage in ALTER
EXTENSION ADD/DROP, since we wouldn't get here for these relkinds; but it
seems wise to clean up better. In passing, extend test logic in test_pg_dump
to exercise the dropped-column code paths here. Since the case is unreachable
at present, there seems no great need to back-patch; hence fix HEAD only.
Kyotaro Horiguchi, with test case and comment adjustments by me Discussion:
https://postgr.es/m/20200417.151831.1153577605111650154.horikyota.ntt@gmail.com
https://git.postgresql.org/pg/commitdiff/3125a5baec1cf6d3aaeb8964bc3b3c49835e0452

- Doc: revise formatting of function/operator tables. The table layout ideas
proposed in commit e894c6183 were not as widely popular as I'd hoped. After
discussion, we've settled on a layout that's effectively a single-column table
with cell contents much like a <varlistentry> description of the function or
operator; though we're not actually using <varlistentry>, because it'd add way
too much vertical space. Instead the effect is accomplished using line-break
processing instructions to separate the description and example(s), plus CSS
or FO customizations to produce indentation of all but the first line in each
cell. While technically this is a bit grotty, it does have the advantage that
we won't need to write nearly as much boilerplate markup. This patch updates
tables 9.30, 9.31, and 9.33 (which were touched by the previous patch) to the
revised style, and additionally converts table 9.10. A lot of work still
remains to do, but hopefully it won't be too controversial. Thanks to Andrew
Dunstan, Pierre Giraud, Robert Haas, Alvaro Herrera, David Johnston, Jonathan
Katz, Isaac Morland for valuable ideas. Discussion:
https://postgr.es/m/8691.1586798003@sss.pgh.pa.us
https://git.postgresql.org/pg/commitdiff/737d69ffc3cfb2e093975411c1becd65ad029478

- Fix race conditions in synchronous standby management. We have repeatedly seen
the buildfarm reach the Assert(false) in SyncRepGetSyncStandbysPriority. This
apparently is due to failing to consider the possibility that the
sync_standby_priority values in shared memory might be inconsistent; but they
will be whenever only some of the walsenders have updated their values after a
change in the synchronous_standby_names setting. That function is vastly too
complex for what it does, anyway, so rewriting it seems better than trying to
apply a band-aid fix. Furthermore, the API of SyncRepGetSyncStandbys is
broken by design: it returns a list of WalSnd array indexes, but there is
nothing guaranteeing that the contents of the WalSnd array remain stable.
Thus, if some walsender exits and then a new walsender process takes over that
WalSnd array slot, a caller might make use of WAL position data that it should
not, potentially leading to incorrect decisions about whether to release
transactions that are waiting for synchronous commit. To fix, replace
SyncRepGetSyncStandbys with a new function SyncRepGetCandidateStandbys that
copies all the required data from shared memory while holding the relevant
mutexes. If the associated walsender process then exits, this data is still
safe to make release decisions with, since we know that that much WAL *was*
sent to a valid standby server. This incidentally means that we no longer
need to treat sync_standby_priority as protected by the SyncRepLock rather
than the per-walsender mutex. SyncRepGetSyncStandbys is no longer used by the
core code, so remove it entirely in HEAD. However, it seems possible that
external code is relying on that function, so do not remove it from the back
branches. Instead, just remove the known-incorrect Assert. When the bug
occurs, the function will return a too-short list, which callers should treat
as meaning there are not enough sync standbys, which seems like a reasonably
safe fallback until the inconsistent state is resolved. Moreover it's
bug-compatible with what has been happening in non-assert builds. We cannot
do anything about the walsender-replacement race condition without an API/ABI
break. The bogus assertion exists back to 9.6, but 9.6 is sufficiently
different from the later branches that the patch doesn't apply at all. I chose
to just remove the bogus assertion in 9.6, feeling that the probability of a
bad outcome from the walsender-replacement race condition is too low to
justify rewriting the whole patch for 9.6. Discussion:
https://postgr.es/m/21519.1585272409@sss.pgh.pa.us
https://git.postgresql.org/pg/commitdiff/f332241a60aa9c0945d74642cb3dbcbc11621154

- Doc: sync functableentry markup choices with website style. Jonathan Katz felt
that slightly different indentation settings made for a better-looking result,
so sync stylesheet-fo.xsl (for PDF) and stylesheet.css (for non-website-style
HTML) with those choices. Discussion:
https://postgr.es/m/31464.1587156281@sss.pgh.pa.us
https://git.postgresql.org/pg/commitdiff/00f4aba46d537452da41e6d398f8f4c915de586d

- Doc: update sections 9.1-9.3 for new function table layout. I took the
opportunity to do some copy-editing in this area as well, and to add some new
material such as a note about BETWEEN's syntactical peculiarities. Of note is
that quite a few of the examples of transcendental functions needed to be
updated, because the displayed output no longer matched what you get on a
modern server. I believe some of these cases are side-effects of the new Ryu
algorithm in float8out. Others appear to be because the examples predate the
addition of type numeric, and were expecting that float8 calculations would be
done although the given syntax would actually lead to calling the numeric
function nowadays.
https://git.postgresql.org/pg/commitdiff/81e83216d5b6826167d5116bd084b7b3a364b0b3

Peter Eisentraut pushed:

- Improve error messages after LoadLibrary(). Move the file name to a format
parameter to ease translatability. Add error code where missing. Make the
wording consistent.
https://git.postgresql.org/pg/commitdiff/0c620a5803afbb667da7ac4bcc91b245a214a80c

- Fix update-unicode target. The normalization-check target needs to be run
last, after moving the newly generated files into place. Also, we need an
additional dependency so that unicode_norm.o is rebuilt first. Otherwise,
norm_test will still test the old files but against the new expected results,
which will probably fail.
https://git.postgresql.org/pg/commitdiff/73afabcdc20e227beb8094efb44753b4de1c5c0a

Amit Kapila pushed:

- Cosmetic fixups for WAL usage work. Reported-by: Justin Pryzby and Euler
Taveira Author: Justin Pryzby and Julien Rouhaud Reviewed-by: Amit Kapila
Discussion:
https://postgr.es/m/CAB-hujrP8ZfUkvL5OYETipQwA=e3n7oqHFU=4ZLxWS_Cza3kQQ@mail.gmail.com
https://git.postgresql.org/pg/commitdiff/ef08ca113fe302b9ddf298964756eadcd41e77a1

- Comments and doc fixes for commit 40d964ec99. Reported-by: Justin Pryzby
Author: Justin Pryzby, with few changes by me Reviewed-by: Amit Kapila and
Sawada Masahiko Discussion:
https://postgr.es/m/20200322021801.GB2563@telsasoft.com
https://git.postgresql.org/pg/commitdiff/a6fea120a7e3858e642bb5e96027f166a1a6f134

- Fix the usage of parallel and full options of vacuum command. Earlier we were
inconsistent in allowing the usage of parallel and full options. Change it
such that we disallow them only when they are combined in a way that we don't
support. In passing, improve the comments in some of the existing tests of
parallel vacuum. Reported-by: Tushar Ahuja Author: Justin Pryzby, Amit Kapila
Reviewed-by: Sawada Masahiko, Michael Paquier, Mahendra Singh Thalor and Amit
Kapila Discussion:
https://postgr.es/m/58c8d171-e665-6fa3-a9d3-d9423b694dae%40enterprisedb.com
https://git.postgresql.org/pg/commitdiff/24d2d38b1eb86c0b410ad0f07f66566a83c6f05c

Robert Haas pushed:

- Rename pg_validatebackup to pg_verifybackup some more. The previous commit
missed an instance. Noriyoshi Shinoda Discussion:
http://postgr.es/m/TU4PR8401MB115291AE850BA7CF1AEB2F0BEEDD0@TU4PR8401MB1152.NAMPRD84.PROD.OUTLOOK.COM
https://git.postgresql.org/pg/commitdiff/7a6b017b34783eb7ee4e93f93a52bde2bb017c8c

- Document the backup manifest file format. Patch by me, at the request of
Andres Freund. Reviewed by Justin Pryzby, Erik Rijkers, Álvaro Herrera, and
Andrew Dunstan. Discussion:
http://postgr.es/m/20200327203225.hcm6ag4grwsiruea@alap3.anarazel.de
https://git.postgresql.org/pg/commitdiff/149f2ae88ab0510be3239caf45699d7373f2c6e8

Andrew Dunstan pushed:

- Print policy name in perlcritic messages. This makes it easier to do a web
search for details of the policy that's been violated, as well as displaying
the name that might be needed for a policy override. Various perlcritic
settings changes are being discussed, but this one should be uncontroversial.
https://git.postgresql.org/pg/commitdiff/8930e43ecd3f683c457865131d7a932401a2188f

- Use perl warnings pragma consistently. We've had a mixture of the warnings
pragma, the -w switch on the shebang line, and no warnings at all. This patch
removes the -w swicth and add the warnings pragma to all perl sources missing
it. It raises the severity of the TestingAndDebugging::RequireUseWarnings
perlcritic policy to level 5, so that we catch any future violations.
Discussion: https://postgr.es/m/20200412074245.GB623763@rfd.leadboat.com
https://git.postgresql.org/pg/commitdiff/7be5d8df1f74b78620167d3abf32ee607e728919

- Use perl's $/ more idiomatically. This replaces a few occurrences of ugly code
with a more clean and idiomatic usage. The problem was highlighted by
perlcritic, but we're not enforcing the policy that led to the discovery.
Discussion: https://postgr.es/m/20200412074245.GB623763@rfd.leadboat.com
https://git.postgresql.org/pg/commitdiff/8f00d84afc0dad577b65df5a313e5306cee3d11f

- Set Perl search path more idiomatically. Back in commits 1df92eeafe,
f884a96819, and 592123efbb I used some hackish code to set the script search
path, unaware despite decades of perl that there was a completely standard way
to do this. This patch changes those cases to use the standard perl FindBin
package.
https://git.postgresql.org/pg/commitdiff/e60c6f6ea17b50833242653fe2781f6d47429540

- Stop requiring an explicit return from perl subroutines. The consensus of the
project appears to be that this provides little benefit and is simply an
annoyance. Discussion: https://postgr.es/m/27481.1586618092@sss.pgh.pa.us
https://git.postgresql.org/pg/commitdiff/0516f94d18c57cc5dce72ff8fb84a05d24a67063

- Only provide new libpq sslpasskey hook for openssl-enabled builds. In commit
4dc6355210 I neglected to put #ifdef USE_OPENSSL around the declarations of
the new items. This is remedied here. Per complaint from Daniel Gustafsson.
https://git.postgresql.org/pg/commitdiff/9e24109f1a4e4d8d1d372b004d6a0dd06e673fe7

- Use a slightly more liberal regex to detect Visual Studio version. Apparently
in some language versions of Visual Studio nmake outputs some material after
the version number and before the end of the line. This has been seen in
Chinese versions. Therefore, we no longer demand that the version string comes
at the end of a line. Per complaint from Cuiping Lin. Backpatch to all live
branches.
https://git.postgresql.org/pg/commitdiff/a9659fb65497af8d2db66101f2ff7233d02997f1

- Only provide openssl_tls_init_hook if building with openssl. This should have
been protected by #ifdef USE_OPENSSL in commit 896fcdb230. Per the real
complaint this time from Daniel Gustafsson.
https://git.postgresql.org/pg/commitdiff/f342d7ad03e61a1cea0339cf0c5aea0d01c3b43e

- Revert "Only provide new libpq sslpasskey hook for openssl-enabled builds".
This reverts commit 9e24109f1a4e4d8d1d372b004d6a0dd06e673fe7. This caused
build errors when building without openssl, and it's simplest just to revert
it.
https://git.postgresql.org/pg/commitdiff/6741cfa5964768279a13e01e46931664fe0903a4

Peter Geoghegan pushed:

- Harmonize nbtree page split point code. An nbtree split point can be thought
of as a point between two adjoining tuples from an imaginary version of the
page being split that includes the incoming/new item (in addition to the items
that really are on the page). These adjoining tuples are called the lastleft
and firstright tuples. The variables that represent split points contained a
field called firstright, which is an offset number of the first data item from
the original page that goes on the new right page. The corresponding tuple
from origpage was usually the same thing as the actual firstright tuple, but
not always: the firstright tuple is sometimes the new/incoming item instead.
This situation seems unnecessarily confusing. Make things clearer by renaming
the origpage offset returned by _bt_findsplitloc() to "firstrightoff". We now
have a firstright tuple and a firstrightoff offset number which are comparable
to the newitem/lastleft tuples and the newitemoff/lastleftoff offset numbers
respectively. Also make sure that we are consistent about how we describe
nbtree page split point state. Push the responsibility for dealing with
pg_upgrade'd !heapkeyspace indexes down to lower level code, relieving
_bt_split() from dealing with it directly. This means that we always have a
palloc'd left page high key on the leaf level, no matter what. This enables
simplifying some of the code (and code comments) within _bt_split(). Finally,
restructure the page split code to make it clearer why suffix truncation
(which only takes place during leaf page splits) is completely different to
the first data item truncation that takes place during internal page splits.
Tuples are marked as having fewer attributes stored in both cases, and the
firstright tuple is truncated in both cases, so it's easy to imagine somebody
missing the distinction.
https://git.postgresql.org/pg/commitdiff/bc3087b626d1073c9b7c9687b334785909ca2237

- Make _bt_insertonpg() more like _bt_split(). It seems like a good idea for
nbtree's retail insert code to be absolutely consistent with nbtree's page
split code for anything that naturally requires equivalent handling. Anything
that concerns inserting newitem (which is handled as part of the page split
atomic action when a page split is required) should work in exactly the same
way. With that in mind, make _bt_insertonpg() handle 'cbuf' in a way that
matches _bt_split().
https://git.postgresql.org/pg/commitdiff/826ee1a019127d611bb0fd22ca878142bfb077ac

- Add defensive "split_only_page" nbtree assertion. Clearly it's not okay for
nbtree to split a page that is the only page on its level, and then find that
it has to split the parent one level up in turn. There is simply no code to
handle the split_only_page case in the _bt_insertonpg() "newitem won't fit"
branch (only the "newitem fits" branch handles split_only_page). Add a
defensive assertion that will fail if a split_only_page call to
_bt_insertonpg() somehow ends up splitting the target/parent page. I
(pgeoghegan) believe that we don't need split_only_page handling for the
"newitem won't fit" branch because anybody calling _bt_insertonpg() like this
would have to hold a lock on the same one and only child page.
https://git.postgresql.org/pg/commitdiff/f762b2feba276a627585cb7e834fb7a1bf4c549d

- Rearrange _bt_insertonpg() "update metapage" code. Nest the "update metapage
as part of insert into root-like page" branch inside the broader "insert into
internal page" branch. This improves readability.
https://git.postgresql.org/pg/commitdiff/80634e3b181562015762b5846583ff67f8037a80

- Remove obsolete "hole in center of page" comment. A comment from the Berkeley
days incorrectly claimed that the page management code cares about the
contents of the hole in the center of the page (at least in the case of the
left half of an nbtree page split). Commit 8fa30f906be added an addendum that
stated that the original comment was "probably obsolete". It's definitely
obsolete, though, so remove the original comment plus the addendum.
https://git.postgresql.org/pg/commitdiff/4a05a6409567719829ec84d7689b41c55009d75f

- Slightly simplify nbtree split point choice loop. Spotted during post-commit
review of the nbtree deduplication commit (commit 0d861bbb).
https://git.postgresql.org/pg/commitdiff/f0ca378d4c139eda99ef14998115c1674dac3fc5

Álvaro Herrera pushed:

- Silence Perl warning. Now that warnings are enabled across the board, this
code that tries to print an undef variable emits one. Silently printing the
empty string achieves the previous behavior. Author: Álvaro Herrera
<alvherre(at)alvh(dot)no-ip(dot)org> Reviewed-by: Andrew Dunstan
<andrew(dot)dunstan(at)2ndquadrant(dot)com> Discussion:
https://postgr.es/m/E1jO1VT-0008Qk-TM@gemulon.postgresql.org
https://git.postgresql.org/pg/commitdiff/e56d717d8a23c48e4ab27b6bd6815c50dc3a378a

Noah Misch pushed:

- Add a wait_for_catchup() before immediate stop of a test master. Per buildfarm
member hoverfly, a slow walsender could make the test fail. Back-patch to
v10, where the test was introduced. Discussion:
https://postgr.es/m/20200414013849.GA886648@rfd.leadboat.com
https://git.postgresql.org/pg/commitdiff/d60cfb6bf2b4812e97271e7b5ba3ad4713406b9d

Michaël Paquier pushed:

- Fix collection of typos and grammar mistakes in the tree, volume 2. This fixes
some comments and documentation new as of Postgres 13, and is a follow-up of
the work done in dd0f37e. Author: Justin Pryzby Discussion:
https://postgr.es/m/20200408165653.GF2228@telsasoft.com
https://git.postgresql.org/pg/commitdiff/8128b0c152a67917535f50738ac26da4f984ddd9

- Fix minor memory leak in pg_dump. A query used to read default ACL information
from the catalogs did not free a set of PQExpBuffer. Oversight in commit
e2090d9, so backpatch down to 9.6. Author: Jie Zhang Reviewed-by: Sawada
Masahiko Discussion:
https://postgr.es/m/05bcbc5857f948efa0b451b85a48ae10@G08CNEXMBPEKD06.g08.fujitsu.local
Backpatch-through: 9.6
https://git.postgresql.org/pg/commitdiff/8f4ee44bcdbcd437f860f2d048125a118335b088

- Disable silently generation of manifests with servers <= 12 in pg_basebackup.
Since 0d8c9c1, pg_basebackup would generate an error if connected to a backend
version older than 12 where backup manifests are not supported. Avoiding this
error is possible by using the --no-manifest option. This error handling
could be confusing for some users, where patching a backup script that
interacts with multiple backend versions would cause the addition of
--no-manifest to potentially not generate a backup manifest even for Postgres
13 and newer versions. As we want to encourage the use of backup manifests as
much as possible, this commit silently disables manifests where not supported,
instead of generating an error. While on it, rework a bit the code to make it
more consistent with the surroundings when generating the BASE_BACKUP command.
Per discussion with Andres Freund, Stephen Frost, Robert Haas, Álvaro Herrera,
Kyotaro Horiguchi, Tom Lane, David Steele, and me. Author: Michael Paquier
Discussion: https://postgr.es/m/20200410080910.GZ1606@paquier.xyz
https://git.postgresql.org/pg/commitdiff/542d7817f774ea9d94798eb95cdf250d4f1527d9

- Fix minor memory leak in pg_basebackup and pg_receivewal. The result of the
query used to retrieve the WAL segment size from the backend was not getting
freed in two code paths. Both pg_basebackup and pg_receivewal exit
immediately if a failure happened on this query, so this was not an actual
problem, but it could be an issue if this code gets used for other tools in
different ways, be they future tools in this code tree or external, existing,
ones. Oversight in commit fc49e24, so backpatch down to 11. Author: Jie
Zhang Discussion:
https://postgr.es/m/970ad9508461469b9450b64027842331@G08CNEXMBPEKD06.g08.fujitsu.local
Backpatch-through: 11
https://git.postgresql.org/pg/commitdiff/198efe774b0d2065d828cf7822376a2871c671fc

Fujii Masao pushed:

- Code review for backup manifest. This commit prevents pg_basebackup from
receiving backup_manifest file when --no-manifest is specified. Previously,
when pg_basebackup was writing a tarfile to stdout, it tried to receive
backup_manifest file even when --no-manifest was specified, and reported an
error. Also remove unused -m option from pg_basebackup. Also fix typo in
BASE_BACKUP command documentation. Author: Fujii Masao Reviewed-by: Michael
Paquier, Robert Haas Discussion:
https://postgr.es/m/01e3ed3a-8729-5aaa-ca84-e60e3ca59db8@oss.nttdata.com
https://git.postgresql.org/pg/commitdiff/a2ac73e7be7adf2a9248d14322d3a5e055ea4fd0

- Add index term for backup manifest in documentation. Author: Fujii Masao
Reviewed-by: Robert Haas Discussion:
https://postgr.es/m/951743d0-fd7e-8e2b-d489-1368a58b7304@oss.nttdata.com
https://git.postgresql.org/pg/commitdiff/4db819ba403901d38c4e4328883412c59061bb58

David Rowley pushed:

- Remove unneeded constraint dependency tracking. It was previously thought that
remove_useless_groupby_columns() needed to keep track of which constraints the
generated plan depended upon, however, this is unnecessary. The confusion
likely arose regarding this because of check_functional_grouping(), which does
need to track the dependency to ensure VIEWs with columns which are
functionally dependant on the GROUP BY remain so. For
remove_useless_groupby_columns(), cached plans will just become invalidated
when the primary key's underlying index is removed through the normal relcache
invalidation code. Here we just remove the unneeded code which records the
dependency and updates the comments. The previous comments claimed that we
could not use UNIQUE constraints for the same optimization due to lack of a
pg_constraint record for NOT NULL constraints (which are required because
NULLs can be duplicated in a unique index). Since we don't actually need a
pg_constraint record to handle the invalidation, it looks like we could add
code to do this in the future. But not today. We're not really fixing any bug
in the code here, this fix is just to set the record straight on UNIQUE
constraints. This code was added back in 9.6, but due to lack of any bug,
we'll not be backpatching this. Reviewed-by: Tom Lane Discussion:
https://postgr.es/m/CAApHDvrdYa=VhOoMe4ZZjZ-G4ALnD-xuAeUNCRTL+PYMVN8OnQ@mail.gmail.com
https://git.postgresql.org/pg/commitdiff/5b736e9cf95793ca6a4c00d291a06dfe3559c8ec

- Fix possible crash with GENERATED ALWAYS columns. In some corner cases, this
could also lead to corrupted values being included in the tuple. Users who
are concerned that they are affected by this should first upgrade and then
perform a base backup of their database and restore onto an off-line server.
They should then query each table with generated columns to ensure there are
no rows where the generated expression does not match a newly calculated
version of the GENERATED ALWAYS expression. If no crashes occur and no rows
are returned then you're not affected. Fixes bug #16369. Reported-by:
Cameron Ezell Discussion:
https://postgr.es/m/16369-5845a6f1bef59884@postgresql.org Backpatch-through:
12 (where GENERATED ALWAYS columns were added.)
https://git.postgresql.org/pg/commitdiff/3cb02e307e350caf3c9099c6f661a95fd00e7e4c

== Pending Patches ==

Yugo Nagata sent in another revision of a patch to implement incremental
maintenance of materialized views.

Fabien COELHO sent in another revision of a patch to pgbench to enable testing
whether a variable exists.

Zeng Wenjing sent in another revision of a patch to implement global temporary
tables.

Peter Eisentraut sent in a patch to improve the error messages about mismatching
relkind.

Julien Rouhaud and Amit Kapila traded patches to fix up the WAL usage
calculation patch.

David Rowley and Andy Fan traded patches to introduce UniqueKeys to determine
RelOptInfo unique properties, and use same to skip DISTINCT / GROUP BY if the
input is already unique.

Asif Rehman sent in two more revisions of a patch to implement parallel
basebackup.

Dilip Kumar sent in two more revisions of a patch to fix an infelicity between
logical_work_mem and logical streaming of large in-progress transactions.

Davinder Singh, Juan José Santamaría Flecha, Amit Kapila, and Ranier Vilela
traded patches to fix a compilation error with Microsoft Visual Studio
2015/2017/2019.

Mark Dilger sent in a patch to add HeapFile related perl modules.

Andrey Borodin and Magnus Hagander traded patches to allow pg_read_all_stats to
read pg_stat_progress_*.

Thomas Munro sent in a patch to implement snapshot_too_old using a timer.

Kyotaro HORIGUCHI sent in a patch to pg restore to handle cfclose error
properly.

Cary Huang sent in another revision of a patch to support sequences in logical
replication.

Dilip Kumar sent in a patch to fix an issue with REPLICA IDENTITY FULL.

Richard Guo sent in a patch to fix an incremental sort crash found by sqlsmith.

Robert Haas and Thomas Munro traded patches to fix old_snapshot_threshold's
time->xid mapping.

Ranier Vilela sent in a patch to nbtree to avert some some unneeded calls and
set vars.

Ranier Vilela sent in a patch to fix a possible overflow on tuplesort.c.

Amit Langote sent in a patch to avoid partial nested append, which could break
run-time partition pruning.

Peter Eisentraut sent in a patch to make pgstattuple_approx accept TOAST tables.

Masahiko Sawada sent in a patch to fix an error in pg_dump involving generated
columns.

Amit Langote sent in a patch to rearrange some code in pgoutput.c, and fix some
partition logical replication TAP tests for better coverage.

Nikita Glukhov sent in a patch to return FALSE instead of NULL in jsonpath
operators.

Kyotaro HORIGUCHI sent in a patch to fix an aclchk cacheref leak.

Masahiro Ikeda sent in a patch to fix the documentation about the way to exit
standby mode.

Peter Geoghegan sent in a patch to redefine split interval to be space-wise.

Thomas Munro sent in a patch to rename xid to xid4.

Pavel Stěhule sent in two revisions of a patch to implement a string_to_table()
function.

Thomas Munro sent in two revisions of a patch to expose oldSnapshotControl, add
a time->xid mapping to contrib/old_snapshot, fix some bugs in
MaintainOldSnapshotTimeMapping, add a pg_clobber_current_snapshot_timestamp(),
and truncate old snapshot XIDs before truncating CLOG.

Noah Misch sent in a patch to revert "When WalSndCaughtUp, sleep only in
WalSndWaitForWal().

Robert Haas sent in a patch to move the server's backup manifest code to a
separate file.

Justin Pryzby sent in two revisions of a patch to fix explain output for
incremental sort, and fix some comment typos in same.

James Coleman sent in a patch to disable mark/restore for incremental sort, and
fix up EXEC_FLAG_REWIND for incremental sort.

Pavel Stěhule sent in a patch to make it possible to use a pager for psql's
\watch command.

Justin Pryzby sent in three more revisions of a patch to fix detaching tables
with inherited row triggers.

Ranier Vilela sent in a patch to remove a few unneeded uses of strlen().

Jeff Davis sent in a patch to fix a performance regression related to
FORTIFY_SOURCE.