== PostgreSQL Weekly News - April 12, 2020 ==

== PostgreSQL Weekly News - April 12, 2020 ==

There is now a "common" yum repository for those things which do not depend
directly on specific versions of PostgreSQL to operate. This reduces build times
for those that do so depend by factoring out the ones that don't.

== PostgreSQL Product News ==

psycopg2 2.8.5, a Python connector for PostgreSQL, released.
http://initd.org/psycopg/articles/2020/04/06/psycopg-285-released/

== PostgreSQL Local ==

PGCon 2020 will take place online on May 26-29, 2020.
https://www.pgcon.org/2020/

PostgresLondon 2020 will be July 7-8, 2020 with an optional training day on
July 6.
http://postgreslondon.org

PG Day Russia will be in Saint Petersburg on July 10, 2020.
https://pgday.ru/en/2020/

FOSS4G 2020, will take place in Calgary, Alberta, Canada August 24-29 2020.
the Call for Papers is currently open at https://2020.foss4g.org/speakers/
https://2020.foss4g.org/

PGDay Ukraine will take place September 5th, 2020 in Lviv at the Bank Hotel.
https://pgday.org.ua/

pgDay Israel 2020 will take place on September 10, 2020 in Tel Aviv.
http://pgday.org.il/

PGDay Austria will take place September 18, 2020 at Schloss Schoenbrunn
(Apothekertrakt) in Vienna. The CfP is open until April 19, 2020 at
https://pgday.at/en/talk-commitee/
https://pgday.at/en/

== PostgreSQL in the News ==

Planet PostgreSQL: http://planet.postgresql.org/

PostgreSQL Weekly News is brought to you this week by David Fetter

Submit news and announcements by Sunday at 3:00pm PST8PDT to david(at)fetter(dot)org(dot)

== Applied Patches ==

Andres Freund pushed:

- Fix recently introduced typo. Reported-By: David Rowley
https://git.postgresql.org/pg/commitdiff/549a3e23c3d618103487161b19dbbf8fd2206a5c

- Use TransactionXmin instead of RecentGlobalXmin in heap_abort_speculative().
There's a very low risk that RecentGlobalXmin could be far enough in the past
to be older than relfrozenxid, or even wrapped around. Luckily the
consequences of that having happened wouldn't be too bad - the page wouldn't
be pruned for a while. Avoid that risk by using TransactionXmin instead. As
that's announced via MyPgXact->xmin, it is protected against wrapping around
(see code comments for details around relfrozenxid). Author: Andres Freund
Discussion:
https://postgr.es/m/20200328213023.s4eyijhdosuc4vcj@alap3.anarazel.de
Backpatch: 9.5-
https://git.postgresql.org/pg/commitdiff/f946069e6827e729857b9f2db06bf27a1c0563ee

- Recompute stack base in forked postmaster children. This is for the benefit of
running postgres under the rr debugger. When using rr signal handlers running
while a syscall is active use an alternative stack. As e.g. bgworkers are
started from within signal handlers, the forked backend then has a different
stack base than postmaster. Previously that subsequently lead to those
processes triggering spurious "stack depth limit exceeded" errors.
Discussion:
https://postgr.es/m/20200327182217.ubrrl32lyfhxfwk5@alap3.anarazel.de
https://git.postgresql.org/pg/commitdiff/fc3f4453a2bc95549682e23600b22e658cb2d6d7

- Fix XLogReader FD leak that makes backends unusable after 2PC usage. Before
the fix every 2PC commit/abort leaked a file descriptor. As the files are
opened using BasicOpenFile(), that quickly leads to the backend running out of
file descriptors. Once enough 2PC abort/commit have caused enough FDs to
leak, any IO in the backend will fail with "Too many open files", as
BasicOpenFilePerm() will have triggered all open files known to fd.c to be
closed. The leak causing the problem at hand is a consequence of 0dc8ead46,
but is only exascerbated by it. Previously most XLogPageReadCB callbacks used
static variables to cache one open file, but after the commit the cache is
private to each XLogReader instance. There never was infrastructure to close
FDs at the time of XLogReaderFree, but the way XLogReader was used limited the
leak to one FD. This commit just closes the during XLogReaderFree() if the FD
is stored in XLogReaderState.seg.ws_segno. This may not be the way to solve
this medium/long term, but at least unbreaks 2PC. Discussion:
https://postgr.es/m/20200406025651.fpzdb5yyb7qyhqko@alap3.anarazel.de
https://git.postgresql.org/pg/commitdiff/91c40548d5f7a9241d9fa344fae8069cfdb92bf2

- snapshot scalability: Move delayChkpt from PGXACT to PGPROC. The goal of
separating hotly accessed per-backend data from PGPROC into PGXACT is to make
accesses fast (GetSnapshotData() in particular). But delayChkpt is not
actually accessed frequently; only when starting a checkpoint. As it is
frequently modified (multiple times in the course of a single transaction),
storing it in the same cacheline as hotly accessed data unnecessarily dirties
a contended cacheline. Therefore move delayChkpt to PGPROC. This is part of
a larger series of patches intending to improve GetSnapshotData() scalability.
It is committed and pushed separately, as it is independently beneficial
(small but measurable win, limited by the other frequent modifications of
PGXACT). Author: Andres Freund Reviewed-By: Robert Haas, Thomas Munro, David
Rowley Discussion:
https://postgr.es/m/20200301083601.ews6hz5dduc3w2se@alap3.anarazel.de
https://git.postgresql.org/pg/commitdiff/75848bc74411130ede23995d0ab1aefb12c4c4b0

Michaël Paquier pushed:

- Preserve clustered index after rewrites with ALTER TABLE. A table rewritten by
ALTER TABLE would lose tracking of an index usable for CLUSTER. This setting
is tracked by pg_index.indisclustered and is controlled by ALTER TABLE, so
some extra work was needed to restore it properly. Note that ALTER TABLE only
marks the index that can be used for clustering, and does not do the actual
operation. Author: Amit Langote, Justin Pryzby Reviewed-by: Ibrar Ahmed,
Michael Paquier Discussion:
https://postgr.es/m/20200202161718.GI13621@telsasoft.com Backpatch-through:
9.5
https://git.postgresql.org/pg/commitdiff/a40caf5f862ca8b7e927b2ab2567e934868e9376

- Refactor cluster.c to use new routine get_index_isclustered(). This new cache
lookup routine has been introduced in a40caf5, and more code paths can
directly use it. Note that in cluster_rel(), the code was returning
immediately if the tuple's entry in pg_index for the clustered index was not
valid. This commit changes the code so as a lookup error is raised instead,
something that could not happen from the start as we check for the existence
of the index beforehand, while holding an exclusive lock on the parent table.
Author: Justin Pryzby Reviewed-by: Álvaro Herrera, Michael Paquier Discussion:
https://postgr.es/m/20200202161718.GI13621@telsasoft.com
https://git.postgresql.org/pg/commitdiff/8ef9451f58ee92d1cdb910e72010dbe75e76f9b8

- Fix crash when using COLLATE in partition bound expressions. Attempting to use
a COLLATE clause with a type that it not collatable in a partition bound
expression could crash the server. This commit fixes the code by adding more
checks similar to what is done when computing index or partition attributes by
making sure that there is a collation iff the type is collatable. Backpatch
down to 12, as 7c079d7 introduced this problem. Reported-by: Alexander Lakhin
Author: Dmitry Dolgov Discussion:
https://postgr.es/m/16325-809194cf742313ab@postgresql.org Backpatch-through:
12
https://git.postgresql.org/pg/commitdiff/c0187869a0f6eb05135d388462522a593ced1b88

- Fix collection of typos and grammar mistakes in the tree. This fixes some
comments and documentation new as of Postgres 13. Author: Justin Pryzby
Discussion: https://postgr.es/m/20200408165653.GF2228@telsasoft.com
https://git.postgresql.org/pg/commitdiff/dd0f37eccecc2db5c0ffafefbf697a2c916e8bc3

Amit Kapila pushed:

- Add the option to report WAL usage in EXPLAIN and auto_explain. This commit
adds a new option WAL similar to existing option BUFFERS in the EXPLAIN
command. This option allows to include information on WAL record generation
added by commit df3b181499 in EXPLAIN output. This also allows the WAL usage
information to be displayed via the auto_explain module. A new parameter
auto_explain.log_wal controls whether WAL usage statistics are printed when an
execution plan is logged. This parameter has no effect unless
auto_explain.log_analyze is enabled. Author: Julien Rouhaud Reviewed-by:
Dilip Kumar and Amit Kapila Discussion:
https://postgr.es/m/CAB-hujrP8ZfUkvL5OYETipQwA=e3n7oqHFU=4ZLxWS_Cza3kQQ@mail.gmail.com
https://git.postgresql.org/pg/commitdiff/33e05f89c53e5a1533d624046bb6fb0da7bb7141

- Allow autovacuum to log WAL usage statistics. This commit allows autovacuum to
log WAL usage statistics added by commit df3b181499. Author: Julien Rouhaud
Reviewed-by: Dilip Kumar and Amit Kapila Discussion:
https://postgr.es/m/CAB-hujrP8ZfUkvL5OYETipQwA=e3n7oqHFU=4ZLxWS_Cza3kQQ@mail.gmail.com
https://git.postgresql.org/pg/commitdiff/b7ce6de93b59852c55d09acdaeebbf5aaf89114e

- Allow parallel create index to accumulate buffer usage stats. Currently, we
don't account for buffer usage incurred by parallel workers for parallel
create index.  This commit allows each worker to record the buffer usage stats
and leader backend to accumulate that stats at the end of the operation.  This
will allow pg_stat_statements to display correct buffer usage stats for
(parallel) create index command. Reported-by: Julien Rouhaud Author: Sawada
Masahiko Reviewed-by: Dilip Kumar, Julien Rouhaud and Amit Kapila
Backpatch-through: 11, where this was introduced Discussion:
https://postgr.es/m/20200328151721.GB12854@nol
https://git.postgresql.org/pg/commitdiff/5c71362174eb56676f8b91c73ec066dd5513fd4b

Peter Eisentraut pushed:

- Add logical replication support to replicate into partitioned tables. Mainly,
this adds support code in logical/worker.c for applying replicated operations
whose target is a partitioned table to its relevant partitions. Author: Amit
Langote <amitlangote09(at)gmail(dot)com> Reviewed-by: Rafia Sabih
<rafia(dot)pghackers(at)gmail(dot)com> Reviewed-by: Peter Eisentraut
<peter(dot)eisentraut(at)2ndquadrant(dot)com> Reviewed-by: Petr Jelinek
<petr(at)2ndquadrant(dot)com> Discussion:
https://www.postgresql.org/message-id/flat/CA+HiwqH=Y85vRK3mOdjEkqFK+E=ST=eQiHdpj43L=_eJMOOznQ(at)mail(dot)gmail(dot)com
https://git.postgresql.org/pg/commitdiff/f1ac27bfda6ce8a399d8001843e9aefff5814f9b

- Allow publishing partition changes via ancestors. To control whether partition
changes are replicated using their own identity and schema or an ancestor's,
add a new parameter that can be set per publication named
'publish_via_partition_root'. This allows replicating a partitioned table
into a different partition structure on the subscriber. Author: Amit Langote
<amitlangote09(at)gmail(dot)com> Reviewed-by: Rafia Sabih <rafia(dot)pghackers(at)gmail(dot)com>
Reviewed-by: Peter Eisentraut <peter(dot)eisentraut(at)2ndquadrant(dot)com> Reviewed-by:
Petr Jelinek <petr(at)2ndquadrant(dot)com> Discussion:
https://www.postgresql.org/message-id/flat/CA+HiwqH=Y85vRK3mOdjEkqFK+E=ST=eQiHdpj43L=_eJMOOznQ(at)mail(dot)gmail(dot)com
https://git.postgresql.org/pg/commitdiff/83fd4532a72179c370e318075a10e0e2aa832024

- createuser: Change a fprintf to pg_log_error.
https://git.postgresql.org/pg/commitdiff/f45b8e51b6838ab820df13983c194f737be48778

- Fix CREATE TABLE LIKE INCLUDING GENERATED column order issue. CREATE TABLE
LIKE INCLUDING GENERATED would fail if a generated column referred to a column
with a higher attribute number. This is because the column mapping mechanism
created the mapping incrementally as columns are added. This was sufficient
for previous uses of that mechanism (omitting dropped columns), and it also
happened to work if generated columns only referred to columns with lower
attribute numbers, but here it failed. This fix is to build the attribute
mapping in a separate loop before processing the columns in detail. Bug:
#16342 Reported-by: Ethan Waldo <ewaldo(at)healthetechs(dot)com> Reviewed-by: Tom
Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us>
https://git.postgresql.org/pg/commitdiff/e92e4a2b68fe877677278c1300db1780956c984e

- Fix relcache reference leak. Introduced by
83fd4532a72179c370e318075a10e0e2aa832024
https://git.postgresql.org/pg/commitdiff/5a1d0c9925fbda9ec434061dee74b1b5d9a16038

- Fix RELCACHE_FORCE_RELEASE issue. Introduced by
83fd4532a72179c370e318075a10e0e2aa832024. To fix, the tuple descriptors need
to be copied into the current memory context. Discussion:
https://www.postgresql.org/message-id/04d78603-edae-9243-9dde-fe3037176a7d@2ndquadrant.com
https://git.postgresql.org/pg/commitdiff/12fb189bfeaf03faea5b6f15544c1f71e9ef4677

Tom Lane pushed:

- Re-stabilize infinite_recurse() test case. Since commit 8f59f6b9c0,
CLOBBER_CACHE_ALWAYS buildfarm members have been failing this test case
because the error message now sometimes includes an error cursor position. It
seems largely just luck that that never happened before, and there are likely
to be more ways it could happen in future. Hence, rather than trying to
prevent it, adjust the test script to suppress that component of the report.
At some point we might need to back-patch this, but refrain until there's a
demonstrated need. (We'd need a different fix before v12, anyway, since
VERBOSITY=sqlstate is a recent thing.) Tom Lane and Andres Freund
Discussion: https://postgr.es/m/30675.1586111599@sss.pgh.pa.us
https://git.postgresql.org/pg/commitdiff/3c8553547b1493c4afdb80393f4a47dbfa019a79

- Fix representation of SORT_TYPE_STILL_IN_PROGRESS. It turns out that the code
did indeed rely on a zeroed TuplesortInstrumentation.sortMethod field to
indicate "this worker never did anything", although it seems the issue only
comes up during certain race-condition-y cases. Hence, rearrange the
TuplesortMethod enum to restore SORT_TYPE_STILL_IN_PROGRESS to having the
value zero, and add some comments reinforcing that that isn't optional. Also
future-proof a loop over the possible values of the enum. sizeof(bits32)
happened to be the correct limit value, but only by purest coincidence. Per
buildfarm and local investigation. Discussion:
https://postgr.es/m/12222.1586223974@sss.pgh.pa.us
https://git.postgresql.org/pg/commitdiff/c7654f6a37792ab9525ff98b710c23b27c7868a5

- Adjust bytea get_bit/set_bit to use int8 not int4 for bit numbering. Since the
existing bit number argument can't exceed INT32_MAX, it's not possible for
these functions to manipulate bits beyond the first 256MB of a bytea value.
Lift that restriction by redeclaring the bit number arguments as int8 (which
requires a catversion bump, hence is not back-patchable). The similarly-named
functions for bit/varbit don't really have a problem because we restrict those
types to at most VARBITMAXLEN bits; hence leave them alone. While here,
extend the encode/decode functions in utils/adt/encode.c to allow dealing with
values wider than 1GB. This is not a live bug or restriction in current
usage, because no input could be more than 1GB, and since none of the encoders
can expand a string more than 4X, the result size couldn't overflow uint32.
But it might be desirable to support more in future, so make the input length
values size_t and the potential-output-length values uint64. Also add some
test cases to improve the miserable code coverage of these functions. Movead
Li, editorialized some by me; also reviewed by Ashutosh Bapat Discussion:
https://postgr.es/m/20200312115135445367128@highgo.ca
https://git.postgresql.org/pg/commitdiff/26a944cf29ba67bb49f42656dd2be98fe2485f5f

- Allow psql's \g and \gx commands to transiently change \pset options. We
invented \gx to allow the "\pset expanded" flag to be forced on for the
duration of one command output, but that turns out to not be nearly enough to
satisfy the demand for variant output formats. Hence, make it possible to
change any pset option(s) for the duration of a single command output, by
writing "option=value ..." inside parentheses, for example \g (format=csv
csv_fieldsep='\t') somefile \gx can now be understood as a shorthand for
including expanded=on inside the parentheses. Patch by me, expanding on a
proposal by Pavel Stehule Discussion:
https://postgr.es/m/CAFj8pRBx9OnBPRJVtfA5ycUpySge-XootAXAsv_4rrkHxJ8eRg@mail.gmail.com
https://git.postgresql.org/pg/commitdiff/b63c293bcbd7439f883cd4cf748f6755df0fbb3c

- Fix circle_in to accept "(x,y),r" as it's advertised to do. Our documentation
describes four allowed input syntaxes for circles, but the regression tests
tried only three ... with predictable consequences. Remarkably, this has been
wrong since the circle datatype was added in 1997, but nobody noticed till
now. David Zhang, with some help from me Discussion:
https://postgr.es/m/332c47fa-d951-7574-b5cc-a8f7f7201202@highgo.ca
https://git.postgresql.org/pg/commitdiff/41a194f49177daf9348bfde2c42e85b806dcee31

- Allow partitionwise join to handle nested FULL JOIN USING cases. This case
didn't work because columns merged by FULL JOIN USING are represented in the
parse tree by COALESCE expressions, and the logic for recognizing a
partitionable join failed to match upper-level join clauses to such
expressions. To fix, synthesize suitable COALESCE expressions and add them to
the nullable_partexprs lists. This is pretty ugly and brute-force, but it
gets the job done. (I have ambitions of rethinking the way outer-join output
Vars are represented, so maybe that will provide a cleaner solution someday.
For now, do this.) Amit Langote, reviewed by Justin Pryzby, Richard Guo, and
myself Discussion:
https://postgr.es/m/CA+HiwqG2WVUGmLJqtR0tPFhniO=H=9qQ+Z3L_ZC+Y3-EVQHFGg@mail.gmail.com
https://git.postgresql.org/pg/commitdiff/981643dcdb70b6ce70d8a08417f71f465f236cb5

- Put back mistakenly removed #include. In commit 4dbcb3f84 I removed some code
from parse_coerce.c, and also removed some apparently-no-longer-needed
#includes. But removing datum.h broke some not-compiled-by-default code.
Discussion: https://postgr.es/m/20200407205436.pyjhddw5bn5upvsu@development
https://git.postgresql.org/pg/commitdiff/7a5d74b7dd4b1324e8a8fff4086d51d4f43b1721

- Fix pg_dump/pg_restore to restore event trigger comments later. Repair an
oversight in commit 8728b2c70: if we're postponing restore of event triggers
to the end, we must also postpone restoring any comments on them, since of
course we cannot create the comments first. (This opens yet another
opportunity for an event trigger to bollix the restore, but there's no help
for that.) Per bug #16346 from Alexander Lakhin. Like the previous commit,
back-patch to all supported branches. Hamid Akhtar and Tom Lane Discussion:
https://postgr.es/m/16346-6210ad7a0ea81be1@postgresql.org
https://git.postgresql.org/pg/commitdiff/a9d70c108786712a1023c65e360602edf7bafbf4

- Cosmetic improvements for default text search parser's ts_headline code. This
code was woefully unreadable and under-commented. Try to improve matters by
adding comments, using some macros to make complicated if-tests more readable,
using boolean type where appropriate, etc. There are a couple of tiny coding
improvements too, but this commit includes (I hope) no behavioral change.
Nonetheless, back-patch as far as 9.6, because a followup bug-fixing commit
depends on this. Discussion:
https://postgr.es/m/16345-2e0cf5cddbdcd3b4@postgresql.org
https://git.postgresql.org/pg/commitdiff/b10f8bb9fd39da44efd411aec5c643b92bd23676

- Fix default text search parser's ts_headline code for phrase queries. This
code could produce very poor results when asked to highlight a string based on
a query using phrase-match operators. The root cause is that hlCover(), which
is supposed to find a minimal substring that matches the query, was written
assuming that word position is not significant. I'm only 95% convinced that
its algorithm was correct even for plain AND/OR queries; but it definitely
fails completely for phrase matches, causing it to possibly not identify a
cover string at all. Hence, rewrite hlCover() with a less-tense algorithm
that just tries all the possible substrings, earlier and shorter ones first.
(This is not as bad as it sounds performance-wise, because all of the string
matching has been done already: the repeated tsquery match checks boil down to
pointer comparisons.) Unfortunately, since that approach produces more
candidate cover strings than before, it also exposes that there were bugs in
the heuristics in mark_hl_words() for selecting a best cover string. Fixes
there include: * Do not apply the ShortWord filter to words that appear in the
query. * Remove a misguided optimization for quickly rejecting a cover. * Fix
order-of-operation bug that could cause computation of a wrong figure of merit
(poslen) when shortening a cover. * Change the preference rule so that
candidate headlines that do not include their whole cover string (after
MaxWords trimming) are lowest priority, since they may not actually satisfy
the user's query. This results in some changes in existing regression test
cases, but they all seem reasonable. Note in particular that the tests
involving strings like "1 2 3" were previously being affected by the ShortWord
filter, masking the normal matching behavior. Per bug #16345 from Augustinas
Jokubauskas; the new test cases are based on that example. Back-patch to 9.6
where phrase search was added to tsquery. Discussion:
https://postgr.es/m/16345-2e0cf5cddbdcd3b4@postgresql.org
https://git.postgresql.org/pg/commitdiff/c9b0c678d30aa3f6bbf996ea15af8acbfcfb2ac8

- Doc: improve documentation about ts_headline() function. Now that I've had my
nose in that code, I thought the docs about it left something to be desired.
https://git.postgresql.org/pg/commitdiff/a4d4f59196ea8745fe4c048085d6d2bd66e8e7d8

- Further cleanup of ts_headline code. Suppress a probably-meaningless
uninitialized-variable warning (induced by my previous patch, I'm sorry to
say). Improve mark_hl_fragments()'s test for overlapping cover strings: it
failed to consider the possibility that the current string is strictly within
another one. That's unlikely given the preceding splitting into MaxWords
fragments, but I don't think it's impossible. Discussion:
https://postgr.es/m/16345-2e0cf5cddbdcd3b4@postgresql.org
https://git.postgresql.org/pg/commitdiff/2e0e409e3cbab4f4ac01a6f70931817cfd2bdcb1

- Further stabilize results of 019_replslot_limit.pl. Depending on specific
values of restart_lsn or pg_current_wal_lsn() is obviously unsafe. The
previous coding tried to dodge this issue by rounding off, but that's not good
enough, as shown by multiple buildfarm members. Nuke all the uses of these
values except for null-ness checks, pending some credible argument why we
should think something else could be usefully stable. Kyotaro Horiguchi,
further modified by me Discussion:
https://postgr.es/m/E1jM1Sa-0003mS-99@gemulon.postgresql.org
https://git.postgresql.org/pg/commitdiff/e083fa34ced0d53807a57482048bb4c135c3d006

- Doc: sync CREATE GROUP syntax synopsis with CREATE ROLE. CREATE GROUP is an
exact alias for CREATE ROLE, and CREATE USER is almost an exact alias, as can
easily be confirmed by checking the code. So the man page syntax descriptions
ought to match up. The last few additions of role options seem to have
forgotten to update create_group.sgml, though. Fix that, and add a naggy
reminder to create_role.sgml in hopes of not forgetting again. Discussion:
https://postgr.es/m/158647836143.655.9853963229391401576@wrigleys.postgresql.org
https://git.postgresql.org/pg/commitdiff/7c91e9055d254524d76a72b35a919b8ff9931802

- Suppress unused-variable warning. Ashutosh Bapat Discussion:
https://postgr.es/m/CAG-ACPWPB8Lc_aFj25eiPFqi31YB5vmaZnb39mbHSf5Yej=miA@mail.gmail.com
https://git.postgresql.org/pg/commitdiff/401418ca6a689f772cbfa1aedc7485cbbcde7a94

- Doc: clarify locking requirements for ALTER TABLE ADD FOREIGN KEY. The docs
explained that a SHARE ROW EXCLUSIVE lock is needed on the referenced table,
but failed to say the same about the table being altered. Since the page says
that ACCESS EXCLUSIVE lock is taken unless otherwise stated, this left readers
with the wrong conclusion. Discussion:
https://postgr.es/m/834603375.3470346.1586482852542@mail.yahoo.com
https://git.postgresql.org/pg/commitdiff/f333d35428c1cba8d35065b6dbb2dd46e18bd929

- Clear dangling pointer to avoid bogus EXPLAIN printout in a corner case.
ExecReScanHashJoin will destroy the join's hash table if it expects that the
inner relation will produce different rows on rescan. Up to now it's not
bothered to clear the additional pointer to that hash table that exists in the
child HashState node. However, it's possible for the query to terminate
without building a fresh hash table (this happens if the outer relation is
found to be empty during the final rescan). So we can end with a dangling
pointer to a deleted hash table. That was harmless originally, but since 9.0
EXPLAIN ANALYZE has used that pointer to print hash table statistics. In
debug builds this reproducibly results in garbage statistics. In non-debug
builds there's frequently no ill effects, but in principle one could get wrong
EXPLAIN ANALYZE output, or perhaps even a crash if free() has released the
hashtable memory back to the OS. To fix, just make sure we clear the
additional pointer when destroying the hash table. In problematic cases,
EXPLAIN ANALYZE will then print no hashtable statistics (reverting to its
pre-9.0 behavior). This isn't ideal, but since the problem manifests only in
unusual corner cases, it's hard to justify taking any risks to do better in
the back branches. A follow-on patch will improve matters in HEAD.
Konstantin Knizhnik and Tom Lane, per diagnosis by Thomas Munro of a trouble
report from Alvaro Herrera. Discussion:
https://postgr.es/m/20200323165059.GA24950@alvherre.pgsql
https://git.postgresql.org/pg/commitdiff/5c27bce7f39ded1f027475221b732bbbc31a2bfe

- Make EXPLAIN report maximum hashtable usage across multiple rescans. Before
discarding the old hash table in ExecReScanHashJoin, capture its statistics,
ensuring that we report the maximum hashtable size across repeated rescans of
the hash input relation. We can repurpose the existing code for reporting
hashtable size in parallel workers to help with this, making the patch pretty
small. This also ensures that if rescans happen within parallel workers, we
get the correct maximums across all instances. Konstantin Knizhnik and Tom
Lane, per diagnosis by Thomas Munro of a trouble report from Alvaro Herrera.
Discussion: https://postgr.es/m/20200323165059.GA24950@alvherre.pgsql
https://git.postgresql.org/pg/commitdiff/969f9d0b4ba574bb8df65683dbf7a09c030f3e67

- Suppress -Wimplicit-fallthrough warning in new LIMIT WITH TIES code. The
placement of the fall-through comment in this code appears not to work to
suppress the warning in recent gcc. Move it to the bottom of the case group,
and add an assertion that we didn't get there through some other code path.
Also improve wording of nearby comments. Julien Rouhaud, comment hacking by
me Discussion:
https://postgr.es/m/CAOBaU_aLdPGU5wCpaowNLF-Q8328iR7mj1yJAhMOVsdLwY+sdg@mail.gmail.com
https://git.postgresql.org/pg/commitdiff/35cb574aa84723f4661e9fc51340130e64cb5dfc

Tomáš Vondra pushed:

- Implement Incremental Sort. Incremental Sort is an optimized variant of
multikey sort for cases when the input is already sorted by a prefix of the
requested sort keys. For example when the relation is already sorted by (key1,
key2) and we need to sort it by (key1, key2, key3) we can simply split the
input rows into groups having equal values in (key1, key2), and only
sort/compare the remaining column key3. This has a number of benefits: -
Reduced memory consumption, because only a single group (determined by
values in the sorted prefix) needs to be kept in memory. This may also
eliminate the need to spill to disk. - Lower startup cost, because
Incremental Sort produce results after each prefix group, which is
beneficial for plans where startup cost matters (like for example queries
with LIMIT clause). We consider both Sort and Incremental Sort, and decide
based on costing. The implemented algorithm operates in two different modes:
- Fetching a minimum number of tuples without check of equality on the
prefix keys, and sorting on all columns when safe. - Fetching all tuples for
a single prefix group and then sorting by comparing only the remaining
(non-prefix) keys. We always start in the first mode, and employ a heuristic
to switch into the second mode if we believe it's beneficial - the goal is to
minimize the number of unnecessary comparions while keeping memory consumption
below work_mem. This is a very old patch series. The idea was originally
proposed by Alexander Korotkov back in 2013, and then revived in 2017. In 2018
the patch was taken over by James Coleman, who wrote and rewrote most of the
current code. There were many reviewers/contributors since 2013 - I've done
my best to pick the most active ones, and listed them in this commit message.
Author: James Coleman, Alexander Korotkov Reviewed-by: Tomas Vondra, Andreas
Karlsson, Marti Raudsepp, Peter Geoghegan, Robert Haas, Thomas Munro, Antonin
Houska, Andres Freund, Alexander Kuzmenkov Discussion:
https://postgr.es/m/CAPpHfdscOX5an71nHd8WSUH6GNOCf=V7wgDaTXdDd9=goN-gfA@mail.gmail.com
Discussion:
https://postgr.es/m/CAPpHfds1waRZ=NOmueYq0sx1ZSCnt+5QJvizT8ndT2=etZEeAQ@mail.gmail.com
https://git.postgresql.org/pg/commitdiff/d2d8a229bc58a2014dce1c7a4fcdb6c5ab9fb8da

- Fix show_incremental_sort_info with force_parallel_mode. When executed with
force_parallel_mode=regress, the function was exiting too early and thus
failed to print the worker stats. Fixed by making it more like show_sort_info.
Discussion:
https://postgr.es/m/CAPpHfds1waRZ=NOmueYq0sx1ZSCnt+5QJvizT8ndT2=etZEeAQ@mail.gmail.com
https://git.postgresql.org/pg/commitdiff/7d6d82a52493ad47c57662d0ac6758da551e87a5

- Fix failures in incremental_sort due to number of workers. The last test in
incremental_sort suite prints a parallel plan, but some of the buildfarm
animals have custom max_parallel_workers_per_gather values, causing failures.
Fixed by setting the GUC to an explicit value. Discussion:
https://postgr.es/m/CAPpHfds1waRZ=NOmueYq0sx1ZSCnt+5QJvizT8ndT2=etZEeAQ@mail.gmail.com
https://git.postgresql.org/pg/commitdiff/23ba3b5ee278847e4fad913b80950edb2838fd35

- Use INT64_FORMAT when formatting int64 values in explain. Per report from
lapwing.
https://git.postgresql.org/pg/commitdiff/4bea576b032d6e5435ef0946194aada314e67691

- Consider Incremental Sort paths at additional places. Commit d2d8a229bc
introduced Incremental Sort, but it was considered only in
create_ordered_paths() as an alternative to regular Sort. There are many other
places that require sorted input and might benefit from considering
Incremental Sort too. This patch modifies a number of those places, but not
all. The concern is that just adding Incremental Sort to any place that
already adds Sort may increase the number of paths considered, negatively
affecting planning time, without any benefit. So we've taken a more
conservative approach, based on analysis of which places do affect a set of
queries that did seem practical. This means some less common queries may not
benefit from Incremental Sort yet. Author: Tomas Vondra Reviewed-by: James
Coleman Discussion:
https://postgr.es/m/CAPpHfds1waRZ=NOmueYq0sx1ZSCnt+5QJvizT8ndT2=etZEeAQ@mail.gmail.com
https://git.postgresql.org/pg/commitdiff/ba3e76cc571eba3dea19c9465ff15ac3ac186576

- Minor improvements in Incremental Sort explain. Some places still used
"Maximum" instead of "Peak" when displaying info about sort space, so fix
that. Also, add a comment clarifying why it's correct to check the number of
full/prefix sort groups. Author: James Coleman Discussion:
https://postgr.es/m/CAPpHfds1waRZ=NOmueYq0sx1ZSCnt+5QJvizT8ndT2=etZEeAQ@mail.gmail.com
https://git.postgresql.org/pg/commitdiff/d22782a5392f6a1cb6cfca34031b93eb9dd2aa03

- Remove debugging elog from pgstat_recv_resetslrucounter. Reported-by: Thomas
Munro
https://git.postgresql.org/pg/commitdiff/9c74ceb20b991f786f71666d4b4d557d2744a567

- Track SLRU page hits in SimpleLruReadPage_ReadOnly. SLRU page hits were
tracked only in SimpleLruReadPage, but that's not enough because we may hit
the page in SimpleLruReadPage_ReadOnly in which case we don't call
SimpleLruReadPage at all. Reported-by: Kuntal Ghosh Discussion:
https://postgr.es/m/20200119143707.gyinppnigokesjok@development
https://git.postgresql.org/pg/commitdiff/2b88fdde30d8e9bf833b75a014189e9148233b85

- Stabilize incremental_sort tests. The test never did ANALYZE on the test
table, so the plans depended on various defaults (e.g. number of groups being
200). This worked most of the time, but with CLOBBER_CACHE_ALWAYS the
autoanalyze often managed to build accurate stats, changing the plan. Fixed
by increasing the size of test tables a bit, making the Sort a bit more
expensive than Incremental Sort. The tests were constructed to test
transitions in the Incremental Sort algorithm, and this change does not break
that. Reviewed-by: James Coleman Discussion:
https://postgr.es/m/CAPpHfds1waRZ=NOmueYq0sx1ZSCnt+5QJvizT8ndT2=etZEeAQ@mail.gmail.com
https://git.postgresql.org/pg/commitdiff/cea09246e57821b8a97a6483a7df6c7345b055ef

Peter Geoghegan pushed:

- Fix nbtree kill_prior_tuple posting list assert. An assertion added by commit
0d861bbb checked that _bt_killitems() only processes a BTScanPosItem whose
heap TID is contained in a posting list tuple when its page offset number
still matches what is on the page (i.e. when it matches the posting list
tuple's current offset number). This was only correct in the common case where
the page can't have changed since we first read it. It was not correct in
cases where we don't drop the buffer pin (and don't need to verify the page
hasn't changed using its LSN). The latter category includes scans involving
unlogged tables, and scans that use a non-MVCC snapshot, per the logic
originally introduced by commit 2ed5b87f. The assertion still seems helpful.
Fix it by taking cases where the page may have been concurrently modified into
account. Reported-By: Anastasia Lubennikova, Alexander Lakhin Discussion:
https://postgr.es/m/c4e38e9a-0f9c-8e53-e639-adf343f94472@postgrespro.ru
https://git.postgresql.org/pg/commitdiff/ce2cee0ade8a6a360322c51201fda1fc25be7773

- Remove nbtree BTreeTupleSetAltHeapTID() function. Since heap TID is supposed
to be just another key attribute to the implementation, it doesn't make much
sense to have separate BTreeTupleSetNAtts() and BTreeTupleSetAltHeapTID()
functions. Merge the two functions together. This slightly simplifies
_bt_truncate().
https://git.postgresql.org/pg/commitdiff/60cbd7751c1ec6ffdf2ffc520ddeb35cb2f2ed70

- Add contrib/amcheck debug message. Add a DEBUG1 message indicating that
verification of the index structure is underway. Also reduce the severity
level of the existing "tree level" debug message to DEBUG1. It should never
have been made DEBUG2. Any B-Tree index with more than a couple of levels will
generally also have so many pages that the per-page DEBUG2 messages will
become completely unmanageable. In passing, add a new "Tip" to the docs that
advises users that run into corruption that the debug messages might provide
useful additional context.
https://git.postgresql.org/pg/commitdiff/20fbb711ef43ef70093378ff5efdf1b6e8cc10d8

- Doc: Fix contrib/amcheck tip. Fixes an oversight in commit 20fbb711.
https://git.postgresql.org/pg/commitdiff/26640c40715c7f2045cf1b7c6753cac40b64d1e8

Thomas Munro pushed:

- Add SQL type xid8 to expose FullTransactionId to users. Similar to xid, but 64
bits wide. This new type is suitable for use in various system views and
administration functions. Reviewed-by: Fujii Masao
<masao(dot)fujii(at)oss(dot)nttdata(dot)com> Reviewed-by: Takao Fujii
<btfujiitkp(at)oss(dot)nttdata(dot)com> Reviewed-by: Yoshikazu Imai
<imai(dot)yoshikazu(at)fujitsu(dot)com> Reviewed-by: Mark Dilger
<mark(dot)dilger(at)enterprisedb(dot)com> Discussion:
https://postgr.es/m/20190725000636.666m5mad25wfbrri%40alap3.anarazel.de
https://git.postgresql.org/pg/commitdiff/aeec457de8a8820368e343e791accffe24dc7198

- Introduce xid8-based functions to replace txid_XXX. The txid_XXX family of
fmgr functions exposes 64 bit transaction IDs to users as int8. Now that we
have an SQL type xid8 for FullTransactionId, define a new set of functions
including pg_current_xact_id() and pg_current_snapshot() based on that. Keep
the old functions around too, for now. It's a bit sneaky to use the same C
functions for both, but since the binary representation is identical except
for the signedness of the type, and since older functions are the ones using
the wrong signedness, and since we'll presumably drop the older ones after a
reasonable period of time, it seems reasonable to switch to FullTransactionId
internally and share the code for both. Reviewed-by: Fujii Masao
<masao(dot)fujii(at)oss(dot)nttdata(dot)com> Reviewed-by: Takao Fujii
<btfujiitkp(at)oss(dot)nttdata(dot)com> Reviewed-by: Yoshikazu Imai
<imai(dot)yoshikazu(at)fujitsu(dot)com> Reviewed-by: Mark Dilger
<mark(dot)dilger(at)enterprisedb(dot)com> Discussion:
https://postgr.es/m/20190725000636.666m5mad25wfbrri%40alap3.anarazel.de
https://git.postgresql.org/pg/commitdiff/4c04be9b05ad2ec5acd27c3417bf075c13cab134

- Support PrefetchBuffer() in recovery. Provide PrefetchSharedBuffer(), a
variant that takes SMgrRelation, for use in recovery. Rename
LocalPrefetchBuffer() to PrefetchLocalBuffer() for consistency. Add a return
value to all of these. In recovery, tolerate and report missing files, so we
can handle relations unlinked before crash recovery began. Also report cache
hits and misses, so that callers can do faster buffer lookups and better I/O
accounting. Reviewed-by: Alvaro Herrera <alvherre(at)2ndquadrant(dot)com>
Reviewed-by: Andres Freund <andres(at)anarazel(dot)de> Discussion:
https://postgr.es/m/CA%2BhUKGJ4VJN8ttxScUFM8dOKX0BrBiboo5uz1cq%3DAovOddfHpA%40mail.gmail.com
https://git.postgresql.org/pg/commitdiff/3985b600f57d75b9743d86430cb5c21370057a23

- Rationalize GetWalRcv{Write,Flush}RecPtr(). GetWalRcvWriteRecPtr() previously
reported the latest *flushed* location. Adopt the conventional terminology
used elsewhere in the tree by renaming it to GetWalRcvFlushRecPtr(), and
likewise for some related variables that used the term "received". Add a new
definition of GetWalRcvWriteRecPtr(), which returns the latest *written*
value. This will allow later patches to use the value for non-data-integrity
purposes, without having to wait for the flush pointer to advance.
Reviewed-by: Alvaro Herrera <alvherre(at)2ndquadrant(dot)com> Reviewed-by: Andres
Freund <andres(at)anarazel(dot)de> Discussion:
https://postgr.es/m/CA%2BhUKGJ4VJN8ttxScUFM8dOKX0BrBiboo5uz1cq%3DAovOddfHpA%40mail.gmail.com
https://git.postgresql.org/pg/commitdiff/d140f2f3e225ea53e2d92ab6833b8c186c90666c

Fujii Masao pushed:

- Prevent archive recovery from scanning non-existent WAL files. Previously when
there were multiple timelines listed in the history file of the recovery
target timeline, archive recovery searched all of them, starting from the
newest timeline to the oldest one, to find the segment to read. That is,
archive recovery had to continuously fail scanning the segment until it
reached the timeline that the segment belonged to. These scans for
non-existent segment could be harmful on the recovery performance especially
when archival area was located on the remote storage and each scan could take
a long time. To address the issue, this commit changes archive recovery so
that it skips scanning the timeline that the segment to read doesn't belong
to. Author: Kyotaro Horiguchi, tweaked a bit by Fujii Masao Reviewed-by:
David Steele, Pavel Suderevsky, Grigory Smolkin Discussion:
https://postgr.es/m/16159-f5a34a3a04dc67e0@postgresql.org Discussion:
https://postgr.es/m/20200129.120222.1476610231001551715.horikyota.ntt@gmail.com
https://git.postgresql.org/pg/commitdiff/4bd0ad9e44be9fbc3ad77747d7672dab1c3df7d9

- Add note in pg_stat_statements documentation about planning statistics. The
added note explains that the numbers of planning and execution in the
statement are not always expected to match because their statistics are
updated at their respective end phase, and only for successful operations.
Author: Pascal Legrand, Julien Rouhaud, tweaked a bit by Fujii Masao
Discussion: https://postgr.es/m/1585857868967-0.post@n3.nabble.com
https://git.postgresql.org/pg/commitdiff/58ad961f19f7eca26e6d60eb07adcffeafd0082e

- Exclude backup_manifest file that existed in database, from BASE_BACKUP. If
there is already a backup_manifest file in the database cluster, it belongs to
the past backup that was used to start this server. It is not correct for the
backup being taken now. So this commit changes pg_basebackup so that it always
skips such backup_manifest file. The backup_manifest file for the current
backup will be injected separately if users want it. Author: Fujii Masao
Reviewed-by: Robert Haas Discussion:
https://postgr.es/m/78f76a3d-1a28-a97d-0394-5c96985dd1c0@oss.nttdata.com
https://git.postgresql.org/pg/commitdiff/1ec50a81ec0acd452c7520de19e607a6de8fba5e

- Fix typo in pg_validatebackup documentation. Author: Fujii Masao Reviewed-by:
Robert Haas Discussion:
https://postgr.es/m/78f76a3d-1a28-a97d-0394-5c96985dd1c0@oss.nttdata.com
https://git.postgresql.org/pg/commitdiff/c4f82a779d2676bfca1694a6f9b5499e6cc5f60f

Álvaro Herrera pushed:

- Support FETCH FIRST WITH TIES. WITH TIES is an option to the FETCH FIRST N
ROWS clause (the SQL standard's spelling of LIMIT), where you additionally get
rows that compare equal to the last of those N rows by the columns in the
mandatory ORDER BY clause. There was a proposal by Andrew Gierth to implement
this functionality in a more powerful way that would yield more features, but
the other patch had not been finished at this time, so we decided to use this
one for now in the spirit of incremental development. Author: Surafel
Temesgen <surafel3000(at)gmail(dot)com> Reviewed-by: Álvaro Herrera
<alvherre(at)alvh(dot)no-ip(dot)org> Reviewed-by: Tomas Vondra
<tomas(dot)vondra(at)2ndquadrant(dot)com> Discussion:
https://postgr.es/m/CALAY4q9ky7rD_A4vf=FVQvCGngm3LOes-ky0J6euMrg=_Se+ag@mail.gmail.com
Discussion: https://postgr.es/m/87o8wvz253.fsf@news-spur.riddles.org.uk
https://git.postgresql.org/pg/commitdiff/357889eb17bb9c9336c4f324ceb1651da616fe57

- Allow users to limit storage reserved by replication slots. Replication slots
are useful to retain data that may be needed by a replication system. But
experience has shown that allowing them to retain excessive data can lead to
the primary failing because of running out of space. This new feature allows
the user to configure a maximum amount of space to be reserved using the new
option max_slot_wal_keep_size. Slots that overrun that space are invalidated
at checkpoint time, enabling the storage to be released. Author: Kyotaro
HORIGUCHI <horiguchi(dot)kyotaro(at)lab(dot)ntt(dot)co(dot)jp> Reviewed-by: Masahiko Sawada
<sawada(dot)mshk(at)gmail(dot)com> Reviewed-by: Jehan-Guillaume de Rorthais
<jgdr(at)dalibo(dot)com> Reviewed-by: Álvaro Herrera <alvherre(at)alvh(dot)no-ip(dot)org>
Discussion:
https://postgr.es/m/20170228.122736.123383594.horiguchi.kyotaro@lab.ntt.co.jp
https://git.postgresql.org/pg/commitdiff/c6550776394e25c1620bc8258427c8f1d448080d

- Appease perlcritic. Food for the gods must always be found somehow, even when
the land starves.
https://git.postgresql.org/pg/commitdiff/7e2ffb3885007183af7b161e046e126be2cfba12

- Remove testing for precise LSN/reserved bytes in new TAP test. Trying to
ensure that a slot's restart_lsn or amount of reserved bytes exactly match
some specific values seems unnecessary, and fragile as shown by failures in
multiple buildfarm members. Discussion:
https://postgr.es/m/20200407232602.GA21559@alvherre.pgsql
https://git.postgresql.org/pg/commitdiff/9e9abed746280086474e2191b8c399b5fd9b0678

Alexander Korotkov pushed:

- Implement waiting for given lsn at transaction start. This commit adds
following optional clause to BEGIN and START TRANSACTION commands. WAIT FOR
LSN lsn [ TIMEOUT timeout ] New clause pospones transaction start till given
lsn is applied on standby. This clause allows user be sure, that changes
previously made on primary would be visible on standby. New shared memory
struct is used to track awaited lsn per backend. Recovery process wakes up
backend once required lsn is applied. Author: Ivan Kartyshov, Anna Akenteva
Reviewed-by: Craig Ringer, Thomas Munro, Robert Haas, Kyotaro Horiguchi
Reviewed-by: Masahiko Sawada, Ants Aasma, Dmitry Ivanov, Simon Riggs
Reviewed-by: Amit Kapila, Alexander Korotkov Discussion:
https://postgr.es/m/0240c26c-9f84-30ea-fca9-93ab2df5f305%40postgrespro.ru
https://git.postgresql.org/pg/commitdiff/0f5ca02f53ac2b211d8518f0882c49284c0c9610

- Revert 0f5ca02f53. 0f5ca02f53 introduces 3 new keywords. It appears to be too
much for relatively small feature. Given now we past feature freeze, it's
already late for discussion of the new syntax. So, revert. Discussion:
https://postgr.es/m/28209.1586294824%40sss.pgh.pa.us
https://git.postgresql.org/pg/commitdiff/1aac32df89eb19949050f6f27c268122833ad036

Etsuro Fujita pushed:

- Allow partitionwise joins in more cases. Previously, the partitionwise join
technique only allowed partitionwise join when input partitioned tables had
exactly the same partition bounds. This commit extends the technique to some
cases when the tables have different partition bounds, by using an advanced
partition-matching algorithm introduced by this commit. For both the input
partitioned tables, the algorithm checks whether every partition of one input
partitioned table only matches one partition of the other input partitioned
table at most, and vice versa. In such a case the join between the tables can
be broken down into joins between the matching partitions, so the algorithm
produces the pairs of the matching partitions, plus the partition bounds for
the join relation, to allow partitionwise join for computing the join.
Currently, the algorithm works for list-partitioned and range-partitioned
tables, but not hash-partitioned tables. See comments in
partition_bounds_merge(). Ashutosh Bapat and Etsuro Fujita, most of
regression tests by Rajkumar Raghuwanshi, some of the tests by Mark Dilger and
Amul Sul, reviewed by Dmitry Dolgov and Amul Sul, with additional review at
various points by Ashutosh Bapat, Mark Dilger, Robert Haas, Antonin Houska,
Amit Langote, Justin Pryzby, and Tomas Vondra Discussion:
https://postgr.es/m/CAFjFpRdjQvaUEV5DJX3TW6pU5eq54NCkadtxHX2JiJG_GvbrCA@mail.gmail.com
https://git.postgresql.org/pg/commitdiff/c8434d64ce03c32e0029417a82ae937f2055268f

David Rowley pushed:

- Add functions to calculate the next power of 2. There are many areas in the
code where we need to determine the next highest power of 2 of a given number.
We tend to always do that in an ad-hoc way each time, generally with some
tight for loop which performs a bitshift left once per loop and goes until it
finds a number above the given number. Here we add two generic functions
which make use of the existing pg_leftmost_one_pos* functions which, when
available, will allow us to calculate the next power of 2 without any looping.
Here we don't add any code which uses these new functions. That will be done
in follow-up commits. Author: David Fetter, with some minor adjustments by me
Reviewed-by: John Naylor, Jesse Zhang Discussion:
https://postgr.es/m/20200114173553.GE32763%40fetter.org
https://git.postgresql.org/pg/commitdiff/f0705bb6286d8a24e08ddd99641264ba947ebd03

- Modify various power 2 calculations to use new helper functions. First pass of
modifying various places that obtain the next power of 2 of a number and make
them use the new functions added in pg_bitutils.h instead. This also removes
the _hash_log2() function. There are no longer any callers in core. Other
users can swap their _hash_log2(n) call to make use of pg_ceil_log2_32(n).
Author: David Fetter, with some minor adjustments by me Reviewed-by: John
Naylor, Jesse Zhang Discussion:
https://postgr.es/m/20200114173553.GE32763%40fetter.org
https://git.postgresql.org/pg/commitdiff/d025cf88ba5a64487ee4a17ef23e8f55b1536606

- Modify additional power 2 calculations to use new helper functions. 2nd pass
of modifying various places which obtain the next power of 2 of a number and
make them use the new functions added in f0705bb62. In passing, also modify
num_combinations(). This can be implemented using simple bitshifting rather
than looping. Reviewed-by: John Naylor Discussion:
https://postgr.es/m/20200114173553.GE32763%40fetter.org
https://git.postgresql.org/pg/commitdiff/02a2e8b442002a698336954633b0ccc4e30061e6

Jeff Davis pushed:

- Create memory context for HashAgg with a reasonable maxBlockSize. If the
memory context's maxBlockSize is too big, a single block allocation can
suddenly exceed work_mem. For Hash Aggregation, this can mean spilling to disk
too early or reporting a confusing memory usage number for EXPLAN ANALYZE.
Introduce CreateWorkExprContext(), which is like CreateExprContext(), except
that it creates the AllocSet with a maxBlockSize that is reasonable in
proportion to work_mem. Right now, CreateWorkExprContext() is only used by
Hash Aggregation, but it may be generally useful in the future. Discussion:
https://postgr.es/m/412a3fbf306f84d8d78c4009e11791867e62b87c.camel@j-davis.com
https://git.postgresql.org/pg/commitdiff/50a38f65177ea7858bc97f71ba0757ba04c1c167

Andrew Dunstan pushed:

- Msys2 tweaks for pg_validatebackup corruption test. 1. Tell Msys2 not to
mangle the tablespace map parameter 2. If rmdir doesn't work, fall back to
trying unlink on the entry in pg_tblspc. Discussion:
https://postgr.es/m/7330a7c7-ce5f-9769-39a1-bdb0b32bb4a6@2ndQuadrant.com
https://git.postgresql.org/pg/commitdiff/c3e4cbaab936a17b579d85c5ff28bcb2251736d0

Andrew Gierth pushed:

- doc: restore intentional typo. Commit ac8623760 "fixed" a typo in an example
of what would happen in the event of a typo. Restore the original typo and add
a comment about its intentionality. Backpatch to 12 where the error was
introduced. Per report from irc user Nicolás Alvarez.
https://git.postgresql.org/pg/commitdiff/8a47b775a16fb4f1e154c0f319a030498e123164

Noah Misch pushed:

- When WalSndCaughtUp, sleep only in WalSndWaitForWal(). Before sleeping,
WalSndWaitForWal() sends a keepalive if MyWalSnd->write < sentPtr. That is
important in logical replication. When the latest physical LSN yields no
logical replication messages (a common case), that keepalive elicits a reply,
and processing the reply updates pg_stat_replication.replay_lsn. WalSndLoop()
lacks that; when WalSndLoop() slept, replay_lsn advancement could stall until
wal_receiver_status_interval elapsed. This sometimes stalled
src/test/subscription/t/001_rep_changes.pl for up to 10s. Discussion:
https://postgr.es/m/20200406063649.GA3738151@rfd.leadboat.com
https://git.postgresql.org/pg/commitdiff/421685812290406daea58b78dfab0346eb683bbb

- Optimize RelationFindReplTupleSeq() for CLOBBER_CACHE_ALWAYS. Specifically,
remember lookup_type_cache() results instead of retrieving them once per
comparison. Under CLOBBER_CACHE_ALWAYS, this reduced
src/test/subscription/t/001_rep_changes.pl elapsed time by an order of
magnitude, which reduced check-world elapsed time by 9%. Discussion:
https://postgr.es/m/20200406085420.GC162712@rfd.leadboat.com
https://git.postgresql.org/pg/commitdiff/328c70997bc3518a50bd9a8ff33de349a7223413

Robert Haas pushed:

- Rename pg_validatebackup to pg_verifybackup. Also, use "verify" rather than
"validate" to refer to the process being undertaken here. Per discussion, that
is a more appropriate term. Discussion:
https://www.postgresql.org/message-id/172c9d9b-1d0a-1b94-1456-376b1e017322@2ndquadrant.com
Discussion:
http://postgr.es/m/CA+TgmobLgMh6p8FmLbj_rv9Uhd7tPrLnAyLgGd2SoSj=qD-bVg@mail.gmail.com
https://git.postgresql.org/pg/commitdiff/dbc60c5593f26dc777a3be032bff4fb4eab1ddd1

== Pending Patches ==

Michaël Paquier sent in a WIP patch to libpq which attempts an SSL connection
first when both sslmode and gssencmode are both set to avert a performance hit
for SSL users.

Ivan N. Taranov sent in a patch to speed up VALGRIND_MEMPOOL_* calls by adding a
GetMemoryChunkCapacity method which returns the size of usable space in the
chunk, using same on VALGRIND_MEMPOOL_* calls, and calling
VALGRIND_MEMPOOL_CHANGED only for really changed chunks.

Anastasia Lubennikova sent in two more revisions of a patch to fix an infelicity
between pg_upgrade and non-standard ACLs.

Grigory Smolkin sent in a patch to fix an issue where archive_recovery was
fetching the wrong segments.

Tom Lane sent in another revision of a patch to fix checksum verification in
base backups for random.

Michael Banck sent in a patch to allow checking base backups in pg_checksums.

Fujii Masao sent in two more revisions of a patch to avoid fetching
out-of-timeline segments.

Michaël Paquier sent in a patch to fix plug a file descriptor leak in xlogreader.

Kyotaro HORIGUCHI and Álvaro Herrera traded patches to add a WAL relief vent for
replication slots.

Tomáš Vondra, Justin Pryzby, and James Coleman traded patches to implement
incremental sort.

Davinder Singh and Ranier Vilela traded patches to fix a compilation error with
Microsoft Visual Studio 2015/2017/2019.

Ivan Kartyshov and Anna Akenteva traded patches to implement BEGIN ... WAIT FOR
... [TIMEOUT ...].

Paul A Jungwirth and Álvaro Herrera traded patches to implement multi-ranges.

Justin Pryzby and Aleksey Kondratov traded patches to enable CLUSTER, VACUUM
FULL and REINDEX to change tablespace on the fly.

Julien Rouhaud sent in another revision of a patch to expose WAL usage counters
in verbose (auto)vacuum output.

Masahiko Sawada sent in three more revisions of a patch to instrument buffer
usage for CREATE INDEX.

James Coleman sent in a patch to fix parallel explain.

Jie Zhang sent in a patch to ensure that PQExpBuffers are destroyed during
pg_dump.

Yuzuko Hosoya sent in another revision of a patch to run autovacuum on
partitioned tables.

Kyotaro HORIGUCHI sent in another revision of a patch to implement the stats
collector using shared memory instead of files.

Zeng Wenjing sent in a patch to make the testing of equivalents of boolean
values more precise in relopt.

Kuntal Ghosh sent in a patch to update stats in SimpleLruReadPage_ReadOnly.

David Cramer sent in another revision of a patch to implement binary support for
the pgoutput logical replication plugin.

Jeff Davis sent in another revision of a patch to make
MemoryContextMemAllocated() more precise.

David Zhang sent in another revision of a patch to ensure that the circle type
parses as documented.

Asif Rehman sent in two more revisions of a patch to implement parallel backup.

Thomas Munro sent in three more revisions of a patch to implement WAL
prefetching.

Andres Freund sent in two more revisions of a patch to improve the scalability
of GetSnapshotData().

Zeng Wenjing sent in five more revisions of a patch to implement global
temporary tables.

Andres Freund sent in another revision of a patch to fix infelicities among
catalog invalidations, catalog scans, and ScanPgRelation().

Amit Kapila and Justin Pryzby traded patches to fix the documentation for
parallel vacuum.

Dmitry Dolgov sent in another revision of a patch to implement index skip scans.

David Fetter sent in another revision of a patch to enable setting pg_hba.conf
permissions from initdb.

Antonin Houska sent in another revision of a patch to make foreign key
reference checks more efficient.

Justin Pryzby sent in another revision of a patch to make explain HashAggregate
report bucket and memory stats.

Justin Pryzby sent in a WIP patch to fix detaching tables with inherited
triggers.

Fujii Masao sent in a patch to clarify what pg_stat_statements is actually
counting in the "calls" field.

Andrew Dunstan sent in a patch to fix some tests backup manifests broke on
Windows.

Michail Nikolaev sent in another revision of a patch to help standbys use hint
bits.

Justin Pryzby sent in two more revisions of a patch to fix some infelicities in
the documentation for 13.

Justin Pryzby sent in three revisions of a patch to Allow specifying "(parallel
0)" with "vacuum(full)", even though full implies parallel 0.

Michaël Paquier sent in a patch to fix some bugs in some of the TID functions
and which caused a segmentation fault using currtid and partitioned tables.

Dilip Kumar sent in another revision of a patch to fix an infelicity between
logical_work_mem and logical streaming of large in-progress transactions.

Peter Eisentraut sent in another revision of a patch to propagate ALTER TABLE
... SET STORAGE to indexes.

Thomas Munro sent in a patch to implement fast DSM segments.

Alexandra Wang and Ashutosh Bapat traded patches to improve the check new
partition bound error position report.

Álvaro Herrera sent in another revision of a patch to add ALTER .. NO DEPENDS
ON, and tab completion for ALTER INDEX ... NO DEPENDS ON.

Masahiko Sawada sent in a patch to fix a bug that manifested as Multiple
FPI_FOR_HINT for the same block during killing btree index items by adding a
check for whether an item was dead to _bt_killitems.

Yugo Nagata sent in another revision of a patch to implement incremental
maintenance of materialized views.

Pavel Stěhule sent in another revision of a patch to implement schema variables.

Nathan Bossart sent in a patch to avoid stop considering pg_init_privs entries
when dumping renamed system schemas.

Jesse Zhang, Denis Smirnov, and Soumyadeep Chakraborty sent in a patch to
properly mark NULL returns in numeric aggregates.

Fujii Masao sent in a patch to ensure a fast exit to locking when no
synchronous replica names are defined.

Robert Haas sent in a patch to rename pg_validatebackup to pg_verifybackup.

Teja Mupparti sent in a patch to fix a bug which can cause redo recovery to
fail(forever) by marking all the buffers as about-to-be-dropped, calling
CacheInvalidateSmgr(), then calling Truncate on the filesystem level.

Pavel Stěhule sent in another revision of a patch to make it possible to
redirect tabular-only output.

Andrew Dunstan sent in a patch to clean up some Perl issues pointed out by
perlcritic.

David Steele sent in a patch to make appropriate exceptions for perlcritic.
Julien Rouhaud sent in two revisions of a patch to add "-Wimplicit-fallthrough"
to the default flags.