== PostgreSQL Weekly News - May 19, 2019 ==

== PostgreSQL Weekly News - May 19, 2019 ==

== PostgreSQL Product News ==

Pgpool-II 4.0.5, 3.7.10, 3.6.17, 3.5.21 and 3.4.24 released.
http://pgpool.net/mediawiki/index.php/Downloads

== PostgreSQL Jobs for May ==

http://archives.postgresql.org/pgsql-jobs/2019-05/

== PostgreSQL Local ==

PGCon 2019 will take place in Ottawa on May 28-31, 2019. Registration is open.
https://www.pgcon.org/2019

pgibz will be held in Ibiza, Spain on June 19-23, 2019. The CfP is open.
https://pgibz.io/

Swiss PGDay 2019 will take place in Rapperswil (near Zurich) on June 28, 2019.
Registration is open.
http://www.pgday.ch/2019/

PostgresLondon 2019 will be July 2-3, 2019 with an optional training day on
July 1.
http://postgreslondon.org

PGConf.Brazil 2019 will take place August 1-3, 2019 in São Paulo.
http://pgconf.com.br

The first Austrian pgDay, will take place September 6, 2019 at the Hilton Garden
Inn in Wiener Neustadt.
https://pgday.at/en/

PostgresConf South Africa 2019 will take place in Johannesburg on October 8-9, 2019
The Call for Papers is open through June 30, 2019.
https://postgresconf.org/conferences/SouthAfrica2019

== PostgreSQL in the News ==

Planet PostgreSQL: http://planet.postgresql.org/

PostgreSQL Weekly News is brought to you this week by David Fetter

Submit news and announcements by Sunday at 3:00pm PST8PDT to david(at)fetter(dot)org(dot)

== Applied Patches ==

Noah Misch pushed:

- Fail pgwin32_message_to_UTF16() for SQL_ASCII messages. The function had been
interpreting SQL_ASCII messages as UTF8, throwing an error when they were
invalid UTF8. The new behavior is consistent with
pg_do_encoding_conversion(). This affects LOG_DESTINATION_STDERR and
LOG_DESTINATION_EVENTLOG, which will send untranslated bytes to write() and
ReportEventA(). On buildfarm member bowerbird, enabling log_connections
caused an error whenever the role name was not valid UTF8. Back-patch to 9.4
(all supported versions). Discussion:
https://postgr.es/m/20190512015615.GD1124997@rfd.leadboat.com
https://git.postgresql.org/pg/commitdiff/d02768ddd1791bab742cf131de726d0a36b3f337

Tom Lane pushed:

- Fix misoptimization of "{1,1}" quantifiers in regular expressions. A bounded
quantifier with m = n = 1 might be thought a no-op. But according to our
documentation (which traces back to Henry Spencer's original man page) it
still imposes greediness, or non-greediness in the case of the non-greedy
variant "{1,1}?", on whatever it's attached to. This turns out not to work
though, because parseqatom() optimizes away the m = n = 1 case without regard
for whether it's supposed to change the greediness of the argument RE. We can
fix this by just not applying the optimization when the greediness needs to
change; the subsequent general cases handle it fine. The three cases in which
we can still apply the optimization are (a) no quantifier, or quantifier does
not impose a preference; (b) atom has no greediness property, implying it
cannot match a variable amount of text anyway; or (c) quantifier's greediness
is same as atom's. Note that in most cases where one of these applies, we'd
have exited earlier in the "not a messy case" fast path. I think it's now
only possible to get to the optimization when the atom involves capturing
parentheses or a non-top-level backref. Back-patch to all supported branches.
I'd ordinarily be hesitant to put a subtle behavioral change into back
branches, but in this case it's very hard to see a reason why somebody would
write "{1,1}?" unless they're trying to get the documented
change-of-greediness behavior. Discussion:
https://postgr.es/m/5bb27a41-350d-37bf-901e-9d26f5592dd0@charter.net
https://git.postgresql.org/pg/commitdiff/8a29ed05302ea18b4be177c2d1181b43706af45f

- Fix misuse of an integer as a bool. pgtls_read_pending is declared to return
bool, but what the underlying SSL_pending function returns is a count of
available bytes. This is actually somewhat harmless if we're using C99 bools,
but in the back branches it's a live bug: if the available-bytes count
happened to be a multiple of 256, it would get converted to a zero char value.
On machines where char is signed, counts of 128 and up could misbehave as
well. The net effect is that when using SSL, libpq might block waiting for
data even though some has already been received. Broken by careless
refactoring in commit 4e86f1b16, so back-patch to 9.5 where that came in. Per
bug #15802 from David Binderman. Discussion:
https://postgr.es/m/15802-f0911a97f0346526@postgresql.org
https://git.postgresql.org/pg/commitdiff/ddf927fb13471bec0cc76794eaf552df8a1178fb

- Improve commentary about hack in is_publishable_class(). The
FirstNormalObjectId test here is a kluge that needs to go away, but the only
substitute we can think of is to add a column to pg_class, which will take
more work than can be handled right now. Add some commentary in the
meanwhile. Discussion: https://postgr.es/m/15150.1557257111@sss.pgh.pa.us
https://git.postgresql.org/pg/commitdiff/e34ee993fbc38c8538f9342c5961e5f61fd45180

- Fix logical replication's ideas about which type OIDs are built-in. Only
hand-assigned type OIDs should be presumed to match across different PG
servers; those assigned during genbki.pl or during initdb are likely to change
due to addition or removal of unrelated objects. This means that the cutoff
should be FirstGenbkiObjectId (in HEAD) or FirstBootstrapObjectId (before
that), not FirstNormalObjectId. Compare postgres_fdw's is_builtin() test.
It's likely that this error has no observable consequence in a
normally-functioning system, since ATM the only affected type OIDs are system
catalog rowtypes and information_schema types, which would not typically be
interesting for logical replication. But you could probably break it if you
tried hard, so back-patch. Discussion:
https://postgr.es/m/15150.1557257111@sss.pgh.pa.us
https://git.postgresql.org/pg/commitdiff/32ebb35128c3d16b64039f6d6774afd96a3d91b9

- In bootstrap mode, use default signal handling for SIGINT etc. Previously, the
code pointed the standard process-termination signals to postgres.c's die().
That would typically result in an attempt to execute a transaction abort,
which is not possible in bootstrap mode, leading to PANIC. This choice seems
to be a leftover from an old code structure in which the same
signal-assignment code was used for many sorts of auxiliary processes,
including interactive standalone backends. It's not very sensible for
bootstrap mode, which has no interest in either interactivity or continuing
after an error. We can get better behavior with less effort by just letting
normal process termination happen, after which the parent initdb process will
clean up. This is basically cosmetic in any case, since initdb will react the
same way whether bootstrap dies on a signal or abort(). Given the lack of
previous complaints, I don't feel a need to back-patch, even though the
behavior is old. Discussion:
https://postgr.es/m/3850b11a.5121.16aaf827e4a.Coremail.thunder1@126.com
https://git.postgresql.org/pg/commitdiff/fb489e4b3195fc33cccc0fd308e5a0ab502cf199

- Fix SQL-style substring() to have spec-compliant greediness behavior. SQL's
regular-expression substring() function is defined to have a pattern argument
that's separated into three subpatterns by escape- double-quote markers; the
function result is the part of the input matching the second subpattern. The
standard makes it clear that if there is ambiguity about how to match the
input to the subpatterns, the first and third subpatterns should be taken to
match the smallest possible amount of text (i.e., they're "non greedy", in the
terms of our regex code). We were not doing it that way: the first subpattern
would eat the largest possible amount of text, causing the function result to
be shorter than what the spec requires. Fix that by attaching explicit
greediness quantifiers to the subpatterns. (This depends on the regex fix in
commit 8a29ed053; before that, this didn't reliably change the regex engine's
behavior.) Also, by adding parentheses around each subpattern, we ensure that
"|" (OR) in the subpatterns behave sanely. Previously, "|" in the first or
third subpatterns didn't work. This patch also makes the function throw error
if you write more than two escape-double-quote markers, and do something sane
if you write just one, and document that behavior. Previously, an odd number
of markers led to a confusing complaint about unbalanced parentheses, while
extra pairs of markers were just ignored. (Note that the spec requires
exactly two markers, but we've historically allowed there to be none, and this
patch preserves the old behavior for that case.) In passing, adjust some
substring() test cases that didn't really prove what they said they were
testing for: they used patterns that didn't match the data string, so that the
output would be NULL whether or not the function was really strict. Although
this is certainly a bug fix, changing the behavior in back branches seems
undesirable: applications could perhaps be depending on the old behavior,
since it's not obviously wrong unless you read the spec very closely. Hence,
no back-patch. Discussion:
https://postgr.es/m/5bb27a41-350d-37bf-901e-9d26f5592dd0@charter.net
https://git.postgresql.org/pg/commitdiff/7c850320d8cfa5503ecec61c2559661b924f7595

- Remove pg_rewind's private logging.h/logging.c files. The existence of these
files became rather confusing with the introduction of a widely-known
logging.h header in commit cc8d41511. (Indeed, there's already some
duplicative #includes here, perhaps betraying such confusion.) The only thing
left in them, after that commit, is a progress-reporting function that's
neither general-purpose nor tied in any way to other logging infrastructure.
Hence, let's just move that function to pg_rewind.c, and get rid of the
separate files. Discussion: https://postgr.es/m/3971.1557787914@sss.pgh.pa.us
https://git.postgresql.org/pg/commitdiff/53ddefbaf8a0493d2c1fa0b18bfaba72da556985

- Move logging.h and logging.c from src/fe_utils/ to src/common/. The original
placement of this module in src/fe_utils/ is ill-considered, because several
src/common/ modules have dependencies on it, meaning that libpgcommon and
libpgfeutils now have mutual dependencies. That makes it pointless to have
distinct libraries at all. The intended design is that libpgcommon is
lower-level than libpgfeutils, so only dependencies from the latter to the
former are acceptable. We already have the precedent that fe_memutils and a
couple of other modules in src/common/ are frontend-only, so it's not
stretching anything out of whack to treat logging.c as a frontend-only module
in src/common/. To the extent that such modules help provide a common
frontend/backend environment for the rest of common/ to use, it's a reasonable
design. (logging.c does not yet provide an ereport() emulation, but one can
dream.) Hence, move these files over, and revert basically all of the
build-system changes made by commit cc8d41511. There are no places that need
to grow new dependencies on libpgcommon, further reinforcing the idea that
this is the right solution. Discussion:
https://postgr.es/m/a912ffff-f6e4-778a-c86a-cf5c47a12933@2ndquadrant.com
https://git.postgresql.org/pg/commitdiff/fc9a62af3f87f4bec1e8c904ea99ae50f3c881ef

- Fix "make clean" to clean out junk files left behind after ssl tests. We
.gitignore'd this junk, but we didn't actually remove it.
https://git.postgresql.org/pg/commitdiff/6d2fba3189608186d1a196a595f21760412ec5e8

- Remove no-longer-used typedef. struct ClonedConstraint is no longer needed, so
delete it. Discussion: https://postgr.es/m/18102.1557947143@sss.pgh.pa.us
https://git.postgresql.org/pg/commitdiff/8a0f0ad54047c8dc1430e521e2dc90f2cd138a9d

- Fix partition pruning to treat stable comparison operators properly.
Cross-type comparison operators in a btree or hash opclass might be only
stable not immutable (this is true of timestamp vs. timestamptz for example).
partprune.c ignored this possibility and would perform plan-time pruning with
them anyway, possibly leading to wrong answers if the environment changed
between planning and execution. To fix, teach gen_partprune_steps() to do
things differently when creating plan-time pruning steps vs. run-time pruning
steps. analyze_partkey_exprs() also needs an extra check, which is rather
annoying but now is not the time to restructure things enough to avoid that.
While at it, simplify the logic for the plan-time case a little by insisting
that the comparison value be a Const and nothing else. This relies on the
assumption that eval_const_expressions will have reduced any immutable
expression to a Const; which is not quite 100% true, but certainly any case
that comes up often enough to be interesting should have simplification logic
there. Also improve a bunch of inadequate/obsolete/wrong comments. Per
discussion of a report from Alan Jackson (though this fixes only one aspect of
that problem). Back-patch to v11 where this code came in. David Rowley, with
some further hacking by me Discussion:
https://postgr.es/m/FAD28A83-AC73-489E-A058-2681FA31D648@tvsquared.com
https://git.postgresql.org/pg/commitdiff/4b1fcb43d070ba8b34ea01d5a657630d76e4b33f

- Fix bogus logic for combining range-partitioned columns during pruning.
gen_prune_steps_from_opexps's notion of how to do this was overly complicated
and underly correct. Per discussion of a report from Alan Jackson (though
this fixes only one aspect of that problem). Back-patch to v11 where this
code came in. Amit Langote Discussion:
https://postgr.es/m/FAD28A83-AC73-489E-A058-2681FA31D648@tvsquared.com
https://git.postgresql.org/pg/commitdiff/3922f10646fc79264228ea3f6a7343cc3f8f528a

- Restructure creation of run-time pruning steps. Previously,
gen_partprune_steps() always built executor pruning steps using all suitable
clauses, including those containing PARAM_EXEC Params. This meant that the
pruning steps were only completely safe for executor run-time (scan start)
pruning. To prune at executor startup, we had to ignore the steps involving
exec Params. But this doesn't really work in general, since there may be
logic changes needed as well --- for example, pruning according to the last
operator's btree strategy is the wrong thing if we're not applying that
operator. The rules embodied in gen_partprune_steps() and its minions are
sufficiently complicated that tracking their incremental effects in other
logic seems quite impractical. Short of a complete redesign, the only safe
fix seems to be to run gen_partprune_steps() twice, once to create executor
startup pruning steps and then again for run-time pruning steps. We can save
a few cycles however by noting during the first scan whether we rejected any
clauses because they involved exec Params --- if not, we don't need to do the
second scan. In support of this, refactor the internal APIs in partprune.c to
make more use of passing information in the GeneratePruningStepsContext
struct, rather than as separate arguments. This is, I hope, the last piece of
our response to a bug report from Alan Jackson. Back-patch to v11 where this
code came in. Discussion:
https://postgr.es/m/FAD28A83-AC73-489E-A058-2681FA31D648@tvsquared.com
https://git.postgresql.org/pg/commitdiff/6630ccad7a25cad32e2d1a6833fb971602cb67fe

- "A void function may not return a value". Per buildfarm.
https://git.postgresql.org/pg/commitdiff/d307954a7d88ed45655d9206bbe1aefc280c09c7

- Make BufFileCreateTemp() ensure that temp tablespaces are set up. If
PrepareTempTablespaces() has never been called in the current transaction,
OpenTemporaryFile() will fall back to using the default tablespace, which is a
bug if the user wanted temp files placed elsewhere. gistInitBuildBuffers()
appears to have this disease already, and it seems like an easy trap for
future coders to fall into. We discussed other ways to close this gap, but
none of them are prettier or more reliable than just having BufFileCreateTemp
do it. In particular, having fd.c do this creates layering issues that we
could do without. Per suggestion from Melanie Plageman. Arguably this is a
bug fix, but nobody seems very excited about back-patching, so change in HEAD
only. Discussion:
https://postgr.es/m/CAAKRu_YwzjuGAmmaw4-8XO=OVFGR1QhY_Pq-t3wjb9ribBJb_Q@mail.gmail.com
https://git.postgresql.org/pg/commitdiff/93f03dad824f14f40519597e5e4a8fe7b6df858e

- ANSI-ify a few straggler K&R-style function definitions. We still had a couple
of these left in ancient src/port/ files. Convert them to modern style in
preparation for switching to a version of pg_bsd_indent that doesn't cope well
with K&R style. Discussion:
https://postgr.es/m/16886.1558104483@sss.pgh.pa.us
https://git.postgresql.org/pg/commitdiff/da71f98efba9aed493a178aa29f25dbca7407eaf

- Revert "postmaster: Start syslogger earlier". This commit reverts
57431a911d3a650451d198846ad3194900666152. While that's still a good idea in
the abstract, we found out that there are multiple crasher bugs in it on
Windows builds, making the logging_collector option unusable on Windows.
There's no time left to fix these issues before 12beta1, so revert the patch
to allow Windows beta testing to proceed. We'll try again at some future date.
Per bug #15804 from Yulian Khodorkovskiy and additional investigation by
Michael Paquier. Discussion:
https://postgr.es/m/15804-3721117bf40fb654@postgresql.org
https://git.postgresql.org/pg/commitdiff/833451552925d0175e1e15128e411ddef9a36996

Michaël Paquier pushed:

- Fix incorrect return value in JSON equality function for scalars.
equalsJsonbScalarValue() uses a boolean as return type, however for one code
path -1 gets returned, which is confusing. The origin of the confusion is
visibly that this code got copy-pasted from compareJsonbScalarValue() since it
has been introduced in d1d50bf. No backpatch, as this is only cosmetic.
Author: Rikard Falkeborn Discussion:
https://postgr.es/m/CADRDgG7mJnek6HNW13f+LF6V=6gag9PM+P7H5dnyWZAv49aBGg@mail.gmail.com
https://git.postgresql.org/pg/commitdiff/1171dbde2daef8f0dcd1dc1e54531a0d8dd34d88

- Fix duplicated words in comments. Author: Stephen Amell Discussion:
https://postgr.es/m/539fa271-21b3-777e-a468-d96cffe9c768@gmail.com
https://git.postgresql.org/pg/commitdiff/7e19929ea29668f3d01280262e407b93042c959a

- Fix typos in documentatoin of GSSAPI encryption. Author: Daniel Gustafsson
Discussion: https://postgr.es/m/5520EDD8-7AC7-4307-8171-400DD1D84FDC@yesql.se
https://git.postgresql.org/pg/commitdiff/41998f90ce66425ca6071ac5e170a19bb1498414

- Fix regression test outputs. 75445c1 has caused various failures in tests
across the tree after updating some error messages, so fix the newly-expected
output. Author: Michael Paquier Reviewed-by: Tom Lane Discussion:
https://postgr.es/m/8332.1558048838@sss.pgh.pa.us
https://git.postgresql.org/pg/commitdiff/6ba500cae6f57acf8f392d9e1671640f7c9d43b3

Bruce Momjian pushed:

- docs: fix typo in mention of MSVC.
https://git.postgresql.org/pg/commitdiff/fefb6a75386b72a40f697523953eafc7cb1c7a5b

- doc: adjust PG 12 release note sections. Tighten section designations.
https://git.postgresql.org/pg/commitdiff/a6927996be59ac329d65d41cf058d0b1b0978eeb

- doc: PG 12 release notes: normalize attribution names. Reported-by: David
Rowley Discussion:
https://postgr.es/m/CAKJS1f-ktEhmQ2zJQ1L1niuJ9KB8WPA-bE-AhGiFsSO6QASB_w@mail.gmail.com
https://git.postgresql.org/pg/commitdiff/f86b0c3c465319b80d71db7d7a9de636950e104a

- doc: properly attibute PG 12 pgbench release note item. Reported-by: Fabien
COELHO Discussion:
https://postgr.es/m/alpine.DEB.2.21.1905130839140.13487@lancre
https://git.postgresql.org/pg/commitdiff/5d971565a7993ba8dcd06d34d1387af48fdbd034

- doc: improve wording of PG 12 releaase note partition item. Reported-by: Amit
Langote Discussion:
https://postgr.es/m/d5267ae5-bd4a-3e96-c21b-56bfa9fec7e8@lab.ntt.co.jp
https://git.postgresql.org/pg/commitdiff/f4125278e3c92d8cdd83c77d0b54f468ee81c750

- doc: Update OID item in PG 12 release notes. Reported-by: Justin Pryzby
Discussion: https://postgr.es/m/20190513174759.GE23251@telsasoft.com
https://git.postgresql.org/pg/commitdiff/0b62f0f2552d895301fdae56aa839c1eaae24a3a

- docs: fix duplicate wording in PG 12 release notes. Reported-by:
nickb(at)imap(dot)cc Discussion:
https://postgr.es/m/6b3414e1-fcef-4ad9-b123-b3ab3702d3db@www.fastmail.com
https://git.postgresql.org/pg/commitdiff/34d40becfa7a5c7343d0079f181597d7e06122e1

- docs: update partition item in PG 12 release notes. Reported-by: Amit Langote
Discussion:
https://postgr.es/m/b7954643-41ef-a174-479d-1f8d4834f40a@lab.ntt.co.jp
https://git.postgresql.org/pg/commitdiff/356c83795aaa39cfd9bca588bfe4f0196710962e

- docs: Indent listitem tags in PG 12 release notes.
https://git.postgresql.org/pg/commitdiff/b71dad22ce8a645a47c01e544f640f35b91bfbd3

- docs: properly indent PG 12 release notes.
https://git.postgresql.org/pg/commitdiff/a429164ef31fc6899cef4d5e4f3f2ce8fc33aa27

- docs: split out sort-skip partition item in PG 12 release notes. Discussion:
https://postgr.es/m/0cf10a27-c6a0-de4a-cd20-ab7493ea7422@lab.ntt.co.jp
https://git.postgresql.org/pg/commitdiff/05685897f07e7ae5684f0247157faad1f782f889

- docs: tighten up PG 12 release note item on 1k partitions.
https://git.postgresql.org/pg/commitdiff/b12db9ff5f09dc60dc05d0e208a5b28728970e55

Etsuro Fujita pushed:

- postgres_fdw: Fix typo in comment.
https://git.postgresql.org/pg/commitdiff/cc866941ad4583f07e5637c145f6f6ee8a573e11

- Fix typo.
https://git.postgresql.org/pg/commitdiff/7d9eca59cf958eb8c8ff47c40003392681a9ce68

Robert Haas pushed:

- Improve comment for att_isnull. The comment implies that a 1 in the null
bitmap indicates a null value, but actually a 0 in the null bitmap indicates a
null value. Try to be more clear. Patch by me; proposed wording reviewed by
Alvaro Herrera and Tom Lane. Discussion:
http://postgr.es/m/CA+TgmobHOP8r6cG+UnsDFMrS30-m=jRrCBhgw-nFkn0k9QnFsg@mail.gmail.com
https://git.postgresql.org/pg/commitdiff/221b377f09a30dd7c74c26c8c1db439afcc59141

Peter Geoghegan pushed:

- Don't leave behind junk nbtree pages during split. Commit 8fa30f906be reduced
the elevel of a number of "can't happen" _bt_split() errors from PANIC to
ERROR. At the same time, the new right page buffer for the split could
continue to be acquired well before the critical section. This was possible
because it was relatively straightforward to make sure that _bt_split() could
not throw an error, with a few specific exceptions. The exceptional cases
were safe because they involved specific, well understood errors, making it
possible to consistently zero the right page before actually raising an error
using elog(). There was no danger of leaving around a junk page, provided
_bt_split() stuck to this coding rule. Commit 8224de4f, which introduced
INCLUDE indexes, added code to make _bt_split() truncate away non-key
attributes. This happened at a point that broke the rule around zeroing the
right page in _bt_split(). If truncation failed (perhaps due to palloc()
failure), that would result in an errant right page buffer with junk contents.
This could confuse VACUUM when it attempted to delete the page, and should be
avoided on general principle. To fix, reorganize _bt_split() so that
truncation occurs before the new right page buffer is even acquired. A junk
page/buffer will not be left behind if _bt_nonkey_truncate()/_bt_truncate()
raise an error. Discussion:
https://postgr.es/m/CAH2-WzkcWT_-NH7EeL=Az4efg0KCV+wArygW8zKB=+HoP=VWMw@mail.gmail.com
Backpatch: 11-, where INCLUDE indexes were introduced.
https://git.postgresql.org/pg/commitdiff/9b42e713761a43e9ade3965285f077e2ba25bbb7

- Doc: Refer to line pointers as item identifiers. An upcoming HEAD-only patch
will standardize the terminology around ItemIdData variables/line pointers,
ending the practice of referring to them as "item pointers". Make the
"Database Page Layout" docs consistent with the new policy. The term "item
identifier" is already used in the same section, so stick with that.
Discussion:
https://postgr.es/m/CAH2-Wz=c=MZQjUzde3o9+2PLAPuHTpVZPPdYxN=E4ndQ2--8ew@mail.gmail.com
Backpatch: All supported branches.
https://git.postgresql.org/pg/commitdiff/08ca9d7feca890e97f77ef1fde02d7542a54ac5e

- Standardize ItemIdData terminology. The term "item pointer" should not be used
to refer to ItemIdData variables, since that is needlessly ambiguous. Only
ItemPointerData/ItemPointer variables should be called item pointers. To fix,
establish the convention that ItemIdData variables should always be referred
to either as "item identifiers" or "line pointers". The term "item
identifier" already predominates in docs and translatable messages, and so
should be the preferred alternative there. Discussion:
https://postgr.es/m/CAH2-Wz=c=MZQjUzde3o9+2PLAPuHTpVZPPdYxN=E4ndQ2--8ew@mail.gmail.com
https://git.postgresql.org/pg/commitdiff/ae7291acbc5ae4c3751615177fc256f9f3399403

- Reverse order of newitem nbtree candidate splits. Commit fab25024, which
taught nbtree to choose candidate split points more carefully, had
_bt_findsplitloc() record all possible split points in an initial pass over a
page that is about to be split. The order that candidate split points were
processed and stored in was assumed to match the offset number order of split
points on an imaginary version of the page that contains the same items as the
original, but also fits newitem (the item that provoked the split precisely
because it didn't fit). However, the order of split points in the final array
was not quite what was expected: the split point that makes newitem the
firstright item came after the split point that makes newitem the lastleft
item -- not before. As a result, _bt_findsplitloc() could get confused about
the leftmost and rightmost tuples among all possible split points recorded for
the page. This seems to have no appreciable impact on the quality of the
final split point chosen by _bt_findsplitloc(), but it's still wrong. To fix,
switch the order in which newitem candidate splits are recorded in. This also
makes it possible to describe candidate split points in terms of which pair of
adjoining tuples enclose the split point within _bt_findsplitloc(), making it
clearer why it's generally safe for _bt_split() to expect lastleft and
firstright tuples.
https://git.postgresql.org/pg/commitdiff/7505da2f459776a397177f7d591991f5591c2812

- Remove obsolete nbtree insertion comment. Remove a Berkeley-era comment above
_bt_insertonpg() that admonishes the reader to grok Lehman and Yao's paper
before making any changes. This made a certain amount of sense back when
_bt_insertonpg() was responsible for most of the things that are now spread
across _bt_insertonpg(), _bt_findinsertloc(), _bt_insert_parent(), and
_bt_split(), but it doesn't work like that anymore. I believe that this
comment alludes to the need to "couple" or "crab" buffer locks as we ascend
the tree as page splits cascade upwards. The nbtree README already explains
this in detail, which seems sufficient. Besides, the changes to page splits
made by commit 40dae7ec537 altered the exact details of how buffer locks are
retained during splits; Lehman and Yao's original algorithm seems to release
the lock on the left child page/buffer slightly earlier than
_bt_insertonpg()/_bt_insert_parent() can.
https://git.postgresql.org/pg/commitdiff/489e431ba56b10f5736fb54a045ca40880f11bbc

- Remove extra nbtree half-dead internal page check. It's not safe for nbtree
VACUUM to attempt to delete a target page whose right sibling is already
half-dead, since that would fail the cross-check when VACUUM attempts to
re-find a downlink to the right sibling in the parent page. Logic to prevent
this from happening was added by commit 8da31837803, which addressed a bug in
the overhaul of page deletion that went into PostgreSQL 9.4 (commit
efada2b8e92). VACUUM was made to check the right sibling page, and back off
when it happened to be half-dead already. However, it is only truly necessary
to do the right sibling check on the leaf level, since that transitively
determines if the deletion target's parent's right sibling page is itself
undergoing deletion. Remove the internal page level check, and add a comment
explaining why the leaf level check alone suffices. The extra check is also
unnecessary due to the fact that internal pages that are marked half-dead are
generally considered corrupt. Commit efada2b8e92 established the principle
that there should never be half-dead internal pages (internal pages pending
deletion are possible, but that status is never directly represented in the
internal page). VACUUM will complain about corruption when it encounters
half-dead internal pages, so VACUUM is bound to raise an error one way or
another when an nbtree index has a half-dead internal page (contrib/amcheck
will also report that the page is corrupt). It's possible that a pg_upgrade'd
9.3 database will still have half-dead internal pages, so it may seem like
there is an argument for leaving the check in place to reliably get a cleaner
error message that advises the user to REINDEX. However, leaf pages are also
deleted in the first phase of deletion prior to PostgreSQL 9.4, so I believe
we won't even attempt to re-find the parent page anyway (we won't have the
fully deleted leaf page as the right sibling of our target page, so we won't
even try to find a downlink for it). Discussion:
https://postgr.es/m/CAH2-Wzm_ntmqJjWLRyKzimFmFvk+BnVAvUpaA4s1h9Ja58woaQ@mail.gmail.com
https://git.postgresql.org/pg/commitdiff/3f58cc6dd8fc821992da7ed8099d283b014fc2dc

Heikki Linnakangas pushed:

- Fix comment on when HOT update is possible. The conditions listed in this
comment have changed several times, and at some point the thing that the "if
so" referred to was negated. The text was OK up to 9.6. It was differently
wrong in v10, v11 and master, so fix in all those versions.
https://git.postgresql.org/pg/commitdiff/e95d550bbbe21f51d329fbae84cbac3545a9e650

- Detect internal GiST page splits correctly during index build. As we descend
the GiST tree during insertion, we modify any downlinks on the way down to
include the new tuple we're about to insert (if they don't cover it already).
Modifying an existing downlink might cause an internal page to split, if the
new downlink tuple is larger than the old one. If that happens, we need to
back up to the parent and re-choose a page to insert to. We used to detect
that situation, thanks to the NSN-LSN interlock normally used to detect
concurrent page splits, but that got broken by commit 9155580fd5. With that
commit, we now use a dummy constant LSN value for every page during index
build, so the LSN-NSN interlock no longer works. I thought that was OK because
there can't be any other backends modifying the index during index build, but
missed that the insertion itself can modify the page we're inserting to. The
consequence was that we would sometimes insert the new tuple to an incorrect
page, one whose downlink doesn't cover the new tuple. To fix, add a flag to
the stack that keeps track of the state while descending tree, to indicate
that a page was split, and that we need to retry the descend from the parent.
Thomas Munro first reported that the contrib/intarray regression test was
failing occasionally on the buildfarm after commit 9155580fd5. The failure was
intermittent, because the gistchoose() function is not deterministic, and
would only occasionally create the right circumstances for this bug to cause
the failure. Patch by Anastasia Lubennikova, with some changes by me to make
it work correctly also when the internal page split also causes the
"grandparent" to be split. Discussion:
https://www.postgresql.org/message-id/CA%2BhUKGJRzLo7tZExWfSbwM3XuK7aAK7FhdBV0FLkbUG%2BW0v0zg%40mail.gmail.com
https://git.postgresql.org/pg/commitdiff/22251686f07f70527aecda22ab5402986884f6f5

Peter Eisentraut pushed:

- Update information_schema for SQL:2016. This is mainly a light renumbering to
match the sections in the standard.
https://git.postgresql.org/pg/commitdiff/eb3a1376c98a1d220354b468f4b09e4a1dca909a

- Update SQL features/conformance information to SQL:2016.
https://git.postgresql.org/pg/commitdiff/037165ca95d854e04c0c28cfa85f1515bd852892

- Update SQL keywords list to SQL:2016. Per previous convention (see
ace397e9d24eddc56e7dffa921f506117b602d78), drop SQL:2008 and only keep the
latest two standards and SQL-92. Note: SQL:2016-2 lists a large number of
non-reserved keywords that are really just information_schema column names
related to new features. Those kinds of thing have not previously been listed
as keywords, and this was apparently done here by mistake, since these
keywords have been removed again in post-2016 working drafts. So in order to
avoid bloating the keywords table unnecessarily, I have omitted these
erroneous keywords here.
https://git.postgresql.org/pg/commitdiff/c29ba98189190288c2251c6073c13c6234479e5c

Andres Freund pushed:

- Add isolation test for INSERT ON CONFLICT speculative insertion failure. This
path previously was not reliably covered. There was some heuristic coverage
via insert-conflict-toast.spec, but that test is not deterministic, and only
tested for a somewhat specific bug. Backpatch, as this is a complicated and
otherwise untested code path. Unfortunately 9.5 cannot handle two waiting
sessions, and thus cannot execute this test. Triggered by a conversion with
Melanie Plageman. Author: Andres Freund Discussion:
https://postgr.es/m/CAAKRu_a7hbyrk=wveHYhr4LbcRnRCG=yPUVoQYB9YO1CdUBE9Q@mail.gmail.com
Backpatch: 9.5-
https://git.postgresql.org/pg/commitdiff/08e2edc0767ab6e619970f165cb34d4673105f23

- Handle table_complete_speculative's succeeded argument as documented. For some
reason both callsite and the implementation for heapam had the meaning
inverted (i.e. succeeded == true was passed in case of conflict). That's
confusing. I (Andres) briefly pondered whether it'd be better to rename
table_complete_speculative's argument to 'bool specConflict' or such, but
decided not to. The 'complete' in the function name for me makes `succeeded`
sound a bit better. Reported-By: Ashwin Agrawal, Melanie Plageman, Heikki
Linnakangas Discussion:
https://postgr.es/m/CALfoeitk7-TACwYv3hCw45FNPjkA86RfXg4iQ5kAOPhR+F1Y4w@mail.gmail.com
https://postgr.es/m/97673451-339f-b21e-a781-998d06b1067c@iki.fi
https://git.postgresql.org/pg/commitdiff/aa4b8c61d2cd57b53be03defb04d59b232a0e150

- tableam: Don't assume that every AM uses md.c style storage. Previously
various parts of the code routed size requests through
RelationGetNumberOfBlocks[InFork]. That works if md.c is used by the AM, but
not otherwise. Add a tableam callback to return the size of the table. As not
every AM will use postgres' BLCKSZ, have it return bytes, and have
RelationGetNumberOfBlocksInFork() round the byte size up into blocks. To
allow code outside of the AM to determine the actual relation size map
InvalidForkNumber the total size of a relation, as not every AM might just
need the postgres defined forks. A few users of RelationGetNumberOfBlocks()
ought to be converted away from that. One case, the use of it to determine
whether a tid is valid, will be fixed in a follow up commit. Others will have
to wait for v13. Author: Andres Freund Discussion:
https://postgr.es/m/20190423225201.3bbv6tbqzkb5w7cw@alap3.anarazel.de
https://git.postgresql.org/pg/commitdiff/7f44ede5941499c4cee13b812dd93335f4005095

- tableam: Avoid relying on relation size to determine validity of tids. Instead
add a tableam callback to do so. To avoid adding per validation overhead, pass
a scan to tuple_tid_valid. In heap's case we'd otherwise incurred a
RelationGetNumberOfBlocks() call for each tid - which'd have added noticable
overhead to nodeTidscan.c. Author: Andres Freund Reviewed-By: Ashwin Agrawal
Discussion:
https://postgr.es/m/20190515185447.gno2jtqxyktylyvs@alap3.anarazel.de
https://git.postgresql.org/pg/commitdiff/147e3722f7e531f15ba389a4d518efe8cd0bd736

Álvaro Herrera pushed:

- More message style fixes. Discussion:
https://postgr.es/m/20190515183005.GA26486@alvherre.pgsql
https://git.postgresql.org/pg/commitdiff/75445c1515fffa581585f1b72e5b5314c1c395ce

Alexander Korotkov pushed:

- Improve documentation for array subscription in jsonpath. Usage of expressions
and multiple ranges in jsonpath array subscription was undocumented. This
commit adds lacking documentation.
https://git.postgresql.org/pg/commitdiff/87bcc02f4f739d76703feee0ff8346b98fa9b907

- Document jsonpath .** accessor with nesting level filter. It appears that some
variants of .** jsonpath accessor are undocumented. In particular
undocumented variants are: .**{level} .**{lower_level to upper_level}
.**{lower_level to last} This commit adds missing documentation for them.
https://git.postgresql.org/pg/commitdiff/e0e3dad770d3eefc4ee6cd562d9c25b61c263a6e

- Fix declarations of couple jsonpath functions. Make jsonb_path_query_array()
and jsonb_path_query_first() use PG_FUNCTION_ARGS macro instead of its
expansion.
https://git.postgresql.org/pg/commitdiff/da24961e9e1d887111d17cc2dcece2e3946ca5a6

== Pending Patches ==

Michaël Paquier sent in another revision of a patch to clean up and refactor
reindexdb.c.

Alexander Kuzmenkov sent in another revision of a patch to remove unneeded
self-joins.

Zhang Lian Zhuang sent in a patch to ensure that pg_stat_database's stats_reset
is only updated by pg_stat_reset.

Yugo Nagata sent in another revision of a patch to implement incremental
maintenance of materialized views.

Stas Kelvich sent in a patch to ensure read-only access to temp tables for 2PC
transactions.

Thomas Munro sent in two revisions of a patch to add tab completion to psql for
CREATE TYPE.

Takeshi Ideriha sent in another revision of a patch to implement another PoC of
MemoryContext for DSA.

Thomas Munro sent in a patch to implement ldapbindpasswdfile.

Ashwin Agrawal sent in another revision of a patch to rename table AM wrappers
to match the corresponding table AM callback names.

Fujii Masao sent in two revisions of a patch to ensure that VACUUM processes 0
and 1 as booleans, as much of the other code does.

Michaël Paquier sent in another revision of a patch to fix some inconsistent
error message wording for REINDEX CONCURRENTLY.

Dhruv Goel sent in a patch to avoid deadlock errors in CREATE INDEX
CONCURRENTLY.

Dilip Kumar sent in another revision of a patch to clean up orphaned files using
undo logs.

Peter Eisentraut sent in another revision of a patch to convert
ExecComputeStoredGenerated to use tuple slots.

Álvaro Herrera sent in a patch to fix more messsages.

Hubert Zhang sent in a patch to replace to replace growEnable with growPenalty
in hashtable.

Masahiko Sawada sent in another revision of a patch to add --index-cleanup and
--truncate options to vacuumdb.

David Fetter sent in two more revisions of a patch to change EXPLAIN's ANALYZE
option to the clearer and shorter EXEC, preserving backward compatibility.

Tom Lane sent in two revisions of a patch to fix bsdindent's treatment of
function declarations.

Fabien COELHO sent in another revision of a patch to add a SHOW_ALL_RESULTS
option to psql for multi-statement cases.

Thomas Munro sent in a patch to avoid hash join batch explosions with extreme
skew and weird stats.

Rushabh Lathia sent in a patch to consider the default_tablespace if the parent
partitioned tablespace is an InvalidOid.

Kyotaro HORIGUCHI sent in two more revisions of a patch to implement a
shared-memory-based stats collector.

Etsuro Fujita sent in a patch to fix estimate_path_cost_size in the PostgreSQL
FDW.

Robert Haas sent in a patch to move heap-specific logic from needs_toast_table
below tableam, split tuptoaster.c into three separate files, create an API for
inserting and deleting rows in TOAST tables, and allow TOAST tables to be
implemented using table AMs other than heap.

Ashwin Agrawal and Robert Haas traded patches to create a TOAST table only if
the table AM needs it.

Alexander Lakhin sent in a patch to fix some unique identifiers/entities.

Jeff Janes sent in a patch to make bitmap-only heap scans more transparent.

Paul A Jungwirth sent in a patch to implement temporal foreign keys.

David Rowley sent in a patch to fix the argument count for the "combine"
functions used in aggregates.