== PostgreSQL Weekly News - October 02 2016 ==

== PostgreSQL Weekly News - October 02 2016 ==

PostgreSQL 9.6 Released!
https://www.postgresql.org/docs/current/static/release-9-6.html

== PostgreSQL Product News ==

pgAdmin4 1.0, a web- and native GUI control center for PostgreSQL, released.
https://www.pgadmin.org/

barman 2.0, a backup and recovery manager for PostgreSQL, released.
http://docs.pgbarman.org/release/2.0/

== PostgreSQL Local ==

Postgres Vision 2016 will be held October 11-13, 2016, in San Francisco.
http://postgresvision.com/

The Korean Users Group will hold its first PGDay event on October 15, 2016 in Seoul.
http://pgday.postgresql.kr/

PostgreSQL Conference Europe will take place in Tallin, Estonia, on
November 1-4, 2016. The schedule has been published.
http://2016.pgconf.eu/registration/

PGDay Austin 2016, will take place on November 12, 2016. Submission deadline
is September 21, 2016 by midnight CST. Details and submission form at:
https://www.postgresql.us/events/2016/austin

PgConf Silicon Valley 2016 will be held on November 14-16, 2016.
http://www.pgconfsv.com/

CHAR(16) will take place in New York, December 6, 2016.
http://charconference.org/

PGDay.IT 2016 will take place in Prato on December the 13th 2016.
http://pgday.it

== PostgreSQL in the News ==

Planet PostgreSQL: http://planet.postgresql.org/

PostgreSQL Weekly News is brought to you this week by David Fetter

Submit news and announcements by Sunday at 3:00pm Pacific time.
Please send English language ones to david(at)fetter(dot)org, German language
to pwn(at)pgug(dot)de, Italian language to pwn(at)itpug(dot)org(dot)

== Applied Patches ==

Tom Lane pushed:

- Refer to OS X as "macOS", except for the port name which is still "darwin".
We weren't terribly consistent about whether to call Apple's OS "OS X" or "Mac
OS X", and the former is probably confusing to people who aren't Apple users.
Now that Apple has rebranded it "macOS", follow their lead to establish a
consistent naming pattern. Also, avoid the use of the ancient project name
"Darwin", except as the port code name which does not seem desirable to
change. (In short, this patch touches documentation and comments, but no
actual code.) I didn't touch contrib/start-scripts/osx/, either. I suspect
those are obsolete and due for a rewrite, anyway. I dithered about whether to
apply this edit to old release notes, but those were responsible for quite a
lot of the inconsistencies, so I ended up changing them too. Anyway, Apple's
being ahistorical about this, so why shouldn't we be?
http://git.postgresql.org/pg/commitdiff/da6c4f6ca88df346573bdada2aa2544510bf167e

- Document has_type_privilege(). Evidently an oversight in commit 729205571.
Back-patch to 9.2 where privileges for types were introduced. Report:
<20160922173517(dot)8214(dot)88959(at)wrigleys(dot)postgresql(dot)org>
http://git.postgresql.org/pg/commitdiff/a4afb2b5c0b409bb175c20104b2ae9d47cf71be6

- Replace the built-in GIN array opclasses with a single polymorphic opclass.
We had thirty different GIN array opclasses sharing the same operators and
support functions. That still didn't cover all the built-in types, nor did it
cover arrays of extension-added types. What we want is a single polymorphic
opclass for "anyarray". There were two missing features needed to make this
possible: 1. We have to be able to declare the index storage type as
ANYELEMENT when the opclass is declared to index ANYARRAY. This just takes a
few more lines in index_create(). Although this currently seems of use only
for GIN, there's no reason to make index_create() restrict it to that. 2. We
have to be able to identify the proper GIN compare function for the index
storage type. This patch proceeds by making the compare function optional in
GIN opclass definitions, and specifying that the default btree comparison
function for the index storage type will be looked up when the opclass omits
it. Again, that seems pretty generically useful. Since the comparison
function lookup is done in initGinState(), making use of the second feature
adds an additional cache lookup to GIN index access setup. It seems unlikely
that that would be very noticeable given the other costs involved, but maybe
at some point we should consider making GinState data persist longer than it
now does --- we could keep it in the index relcache entry, perhaps. Rather
fortuitously, we don't seem to need to do anything to get this change to play
nice with dump/reload or pg_upgrade scenarios: the new opclass definition is
automatically selected to replace existing index definitions, and the on-disk
data remains compatible. Also, if a user has created a custom opclass
definition for a non-builtin type, this doesn't break that, since CREATE INDEX
will prefer an exact match to opcintype over a match to ANYARRAY. However, if
there's anyone out there with handwritten DDL that explicitly specifies
_bool_ops or one of the other replaced opclass names, they'll need to adjust
that. Tom Lane, reviewed by Enrique Meneses Discussion:
<14436(dot)1470940379(at)sss(dot)pgh(dot)pa(dot)us>
http://git.postgresql.org/pg/commitdiff/fdc9186f7ed1ead827509584f3b763f8dc332c43

- Fix newly-introduced issues in pgbench. The result of FD_ISSET() doesn't
necessarily fit in a bool, though assigning it to one might accidentally work
depending on platform and which socket FD number is being inquired of.
Rewrite to test it with if(), rather than making any specific assumption about
the result width, to match the way every other such call in PG is written.
Don't break out of the input_mask-filling loop after finding the first client
that we're waiting for results from. That mostly breaks parallel query
management. Also, if we choose not to call select(), be sure to clear out any
bits the mask-filling loop might have set, so that we don't accidentally call
doCustom for clients we don't know have input. Doing so would likely be
harmless, but it's a waste of cycles and doesn't seem to be intended. Make
this_usec wide enough. (Yeah, the value would usually fit in an int, but then
why are we using int64 everywhere else?) Minor cosmetic adjustments, mostly
comment improvements. Problems introduced by commit 12788ae49. The first
issue was discovered by buildfarm testing, the others by code review.
http://git.postgresql.org/pg/commitdiff/9779bda86c026e645773a3308f9169c7c0791f7c

- Improve contrib/cube's handling of zero-D cubes, infinities, and NaNs. It's
always been possible to create a zero-dimensional cube by converting from a
zero-length float8 array, but cube_in failed to accept the '()' representation
that cube_out produced for that case, resulting in a dump/reload hazard. Make
it accept the case. Also fix a couple of other places that didn't behave
sanely for zero-dimensional cubes: cube_size would produce 1.0 when surely the
answer should be 0.0, and g_cube_distance risked a divide-by-zero failure.
Likewise, it's always been possible to create cubes containing float8 infinity
or NaN coordinate values, but cube_in couldn't parse such input, and cube_out
produced platform-dependent spellings of the values. Convert them to use
float8in_internal and float8out_internal so that the behavior will be the same
as for float8, as we recently did for the core geometric types (cf commit
50861cd68). As in that commit, I don't pretend that this patch fixes all
insane corner-case behaviors that may exist for NaNs, but it's a step forward.
(This change allows removal of the separate cube_1.out and cube_3.out
expected-files, as the platform dependency that previously required them is
now gone: an underflowing coordinate value will now produce an error not plus
or minus zero.) Make errors from cube_in follow project conventions as to
spelling ("invalid input syntax for cube" not "bad cube representation") and
errcode (INVALID_TEXT_REPRESENTATION not SYNTAX_ERROR). Also a few marginal
code cleanups and comment improvements. Tom Lane, reviewed by Amul Sul
Discussion: <15085(dot)1472494782(at)sss(dot)pgh(dot)pa(dot)us>
http://git.postgresql.org/pg/commitdiff/f31a931fade868d788ef4480c59753a2d5059246

- Redesign parallel dump/restore's wait-for-workers logic. The
ListenToWorkers/ReapWorkerStatus APIs were messy and hard to use. Instead,
make DispatchJobForTocEntry register a callback function that will take care
of state cleanup, doing whatever had been done by the caller of
ReapWorkerStatus in the old design. (This callback is essentially just the
old mark_work_done function in the restore case, and a trivial test for worker
failure in the dump case.) Then we can have ListenToWorkers call the callback
immediately on receipt of a status message, and return the worker to WRKR_IDLE
state; so the WRKR_FINISHED state goes away. This allows us to design a
unified wait-for-worker-messages loop: WaitForWorkers replaces
EnsureIdleWorker and EnsureWorkersFinished as well as the mess in
restore_toc_entries_parallel. Also, we no longer need the fragile API spec
that the caller of DispatchJobForTocEntry is responsible for ensuring there's
an idle worker, since DispatchJobForTocEntry can just wait until there is one.
In passing, I got rid of the ParallelArgs struct, which was a net negative in
terms of notational verboseness, and didn't seem to be providing any
noticeable amount of abstraction either. Tom Lane, reviewed by Kevin Grittner
Discussion: <1188(dot)1464544443(at)sss(dot)pgh(dot)pa(dot)us>
http://git.postgresql.org/pg/commitdiff/b7b8cc0cfcf1c956b752f3e25894f9ad607583b7

- Rationalize parallel dump/restore's handling of worker cmd/status messages.
The existing APIs for creating and parsing command and status messages are
rather messy; for example, archive-format modules have to provide code for
constructing command messages, which is entirely pointless since the code to
read them is hard-wired in WaitForCommands() and hence no format-specific
variation is actually possible. But there's little foreseeable reason to need
format-specific variation anyway. The situation for status messages is no
better; at least those are both constructed and parsed by format-specific
code, but said code is quite redundant since there's no actual need for
format-specific variation. To add insult to injury, the first API involves
returning pointers to static buffers, which is bad, while the second involves
returning pointers to malloc'd strings, which is safer but randomly
inconsistent. Hence, get rid of the MasterStartParallelItem and
MasterEndParallelItem APIs, and instead write centralized functions that
construct and parse command and status messages. If we ever do need more
flexibility, these functions can be the standard implementations of
format-specific callback methods, but that's a long way off if it ever
happens. Tom Lane, reviewed by Kevin Grittner Discussion:
<17340(dot)1464465717(at)sss(dot)pgh(dot)pa(dot)us>
http://git.postgresql.org/pg/commitdiff/fb03d08a89e81a68585f17fd8e7f21c618f4e851

- Make struct ParallelSlot private within pg_dump/parallel.c. The only field of
this struct that other files have any need to touch is the pointer to the
TocEntry a worker is working on. (Well, pg_backup_archiver.c is actually
looking at workerStatus too, but that can be finessed by specifying that the
TocEntry pointer is NULL for a non-busy worker.) Hence, move out the TocEntry
pointers to a separate array within struct ParallelState, and then we can make
struct ParallelSlot private. I noted the possibility of this previously, but
hadn't got round to actually doing it. Discussion:
<1188(dot)1464544443(at)sss(dot)pgh(dot)pa(dot)us>
http://git.postgresql.org/pg/commitdiff/0109ab27609c0d58c1eddc6b799077d0968083de

- Disallow pushing volatile quals past set-returning functions. Pushing an
upper-level restriction clause into an unflattened subquery-in-FROM is okay
when the subquery contains no SRFs in its targetlist, or when it does but the
SRFs are unreferenced by the clause *and the clause is not volatile*.
Otherwise, we're changing the number of times the clause is evaluated, which
is bad for volatile quals, and possibly changing the result, since a volatile
qual might succeed for some SRF output rows and not others despite not
referencing any of the changing columns. (Indeed, if the clause is something
like "random() > 0.5", the user is probably expecting exactly that behavior.)
We had most of these restrictions down, but not the one about the upper clause
not being volatile. Fix that, and add a regression test to illustrate the
expected behavior. Although this is definitely a bug, it doesn't seem like
back-patch material, since possibly some users don't realize that the broken
behavior is broken and are relying on what happens now. Also, while the added
test is quite cheap in the wake of commit a4c35ea1c, it would be much more
expensive (or else messier) in older branches. Per report from Tom van
Tilburg. Discussion:
<CAP3PPDiucxYCNev52=YPVkrQAPVF1C5PFWnrQPT7iMzO1fiKFQ(at)mail(dot)gmail(dot)com>
http://git.postgresql.org/pg/commitdiff/72daabc7a3e75788df862104b8f723513c2471ae

- Make to_timestamp() and to_date() range-check fields of their input.
Historically, something like to_date('2009-06-40','YYYY-MM-DD') would return
'2009-07-10' because there was no prohibition on out-of-range month or day
numbers. This has been widely panned, and it also turns out that Oracle
throws an error in such cases. Since these functions are nominally
Oracle-compatibility features, let's change that. There's no particular
restriction on year (modulo the fact that the scanner may not believe that
more than 4 digits are year digits, a matter to be addressed separately if at
all). But we now check month, day, hour, minute, second, and
fractional-second fields, as well as day-of-year and second-of-day fields if
those are used. Currently, no checks are made on ISO-8601-style week numbers
or day numbers; it's not very clear what the appropriate rules would be there,
and they're probably so little used that it's not worth sweating over. Artur
Zakirov, reviewed by Amul Sul, further adjustments by me Discussion:
<1873520224(dot)1784572(dot)1465833145330(dot)JavaMail(dot)yahoo(at)mail(dot)yahoo(dot)com> See-Also:
<57786490(dot)9010201(at)wars-nicht(dot)de>
http://git.postgresql.org/pg/commitdiff/d3cd36a133d96ad5578b6c10279b55fd5b538093

- Rationalize format-picture caching logic in formatting.c. Add a validity flag
to DCHCacheEntry and NUMCacheEntry entries, and do not set it true until after
we've parsed the supplied format string. This allows dealing with possible
errors while parsing the format without the baroque hack that was there before
(which only covered errors within NUMDesc_prepare, anyway). We can get rid of
the PG_TRY in NUMDesc_prepare, as well as last_NUMCacheEntry and
NUM_cache_remove. (Essentially, this reverts commit ff783fbae in favor of a
less fragile solution; the problems with that approach are well illustrated by
later hacking such as 55f927a46.) In passing, define the size of these caches
as DCH_CACHE_ENTRIES not DCH_CACHE_FIELDS + 1 (whoever thought that was a good
definition?) and likewise for the NUM cache. Also const-ify format string
parameters where convenient, and merge duplicated cache lookup logic. This is
primarily driven by a proposed patch from Artur Zakirov, which introduced some
ereport's into format string parsing for the datetime case. He proposed
preventing the creation of invalid cache entries by parsing the format string
first into a local-variable array, and then copying that to a cache entry.
That seemed a bit ugly to me, and anyway randomly different from the way the
identical problem had been solved for the numeric case. Let's make the two
sets of code more similar not less so. I'm not sure whether we'll adopt the
new error conditions Artur proposes, but this patch seems like good code
cleanup and future-proofing in any case. The existing code is critically (and
undocumented-ly) dependent on no elog being thrown out of several nontrivial
functions, which is trouble waiting to happen, though it doesn't seem to be
actively broken today. Discussion:
<b2a39359-3282-b402-f4a3-057aae500ee7(at)postgrespro(dot)ru>
http://git.postgresql.org/pg/commitdiff/83bed06be4e808f3da30f99b6c91e9efda3961ad

- Allow contrib/file_fdw to read from a program, like COPY FROM PROGRAM. This
patch just exposes COPY's FROM PROGRAM option in contrib/file_fdw. There
don't seem to be any security issues with that that are any worse than what
already exist with file_fdw and COPY; as in the existing cases, only
superusers are allowed to control what gets executed. A regression test case
might be nice here, but choosing a 100% portable command to run is hard. (We
haven't got a test for COPY FROM PROGRAM itself, either.) Corey Huinker and
Adam Gomaa, reviewed by Amit Langote Discussion:
<CADkLM=dGDGmaEiZ=UDepzumWg-CVn7r8MHPjr2NArj8S3TsROQ(at)mail(dot)gmail(dot)com>
http://git.postgresql.org/pg/commitdiff/8e91e12bc3af85ba2287866669268f6825d2cc03

- Fix multiple portability issues in pg_upgrade's rewriteVisibilityMap(). This
is new code in 9.6, and evidently we missed out testing it as thoroughly as it
should have been. Bugs fixed here: 1. Use binary not text mode to open the
files on Windows. Before, if the visibility map chanced to contain two bytes
that looked like \r\n, Windows' read() would convert that to \n, which both
corrupts the map data and causes the file to look shorter than it should.
Unless you were *very* unlucky and had an exact multiple of 8K such
occurrences in each VM file, this would cause pg_upgrade to report a failure,
though with a rather obscure error message. 2. The code for copying rebuilt
bytes into the output was simply wrong. It chanced to work okay on
little-endian machines but would emit the bytes in the wrong order on
big-endian, leading to silent corruption of the visibility map data. 3. The
code was careless about alignment of the working buffers. Given all three of
an alignment-picky architecture, a compiler that chooses to put the
new_vmbuf[] local variable at an odd starting address, and a checksum-enabled
database, pg_upgrade would dump core. Point one was reported by Thomas
Kellerer, the other two detected by code-reading. Point two is much the
nastiest of these issues from an impact standpoint, though fortunately it
affects only a minority of users. The Windows issue will definitely bite
people, but it seems quite unlikely that there would be undetected corruption
from that. In addition, I failed to resist the temptation to do some minor
cosmetic adjustments, mostly improving the comments. It would be a good idea
to try to improve the error reporting here, but that seems like material for a
separate patch. -------
http://git.postgresql.org/pg/commitdiff/5afcd2aa740f87fa6cec7c6693a4891b9df2b67d

- Improve error reporting in pg_upgrade's file copying/linking/rewriting. The
previous design for this had copyFile(), linkFile(), and
rewriteVisibilityMap() returning strerror strings, with the caller producing
one-size-fits-all error messages based on that. This made it impossible to
produce messages that described the failures with any degree of precision,
especially not short-read problems since those don't set errno at all. Since
pg_upgrade has no intention of continuing after any error in this area, let's
fix this by just letting these functions call pg_fatal() for themselves,
making it easy for each point of failure to have a suitable error message.
Taking this approach also allows dropping cleanup code that was unnecessary
and was often rather sloppy about preserving errno. To not lose relevant info
that was reported before, pass in the schema name and table name of the
current table so that they can be included in the error reports. An
additional problem was the use of getErrorText(), which was flat out wrong for
all but a couple of call sites, because it unconditionally did
"_dosmaperr(GetLastError())" on Windows. That's only appropriate when
reporting an error from a Windows-native API, which only a couple of the
callers were actually doing. Thus, even the reported strerror string would be
unrelated to the actual failure in many cases on Windows. To fix, get rid of
getErrorText() altogether, and just have call sites do strerror(errno)
instead, since that's the way all the rest of our frontend programs do it.
Add back the _dosmaperr() calls in the two places where that's actually
appropriate. In passing, make assorted messages hew more closely to project
style guidelines, notably by removing initial capitals in
not-complete-sentence primary error messages. (I didn't make any effort to
clean up places I didn't have another reason to touch, though.) Per discussion
of a report from Thomas Kellerer. Back-patch to 9.6, but no further; given
the relative infrequency of reports of problems here, it's not clear it's
worth adapting the patch to older branches. Patch by me, but with credit to
Alvaro Herrera for spotting the issue with getErrorText's misuse of
_dosmaperr(). Discussion: <nsjrbh$8li$1(at)blaine(dot)gmane(dot)org>
http://git.postgresql.org/pg/commitdiff/f002ed2b8e45286fe73a36e119cba2016ea0de19

- Fix misplacement of submake-generated-headers prerequisites. The sequence
"configure; cd src/pl/plpython; make -j" failed due to trying to compile
plpython's .o files before the generated headers finished building. (This is
an important real-world case, since it's the typical second step when building
both plpython2 and plpython3.) This happens because the
submake-generated-headers target is not placed in a way to make it a
prerequisite to compiling the .o files. Fix that. Checking other uses of
submake-generated-headers, I noted that the one attached to pg_regress was
similarly misplaced; but it's actually not needed at all for pg_regress.o,
rather regress.o, so move it to be a prerequisite of that. Back-patch to 9.6
where submake-generated-headers was introduced (by commit 548af97fc). It's
not immediately clear to me why the previous coding didn't have the same
issue; but since we've not had field reports of plpython make failing, leave
it alone in the older branches. Pavel Raiskup and Tom Lane Discussion:
<1925924(dot)izSMJEZO3x(at)unused-4-107(dot)brq(dot)redhat(dot)com>
http://git.postgresql.org/pg/commitdiff/7107d58ec5a3c45967e77525809612a5f89b97f3

- Fix misstatement in comment in Makefile.shlib. There is no need for "all:
all-lib" to be placed before inclusion of Makefile.shlib. Makefile.global is
what ensures that "all" is the default target, and we already document that
that has to be included first. Per comment from Pavel Raiskup. Discussion:
<1925924(dot)izSMJEZO3x(at)unused-4-107(dot)brq(dot)redhat(dot)com>
http://git.postgresql.org/pg/commitdiff/ea046f08d1bcee56c3bedfa16a05c38a0515b41d

- Copy-editing for contrib/pg_visibility documentation. Add omitted names for
some function parameters. Fix some minor grammatical issues.
http://git.postgresql.org/pg/commitdiff/33596edf09516a7cab65914e16cfd6adf9fc55d1

- Fix bugs in contrib/pg_visibility. collect_corrupt_items() failed to
initialize tuple.t_self. While HeapTupleSatisfiesVacuum() doesn't actually
use that value, it does Assert that it's valid, so that the code would dump
core if ip_posid chanced to be zero. (That's somewhat unlikely, which
probably explains how this got missed. In any case it wouldn't matter for
field use.) Also, collect_corrupt_items was returning the wrong TIDs, that is
the contents of t_ctid rather than the tuple's own location. This would be
the same thing in simple cases, but it could be wrong if, for example, a past
update attempt had been rolled back, leaving a live tuple whose t_ctid doesn't
point at itself. Also, in pg_visibility(), guard against trying to read a
page past the end of the rel. The VM code handles inquiries beyond the end of
the map by silently returning zeroes, and it seems like we should do the same
thing here. I ran into the assertion failure while using pg_visibility to
check pg_upgrade's behavior, and then noted the other problems while reading
the code. Report: <29043(dot)1475288648(at)sss(dot)pgh(dot)pa(dot)us>
http://git.postgresql.org/pg/commitdiff/9a109452da1b923e183f20fcf5516984d448ece9

- Do ClosePostmasterPorts() earlier in SubPostmasterMain(). In standard Unix
builds, postmaster child processes do ClosePostmasterPorts immediately after
InitPostmasterChild, that is almost immediately after being spawned. This is
important because we don't want children holding open the postmaster's end of
the postmaster death watch pipe. However, in EXEC_BACKEND builds,
SubPostmasterMain was postponing this responsibility significantly, in order
to make it slightly more convenient to pass the right flag value to
ClosePostmasterPorts. This is bad, particularly seeing that
process_shared_preload_libraries() might invoke nearly-arbitrary code.
Rearrange so that we do it as soon as we've fetched the socket FDs via
read_backend_variables(). Also move the comment explaining about
randomize_va_space to before the call of PGSharedMemoryReAttach, which is
where it's relevant. The old placement was appropriate when the reattach
happened inside CreateSharedMemoryAndSemaphores, but that was a long time ago.
Back-patch to 9.3; the patch doesn't apply cleanly before that, and it doesn't
seem worth a lot of effort given that we've had no actual field complaints
traceable to this. Discussion: <4157(dot)1475178360(at)sss(dot)pgh(dot)pa(dot)us>
http://git.postgresql.org/pg/commitdiff/3b90e38c5d592ea8ec8236287dd5c749fc041728

- Avoid leaking FDs after an fsync failure. Fixes errors introduced in commit
bc34223bc, as detected by Coverity. In passing, report ENOSPC for a short
write while padding a new wal file in open_walfile, make certain that
close_walfile closes walfile in all cases, and improve a couple of comments.
Michael Paquier and Tom Lane
http://git.postgresql.org/pg/commitdiff/728ceba938dfadb204a4854bb76ae3b11b635401

- Add ALTER EXTENSION ADD/DROP ACCESS METHOD, and use it in pg_upgrade. Without
this, an extension containing an access method is not properly dumped/restored
during pg_upgrade --- the AM ends up not being a member of the extension after
upgrading. Another oversight in commit 473b93287, reported by Andrew Dunstan.
Report: <f7ac29f3-515c-2a44-21c5-ec925053265f(at)dunslane(dot)net>
http://git.postgresql.org/pg/commitdiff/e8bdee2770ff52aab208bc6f8965a4a01979d0aa

Peter Eisentraut pushed:

- Fix some typos in comment.
http://git.postgresql.org/pg/commitdiff/440c8d1bbc8c62d225ab7c3e30e0a7db2639cd0f

- Fix CRC check handling in get_controlfile. The previous patch broke this by
returning NULL for a failed CRC check, which pg_controldata would then try to
read. Fix by returning the result of the CRC check in a separate argument.
Michael Paquier and myself
http://git.postgresql.org/pg/commitdiff/e79e6c4da152154544913b38354b98d07b65e411

- Remove dead line of code.
http://git.postgresql.org/pg/commitdiff/967ed9205bfcd89c1dde1e85735b32b404721399

- Exclude additional directories in pg_basebackup. The list of files and
directories that pg_basebackup excludes from the backup was somewhat
incomplete and unorganized. Change that with having the exclusion driven from
tables. Clean up some code around it. Also document the exclusions in more
detail so that users of pg_start_backup can make use of it as well. The
contents of these directories are now excluded from the backup: pg_dynshmem,
pg_notify, pg_serial, pg_snapshots, pg_subtrans Also fix a bug that a
pg_repl_slot or pg_stat_tmp being a symlink would cause a corrupt tar header
to be created. Now such symlinks are included in the backup as empty
directories. Bug found by Ashutosh Sharma <ashu(dot)coek88(at)gmail(dot)com>. From:
David Steele <david(at)pgmasters(dot)net> Reviewed-by: Michael Paquier
<michael(dot)paquier(at)gmail(dot)com>
http://git.postgresql.org/pg/commitdiff/6ad8ac6026287e3ccbc4d606b6ab6116ccc0eec8

- pg_basebackup pg_receivexlog: Issue fsync more carefully. Several places
weren't careful about fsyncing in the way. See 1d4a0ab1 and 606e0f98 for
details about required fsyncs. This adds a couple of functions in src/common/
that have an equivalent in the backend: durable_rename(), fsync_parent_path()
From: Michael Paquier <michael(dot)paquier(at)gmail(dot)com>
http://git.postgresql.org/pg/commitdiff/bc34223bc1e2c51dff2007b3d3bd492a09b5a491

- Switch pg_basebackup commands in Postgres.pm to use --nosync. On slow
machines, this greatly reduces the I/O pressure induced by the tests. From:
Michael Paquier <michael(dot)paquier(at)gmail(dot)com>
http://git.postgresql.org/pg/commitdiff/728a3e73e96173bb0b0b2afc0da061991f9e3d5d

- pg_basebackup: Add --nosync option. This is useful for testing, similar to
initdb's --nosync. From: Michael Paquier <michael(dot)paquier(at)gmail(dot)com>
http://git.postgresql.org/pg/commitdiff/6ed2d8584cc680a2d6898480de74a57cd96176b5

- Move fsync routines of initdb into src/common/. The intention is to used
those in other utilities such as pg_basebackup and pg_receivexlog. From:
Michael Paquier <michael(dot)paquier(at)gmail(dot)com>
http://git.postgresql.org/pg/commitdiff/bf5bb2e85b6492c7245b9446efaf43d52a98db13

- Fix compiler warnings. This was missed in
bf5bb2e85b6492c7245b9446efaf43d52a98db13, because the code is only visible
under PG_FLUSH_DATA_WORKS.
http://git.postgresql.org/pg/commitdiff/f2af8dc5ba8801c126d877a52076d441cac78646

- Add missing include files to configure tests. atoi() needs stdlib.h strcmp()
needs string.h Reviewed-by: Heikki Linnakangas <hlinnaka(at)iki(dot)fi> Reviewed-by:
Thomas Munro <thomas(dot)munro(at)enterprisedb(dot)com>
http://git.postgresql.org/pg/commitdiff/a5da81359dbddfee49ea7800b43d99c204b4d31f

- Remove unnecessary prototypes. Prototypes for functions implementing
V1-callable functions are no longer necessary. Reviewed-by: Heikki
Linnakangas <hlinnaka(at)iki(dot)fi> Reviewed-by: Thomas Munro
<thomas(dot)munro(at)enterprisedb(dot)com>
http://git.postgresql.org/pg/commitdiff/0665023b4435a469e42289d7065c436967a022b6

- Use return instead of exit() in configure. Using exit() requires stdlib.h,
which is not included. Use return instead. Also add return type for main().
Reviewed-by: Heikki Linnakangas <hlinnaka(at)iki(dot)fi> Reviewed-by: Thomas Munro
<thomas(dot)munro(at)enterprisedb(dot)com>
http://git.postgresql.org/pg/commitdiff/1c0cf52b39ca3a9a79661129cff918dc000a55eb

- Fix use of offsetof(). Using offsetof() with a run-time computed argument is
not allowed in either C or C++. Apparently, gcc allows it, but g++ doesn't.
Reviewed-by: Heikki Linnakangas <hlinnaka(at)iki(dot)fi> Reviewed-by: Thomas Munro
<thomas(dot)munro(at)enterprisedb(dot)com>
http://git.postgresql.org/pg/commitdiff/f1a469c9f1cfeab9f9c7d4a5d3e75892e7b6f60c

- Fix breakage in previous change.
http://git.postgresql.org/pg/commitdiff/cd03890d0b5d9cbcfca169195d71f1693f84cafa

- Separate enum from struct. Otherwise the enum symbols are not visible outside
the struct in C++. Reviewed-by: Thomas Munro <thomas(dot)munro(at)enterprisedb(dot)com>
http://git.postgresql.org/pg/commitdiff/330b48b94b53bcbbc490f952d6798d5ab637721a

- Set log_line_prefix and application name in test drivers. Before pg_regress
runs psql, set the application name to the test name. Similarly, set the
application name to the test file name in the TAP tests. Also, set a default
log_line_prefix that show the application name, as well as the PID and a time
stamp. That way, the server log output can be correlated to the test input
files, making debugging a bit easier.
http://git.postgresql.org/pg/commitdiff/a4327296df7366ecc657b706a9b5e87aa921311a

Álvaro Herrera pushed:

- Include <sys/select.h> where needed. <sys/select.h> is required by
POSIX.1-2001 to get the prototype of select(2), but nearly no systems enforce
that because older standards let you get away with including some other
headers. Recent OpenBSD hacking has removed that frail touch of friendliness,
however, which broke some compiles; fix all the way back to 9.1 by adding the
required standard. Only vacuumdb.c was reported to fail, but it seems easier
to fix the whole lot in a fell swoop. Per bug #14334 by Sean Farrell.
http://git.postgresql.org/pg/commitdiff/51c3e9fade76c12e4aa37bffdf800bbf74fb3fb1

- Silence compiler warnings. Reported by Peter Eisentraut. Coding suggested by
Tom Lane.
http://git.postgresql.org/pg/commitdiff/b82d5a2c7c9dd1a01041feef2834db27a46fb2d3

Heikki Linnakangas pushed:

- Refactor script execution state machine in pgbench. The doCustom() function
had grown into quite a mess. Rewrite it, in a more explicit state machine
style, for readability. This also fixes one minor bug: if a script consisted
entirely of meta commands, doCustom() never returned to the caller, so
progress reports with the -P option were not printed. I don't want to
backpatch this refactoring, and the bug is quite insignificant, so only commit
this to master, and leave the bug unfixed in back-branches. Review and
original bug report by Fabien Coelho. Discussion:
<alpine(dot)DEB(dot)2(dot)20(dot)1607090850120(dot)3412(at)sto>
http://git.postgresql.org/pg/commitdiff/12788ae49e1933f463bc59a6efe46c4a01701b76

- Turn password_encryption GUC into an enum. This makes the parameter easier to
extend, to support other password-based authentication protocols than MD5.
(SCRAM is being worked on.) The GUC still accepts on/off as aliases for "md5"
and "plain", although we may want to remove those once we actually add support
for another password hash type. Michael Paquier, reviewed by David Steele,
with some further edits by me. Discussion:
<CAB7nPqSMXU35g=W9X74HVeQp0uvgJxvYOuA4A-A3M+0wfEBv-w(at)mail(dot)gmail(dot)com>
http://git.postgresql.org/pg/commitdiff/babe05bc2b781eb3eb84a18d7010d08277e2e399

- Don't bother to lock bufmgr partitions in pg_buffercache. That makes the view
a lot less disruptive to use on a production system. Without the locks, you
don't get a consistent snapshot across all buffers, but that's OK. It wasn't a
very useful guarantee in practice. Ivan Kartyshov, reviewed by Tomas Vondra
and Robert Haas. Discusssion:
<f9d6cab2-73a7-7a84-55a8-07dcb8516ae5(at)postgrespro(dot)ru>
http://git.postgresql.org/pg/commitdiff/6e654546fb61f62cc982d0c8f62241b3b30e7ef8

Robert Haas pushed:

- Fix dangling pointer problem in ReorderBufferSerializeChange. Commit
3fe3511d05127cc024b221040db2eeb352e7d716 introduced a new case into this
function, but neglected to ensure that the "ondisk" pointer got updated after
a possible reallocation as the code does in other cases. Stas Kelvich, per
diagnosis by Konstantin Knizhnik.
http://git.postgresql.org/pg/commitdiff/308985b0b404a5891a1a629f38cc46c2b2dcb4be

- worker_spi: Call pgstat_report_stat. Without this, statistics changes
accumulated by the worker never get reported to the stats collector, which is
bad. Julien Rouhaud
http://git.postgresql.org/pg/commitdiff/4929704acb3b4cdde8e890af977cdacaf1ac2ccb

Stephen Frost pushed:

- Remove superuser checks in pgstattuple. Now that we track initial privileges
on extension objects and changes to those permissions, we can drop the
superuser() checks from the various functions which are part of the
pgstattuple extension and rely on the GRANT system to control access to those
functions. Since a pg_upgrade will preserve the version of the extension
which existed prior to the upgrade, we can't simply modify the existing
functions but instead need to create new functions which remove the checks and
update the SQL-level functions to use the new functions (and to REVOKE EXECUTE
rights on those functions from PUBLIC). Thanks to Tom and Andres for adding
support for extensions to follow update paths (see: 40b449a), allowing this
patch to be much smaller since no new base version script needed to be
included. Approach suggested by Noah. Reviewed by Michael Paquier.
http://git.postgresql.org/pg/commitdiff/fd321a1dfd64d30bf1652ea6b39b654304f68ae4

Magnus Hagander pushed:

- Retry opening new segments in pg_xlogdump --follow. There is a small window
between when the server closes out the existing segment and the new one is
created. Put a loop around the open call in this case to make sure we wait for
the new file to actually appear.
http://git.postgresql.org/pg/commitdiff/3d39244e6e7374febff59eba338f7a82f696a618

== Pending Patches ==

Takayuki Tsunakawa sent in two revisions of a patch to implement huge_pages on
Windows.

Michaël Paquier and Heikki Linnakangas traded patch flocks to implement SCRAM
password auth.

Pavel Stěhule sent in another revision of a patch to add a \setfileref command
to psql.

Pavel Stěhule sent in another revision of a patch to add xmltable().

Vitaly Burovoy sent in a patch to detect supported SET parameters when
pg_restore is run.

Vinayak Pokale support prepared transactions with foreign servers.

Peter Eisentraut and Michaël Paquier traded patches to fix CRC check handling in
get_controlfile.

Stephen Frost and Jeevan Chalke traded patches to add support for restrictive
RLS policies.

Jesper Pedersen and Peter Eisentraut traded patches to add hash index support
to pageinspect.

Kyotaro HORIGUCHI and Michaël Paquier traded patches to track LSN on lightly
loaded systems to reduce the number of checkpoints.

Tomas Vondra sent in another revision of a patch to add two slab-like memory
allocators.

Fabien COELHO sent in three more revisions of a patch to add more operators and
functions to pgbench.

Fabien COELHO sent in another revision of a patch to enable storing select
results into variables.

Rajkumar Raghuwanshi sent in a patch to add test cases for partition-wise joins
on declarative partitions.

Daniel Gustafsson sent in a patch to make flex/bison checks stricter in Git
trees.

Mark Dilger sent in a patch to stop casting away const in comparators.

Mark Dilger sent in a patch to stop casting away const in regex.

Mark Dilger sent in a patch to avoid casting away const when using the FileName
typedef.

Takayuki Tsunakawa sent in a patch to allow SIGQUIT to prompt a fast shutdown of
the stats collector without leaving partly-written stats files behind.

Mark Dilger sent in a WIP patch to const-ify xlog_outdesc() and
rm_redo_error_callback().

David Steele and Michaël Paquier traded patches to exclude additional
directories in pg_basebackup.

Amit Langote sent in three more revisions of a patch to implement declarative
partitions.

Michaël Paquier sent in four more revisions of a patch to track wait event for
latches.

Victor Wagner sent in two more revisions of a patch to implement failover at the
libpq connect level.

Mark Dilger sent in a patch to tidy up some handling of pointers in
rangetypes_spgist.c, tsgistidx.c, and tsrank.c.

David Cramer and Heikki Linnakangas traded patches to add support for
multi-dimensional arrays to PL/PythonU.

Mithun Cy sent in another revision of a patch to cache hash index meta pages.

Emre Hasegeli sent in another revision of a patch to fix floating point
comparison inconsistencies in the geometric types.

Ivan Kartyshov sent in another revision of a patch to make pg_buffercache more
efficient with large shared memory.

Etsuro Fujita sent in two more revisions of a patch to push down more full joins
to the PostgreSQL FDW.

David Fetter and Thomas Munro traded patches to add a hook and corresponding
GUCs to disable UPDATEs and DELETEs that lack a WHERE clause.

Michaël Paquier sent in another revision of a patch to rename pg_xlog to pg_wal
and rename pg_clog to pg_transaction.

Heikki Linnakangas sent in two more revisions of a patch to change the way use
memory for larger read buffers in logtape.c rather than pre-reading tuples into
SortTuple slots during merge.

Magnus Hagander sent in two more revisions of a patch to enable pg_basebackup to
stream xlog to tar.

Kyotaro HORIGUCHI sent in another revision of a patch to fix wal level minimal.

Ashutosh Bapat sent in another revision of a patch to fix an issue where
altering foreign table was not invalidating prepare statement execution plans
that it should have.

Jeevan Chalke sent in another revision of a patch to enable pushing aggregates
to a FDW.

Aleksander Alekseev sent in a patch to rename md5Salt to pwsalt.

Corey Huinker sent in a PoC patch to make a copy() set-returning function.

Stephen Frost sent in another revision of a patch to fix some infelicities
between COPY and RLS.

Andres Freund sent in another revision of a patch to add a macro-customizable
hashtable for bitmapscan and aggregation performance.

Daniel Vérité sent in another revision of a patch to fix some pg_dump / copy
bugs with long lines.

Dmitry Dolgov sent in another revision of a patch to add a generic type
subscription.

Emre Hasegeli sent in a patch to add <@, @>, <<@, and @>> operator symbols for
the inet datatype to replace <<=, >>=, <<, and >>.

Christoph Berg sent in a patch to ensure that the default log_line prefix is not
empty.