== PostgreSQL Weekly News - January 28 2007 ==

== PostgreSQL Weekly News - January 28 2007 ==

Another Lively Discussion is going on on -hackers about improvements
to autovacuum.

Andy Astor, CEO of EnterpriseDB has vociferously denied that there was
a policy of gathering email addresses from the PostgreSQL lists and
then sending them marketing materials.

== PostgreSQL Product News ==

Libpqxx 2.6.9 pre-release is out.
http://thaiopensource.org/~jtv/tmp/pqxx/

SQL/PSM 0.1 released.
http://postgresql.interweb.cz/index.php/SQL/PSM

DBI-Link 2.0.0 released.
http://pgfoundry.org/projects/dbi-link/

Another PostgreSQL Diff Tool 1.0.0_beta16 released.
http://pgfoundry.org/projects/apgdiff/

PostgresDAC 2.3.7 released.
http://microolap.com/products/connectivity/postgresdac/download/

Mammoth Replicator 8.0.10-1.7 released.
https://www.commandprompt.com/products/mammothreplicator/

== PostgreSQL Jobs for January ==

http://archives.postgresql.org/pgsql-jobs/2007-01/threads.php

== PostgreSQL Local ==

Major contributor Gavin Sherry and Postgres-R's lead Markus will be
speaking at the SFPUG meeting in San Francisco at Hyperic LLC's
headquarters.
http://postgresql.meetup.com/1/calendar/5380885/

PostgreSQLFr will have a booth at SolutionsLinux 2007, in Paris
(France) from 30th january to 1st February, with special guests Devrim
GUNDUZ and Magnus Hagander. Jean-Paul Argudo, treasurer of the
PostgreSQLFr non-profit assocation, will lead a session about FLOSS
databases. More information in French here:
http://www.solutionslinux.fr/index.php

There will be a PostgreSQL booth at FOSDEM on February 24 and 25 in
Brussels, Belgium. Many of the usual suspects from the EU PostgreSQL
communities will be there. Contact de(at)postgresql(dot)org to participate.
http://www.fosdem.org/2007/

The Italian PostgreSQL community will be holding a PostgreSQL day this
summer. Bookmark the link below to participate.
http://www.pgday.it

== PostgreSQL in the News ==

Planet PostgreSQL: http://www.planetpostgresql.org/

General Bits, Archives and occasional new articles:
http://www.varlena.com/GeneralBits/

PostgreSQL Weekly News is brought to you this week by David Fetter
and Devrim GUNDUZ

To get your submission into the upcoming issue, get it to
david(at)fetter(dot)org by Sunday at 3:00pm Pacific Time.

== Applied Patches ==

Andrew Dunstan committed:

- Fix plpython MSVC build in non-debug mode.

- Fix path problem in MSVC bison wrapper. per Joachim Wieland.

- In PL/Perl, remove unnecessary and now inaccurate cast which I
should have removed with other old code.

- In PL/Perl, allow args to spi_prepare to be standard type aliaes as
well as those known in pg_type. Fixes bug #2917. Add some
regression tests for these cases.

Michael Meskes committed:

- In ecpg, fixed expected files, so they are in sync with tests again.

- In ecpg, removed compiler warning due to unneeded unsigned
declaration and removed regression test that triggers those libc
precision bugs on some archs.

Neil Conway committed:

- Rename the uuid_t type to pg_uuid_t, to avoid a conflict with any
definitions of uuid_t that may be provided by the system headers.
This should hopefully fix the Win32 build problems reported by
Magnus.

- Gevik Babakhani's patch which add a new builtin type, "uuid". This
implements a UUID type, similar to that defined in RFC 4122. This
patch includes the basic implementation, plus regression tests.
Documentation and perhaps some additional functionality will come
later. Catversion bumped. Review from Peter Eisentraut, Tom Lane,
and Neil Conway.

- Magnus Hagander's patch which squelches some VC++ compiler warnings.
Mark float literals with the "f" suffix, to distinguish them from
doubles. Make some function declarations and definitions use the
"const" qualifier for arguments consistently. Ignore warning 4102
("unreferenced label"), because such warnings are always emitted by
bison-generated code.

- Magnus Hagander's patch which adds a setlocal command to the
beginning of build.bat. This is required to deal with buildenv.bat
properly, so that PATH (for example) doesn't expand infintly. Per
report from Joachim Wieland.

- Magnus Hagander's patch which adds a setlocal command to the
beginning of build.bat. This is required to deal with buildenv.bat
properly, so that PATH (for example) doesn't expand infintly. Per
report from Joachim Wieland.

Peter Eisentraut committed:

- In pgsql/src/backend/utils/adt/xml.c, reactivate libxml memory
management via palloc, now that I think I've classified the
conditions under which this is safe to do (see source code comment).

- In pgsql/src/backend/utils/adt/xml.c, add trailing zero byte in
Unicode codepoint conversion.

- Various fixes in the logic of XML functions: Add new SQL command
SET XML OPTION (also available via regular GUC) to control the
DOCUMENT vs. CONTENT option in implicit parsing and serialization
operations. Subtle corrections in the handling of the standalone
property in xmlroot(). Allow xmlroot() to work on content
fragments. Subtle corrections in the handling of the version
property in xmlconcat(). Code refactoring for producing XML
declarations.

- Simplify handling of XML error messages: Just use the string
provided by libxml as the detail message. As per
<http://archives.postgresql.org/pgsql-hackers/2006-12/msg01087.php>.
For converting error codes to messages, we only need to cover those
codes that we raise ourselves now.

Alvaro Herrera committed:

- Magnus Hagander's patch which is required for vcbuild to work after
the changes to pg_proc.h (it's the same changes that are in
gen_fmgrtab.sh in the unix build).

Tom Lane committed:

- Improve hash join to discard input tuples immediately if they can't
match because they contain a null join key (and the join operator is
known strict). Improves performance significantly when the inner
relation contains a lot of nulls, as per bug #2930.

- Remove unnecessary checkpoint from PL regression tests. This was
once handy to prevent core dump files from disappearing, but it's
useless now because (a) we don't drop core in individual DB
subdirectories anymore, and (b) CREATE DATABASE forces an internal
checkpoint anyway.

- Remove some unnecessary conversion work in build_regtype_array().

- Repair oversight in creation of "append relations": we should set up
rel->tuples as well as rel->rows, since some estimation functions
expect both to be valid in every baserel. Per report from Dave
Dutcher.

- Make some small improvements in the accuracy of plpgsql's error
location reports; inspired by the misleading CONTEXT lines shown in
recent bug report from Stefan Kaltenbrunner. Also, allow
statement-type names shown in these messages to be translated.

- Fix up plpgsql's "simple expression" evaluation mechanism so that it
behaves safely in the presence of subtransactions. To ensure that
any ExprContext shutdown callbacks are called at the right times, we
have to have a separate EState for each level of subtransaction.
Per "TupleDesc reference leak" bug report from Stefan Kaltenbrunner.
Although I'm convinced the code is wrong as far back as 8.0, it
doesn't seem that there are any ways for the problem to really
manifest before 8.2: AFAICS, 8.0 and 8.1 only use the
ExprContextCallback mechanism to handle set-returning functions,
which cannot usefully be executed in a "simple expression" anyway.
Hence, no backpatch before 8.2 --- the risk of unforeseen breakage
seems to outweigh the chance of fixing something.

- Drat, can't fit an additional argument into log_error. Is it worth
an sprintf pushup to be sure we can report something useful for
out-of-range exitstatus?

- Clean up broken usage of HAVE_DECL_SYS_SIGLIST and
inconsistent/poorly formatted error messages.

- Add a delay at the start of the stats test, to let any prior stats
activity quiesce. Possibly this will fix the large increase in
non-reproducible stats test failures we've noted since turning on
stats_row_level by default.

- Dept of second thoughts: the IQ of estimate_array_length() needs to
be kept on par with that of scalararraysel(), else estimates that
should track might not. Hence teach it about binary-compatible
cases, too.

- Fix scalararraysel() to cope with binary-compatible cases, such as
text[] versus varchar[]. This oversight probably explains Ryan
Holmes' recent complaint --- he was getting a generic selectivity
estimate instead of anything intelligent.

- Correct an old logic error in btree page splitting: when considering
a split exactly at the point where we need to insert a new item, the
calculation used the wrong size for the "high key" of the new left
page. This could lead to choosing an unworkable split, resulting in
"PANIC: failed to add item to the left sibling" (or "right sibling")
failure. Although this bug has been there a long time, it's very
difficult to trigger a failure before 8.2, since there was generally
a lot of free space on both sides of a chosen split. In 8.2, where
the user-selected fill factor determines how much free space the
code tries to leave, an unworkable split is much more likely.
Report by Joe Conway, diagnosis and fix by Heikki Linnakangas.

- Back-port changes of Jan 16 and 17 to "revoke" pending fsync
requests during DROP TABLE and DROP DATABASE. Should prevent
unexpected "permission denied" failures on Windows, and is cleaner
on other platforms too since we no longer have to take it on faith
that ENOENT is okay during an fsync attempt. Patched as far back as
8.1; per recent discussion I think we are not going to worry about
Windows-specific issues in 8.0 anymore.

- On Windows, use pgwin32_waitforsinglesocket() instead of select() to
wait for input in the stats collector. Our select() emulation is
apparently buggy for UDP sockets :-(. This should resolve problems
with stats collection (and hence autovacuum) failing under more than
minimal load. Diagnosis and patch by Magnus Hagander.

- Get pg_utf_mblen(), pg_utf2wchar_with_len(), and utf2ucs() all on
the same page about the maximum UTF8 sequence length we support (4
bytes since 8.1, 3 before that). pg_utf2wchar_with_len never got
updated to support 4-byte characters at all, and in any case had a
buffer-overrun risk in that it could produce multiple pg_wchars from
what mblen claims to be just one UTF8 character. The only reason we
don't have a major security hole is that most callers allocate
worst-case output buffers; the sole exception in released versions
appears to be pre-8.2 iwchareq() (ie, ILIKE), which can be crashed
due to zeroing out its return address --- but AFAICS that can't be
exploited for anything more than a crash, due to inability to
control what gets written there. Per report from James Russell and
Michael Fuhr. Pre-8.1 the risk is much less, but I still think
pg_utf2wchar_with_len's behavior given an incomplete final character
risks buffer overrun, so back-patch that logic change anyway. This
patch also makes sure that UTF8 sequences exceeding the supported
length (whichever it is) are consistently treated as error cases,
rather than being treated like a valid shorter sequence in some
places.

- Relax an Assert() that has been found to be too strict in some
situations involving unions of types having typmods. Variants of
the failure are known to occur in 8.1 and up; not sure if it's
possible in 8.0 and 7.4, but since the code exists that far back,
I'll just patch 'em all. Per report from Brian Hurt.

- Update xindex.sgml to discuss operator families.

- Update pg_dump to support dumping operator families.

- Add CREATE/ALTER/DROP OPERATOR FAMILY commands, also COMMENT ON
OPERATOR FAMILY; and add FAMILY option to CREATE OPERATOR CLASS to
allow adding a class to a pre-existing family. Per previous
discussion. Man, what a tedious lot of cutting and pasting ...

- Put back planner's ability to cache the results of
mergejoinscansel(), which I had removed in the first cut of the
EquivalenceClass rewrite to simplify that patch a little. But it's
still important --- in a four-way join problem mergejoinscansel()
was eating about 40% of the planning time according to gprof. Also,
improve the EquivalenceClass code to re-use join RestrictInfos
rather than generating fresh ones for each join considered. This
saves some memory space but more importantly improves the
effectiveness of caching planning info in RestrictInfos.

Bruce Momjian committed:

- Use autoconf build-in sys_siglist macro AC_DECL_SYS_SIGLIST, rather
than create our own.

- Add signal.h for sys_siglist reference.

- Use sys_siglist[] to print out signal names for signal exits, rather
than just numbers.

- Modify SGML makefile to allow 'gmake draft' to build draft html, so
'draft' is now a target and no longer a modifier.

- Add to TODO: "Enforce typmod for function inputs, function results
and parameters for spi_prepare'd statements called from PLs."

- Add to TODO: "Consider having the background writer update the
transaction status hint bits before writing out the page."

- Add to TODO: "Consider increasing NUM_CLOG_BUFFERS."

- Update installation wording for an upgrade to state that
dump/restore should not be done, per Peter.

- Control openjade draft-mode by variable DRAFT, rather than whether
the version tag is 'devel'.

- Update wording of installation when upgrading, to distinguish major
release upgrades more clearly from minor ones.

- Modify draft SGML instructions to use DRAFT=Y because recursion was
causing html to be called twice --- no way to exit the makefile
after the recursion returns.

- Update BSD/OS platform for 8.2.

- In TODO, add URL for shrinking tuple to just its headers.

- Update Win32 exception comment.

- Update pg_dumpall -f option description.

- In TODO, mark as done, "Add a GUC variable to control the tablespace
for temporary objects pg_dumpall" per Albert Cervera Areny's patch.

- Albert Cervera Areny's patch with documentation by Jaime Casanova to
add GUC temp_tablespaces to provide a default location for temporary
objects.

- Properly detoast access to bytea field pg_trigger.tgargs. Old code
might cause server crash. Backpatch to 8.2.X.

- Dave Page's patch which adds 'output file' option for pg_dumpall,
especially useful for Win32, where output redirection of child
processes (pg_dump) doesn't work.

- Remove developers list from TODO list now that we have URLs to
reference discussions.

- In TODO, mark as done, "Add -f to pg_dumpall" per Dave Page's patch.

- Dave Page patch which allows pg_dumpall to specify a database name
rather than the default 'template1'.

- Dave Page's patch which adds --tablespaces-only and --roles-only
options to pg_dumpall.

- Simon Riggs's patch to prevent WAL logging when COPY is done in the
same transaction that created it.

- Magnus Hagander's patch that disables the building of ecpg when
using MSVC if pthreads is not specified.

- In pgsql/src/include/port/win32.h, add comment about exception lists
in both winnt.h and ntstatus.h.

- Remove newline from error message because URL is gone.

- Back out use of FormatMessage(), does error values, not exception
values. Point to /include/ntstatus.h for an exception list, rather
than a URL.

- Print meaningfull error text for abonormal process exit on Win32,
rather than hex codes, using FormatMessage().

- In FAQ, update Bruce Momjian's email address.

- Use errhint() for WIN32 SIGTERM message, where possible.

- In pgsql/src/include/port/win32.h, put hex value in upper case.

- When system() fails in Win32, report it as an exception, print the
exception value in hex, and give a URL where the value can be
looked-up.

== Rejected Patches (for now) ==

Guido Goldstein's patch to fix PL/PythonU functions; return
true/false for boolean, rather than 1/0. This helps when creating
trigger functions that output SQL. It also broke most machines on the
buildfarm because it only works with Python 2.3 or higher.

Magnus Hagander's first version of his patch to clean up warnings
generated by VC++. It was too large and invasive, and had some
thinkos. He is working on the next version.

== Pending Patches ==

Arul Shaji sent in another version of his implementation of
pg_get_domaindef.

Gevik Babakhani sent in another version of his uuid patch. Peter
Eisentraut gave some feedback. Gevik sent in another version per the
comments and added tests.

Jeremy Drake sent in a couple of versions of a patch which allows
unprivileged users to CREATE LANGUAGE for trusted languages already
present in pg_pltemplate.

Teodor Sigaev sent in a doc patch which updates the docbook
instructions for FreeBSD, and another which adds a table of
pairwise compatibilities between the different lock modes.

Andrew Dunstan sent in a patch which lets people use the aliases in
pg_type for parameters in PL/Tcl and PL/PythonU, e.g. integer
instead of int4.

Pavan Deolasee sent in a patch which should marginally improve the
ctid chain followin code path when current and the next tuple in the
chain are in the same block.

Alvaro Herrera sent in a patch which separates autovacuum in two
families of processes: one is the "launcher", in charge of examining
statistics and deciding when to start a worker. The other is the
worker, which is started by the postmaster under command of the
launcher, and processes what the launcher tells it to process (by way
of setting info up in shared memory).

Pavel Stehule sent in a patch to add ANSI SQL scrollable cursor
support to SPI and PL/PgSQL.

Jeremy Drake sent in another version of his patch which lets you
install PLs without being superuser.