Re: [postgis-users] ERROR: array size exceeds themaximumallowed(134217727)

On Mon, Mar 22, 2010 at 4:45 PM, Paul Ramsey <pramsey(at)cleverelephant(dot)ca> wrote:
> Did you already try replacing your postgis functions with array_agg
> calls to see if we can push the problem back over the fence to pgsql
> land?

On Sat, Mar 20, 2010 at 9:17 AM, Mike Leahy <mgleahy(at)alumni(dot)uwaterloo(dot)ca> wrote:
> Running this query on various data will produce one of two results.  One is
> the error mentioned in the subject (ERROR:  array size exceeds the maximum
> allowed (134217727)).  I can find very little information on this error.  The
> other outcome is that it often causes the PostgreSQL backend to segfault (see
> gdboutput.txt).

gdboutput.txt:

Program terminated with signal 11, Segmentation fault.
#0 0x00007fa4be23615b in pfree () from
/usr/lib/postgresql/8.4/bin/postgres
#1 0x00007fa4be18091b in makeMdArrayResult () from
/usr/lib/postgresql/8.4/bin/postgres
#2 0x00007fa4b7f038bc in pgis_accum_finalfn () from
/usr/lib/postgresql/8.4/lib/postgis-1.5.so
#3 0x00007fa4b7f039ee in pgis_geometry_union_finalfn () from
/usr/lib/postgresql/8.4/lib/postgis-1.5.so

Any chance you can generate one of these seg faults from a build with
symbols and with assertions enabled? It might catch the problem
earlier and provide more info.

IIRC there were some memory management changes which required changes
to array_agg() and which had some risk of causing problems for other
sites with similar coding. Is it possible your'e missing some of these
changes? I'm having trouble tracking them all down but at least
there's these:

commit 3d332de2eab8a01c0ef3f58ea69de2010fe8a1e1
Author: Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us>
Date: Thu Jul 23 20:45:27 2009 +0000

In a non-hashed Agg node, reset the "aggcontext" at group
boundaries, instead
of individually pfree'ing pass-by-reference transition values. This should
be at least as fast as the prior coding, and it has the major advantage of
clearing out any working data an aggregate function may have stored in or
underneath the aggcontext. This avoids memory leakage when an aggregate
such as array_agg() is used in GROUP BY mode. Per report from Chris Spotts.

Back-patch to 8.4. In principle the problem could arise in prior versions,
but since they didn't have array_agg the issue seems not critical.

commit 7f83b61cc26eeac0c5a09add49f6cf899f87fc0b
Author: Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us>
Date: Sat Jun 20 18:45:28 2009 +0000

Fix things so that array_agg_finalfn does not modify or free its input
ArrayBuildState, per trouble report from Merlin Moncure. By adopting
this fix, we are essentially deciding that aggregate final-functions
should not modify their inputs ever. Adjust documentation and comments
to match that conclusion.

--
greg