Re: memory usage of pg_upgrade

On Mon, Sep 9, 2013 at 06:39:39PM -0400, Andrew Dunstan wrote:
>
> On 09/09/2013 06:20 PM, Jeff Janes wrote:
> >pg_upgrade reserves 5 times MAXPGPATH, or 5120 characters, for the
> >tablespace name of every object (table, toast table, index) in the
> >database being upgraded. This adds up pretty quickly when there is a
> >very large number of objects. It could be changed to char* to a
> >separately allocated name that takes only as much space it needs. But
> >maybe it would be better to point into os_info.old_tablespaces or
> >something like that, as surely there are not going to be one
> >independent file space per object.
> >
> >
> >typedef struct
> >{
> > ...
> > char tablespace[MAXPGPATH];
> >} RelInfo;
> >
> >The struct FileNameMap has 4 more .
> >
> >Since there seems to be some interest in improving the scalability of
> >pg_upgrade, this is one of the things to consider fixing. What is the
> >best way to do it?
>
>
> Send in a patch :-)
>
> We recently ripped out some uses of statically sized strings in the
> parallel code and replaced them with pointers to palloc'ed strings.
> So there is good precedent for this. See <https://github.com/postgres/postgres/commit/910d3a458c15c1b4cc518ba480be2f712f42f179>
>
> In the case of tablespaces, I should have thought you could keep a
> hash table of the names and just store an entry id in the table
> structure. But that's just my speculation without actually looking
> at the code, so don't take my word for it :-)

Yes, please feel free to improve the code. I improved pg_upgrade CPU
usage for a lerge number of objects, but never thought to look at memory
usage. It would be a big win to just palloc/pfree the memory, rather
than allocate tones of memory. If you don't get to it, I will in a few
weeks.

--
Bruce Momjian <bruce(at)momjian(dot)us> http://momjian.us
EnterpriseDB http://enterprisedb.com

+ It's impossible for everything to be true. +