Re: pg_upgrade failing for 200+ million Large Objects

From: Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us>
To: Alexander Korotkov <aekorotkov(at)gmail(dot)com>
Cc: Justin Pryzby <pryzby(at)telsasoft(dot)com>, Nathan Bossart <nathandbossart(at)gmail(dot)com>, Michael Banck <mbanck(at)gmx(dot)net>, Laurenz Albe <laurenz(dot)albe(at)cybertec(dot)at>, vignesh C <vignesh21(at)gmail(dot)com>, "Kumar, Sachin" <ssetiya(at)amazon(dot)com>, Robins Tharakan <tharakan(at)gmail(dot)com>, Jan Wieck <jan(at)wi3ck(dot)info>, Bruce Momjian <bruce(at)momjian(dot)us>, Andrew Dunstan <andrew(at)dunslane(dot)net>, Magnus Hagander <magnus(at)hagander(dot)net>, Peter Eisentraut <peter(dot)eisentraut(at)enterprisedb(dot)com>, pgsql-hackers(at)postgresql(dot)org
Subject: Re: pg_upgrade failing for 200+ million Large Objects
Date: 2024-07-26 23:06:21
Message-ID: 1873872.1722035181@sss.pgh.pa.us
Views: Raw Message | Whole Thread | Download mbox | Resend email
Thread:
Lists: pgsql-hackers

Alexander Korotkov <aekorotkov(at)gmail(dot)com> writes:
> On Sat, Jul 27, 2024 at 1:37 AM Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us> wrote:
>> It's fairly easy to fix things so that this example doesn't cause
>> that to happen: we just need to issue these updates as one command
>> not N commands per table.

> I was thinking about counting actual number of queries, not TOC
> entries for transaction number as a more universal solution. But that
> would require usage of psql_scan() or writing simpler alternative for
> this particular purpose. That looks quite annoying. What do you
> think?

The assumption underlying what we're doing now is that the number
of SQL commands per TOC entry is limited. I'd prefer to fix the
code so that that assumption is correct, at least in normal cases.
I confess I'd not looked closely enough at the binary-upgrade support
code to realize it wasn't correct already :-(. If we go that way,
we can fix this while also making pg_upgrade faster rather than
slower. I also expect that it'll be a lot simpler than putting
a full SQL parser in pg_restore.

regards, tom lane

In response to

Responses

Browse pgsql-hackers by date

  From Date Subject
Next Message Jeff Davis 2024-07-26 23:47:23 Re: MAINTAIN privilege -- what do we need to un-revert it?
Previous Message Alexander Korotkov 2024-07-26 22:55:00 Re: pg_upgrade failing for 200+ million Large Objects