Re: Lots of memory allocated when reassigning Large Objects

Le lun. 29 nov. 2021 à 20:39, Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us> a écrit :

> Guillaume Lelarge <guillaume(at)lelarge(dot)info> writes:
> > I've tried Justin's patch but it didn't help with my memory allocation
> > issue. FWIW, I attach the patch I used in v14.
>
> [ looks closer ... ] Ah, that patch is a bit buggy: it fails to do the
> right thing in the cases where the loop does a "continue". The attached
> revision seems to behave properly.
>
> I still see a small leakage, which I think is due to accumulation of
> pending sinval messages for the catalog updates. I'm curious whether
> that's big enough to be a problem for Guillaume's use case. (We've
> speculated before about bounding the memory used for pending sinval
> in favor of just issuing a cache reset when the list would be too
> big. But nobody's done anything about it, suggesting that people
> seldom have a problem in practice.)
>
>
I've tried your patch with my test case. It still uses a lot of memory.
Actually even more.

I have this with the log_statement_stats:

1185072 kB max resident size

And I have this with the log-memory-contexts function:

LOG: Grand total: 1007796352 bytes in 320 blocks; 3453512 free (627
chunks); 1004342840 used

Contrary to Justin's patch, the shdepReassignOwned doesn't seem to be used.
I don't get any shdepReassignOwned line in the log file. I tried multiple
times to avoid any mistake on my part, but got same result.

>> DROP OWNED BY likely has similar issues.
>
> > Didn't try it, but it wouldn't be a surprise.
>
> I tried just changing the REASSIGN to a DROP in Justin's example,
> and immediately hit
>
> ERROR: out of shared memory
> HINT: You might need to increase max_locks_per_transaction.
>
> thanks to the per-object locks we try to acquire. So I'm not
> sure that the DROP case can reach an interesting amount of
> local memory leaked before it runs out of lock-table space.
>
>
I've hit the same issue when I tried my ALTER LARGE OBJECT workaround in
one transaction.

--
Guillaume.