RE: BUG #18569: Memory leak in Postgres Enterprise server

HI @Daniel Gustafsson<mailto:daniel(at)yesql(dot)se>

Postgres server is restarting multiple times when we run the use case with 3 JOBS. We have enough resources and never seen CPU and memory is gong high.

Please find Postgres parameters along with application logs.

MEMORY : 128GB
CPU : 32 Cores.

max_connections = 200
shared_buffers = 32GB
effective_cache_size = 96GB
maintenance_work_mem = 2GB
checkpoint_completion_target = 0.9
wal_buffers = 16MB
default_statistics_target = 100
random_page_cost = 1.1
effective_io_concurrency = 200
work_mem = 41943kB
huge_pages = try
min_wal_size = 1GB
max_wal_size = 4GB
max_worker_processes = 32
max_parallel_workers_per_gather = 4
max_parallel_workers = 32
max_parallel_maintenance_workers = 4

Madhusudhan | GOC Database Engineer
Informatica Global Operation Center
Office: Mob: +91 9502034172
Email: DL_GOC_Database_Services(at)informatica(dot)com<mailto:DL_GOC_Database_Services(at)informatica(dot)com>
[cid:image002(dot)png(at)01DB0B71(dot)B4D6AF20]<http://www.informatica.com/>

From: Mathias, Renci <rmathias(at)informatica(dot)com>
Sent: Monday, August 19, 2024 12:50 PM
To: Yeddula, Madhusudhan reddy [CONTINGENT WORKER] <myeddula(at)informatica(dot)com>; Sahu, Abhisek Kumar <absahu(at)informatica(dot)com>; Daniel Gustafsson <daniel(at)yesql(dot)se>; pgsql-bugs(at)lists(dot)postgresql(dot)org; Sachan, Vishal <vsachan(at)informatica(dot)com>
Cc: Kumar, Gaurav <gaurkumar(at)informatica(dot)com>; Nayak, Deepak Vaikunta <dnayak(at)informatica(dot)com>
Subject: RE: BUG #18569: Memory leak in Postgres Enterprise server

Hi @Yeddula, Madhusudhan reddy<mailto:myeddula(at)informatica(dot)com>,

Post re-running the operations, the repository experienced downtime after 32 hours and the PID was subsequently changed.

Here are the details of the run:

* Run Start Time: 14th Aug 8:00 PM
* Operation Failure: 15th Aug 8:21 AM (The repository PID 16320 did not change and the operation hang)
* All Operations Hang: 16th Aug 10:38 AM
* Repository PID Change: 16th Aug 11:01 AM (PID changed to 10544)

Below is the screenshot for your reference with reference to Operation Failure: 15th Aug 8:21 AM
[cid:image001(dot)png(at)01DB0B6D(dot)C4BF3E10]

Attached is the file with reference to Repository Logs.

Could you please investigate the DB server logs on your end and address the queries raised by the PostgreSQL team? Your prompt attention to this issue would be of great help.

Thanks,

Renci

-----Original Message-----
From: Yeddula, Madhusudhan reddy [CONTINGENT WORKER] <myeddula(at)informatica(dot)com<mailto:myeddula(at)informatica(dot)com>>
Sent: Wednesday, August 14, 2024 2:22 PM
To: Sahu, Abhisek Kumar <absahu(at)informatica(dot)com<mailto:absahu(at)informatica(dot)com>>; Mathias, Renci <rmathias(at)informatica(dot)com<mailto:rmathias(at)informatica(dot)com>>; Daniel Gustafsson <daniel(at)yesql(dot)se<mailto:daniel(at)yesql(dot)se>>; pgsql-bugs(at)lists(dot)postgresql(dot)org<mailto:pgsql-bugs(at)lists(dot)postgresql(dot)org>; Sachan, Vishal <vsachan(at)informatica(dot)com<mailto:vsachan(at)informatica(dot)com>>
Cc: Kumar, Gaurav <gaurkumar(at)informatica(dot)com<mailto:gaurkumar(at)informatica(dot)com>>; Nayak, Deepak Vaikunta <dnayak(at)informatica(dot)com<mailto:dnayak(at)informatica(dot)com>>
Subject: RE: BUG #18569: Memory leak in Postgres Enterprise server

HI @Sahu, Abhisek Kumar

Have done analysis from my end and changed postgres parameters in postgresql.conf file regarding Memory/CPU and other parameters as well. Kindly re run the repo and let me know if it is getting failed with same error.

Madhusudhan | GOC Database Engineer

Informatica Global Operation Center

Office: Mob: +91 9502034172

Email: DL_GOC_Database_Services(at)informatica(dot)com<mailto:DL_GOC_Database_Services(at)informatica(dot)com>

-----Original Message-----

From: Sahu, Abhisek Kumar <absahu(at)informatica(dot)com<mailto:absahu(at)informatica(dot)com>>

Sent: Monday, August 12, 2024 11:08 AM

To: Yeddula, Madhusudhan reddy [CONTINGENT WORKER] <myeddula(at)informatica(dot)com<mailto:myeddula(at)informatica(dot)com>>; Mathias, Renci <rmathias(at)informatica(dot)com<mailto:rmathias(at)informatica(dot)com>>; Daniel Gustafsson <daniel(at)yesql(dot)se<mailto:daniel(at)yesql(dot)se>>; pgsql-bugs(at)lists(dot)postgresql(dot)org<mailto:pgsql-bugs(at)lists(dot)postgresql(dot)org>; Sachan, Vishal <vsachan(at)informatica(dot)com<mailto:vsachan(at)informatica(dot)com>>

Cc: Kumar, Gaurav <gaurkumar(at)informatica(dot)com<mailto:gaurkumar(at)informatica(dot)com>>; Nayak, Deepak Vaikunta <dnayak(at)informatica(dot)com<mailto:dnayak(at)informatica(dot)com>>

Subject: RE: BUG #18569: Memory leak in Postgres Enterprise server

Hi Daniel,

Kindly let us know if you have any updates.

Regards

Abhisek Kumar Sahu

Senior Manager

Main: +91 80 40203000 Ext : 3135

Mobile: 9886680490

Email: absahu(at)informatica(dot)com<mailto:absahu(at)informatica(dot)com>

-----Original Message-----

From: Yeddula, Madhusudhan reddy [CONTINGENT WORKER] <myeddula(at)informatica(dot)com<mailto:myeddula(at)informatica(dot)com>>

Sent: Wednesday, August 7, 2024 4:45 PM

To: Mathias, Renci <rmathias(at)informatica(dot)com<mailto:rmathias(at)informatica(dot)com>>; Daniel Gustafsson <daniel(at)yesql(dot)se<mailto:daniel(at)yesql(dot)se>>; pgsql-bugs(at)lists(dot)postgresql(dot)org<mailto:pgsql-bugs(at)lists(dot)postgresql(dot)org>; Sahu, Abhisek Kumar <absahu(at)informatica(dot)com<mailto:absahu(at)informatica(dot)com>>; Sachan, Vishal <vsachan(at)informatica(dot)com<mailto:vsachan(at)informatica(dot)com>>

Cc: Kumar, Gaurav <gaurkumar(at)informatica(dot)com<mailto:gaurkumar(at)informatica(dot)com>>; Nayak, Deepak Vaikunta <dnayak(at)informatica(dot)com<mailto:dnayak(at)informatica(dot)com>>

Subject: RE: BUG #18569: Memory leak in Postgres Enterprise server

HI @Daniel Gustafsson

Your report mentions "Postgres Enterprise Server", are you running a product from a vendor like EDB

ANS : We are not using Postgres Enterprise Server from EDB.

Are you running postgres installed by yourself via a package manager (or similar)?

Ans : Installed postgres myself through YUM menthod

If you are buying postgres from a vendor then please contact them, if not you are in the right place.

We did not buy postgres from any vendor .

NOTE : We are using opensource postgresql 16.2 server on RHEL9.

Madhusudhan | GOC Database Engineer

Informatica Global Operation Center

Office: Mob: +91 9502034172

Email: DL_GOC_Database_Services(at)informatica(dot)com<mailto:DL_GOC_Database_Services(at)informatica(dot)com>

-----Original Message-----

From: Mathias, Renci <rmathias(at)informatica(dot)com<mailto:rmathias(at)informatica(dot)com>>

Sent: Wednesday, August 7, 2024 11:05 AM

To: Daniel Gustafsson <daniel(at)yesql(dot)se<mailto:daniel(at)yesql(dot)se>>; pgsql-bugs(at)lists(dot)postgresql(dot)org<mailto:pgsql-bugs(at)lists(dot)postgresql(dot)org>; Yeddula, Madhusudhan reddy [CONTINGENT WORKER] <myeddula(at)informatica(dot)com<mailto:myeddula(at)informatica(dot)com>>; Sahu, Abhisek Kumar <absahu(at)informatica(dot)com<mailto:absahu(at)informatica(dot)com>>; Sachan, Vishal <vsachan(at)informatica(dot)com<mailto:vsachan(at)informatica(dot)com>>

Cc: Kumar, Gaurav <gaurkumar(at)informatica(dot)com<mailto:gaurkumar(at)informatica(dot)com>>

Subject: RE: BUG #18569: Memory leak in Postgres Enterprise server

Hi @Daniel,

The log files are attached for your reference.

Please let us know if you have any queries.

Thanks,

Renci Monthi Mathias | QA Engineer

Bagmane Tech Park, C V Raman Nagar, Bangalore - 560093

Email: rmathias(at)informatica(dot)com<mailto:rmathias(at)informatica(dot)com>

-----Original Message-----

From: Sahu, Abhisek Kumar <absahu(at)informatica(dot)com<mailto:absahu(at)informatica(dot)com>>

Sent: Tuesday, August 6, 2024 5:31 PM

To: Daniel Gustafsson <daniel(at)yesql(dot)se<mailto:daniel(at)yesql(dot)se>>; pgsql-bugs(at)lists(dot)postgresql(dot)org<mailto:pgsql-bugs(at)lists(dot)postgresql(dot)org>; Yeddula, Madhusudhan reddy [CONTINGENT WORKER] <myeddula(at)informatica(dot)com<mailto:myeddula(at)informatica(dot)com>>; Sachan, Vishal <vsachan(at)informatica(dot)com<mailto:vsachan(at)informatica(dot)com>>

Cc: Mathias, Renci <rmathias(at)informatica(dot)com<mailto:rmathias(at)informatica(dot)com>>; Kumar, Gaurav <gaurkumar(at)informatica(dot)com<mailto:gaurkumar(at)informatica(dot)com>>

Subject: RE: BUG #18569: Memory leak in Postgres Enterprise server

Hi Daniel,

Thank you for your response.

@Sachan, Vishal please attach all the logs, including the ODL tracing.

Adding @Yeddula, Madhusudhan reddy [CONTINGENT WORKER] DBA to answer the below queries.

Regards

Abhisek

-----Original Message-----

From: Daniel Gustafsson <daniel(at)yesql(dot)se<mailto:daniel(at)yesql(dot)se>>

Sent: Tuesday, August 6, 2024 4:45 PM

To: Sahu, Abhisek Kumar <absahu(at)informatica(dot)com<mailto:absahu(at)informatica(dot)com>>; pgsql-bugs(at)lists(dot)postgresql(dot)org<mailto:pgsql-bugs(at)lists(dot)postgresql(dot)org>

Subject: Re: BUG #18569: Memory leak in Postgres Enterprise server

Caution: This email originated from outside of the organization. Review for Phishing!

> On 6 Aug 2024, at 10:34, PG Bug reporting form <noreply(at)postgresql(dot)org<mailto:noreply(at)postgresql(dot)org>> wrote:

> PostgreSQL Server/Client Version: PostgreSQL 16.2

Your report mentions "Postgres Enterprise Server", are you running a product from a vendor like EDB or are you running postgres installed by yourself via a package manager (or similar)? If you are buying postgres from a vendor then please contact them, if not you are in the right place.

Upgrading to the latest 16.3 is also recommended even though there are no references to memory leak fixes in the release notes.

> We have also attached our Repository logs, PostgreSQL server logs and

> DB logs.

No, there are no logs attached (I don't think the bugreporting form support attaching any files). Please provide additional information as there is nothing to go in the report.

--

Daniel Gustafsson