Recommendations on Improving the debezium performance even on medium workload

From: Durgamahesh Manne <maheshpostgres9(at)gmail(dot)com>
To: pgsql-general(at)lists(dot)postgresql(dot)org, pgsql-in-general(at)postgresql(dot)org, Greg Sabino Mullane <htamfids(at)gmail(dot)com>, "kdg(dot)dev(at)gmail(dot)com" <kdg(dot)dev(at)gmail(dot)com>
Subject: Recommendations on Improving the debezium performance even on medium workload
Date: 2024-09-12 19:51:55
Message-ID: CAJCZkoKbfzFSfTemPiZN-mBrgy7ob=9gFq-5BnNEbHsH3Fe-dA@mail.gmail.com
Views: Raw Message | Whole Thread | Download mbox | Resend email
Thread:
Lists: pgsql-general pgsql-in-general

Hi Team

How do we improve the debezium performance?
Recommendations at kafka configuration side
Agenda is to minimize the lag during moderate or high work load on db

default > poll.interval.ms = 500ms
Recommended value for balanced performance > 1000ms
Recommended value for high throughput > 1000ms to 5000ms

default > max.batch.size = 2048 allocated > 2560
Recommended value for balanced performance > 500 to 1000
Recommended value for high throughput > 2000 or 5000

default > max.queue.size = 8192 allocated > 10000
Recommended value for balanced performance > 7500 (1000 to 10000)
Recommended value for high throughput > 10000

based on kafka load we can optimize debezium connector performance

default > heartbeat.interval.ms = 0 allocated > 5000ms
Recommended value for this > 5000ms to 10000ms

Are these helpful ?

Need auto connectivity even on connector(slot) failure

please let me know your recommendations please

Regards,

Durga Mahesh

Browse pgsql-general by date

  From Date Subject
Next Message Durgamahesh Manne 2024-09-12 20:04:40 pglogical selective child replication between different partition interval tables
Previous Message Ron Johnson 2024-09-12 19:43:59 Re: Better way to process records in bash?

Browse pgsql-in-general by date

  From Date Subject
Next Message Durgamahesh Manne 2024-09-12 20:04:40 pglogical selective child replication between different partition interval tables
Previous Message Durgamahesh Manne 2024-09-12 17:03:10 Re: Recommendations on improving the insert on conflict do nothing performance