Re: bgwrite process is too lazy

From: wenhui qiu <qiuwenhuifx(at)gmail(dot)com>
To: Andres Freund <andres(at)anarazel(dot)de>
Cc: Tomas Vondra <tomas(at)vondra(dot)me>, Tony Wayne <anonymouslydark3(at)gmail(dot)com>, Laurenz Albe <laurenz(dot)albe(at)cybertec(dot)at>, pgsql-hackers <pgsql-hackers(at)postgresql(dot)org>
Subject: Re: bgwrite process is too lazy
Date: 2024-10-07 06:07:40
Message-ID: CAGjGUAJMcdkRqovWM+x43e5omU48R355WB0Go=iiPrzCW_xtng@mail.gmail.com
Views: Raw Message | Whole Thread | Download mbox | Resend email
Thread:
Lists: pgsql-hackers

Hi Andres Freund
Thank you for explanation

> I doubt that slowdown is caused by bgwriter not being active enough. I
suspect
> what you're seeing is one or more of:

> a) The overhead of doing full page writes (due to increasing the WAL
> volume). You could verify whether that's the case by turning
> full_page_writes off (but note that that's not generally safe!) or see if
> the overhead shrinks if you set wal_compression=zstd or
wal_compression=lz4
> (don't use pglz, it's too slow).

> b) The overhead of renaming WAL segments during recycling. You could see
if
> this is related by specifying --wal-segsize 512 or such during initdb.
I am aware of these optimizations, and these optimizations only mitigate
the impact, I didn't turn on wal log compression on purpose during stress
test ,

shared_buffers = '32GB'
bgwriter_delay = '10ms'
bgwriter_lru_maxpages = '8192'
bgwriter_lru_multiplier = '10.0'
wal_buffers = '64MB'
checkpoint_completion_target = '0.999'
checkpoint_timeout = '600'
max_wal_size = '32GB'
min_wal_size = '16GB'

I think in business scenarios where there are many reads and few writes, it
is indeed desirable to keep as many dirty pages in memory as
possible,However, in scenarios such as push systems and task scheduling
systems, which also have a lot of reads and writes, the impact of
checkpoints will be more obvious,Adaptive bgwrite or bgwrite triggered when
a dirty page reaches a certain watermark eliminates the impact of
checkpoints on performance jitter.From what I understand, quite a few
commercial databases residing in postgresql have added the adaptive refresh
dirty page feature, and from their internal reports, the whole stress
testing process was very smooth! Since it's a trade secret, I don't know
how they implemented this feature.

In response to

Browse pgsql-hackers by date

  From Date Subject
Next Message Hunaid Sohail 2024-10-07 06:17:01 Re: Psql meta-command conninfo+
Previous Message Zhijie Hou (Fujitsu) 2024-10-07 05:57:26 RE: BUG #18641: Logical decoding of two-phase commit fails with TOASTed default values