bgwrite process is too lazy

From: wenhui qiu <qiuwenhuifx(at)gmail(dot)com>
To: PostgreSQL Hackers <pgsql-hackers(at)lists(dot)postgresql(dot)org>
Subject: bgwrite process is too lazy
Date: 2024-10-02 08:48:01
Message-ID: CAGjGUALHnmQFXmBYaFCupXQu7nx7HZ79xN29+HoE5s-USqprUg@mail.gmail.com
Views: Raw Message | Whole Thread | Download mbox | Resend email
Thread:
Lists: pgsql-hackers

Hi hackers
Whenever I check the checkpoint information in a log, most dirty pages
are written by the checkpoint process,I tried to read the source code to
find out why, I think reducing this value will make bgwrite process write
more dirty pages (min_scan_buffers = (int) (NBuffers /
(scan_whole_pool_milliseconds / BgWriterDelay));) ,Here's how I tested
it, Originally, I wanted to write a patch, and I listened to the expert
community leaders first

src/backend/postmaster/bgwriter.c
[image: image.png]

I see a line of comments in (src/backend/storage/buffer/bufmgr.c )

[image: image.png]

This is a simple test I did myself
```
test methods
pgbench -i -s 10000 dbname -U postgres
pgbench -U postgres dbname -c 96 -j 96 -P 1 -T 900
test env
Intel(R) Xeon(R) Platinum 8336C CPU @ 2.30GHz 128 threads 64 cores
512G
ssd:1.8 T
postgresql version 17 GA

postgresql parameters
cat postgresql.auto.conf
listen_addresses = '*'
port = '5430'
max_connections = '1000'
unix_socket_directories = '/tmp,.'
unix_socket_permissions = '0700'
password_encryption = 'md5'
shared_buffers = '32GB'
maintenance_work_mem = '2GB'
autovacuum_work_mem = '2GB'
vacuum_buffer_usage_limit = '16GB'
max_files_per_process = '60000'
vacuum_cost_limit = '10000'
bgwriter_delay = '10ms'
bgwriter_lru_maxpages = '8192'
bgwriter_lru_multiplier = '10.0'
max_worker_processes = '32'
max_parallel_workers_per_gather = '8'
max_parallel_maintenance_workers = '8'
max_parallel_workers = '32'
wal_buffers = '64MB'
checkpoint_completion_target = '0.999'
max_wal_size = '32GB'
min_wal_size = '16GB'
archive_mode = 'on'
archive_command = '/bin/date'
wal_keep_size = '32GB'
effective_cache_size = '512GB'
log_destination = 'csvlog'
logging_collector = 'on'
log_filename = 'postgresql-%a_%H.log'
log_truncate_on_rotation = 'on'
log_lock_waits = 'on'
autovacuum_max_workers = '10'
autovacuum_naptime = '1min'
autovacuum_vacuum_scale_factor = '0.02'
autovacuum_vacuum_insert_scale_factor = '0.02'
autovacuum_analyze_scale_factor = '0.01'
checkpoint_timeout = '600'

test report

original value
scan_whole_pool_milliseconds = 60000.0
scan_whole_pool_milliseconds = 30000.0
scan_whole_pool_milliseconds = 10000.0
buffers_clean
2731311
7247872
13271584
17406338
num_requested
35
27
28
24
write_time
1110084
855959
1176642
870992
sync_time
1285
1110
746
419
buffers_written
49208974
42617458
35153293
31586820
tps
33153.438460
41242.271814
34969.554408
37177.879583

```

Thanks

Responses

Browse pgsql-hackers by date

  From Date Subject
Next Message Peter Smith 2024-10-02 08:54:32 Re: Conflict Detection and Resolution
Previous Message jian he 2024-10-02 08:41:35 Re: not null constraints, again