Re: Trim the heap free memory

Hi Rafia,

I have made the necessary adjustment by replacing the inclusion of malloc.h
with stdlib.h in the relevant codebase. This change should address the
previous concerns regarding memory allocation functions.

Could you please perform another round of testing to ensure that everything
is functioning as expected with this modification?

Thank you for your assistance.

Best regards, Shawn

Rafia Sabih <rafia(dot)pghackers(at)gmail(dot)com> 于2024年9月11日周三 18:25写道：

> Unfortunately, I still see a compiling issue with this patch,
>
> memtrim.c:15:10: fatal error: 'malloc.h' file not found
> #include <malloc.h>
> ^~~~~~~~~~
> 1 error generated.
>
> On Wed, 28 Aug 2024 at 12:54, shawn wang <shawn(dot)wang(dot)pg(at)gmail(dot)com> wrote:
>
>> Hi Ashutosh,
>>
>> Ashutosh Bapat <ashutosh(dot)bapat(dot)oss(at)gmail(dot)com> 于2024年8月26日周一 19:05写道：
>>
>>> Hi Shawn,
>>> It will be good to document usage of this function. Please add
>>> document changes in your patch. We need to document the impact of this
>>> function so that users can judiciously decide whether or not to use
>>> this function and under what conditions. Also they would know what to
>>> expect when they use this function.
>>
>>
>> I have already incorporated the usage of this function into the new patch.
>>
>>
>> Currently, there is no memory information that can be extremely accurate
>> to
>> reflect whether a trim operation should be performed. Here are two
>> conditions
>> that can be used as references:
>> 1. Check the difference between the process's memory usage (for example,
>> the top command, due to the relationship with shared memory, it is
>> necessary
>> to subtract SHR from RES) and the statistics of the memory context. If the
>> difference is very large, this function should be used to release memory;
>> 2. Execute malloc_stats(). If the system bytes are greater than the
>> in-use bytes, this indicates that this function can be used to release
>> memory.
>>
>>>
>>>
>> Running it after a query finishes is one thing but that can't be
>>> guaranteed because of the asynchronous nature of signal handlers.
>>> malloc_trim() may be called while a query is being executed. We need
>>> to assess that impact as well.
>>>
>>> Can you please share some numbers - TPS, latency etc. with and without
>>> this function invoked during a benchmark run?
>>>
>>
>> I have placed malloc_trim() at the end of the exec_simple_query function,
>> so that malloc_trim() is executed once for each SQL statement executed. I
>> used pgbench to reproduce the performance impact,
>> and the results are as follows.
>> *Database preparation：*
>>
>>> create database testc;
>>> create user t1;
>>> alter database testc owner to t1;
>>> ./pgbench testc -U t1 -i -s 100
>>> ./pgbench testc -U t1 -S -c 100 -j 100 -T 600
>>
>> *Without Trim*:
>>
>>> $./pgbench testc -U t1 -S -c 100 -j 100 -T 600
>>> pgbench (18devel)
>>> starting vacuum...end.
>>> transaction type: <builtin: select only>
>>> scaling factor: 100
>>> query mode: simple
>>> number of clients: 100
>>> number of threads: 100
>>> maximum number of tries: 1
>>> duration: 600 s
>>> number of transactions actually processed: 551984376
>>> number of failed transactions: 0 (0.000%)
>>> latency average = 0.109 ms
>>> initial connection time = 23.569 ms
>>> tps = 920001.842189 (without initial connection time)
>>
>> *With Trim :*
>>
>>> $./pgbench testc -U t1 -S -c 100 -j 100 -T 600
>>> pgbench (18devel)
>>> starting vacuum...end.
>>> transaction type: <builtin: select only>
>>> scaling factor: 100
>>> query mode: simple
>>> number of clients: 100
>>> number of threads: 100
>>> maximum number of tries: 1
>>> duration: 600 s
>>> number of transactions actually processed: 470690787
>>> number of failed transactions: 0 (0.000%)
>>> latency average = 0.127 ms
>>> initial connection time = 23.632 ms
>>> tps = 784511.901558 (without initial connection time)
>>
>>
>
> --
> Regards,
> Rafia Sabih
>