Re: Enhancing Memory Context Statistics Reporting

On Wed, Nov 20, 2024 at 2:39 PM Rahila Syed <rahilasyed90(at)gmail(dot)com> wrote:
>
> Hi,
>
>> To achieve both completeness and avoid writing to a file, I can consider
>> displaying the numbers for the remaining contexts as a cumulative total
>> at the end of the output.
>>
>> Something like follows:
>> ```
>> postgres=# select * from pg_get_process_memory_contexts('237244', false);
>> name | ident | type | path | total_bytes | tot
>> al_nblocks | free_bytes | free_chunks | used_bytes | pid
>> ---------------------------------------+------------------------------------------------+----------+--------------+-------------+----
>> -----------+------------+-------------+------------+--------
>> TopMemoryContext | | AllocSet | {0} | 97696 |
>> 5 | 14288 | 11 | 83408 | 237244
>> search_path processing cache | | AllocSet | {0,1} | 8192 |
>> 1 | 5328 | 7 | 2864 | 237244
>> Remaining contexts total: 23456 bytes (total_bytes) , 12345(used_bytes), 11,111(free_bytes)
>>
>> ```
>
>
> Please find attached an updated patch with this change. The file previously used to
> store spilled statistics has been removed. Instead, a cumulative total of the
> remaining/spilled context statistics is now stored in the DSM segment, which is
> displayed as follows.
>
> postgres=# select * from pg_get_process_memory_contexts('352966', false);
> name | ident | type | path | total_bytes | total_nblocks | free_bytes | free_chunks | used_bytes | pi
> d
> ------------------------------+-------+----------+--------+-------------+---------------+------------+-------------+------------+----
> ----
> TopMemoryContext | | AllocSet | {0} | 97696 | 5 | 14288 | 11 | 83408 | 352
> 966
> .
> .
> .
> MdSmgr | | AllocSet | {0,18} | 8192 | 1 | 7424 | 0 | 768 | 352
> 966
> Remaining Totals | | | | 1756016 | 188 | 658584 | 132 | 1097432 | 352
> 966
> (7129 rows)
> -----
>
> I believe this serves as a good compromise between completeness
> and avoiding the overhead of file handling. However, I am open to
> reintroducing file handling if displaying the complete statistics of the
> remaining contexts prove to be more important.
>
> All the known bugs in the patch have been fixed.
>
> In summary, one DSA per PostgreSQL process is used to share
> the statistics of that process. A DSA is created by the first client
> backend that requests memory context statistics, and it is pinned
> for all future requests to that process.
> A handle to this DSA is shared between the client and the publishing
> process using fixed shared memory. The fixed shared memory consists
> of an array of size MaxBackends + auxiliary processes, indexed
> by procno. Each element in this array is less than 100 bytes in size.
>
> A PostgreSQL process uses a condition variable to signal a waiting client
> backend once it has finished publishing the statistics. If, for some reason,
> the signal is not sent, the waiting client backend will time out.

How does the process know that the client backend has finished reading
stats and it can be refreshed? What happens, if the next request for
memory context stats comes before first requester has consumed the
statistics it requested?

Does the shared memory get deallocated when the backend which
allocated it exits?

>
> When statistics of a local backend is requested, this function returns the following
> WARNING and exits, since this can be handled by an existing function which
> doesn't require a DSA.
>
> WARNING: cannot return statistics for local backend
> HINT: Use pg_get_backend_memory_contexts instead

How about using pg_get_backend_memory_contexts() for both - local as
well as other backend? Let PID argument default to NULL which would
indicate local backend, otherwise some other backend?

--
Best Wishes,
Ashutosh Bapat