pgsql: Compress TID lists when writing GIN tuples to disk

From: Tomas Vondra <tomas(dot)vondra(at)postgresql(dot)org>
To: pgsql-committers(at)lists(dot)postgresql(dot)org
Subject: pgsql: Compress TID lists when writing GIN tuples to disk
Date: 2025-03-04 18:02:15
Message-ID: E1tpWat-000mNv-2Q@gemulon.postgresql.org
Views: Raw Message | Whole Thread | Download mbox | Resend email
Thread:
Lists: pgsql-committers

Compress TID lists when writing GIN tuples to disk

When serializing GIN tuples to tuplesorts during parallel index builds,
we can significantly reduce the amount of data by compressing the TID
lists. The GIN opclasses may produce a lot of data (depending on how
many keys are extracted from each row), and the TID compression is very
efficient and effective.

If the number of distinct keys is high, the first worker pass (reading
data from the table and writing them into a private tuplesort) may not
benefit from the compression very much. It is likely to spill data to
disk before the TID lists get long enough for the compression to help.
The second pass (writing the merged data into the shared tuplesort) is
more likely to benefit from compression.

The compression can be seen as a way to reduce the amount of disk space
needed by the parallel builds, because the data is written twice. First
into the per-worker tuplesorts, then into the shared tuplesort.

Author: Tomas Vondra
Reviewed-by: Matthias van de Meent, Andy Fan, Kirill Reshke
Discussion: https://postgr.es/m/6ab4003f-a8b8-4d75-a67f-f25ad98582dc%40enterprisedb.com

Branch
------
master

Details
-------
https://git.postgresql.org/pg/commitdiff/0b2a45a5d1f2b088640a7eb7817ca6a0513a2717

Modified Files
--------------
src/backend/access/gin/gininsert.c | 116 ++++++++++++++++++++++++++++++-------
src/tools/pgindent/typedefs.list | 1 +
2 files changed, 95 insertions(+), 22 deletions(-)

Browse pgsql-committers by date

  From Date Subject
Next Message Masahiko Sawada 2025-03-04 18:41:04 pgsql: Refactor Copy{From|To}GetRoutine() to use pass-by-reference argu
Previous Message Tom Lane 2025-03-04 17:58:20 pgsql: Add .gitignore entry for ecpg test detritus.