Fix receiving large legal tsvector from binary format

From: Ерохин Денис Владимирович <erohin-d(at)datagile(dot)ru>
To: <pgsql-hackers(at)postgresql(dot)org>
Subject: Fix receiving large legal tsvector from binary format
Date: 2023-09-29 13:36:27
Message-ID: 009801d9f2d9$f29730c0$d7c59240$@datagile.ru
Views: Raw Message | Whole Thread | Download mbox | Resend email
Thread:
Lists: pgsql-hackers

Hello,

There is a problem on receiving large tsvector from binary format with
getting error "invalid tsvector: maximum total lexeme length exceeded".
Required simple steps to reproduce problem:

- Make table with one column of type 'tsvector'

- Add row with large tsvector (900 Kb will be enougth)

- Save table to file with "copy to with binary"

- Restore table from file with "copy from with binary"

At last step we can't restore a legal tsvector because there is error of
required memory size calculation for tsvector during reading of its
binary.

It seems function "tsvectorrecv" can be improved:

v1-0001-Fix-receiving-of-big-tsvector-from-binary - patch fixes problem of
required size calculation for tsvector where wrong type is used (WordEntry
instead of WordEntryPos). Size of wrong type is bigger than type of needed
type therefore total size turns out larger than needed. So next test of
that size fails during maximum size check.

v1-0002-Replace-magic-one-in-tsvector-binary-receiving - patch removes
magic ones from code. Those ones are used to add "sizeof(uint16)" to
required size where number of WordEntryPos'es will be stored. Now it works
only because size of WordEntryPos is equal to size of uint16. But it is
not obviously during code reading and causes question "what for this magic
one is?"

v1-0003-Optimize-memory-allocation-during-tsvector-binary - patch makes
memory allocation more accuracy and rarely. It seems memory to store
tsvector's data is allocated redundant and reallocations happen more often
than necessary.

Best regards,

Denis Erokhin

Jatoba

Attachment Content-Type Size
v1-0001-Fix-receiving-of-big-tsvector-from-binary.patch application/octet-stream 753 bytes
v1-0002-Replace-magic-one-in-tsvector-binary-receiving.patch application/octet-stream 1.1 KB
v1-0003-Optimize-memory-allocation-during-tsvector-binary.patch application/octet-stream 1.5 KB

Responses

Browse pgsql-hackers by date

  From Date Subject
Next Message Melanie Plageman 2023-09-29 13:38:56 Re: how to manage Cirrus on personal repository
Previous Message Amit Kapila 2023-09-29 12:02:52 Re: pg_upgrade and logical replication