Re: allocation limit for encoding conversion

Hi,

On September 24, 2019 3:09:28 PM PDT, Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us> wrote:
>Andres Freund <andres(at)anarazel(dot)de> writes:
>> On 2019-09-24 16:19:41 -0400, Tom Lane wrote:
>>> Andres Freund <andres(at)anarazel(dot)de> writes:
>>>> It's probably too big a hammer for this specific case, but I think
>at
>>>> some point we ought to stop using fixed size allocations for this
>kind
>>>> of work. Instead we should use something roughly like our
>StringInfo,
>>>> except that when exceeding the current size limit, the overflowing
>data
>>>> is stored in a separate allocation. And only once we actually need
>the
>>>> data in a consecutive form, we allocate memory that's large enough
>to
>>>> store the all the separate allocations in their entirety.
>
>>> That sounds pretty messy :-(.
>
>> I don't think it's too bad - except for now allowing the .data member
>of
>> such a 'chunked' StringInfo to be directly accessible, it'd be just
>> about the same interface as the current stringinfo.
>
>I dunno. What you're describing would be a whole lotta work, and it'd
>break a user-visible API, and no amount of finagling is going to
>prevent
>it from making conversions somewhat slower, and the cases where it
>matters
>to not preallocate a surely-large-enough buffer are really few and far
>between. I have to think that we have better ways to spend our time.

It'd not just avoid the overallocation, but also avoid the strlen and memcpy afterwards at the callsites, as well as the separate allocation. So I'd bet it'd be almost always a win.

Andres

--
Sent from my Android device with K-9 Mail. Please excuse my brevity.