From: | Pavel Stehule <pavel(dot)stehule(at)gmail(dot)com> |
---|---|
To: | Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us> |
Cc: | pgsql-committers(at)lists(dot)postgresql(dot)org |
Subject: | Re: pgsql: Refactor pg_get_line() to expose an alternative StringInfo-based |
Date: | 2020-09-07 04:44:41 |
Message-ID: | CAFj8pRC2ixEeNK-a9XQwVrP6Vf3KhEG42ckq+gZZvmDTnnGQxA@mail.gmail.com |
Views: | Raw Message | Whole Thread | Download mbox | Resend email |
Thread: | |
Lists: | pgsql-committers |
Hi
ne 6. 9. 2020 v 20:13 odesílatel Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us> napsal:
> Refactor pg_get_line() to expose an alternative StringInfo-based API.
>
> Letting the caller provide a StringInfo to read into is helpful when
> the caller needs to merge lines or otherwise modify the data after
> it's been read. Notably, now the code added by commit 8f8154a50
> can use pg_get_line_append() instead of having its own copy of that
> logic. A follow-on commit will also make use of this.
>
> Also, since StringInfo buffers are a minimum of 1KB long, blindly
> using pg_get_line() in a loop can eat a lot more memory than one would
> expect. I discovered for instance that commit e0f05cd5b caused initdb
> to consume circa 10MB to read postgres.bki, even though that's under
> 1MB worth of data. A less memory-hungry alternative is to re-use the
> same StringInfo for all lines and pg_strdup the results.
>
>
I tried to reuse this new API in pg_dump.c, and I had a problem with
private struct StringInfo.
Inside initdb.c there is a access to this structure via #include
"access/xlog_internal.h" and it is little bit unreadable
maybe there should be included directly with #include "lib/stringinfo.h" ?
Regards
Pavel
> Discussion: https://postgr.es/m/1315832.1599345736@sss.pgh.pa.us
>
> Branch
> ------
> master
>
> Details
> -------
>
> https://git.postgresql.org/pg/commitdiff/8e3c58e6e459b285d37edb6129e412ed25cd90c1
>
> Modified Files
> --------------
> src/backend/libpq/hba.c | 40 +++++++++-----------------
> src/bin/initdb/initdb.c | 12 ++++++--
> src/common/pg_get_line.c | 70
> +++++++++++++++++++++++++++++++--------------
> src/include/common/string.h | 3 ++
> 4 files changed, 75 insertions(+), 50 deletions(-)
>
>
From | Date | Subject | |
---|---|---|---|
Next Message | Michael Paquier | 2020-09-07 05:36:16 | pgsql: doc: Tweak sentence for pg_checksums when enabling checksums |
Previous Message | Amit Kapila | 2020-09-07 02:48:50 | pgsql: Add additional tests to test streaming of in-progress transactio |