Re: New "raw" COPY format

From: "Joel Jacobson" <joel(at)compiler(dot)org>
To: "Tatsuo Ishii" <ishii(at)postgresql(dot)org>
Cc: pgsql-hackers(at)lists(dot)postgresql(dot)org
Subject: Re: New "raw" COPY format
Date: 2024-10-15 01:35:46
Message-ID: 79ccab75-cf78-4fcf-b9c0-493053dc75c0@app.fastmail.com
Views: Raw Message | Whole Thread | Download mbox | Resend email
Thread:
Lists: pgsql-hackers

On Mon, Oct 14, 2024, at 21:59, Joel Jacobson wrote:
> On Mon, Oct 14, 2024, at 10:51, Joel Jacobson wrote:
>> On Mon, Oct 14, 2024, at 10:07, Joel Jacobson wrote:
>>> Attached is a first draft implementation of the new proposed COPY "raw" format.
>>>
>>> The first two patches are just the bug fix in HEAD, reported separately:
>>> https://commitfest.postgresql.org/50/5297/

I noticed tests were failing in cfbot,
which surprised me since tests were passing locally,
but it was due to me not running the full test suite.

Sorry about the noise. I'm not running the full test suite,
with tap and `meson test --num-processes 32`,
so hopefully I won't cause cfbot failures as often any longer.

(The bug was due to an invalid assert; I wrongly assumed
Assert(opts_out->quote) would be sane inside the
if (opts_out->default_print) branch, but it is of course
wrong, since quote is NULL for the text format,
and the quote check was only performed if format was CSV,
so the assert was unnecessary and invalid).

I've also change the :filename I'm using in copy2.sql
for the raw format tests, to not be the same as in copy.sql,
since I guess this could cause problems with tests
running in parallell.

I've now split the reorganization of ProcessCopyOptions,
into separate easily small steps to make it easier to review.

Each step breaks out the validation of a COPY option,
into its own section, except for DELIMITER and NULL
that had to be changed together in a single commit.

/Joel

Attachment Content-Type Size
v7-0001-Fix-thinko-in-tests-for-COPY-options-force_not_null-.patch application/octet-stream 4.6 KB
v7-0002-Fix-validation-of-FORCE_NOT_NULL-FORCE_NULL-for-all-.patch application/octet-stream 5.0 KB
v7-0003-Replace-binary-flags-binary-and-csv_mode-with-format.patch application/octet-stream 18.6 KB
v7-0004-Set-default-format-if-not-specified.patch application/octet-stream 929 bytes
v7-0005-Separate-DELIMITER-and-NULL-option-validation-into-t.patch application/octet-stream 9.2 KB
v7-0006-Separate-QUOTE-option-validation-into-its-own-sectio.patch application/octet-stream 2.4 KB
v7-0007-Separate-ESCAPE-option-validation-into-its-own-secti.patch application/octet-stream 2.5 KB
v7-0008-Separate-DEFAULT-option-validation-into-its-own-sect.patch application/octet-stream 5.0 KB
v7-0009-Separate-HEADER-option-validation-into-its-own-secti.patch application/octet-stream 1.5 KB
v7-0010-Separate-FORCE_QUOTE-option-validation-into-its-own-.patch application/octet-stream 2.2 KB
v7-0011-Separate-FORCE_NOT_NULL-option-validation-into-its-o.patch application/octet-stream 2.2 KB
v7-0012-Separate-FORCE_NULL-option-validation-into-its-own-s.patch application/octet-stream 2.2 KB
v7-0013-Separate-FREEZE-option-validation-into-its-own-secti.patch application/octet-stream 1.6 KB
v7-0014-Separate-ON_ERROR-option-validation-into-its-own-sec.patch application/octet-stream 1.4 KB
v7-0015-Separate-REJECT_LIMIT-option-validation-into-its-own.patch application/octet-stream 2.0 KB
v7-0016-Add-raw-COPY-format-support-for-unstructured-text-da.patch application/octet-stream 40.1 KB

In response to

Responses

Browse pgsql-hackers by date

  From Date Subject
Next Message Corey Huinker 2024-10-15 01:46:58 Re: Statistics Import and Export
Previous Message Yugo NAGATA 2024-10-15 01:10:36 Re: Doc: typo in config.sgml