Re: Emitting JSON to file using COPY TO

From: "Daniel Verite" <daniel(at)manitou-mail(dot)org>
To: "Andrew Dunstan" <andrew(at)dunslane(dot)net>
Cc: Joe Conway <mail(at)joeconway(dot)com>, Davin Shearer <davin(at)apache(dot)org>, PostgreSQL-development <pgsql-hackers(at)postgresql(dot)org>
Subject: Re: Emitting JSON to file using COPY TO
Date: 2023-12-06 18:59:11
Message-ID: b3a73eca-ec3b-4b64-b6c2-82418bb16701@manitou-mail.org
Views: Raw Message | Whole Thread | Download mbox | Resend email
Thread:
Lists: pgsql-general pgsql-hackers

Andrew Dunstan wrote:

> IMNSHO, we should produce either a single JSON
> document (the ARRAY case) or a series of JSON documents, one per row
> (the LINES case).

"COPY Operations" in the doc says:

" The backend sends a CopyOutResponse message to the frontend, followed
by zero or more CopyData messages (always one per row), followed by
CopyDone".

In the ARRAY case, the first messages with the copyjsontest
regression test look like this (tshark output):

PostgreSQL
Type: CopyOut response
Length: 13
Format: Text (0)
Columns: 3
Format: Text (0)
PostgreSQL
Type: Copy data
Length: 6
Copy data: 5b0a
PostgreSQL
Type: Copy data
Length: 76
Copy data:
207b226964223a312c226631223a226c696e652077697468205c2220696e2069743a2031…

The first Copy data message with contents "5b0a" does not qualify
as a row of data with 3 columns as advertised in the CopyOut
message. Isn't that a problem?

At least the json non-ARRAY case ("json lines") doesn't have
this issue, since every CopyData message corresponds effectively
to a row in the table.

[1] https://www.postgresql.org/docs/current/protocol-flow.html#PROTOCOL-COPY

Best regards,
--
Daniel Vérité
https://postgresql.verite.pro/
Twitter: @DanielVerite

In response to

Responses

Browse pgsql-general by date

  From Date Subject
Next Message Joe Conway 2023-12-06 19:47:25 Re: Emitting JSON to file using COPY TO
Previous Message Holger Jakobs 2023-12-06 18:22:26 Re: Disable autocommit inside dbeaver

Browse pgsql-hackers by date

  From Date Subject
Next Message Matthias van de Meent 2023-12-06 19:14:24 Re: Bug in nbtree optimization to skip > operator comparisons (or < comparisons in backwards scans)
Previous Message Peter Geoghegan 2023-12-06 18:54:45 Re: Bug in nbtree optimization to skip > operator comparisons (or < comparisons in backwards scans)