Re: Invalid field size

Tom Lane wrote:

> Moreno Andreo <moreno(dot)andreo(at)evolu-s(dot)it> writes:
> > So the hint is to abandon manual COPY and let pg_dump do the hard work?
>
> If it is a newline-conversion problem, compressed pg_dump archives would
> be just as subject to corruption as your binary COPY file is.

It's mentioned in [1] that the signature at the beginning of these files
embed a CRLF to detect this newline-conversion problem early on,
so I would expect COPY IN to stumble on a corrupted signature
and abort earlier in the process, if that conversion occurred.
Instead the report says it fails after a number of tuples:

> ERROR: invalid field size
> CONTEXT: COPY tab, line 619, column thumbnail

[1] https://www.postgresql.org/docs/current/static/sql-copy.htm

The file header consists of 15 bytes of fixed fields, followed by a
variable-length header extension area. The fixed fields are:

Signature
11-byte sequence PGCOPY\n\377\r\n\0 — note that the zero byte is a
required part of the signature. (The signature is designed to allow
easy identification of files that have been munged by a
non-8-bit-clean transfer. This signature will be changed by
end-of-line-translation filters, dropped zero bytes, dropped high
bits, or parity changes.)
...

Best regards,
--
Daniel Vérité
PostgreSQL-powered mailer: http://www.manitou-mail.org
Twitter: @DanielVerite