Re: BUG #7914: pg_dump aborts occasionally

Hi Tom,

> s-morita(at)beingcorp(dot)co(dot)jp writes:
>> I encountered a strange error during pg_dump as below:
>> ----
>> pg_dump: Dumping the contents of table "t_file_data" failed:
>> PQgetCopyData()
>> failed.
>> pg_dump: Error message from server: lost synchronization with server: got
>> message type "d", length 21861
>
> Hm. Can you create a reproducible test case for this?

This issue does not always happen.
It seems to depend on the environment and the load.
My test environment is actually a virual one as follows:
Hypervisor: VMware vSphere Hypervisor 5.1
Allocated vCPUs: 1
Allocated Memory: 4GB
OS: Windows Server 2012

1. Create test data using the script below.
2. Run pg_dump:
pg_dump --host localhost --port 5432 --username postgres
--format custom --verbose --file "..."
--table public.t_data test_db

----
CREATE TABLE t_data
(
c_id integer NOT NULL,
c_data bytea,
CONSTRAINT t_data_pkey PRIMARY KEY (c_id)
)
WITH (
OIDS=FALSE
);

CREATE LANGUAGE plpgsql;

CREATE FUNCTION create_data(n INT) RETURNS VOID AS
$$
BEGIN
FOR i IN 1..n LOOP
INSERT INTO t_data VALUES (i, rpad('', 20000, 'x')::bytea);
END LOOP;
END;
$$
LANGUAGE plpgsql;

SELECT create_data(300000);
----

>> I think that pqCheckInBufferSpace() should be called only when inBuffer
>> is
>> left-justified like this:
>> 1432: if (conn->inStart == 0 && pqCheckInBufferSpace(conn->inCursor
>> +
>> (size_t) msgLength - 4,
>
> I think changing that code that way would break the considerations
> mentioned in pqParseInput3; nor is it clear why this would help, anyway.
> It doesn't seem likely that the buffer is so large that it can't be
> enlarged by another 20K.

I think this issue happens when pg_dump is slower than the backend
for some reason.
Actually, pg_dump's memory consumption glows over 500MB
when this issue is happening.

If pg_dump is slower than the backend for some reason
and each message is not exactly 8K,
inBuffer will frequently overflow in getCopyDataMessage().

This will cause frequent calls to pqCheckInBufferSpace()
without left-justification.
Each call will double inBuffer and inBuffer will rapidly glow
even if each message is 20K.

> regards, tom lane

Regards,
Shin-ichi MORITA