Re: BUG #17791: Assert on procarray.c

Hi Andreas,

> Are there any log messages around the crashes? I'm wondering whether we're
> erroring out while aborting, to then abort again, or such. Which will often
> elicit warning/error messages.

Thanks for taking a look and possibly you're correct with your
assumption. I mean I see a ton of FATALs but let me know if I am
mistaken in assuming them to be harmless (since they just convey that
the client's gone away)?

Nonetheless, I have provided error logs going back till Oct 22 just in
case the engine can recover from some of those scenarios. Two things
about the test scenario that may be relevant:

1. Since performance was the least of my worries, the postgres server
and the client workload are on the same box. Add dblink / FDW to this
mix, and it is easy to end up with a ton of loopback connections
(think SELECT dblink_conect() FROM pg_catalog.pg_class) - IMO
noteworthy, since there are a ton of "Broken pipe"s and one instance
of 'too many file descriptors'.

2. All versions are subjected to similar workload and it is possible
that v13+ has generally improved in this area, and thus this possibly
crashes less? Unsure.

Hope it helps.

################################################

2023-02-13 10:10:08.474 UTC [3918683] LOG: could not send data to
client: Broken pipe
2023-02-13 10:10:08.474 UTC [3918683] FATAL: connection to client lost
2023-02-13 10:10:15.001 UTC [3918764] LOG: could not send data to
client: Broken pipe
2023-02-13 10:10:15.001 UTC [3918764] FATAL: connection to client lost
2023-02-13 10:10:25.268 UTC [3918886] LOG: could not send data to
client: Broken pipe
2023-02-13 10:10:25.269 UTC [3918886] FATAL: connection to client lost
2023-02-13 10:10:26.261 UTC [3918897] LOG: could not send data to
client: Broken pipe
2023-02-13 10:10:26.261 UTC [3918897] FATAL: connection to client lost
2023-02-13 10:10:29.503 UTC [3918930] LOG: could not send data to
client: Broken pipe
2023-02-13 10:10:29.503 UTC [3918930] FATAL: connection to client lost
TRAP: FailedAssertion("!(((allPgXact[proc->pgprocno].xid) !=
((TransactionId) 0)))", File: "procarray.c", Line: 430)
2023-02-13 10:10:41.629 UTC [3918961] LOG: could not receive data
from client: Connection reset by peer
2023-02-13 10:10:43.008 UTC [3777407] LOG: server process (PID
3911829) was terminated by signal 6: Aborted
2023-02-13 10:10:43.008 UTC [3777407] DETAIL: Failed process was
running: select
cast(nullif(sample_0.a,
(select line_number from pg_catalog.pg_hba_file_rules
limit 1 offset 3)
) as int4) as c0,
pg_catalog.pg_is_in_recovery() as c1
from
public.fprt2 as sample_0 tablesample bernoulli (1.2)
where public.test_found() > pg_catalog.pg_is_wal_replay_paused()
limit 111
2023-02-13 10:10:43.008 UTC [3777407] LOG: terminating any other
active server processes
2023-02-13 10:10:43.340 UTC [3777407] LOG: all server processes
terminated; reinitializing
2023-02-13 10:10:43.472 UTC [3919088] LOG: database system was
interrupted; last known up at 2023-02-13 10:09:22 UTC
2023-02-13 10:10:43.534 UTC [3919088] LOG: database system was not
properly shut down; automatic recovery in progress
2023-02-13 10:10:43.540 UTC [3919088] LOG: redo starts at 2C/473107C8
2023-02-13 10:10:44.043 UTC [3919094] FATAL: the database system is
in recovery mode

################################################

2023-02-13 13:48:00.083 UTC [4073] LOG: could not send data to
client: Broken pipe
2023-02-13 13:48:00.083 UTC [4073] FATAL: connection to client lost
2023-02-13 13:48:07.916 UTC [4174] LOG: could not send data to
client: Broken pipe
2023-02-13 13:48:07.916 UTC [4174] FATAL: connection to client lost
2023-02-13 13:48:16.689 UTC [4193172] LOG: duration: 247511.694 ms
statement: REINDEX INDEX sqithsch4.chr_c_idx4;
2023-02-13 13:48:17.247 UTC [4192307] LOG: duration: 250514.066 ms
statement: REINDEX INDEX sqithsch7.chr_c_idx4;
2023-02-13 13:48:19.108 UTC [4193744] LOG: duration: 248244.637 ms
statement: REINDEX INDEX sqithsch8.chr_c_idx4;
2023-02-13 13:48:19.660 UTC [4191898] LOG: duration: 253850.202 ms
statement: REINDEX INDEX sqithsch25.chr_c_idx4;
2023-02-13 13:48:20.607 UTC [4193307] LOG: duration: 250962.064 ms
statement: REINDEX INDEX sqithsch1.chr_c_idx4;
2023-02-13 13:48:20.857 UTC [4321] LOG: could not send data to
client: Broken pipe
2023-02-13 13:48:20.857 UTC [4321] FATAL: connection to client lost
2023-02-13 13:48:24.460 UTC [4377] LOG: could not send data to
client: Broken pipe
2023-02-13 13:48:24.460 UTC [4377] FATAL: connection to client lost
2023-02-13 13:48:25.059 UTC [4194142] LOG: duration: 251910.592 ms
statement: REINDEX INDEX sqithsch17.chr_c_idx4_ccnew;
2023-02-13 13:48:33.644 UTC [460] LOG: duration: 248070.798 ms
statement: REINDEX INDEX sqithsch29.chr_c_idx4;
2023-02-13 13:48:33.739 UTC [632] LOG: duration: 240490.057 ms
statement: REINDEX INDEX sqithsch14.chr_c_idx4;
2023-02-13 13:48:50.101 UTC [4664] LOG: could not send data to
client: Broken pipe
2023-02-13 13:48:50.101 UTC [4664] FATAL: connection to client lost
2023-02-13 13:48:55.741 UTC [4725] LOG: could not send data to
client: Broken pipe
2023-02-13 13:48:55.741 UTC [4725] FATAL: connection to client lost
2023-02-13 13:48:55.991 UTC [2434] LOG: duration: 192161.947 ms
statement: REINDEX INDEX sqithsch12.chr_c_idx4;
2023-02-13 13:48:56.021 UTC [4737] LOG: could not send data to
client: Broken pipe
2023-02-13 13:48:56.021 UTC [4737] FATAL: connection to client lost
TRAP: FailedAssertion("!(((allPgXact[proc->pgprocno].xid) !=
((TransactionId) 0)))", File: "procarray.c", Line: 430)
2023-02-13 13:48:56.479 UTC [2349] LOG: duration: 193350.243 ms
statement: REINDEX INDEX sqithsch27.chr_c_idx4;
2023-02-13 13:48:56.679 UTC [2438] LOG: duration: 192801.918 ms
statement: REINDEX INDEX sqithsch13.chr_c_idx4_ccnew_ccnew;
2023-02-13 13:48:58.428 UTC [4748] LOG: could not receive data from
client: Connection reset by peer
2023-02-13 13:48:58.677 UTC [4773] LOG: could not send data to
client: Broken pipe
2023-02-13 13:48:58.677 UTC [4773] FATAL: connection to client lost
2023-02-13 13:49:01.002 UTC [1487] LOG: duration: 232718.744 ms
statement: REINDEX INDEX sqithsch28.chr_c_idx4;
2023-02-13 13:49:02.393 UTC [4806] FATAL: terminating connection due
to administrator command
2023-02-13 13:49:02.393 UTC [4806] STATEMENT: ALTER TABLE ONLY sqithsch18.chr
ALTER COLUMN c SET DATA TYPE TEXT USING sqithsch18.chr.c
2023-02-13 13:49:02.470 UTC [3777407] LOG: background worker
"parallel worker" (PID 4806) exited with exit code 1
2023-02-13 13:49:02.787 UTC [832] LOG: duration: 255613.229 ms
statement: REINDEX INDEX sqithsch13.chr_c_idx4;
2023-02-13 13:49:02.961 UTC [3777407] LOG: server process (PID
4149811) was terminated by signal 6: Aborted
2023-02-13 13:49:02.961 UTC [3777407] DETAIL: Failed process was
running: delete from sqithsch24.dt
where
(pg_catalog.getdatabaseencoding() is not NULL)
or (cast(coalesce((select c2 from import_dest2.t3 limit 1 offset 4)
,
cast(coalesce((select c2 from import_dest1.t3 limit 1 offset 4)
,
cast(nullif((select c2 from import_dest2.t3 limit 1 offset 37)
,
cast(null as public.typ1)) as public.typ1)) as
public.typ1)) as public.typ1) *>= cast(nullif(cast(null as
public.typ1),
cast(null as public.typ1)) as public.typ1))
returning
sqithsch24.dt.d as c0,
public.namedparmcursor_test8() as c1,
pg_catalog.pg_current_wal_flush_lsn() as c2,
pg_catalog.now() as c3,
pg_catalog.session_user() as c4
2023-02-13 13:49:02.961 UTC [3777407] LOG: terminating any other
active server processes
2023-02-13 13:49:03.264 UTC [3777407] LOG: all server processes
terminated; reinitializing
2023-02-13 13:49:03.399 UTC [4848] LOG: database system was
interrupted; last known up at 2023-02-13 13:47:57 UTC
2023-02-13 13:49:04.043 UTC [4849] FATAL: the database system is in
recovery mode
2023-02-13 13:49:04.046 UTC [4850] FATAL: the database system is in
recovery mode

################################################

2022-12-09 15:41:07.667 UTC [515871] LOG: could not send data to
client: Broken pipe
2022-12-09 15:41:07.686 UTC [515871] FATAL: connection to client lost
2022-12-09 15:41:14.794 UTC [516872] LOG: could not send data to
client: Broken pipe
2022-12-09 15:41:14.794 UTC [516872] FATAL: connection to client lost
2022-12-09 15:41:54.815 UTC [517815] LOG: could not send data to
client: Broken pipe
2022-12-09 15:41:54.815 UTC [517815] FATAL: connection to client lost
2022-12-09 15:42:09.856 UTC [519130] LOG: could not send data to
client: Broken pipe
2022-12-09 15:42:09.856 UTC [519130] FATAL: connection to client lost
2022-12-09 15:42:11.424 UTC [519753] LOG: could not send data to
client: Broken pipe
2022-12-09 15:42:11.424 UTC [519753] FATAL: connection to client lost
2022-12-09 15:42:37.653 UTC [521507] LOG: could not send data to
client: Broken pipe
2022-12-09 15:42:37.653 UTC [521507] FATAL: connection to client lost
2022-12-09 15:43:34.469 UTC [522743] LOG: could not send data to
client: Broken pipe
2022-12-09 15:43:34.469 UTC [522743] FATAL: connection to client lost
TRAP: FailedAssertion("!(((allPgXact[proc->pgprocno].xid) !=
((TransactionId) 0)))", File: "procarray.c", Line: 430)
2022-12-09 15:43:44.272 UTC [522120] LOG: could not receive data from
client: Connection reset by peer
2022-12-09 15:43:47.381 UTC [336681] LOG: server process (PID 522114)
was terminated by signal 6: Aborted
2022-12-09 15:43:47.381 UTC [336681] DETAIL: Failed process was running: select
public.test_found() as c0,
public.dblink_current_query() as c1,
case when pg_catalog.pg_last_wal_receive_lsn() <
pg_catalog.pg_current_wal_lsn() then ref_0.random else ref_0.random
end
as c2,
pg_catalog.phraseto_tsquery(
cast(pg_catalog.get_current_ts_config() as regconfig),
cast(cast(coalesce(public.dblink_connect_u(
cast(pg_catalog.timeofday() as text),
cast(public.connection_parameters() as text)),
pg_catalog.to_char(
cast(cast(null as "interval") as "interval"),
cast(pg_catalog.current_query() as text))) as text) as
text)) as c3
from
public.random_tbl as ref_0
right join sch3.chr_fdw as ref_1
on (cast(null as tsquery) <@ (select tsquery from
public.tab_core_types limit 1 offset 5)
)
where (pg_catalog.int8send(
cast(cast(null as int8) as int8)) ~~ cast(coalesce(public.hmac(
cast(pg_catalog.cstring_send(
cast(pg_catalog.regroleout(
cast(cast(null as regrole) as regrole)) as cstrin
2022-12-09 15:43:47.381 UTC [336681] LOG: terminating any other
active server processes
2022-12-09 15:43:47.387 UTC [522266] LOG: could not send data to
client: Broken pipe

################################################

2022-12-11 12:35:53.840 UTC [1739971] LOG: out of file descriptors:
Too many open files; release and retry
2022-12-11 12:35:53.840 UTC [1739971] STATEMENT: select
public.lquery_in(
cast(public.citextout(
cast(public.citext(
cast(pg_catalog.bpchar(
cast(cast(null as bpchar) as bpchar),
cast(subq_0.c1 as int4),
cast(case when (62 is NULL)
and ((true)
or (true)) then (select pg_catalog.bool_or(b)
from sch23.bool)
else (select pg_catalog.bool_or(b) from sch23.bool)
end
as bool)) as bpchar)) as public.citext)) as
cstring)) as c0,
public.reraise_test() as c1,
sample_4.c1 as c2,
pg_catalog.tstzrange(
cast(pg_catalog.clock_timestamp() as timestamptz),
cast(pg_catalog.pg_stat_get_bgwriter_stat_reset_time() as
timestamptz),
cast((select t from sch32.txt limit 1 offset 3)
as text)) as c3
from
sch8.txt as ref_0
inner join (select
sample_0.a as c0,
sample_0.a as c1,
sample_0.a as c2,
sample_0.a as c3,
sample_0.a as c4,
sample_0.a as c5,
sample_0.a as c6
from
fkpart5.fk4 as sample_0 tablesample system (3.8)
where sample_0.a is NULL
limit 177) as subq_0
on (cast(null as public.ltree) = (select t from
public.ltreetest limit 1 offset 4)
)
left join (select
ref_1.t as c0,
subq_1.c5 as c1,
66 as c2
from
sch8.vec_fdw as ref_1,
lateral (select
sample_1.even as c0,
sample_1.two as c1,
sample_1.fivethous as c2,
sample_1.fivethous as c3,
sample_1.two as c4,
ref_1.o as c5,
ref_1.i as c6
from
public.tenk2 as sample_1 tablesample system (5.7)
where (select public.logging_agg_strict(t)
from sch23.txt)
~* cast(null as text)
limit 139) as subq_1
where cast(null as uuid) < cast(null as uuid)
limit 139) as subq_2
on (subq_0.c5 is NULL)
inner join public.plt1 as sample_2 tablesample system (0.6)
right join public.fprt1 as sample_3 tablesample system (6.5)
right join (select
ref_2.f as c0,
ref_2.f as c1,
ref_2.f as c2
from
sch29.flot as ref_2
where cast(null as jsonb) <= cast(null as jsonb)
limit 62) as subq_3
right join import_source.t5 as sample_4 tablesample
system (3.3)
inner join import_dest2.t1 as ref_3
right join public.table_with_pkey as sample_5
tablesample system (1.3)
on (cast(null as bpchar) ~~* (select t from
sch27.txt limit 1 offset 4)
)
on (true)
on ((select timestamptz from public.tab_core_types
limit 1 offset 1)
<> (select timestamptzcol from
public.brintest limit 1 offset 3)
)
on (sample_4.c2 > sample_4.c2)
on (sample_2.b = ref_3.c1 )
on (subq_2.c2 = sample_4.c1 )
where false
limit 84
2022-12-11 12:35:53.845 UTC [1806036] LOG: could not send data to
client: Broken pipe
2022-12-11 12:35:53.845 UTC [1806036] FATAL: connection to client lost
2022-12-11 12:35:54.357 UTC [1806042] LOG: could not send data to
client: Broken pipe
2022-12-11 12:35:54.357 UTC [1806042] FATAL: connection to client lost
2022-12-11 12:35:54.969 UTC [1806049] LOG: could not send data to
client: Broken pipe
2022-12-11 12:35:54.969 UTC [1806049] FATAL: connection to client lost
2022-12-11 12:35:55.018 UTC [1806050] LOG: could not send data to
client: Broken pipe
2022-12-11 12:35:55.018 UTC [1806050] FATAL: connection to client lost
2022-12-11 12:35:55.123 UTC [1806051] LOG: could not send data to
client: Broken pipe
2022-12-11 12:35:55.123 UTC [1806051] FATAL: connection to client lost
2022-12-11 12:35:55.610 UTC [1806052] LOG: could not send data to
client: Broken pipe
2022-12-11 12:35:55.610 UTC [1806052] FATAL: connection to client lost
2022-12-11 12:35:55.730 UTC [1806053] LOG: could not send data to
client: Broken pipe
2022-12-11 12:35:55.730 UTC [1806053] FATAL: connection to client lost
2022-12-11 12:35:55.817 UTC [1806058] LOG: could not send data to
client: Broken pipe
2022-12-11 12:35:55.817 UTC [1806058] FATAL: connection to client lost
2022-12-11 12:35:57.141 UTC [1806060] LOG: could not send data to
client: Broken pipe
2022-12-11 12:35:57.141 UTC [1806060] FATAL: connection to client lost
2022-12-11 12:35:57.753 UTC [1806066] LOG: could not send data to
client: Broken pipe
2022-12-11 12:35:57.753 UTC [1806066] FATAL: connection to client lost
2022-12-11 12:35:58.164 UTC [1806072] LOG: could not send data to
client: Broken pipe
2022-12-11 12:35:58.164 UTC [1806072] FATAL: connection to client lost
2022-12-11 12:35:58.824 UTC [1806080] LOG: could not send data to
client: Broken pipe
2022-12-11 12:35:58.824 UTC [1806080] FATAL: connection to client lost
2022-12-11 12:35:58.965 UTC [1806081] LOG: could not send data to
client: Broken pipe
2022-12-11 12:35:58.965 UTC [1806081] FATAL: connection to client lost
2022-12-11 12:35:59.260 UTC [1806082] LOG: could not send data to
client: Broken pipe
2022-12-11 12:35:59.260 UTC [1806082] FATAL: connection to client lost
2022-12-11 12:36:00.604 UTC [1806099] LOG: could not send data to
client: Broken pipe
2022-12-11 12:36:00.604 UTC [1806099] FATAL: connection to client lost
2022-12-11 12:36:01.960 UTC [1806118] LOG: could not send data to
client: Broken pipe
2022-12-11 12:36:01.960 UTC [1806118] FATAL: connection to client lost
2022-12-11 12:36:03.109 UTC [1806125] LOG: could not send data to
client: Broken pipe
2022-12-11 12:36:03.109 UTC [1806125] FATAL: connection to client lost
2022-12-11 12:36:03.194 UTC [1806126] LOG: could not send data to
client: Broken pipe
2022-12-11 12:36:03.194 UTC [1806126] FATAL: connection to client lost
2022-12-11 12:36:04.707 UTC [1806235] LOG: could not send data to
client: Broken pipe
2022-12-11 12:36:04.707 UTC [1806235] FATAL: connection to client lost
2022-12-11 12:36:05.296 UTC [1806241] LOG: could not send data to
client: Broken pipe
2022-12-11 12:36:05.296 UTC [1806241] FATAL: connection to client lost
2022-12-11 12:36:05.484 UTC [1806242] LOG: could not send data to
client: Broken pipe
2022-12-11 12:36:05.484 UTC [1806242] FATAL: connection to client lost
2022-12-11 12:36:05.620 UTC [1806243] LOG: could not send data to
client: Broken pipe
2022-12-11 12:36:05.620 UTC [1806243] FATAL: connection to client lost
2022-12-11 12:36:05.663 UTC [1806244] LOG: could not send data to
client: Broken pipe
2022-12-11 12:36:05.663 UTC [1806244] FATAL: connection to client lost
2022-12-11 12:36:05.784 UTC [1806245] LOG: could not send data to
client: Broken pipe
2022-12-11 12:36:05.784 UTC [1806245] FATAL: connection to client lost
2022-12-11 12:36:05.863 UTC [1806246] LOG: could not send data to
client: Broken pipe
2022-12-11 12:36:05.863 UTC [1806246] FATAL: connection to client lost
2022-12-11 12:36:05.981 UTC [1806252] LOG: could not send data to
client: Broken pipe
2022-12-11 12:36:05.981 UTC [1806252] FATAL: connection to client lost
2022-12-11 12:36:06.020 UTC [1806253] LOG: could not send data to
client: Broken pipe
2022-12-11 12:36:06.020 UTC [1806253] FATAL: connection to client lost
2022-12-11 12:36:06.161 UTC [1806254] LOG: could not send data to
client: Broken pipe
2022-12-11 12:36:06.161 UTC [1806254] FATAL: connection to client lost
2022-12-11 12:36:06.313 UTC [1806255] LOG: could not send data to
client: Broken pipe
2022-12-11 12:36:06.313 UTC [1806255] FATAL: connection to client lost
2022-12-11 12:36:07.639 UTC [1806262] LOG: could not send data to
client: Broken pipe
2022-12-11 12:36:07.639 UTC [1806262] FATAL: connection to client lost
2022-12-11 12:36:07.880 UTC [1806263] LOG: could not send data to
client: Broken pipe
2022-12-11 12:36:07.880 UTC [1806263] FATAL: connection to client lost
2022-12-11 12:36:08.529 UTC [1806269] LOG: could not send data to
client: Broken pipe
2022-12-11 12:36:08.529 UTC [1806269] FATAL: connection to client lost
2022-12-11 12:36:10.989 UTC [1806366] LOG: could not send data to
client: Broken pipe
2022-12-11 12:36:10.989 UTC [1806366] FATAL: connection to client lost
2022-12-11 12:36:11.936 UTC [1806457] LOG: could not send data to
client: Broken pipe
2022-12-11 12:36:11.936 UTC [1806457] FATAL: connection to client lost
2022-12-11 12:36:12.298 UTC [1806458] LOG: could not send data to
client: Broken pipe
2022-12-11 12:36:12.298 UTC [1806458] FATAL: connection to client lost
2022-12-11 12:36:12.688 UTC [1806467] LOG: could not send data to
client: Broken pipe
2022-12-11 12:36:12.688 UTC [1806467] FATAL: connection to client lost
2022-12-11 12:36:13.005 UTC [1806473] LOG: could not send data to
client: Broken pipe
2022-12-11 12:36:13.005 UTC [1806473] FATAL: connection to client lost
2022-12-11 12:36:14.021 UTC [1806480] LOG: could not send data to
client: Broken pipe
2022-12-11 12:36:14.021 UTC [1806480] FATAL: connection to client lost
2022-12-11 12:36:14.357 UTC [1806481] LOG: could not send data to
client: Broken pipe
2022-12-11 12:36:14.357 UTC [1806481] FATAL: connection to client lost
2022-12-11 12:36:14.602 UTC [1806482] LOG: could not send data to
client: Broken pipe
2022-12-11 12:36:14.602 UTC [1806482] FATAL: connection to client lost
2022-12-11 12:36:15.333 UTC [1806489] LOG: could not send data to
client: Broken pipe
2022-12-11 12:36:15.333 UTC [1806489] FATAL: connection to client lost
2022-12-11 12:36:15.989 UTC [1806493] LOG: could not send data to
client: Broken pipe
2022-12-11 12:36:15.989 UTC [1806493] FATAL: connection to client lost
2022-12-11 12:36:16.245 UTC [1806515] LOG: could not send data to
client: Broken pipe
2022-12-11 12:36:16.245 UTC [1806515] FATAL: connection to client lost
2022-12-11 12:36:16.317 UTC [1806532] LOG: could not send data to
client: Broken pipe
2022-12-11 12:36:16.317 UTC [1806532] FATAL: connection to client lost
2022-12-11 12:36:16.862 UTC [1806609] LOG: could not send data to
client: Broken pipe
2022-12-11 12:36:16.873 UTC [1806609] FATAL: connection to client lost
2022-12-11 12:36:17.170 UTC [1806680] LOG: could not send data to
client: Broken pipe
2022-12-11 12:36:17.170 UTC [1806680] FATAL: connection to client lost
2022-12-11 12:36:17.405 UTC [1806738] LOG: could not send data to
client: Broken pipe
2022-12-11 12:36:17.405 UTC [1806738] FATAL: connection to client lost
2022-12-11 12:36:18.061 UTC [1806886] LOG: could not send data to
client: Broken pipe
2022-12-11 12:36:18.061 UTC [1806886] FATAL: connection to client lost
2022-12-11 12:36:18.320 UTC [1806938] LOG: could not send data to
client: Broken pipe
2022-12-11 12:36:18.320 UTC [1806938] FATAL: connection to client lost
2022-12-11 12:36:18.843 UTC [1807035] LOG: could not send data to
client: Broken pipe
2022-12-11 12:36:18.843 UTC [1807035] FATAL: connection to client lost
2022-12-11 12:36:19.209 UTC [1807113] LOG: could not send data to
client: Broken pipe
2022-12-11 12:36:19.209 UTC [1807113] FATAL: connection to client lost
2022-12-11 12:36:20.263 UTC [1807296] LOG: could not send data to
client: Broken pipe
2022-12-11 12:36:20.263 UTC [1807296] FATAL: connection to client lost
2022-12-11 12:36:21.356 UTC [1807495] LOG: could not send data to
client: Broken pipe
2022-12-11 12:36:21.356 UTC [1807495] FATAL: connection to client lost
2022-12-11 12:36:21.458 UTC [1807521] LOG: could not send data to
client: Broken pipe
2022-12-11 12:36:21.458 UTC [1807521] FATAL: connection to client lost
2022-12-11 12:36:22.489 UTC [1807708] LOG: could not send data to
client: Broken pipe
2022-12-11 12:36:22.489 UTC [1807708] FATAL: connection to client lost
2022-12-11 12:36:23.671 UTC [1807957] LOG: could not send data to
client: Broken pipe
2022-12-11 12:36:23.671 UTC [1807957] FATAL: connection to client lost
2022-12-11 12:36:23.748 UTC [1807970] LOG: could not send data to
client: Broken pipe
2022-12-11 12:36:23.748 UTC [1807970] FATAL: connection to client lost
2022-12-11 12:36:24.541 UTC [1807983] LOG: could not send data to
client: Broken pipe
2022-12-11 12:36:24.541 UTC [1807983] FATAL: connection to client lost
2022-12-11 12:36:24.702 UTC [1808144] LOG: could not send data to
client: Broken pipe
2022-12-11 12:36:24.702 UTC [1808144] FATAL: connection to client lost
2022-12-11 12:36:24.894 UTC [1808179] LOG: could not send data to
client: Broken pipe
2022-12-11 12:36:24.894 UTC [1808179] FATAL: connection to client lost
2022-12-11 12:36:25.774 UTC [1808478] FATAL: sorry, too many clients already
2022-12-11 12:36:25.776 UTC [1808477] FATAL: sorry, too many clients already
2022-12-11 12:36:25.776 UTC [1808479] FATAL: sorry, too many clients already
2022-12-11 12:36:25.778 UTC [1808485] FATAL: sorry, too many clients already
2022-12-11 12:36:25.779 UTC [1808487] FATAL: sorry, too many clients already
2022-12-11 12:36:25.796 UTC [1808498] FATAL: sorry, too many clients already
2022-12-11 12:36:26.413 UTC [1808565] FATAL: sorry, too many clients already
2022-12-11 12:36:26.561 UTC [1808566] FATAL: sorry, too many clients already
2022-12-11 12:36:26.711 UTC [1808567] FATAL: sorry, too many clients already
2022-12-11 12:36:27.001 UTC [1808568] FATAL: sorry, too many clients already
2022-12-11 12:36:27.345 UTC [1808574] FATAL: sorry, too many clients already
2022-12-11 12:36:27.461 UTC [1808575] FATAL: sorry, too many clients already
2022-12-11 12:36:27.569 UTC [1808576] FATAL: sorry, too many clients already
2022-12-11 12:36:27.653 UTC [1808577] FATAL: sorry, too many clients already
2022-12-11 12:36:27.811 UTC [1808578] FATAL: sorry, too many clients already
2022-12-11 12:36:28.213 UTC [1808584] LOG: could not send data to
client: Broken pipe
2022-12-11 12:36:28.213 UTC [1808584] FATAL: connection to client lost
TRAP: FailedAssertion("!(((allPgXact[proc->pgprocno].xid) !=
((TransactionId) 0)))", File: "procarray.c", Line: 430)
2022-12-11 12:36:29.403 UTC [1808585] LOG: could not receive data
from client: Connection reset by peer
2022-12-11 12:36:30.450 UTC [1632383] LOG: server process (PID
1739971) was terminated by signal 6: Aborted
2022-12-11 12:36:30.450 UTC [1632383] DETAIL: Failed process was
running: insert into sch18.pnt ( p ) values (
(select p from sch20.pnt_fdw limit 1 offset 79)
)
2022-12-11 12:36:30.450 UTC [1632383] LOG: terminating any other
active server processes

################################################

2022-11-30 12:27:27.523 UTC [271497] LOG: out of file descriptors:
Too many open files; release and retry
2022-11-30 12:27:27.523 UTC [271497] STATEMENT: update sch18.dt set
d = cast(nullif(sch18.dt.d,
sch18.dt.d) as date)
returning
pg_catalog.pg_client_encoding() as c0,
cast(coalesce(pg_catalog.pg_conf_load_time(),
case when pg_catalog.random() is NULL then
pg_catalog.pg_stat_get_last_autovacuum_time(
cast(pg_catalog.pg_my_temp_schema() as oid)) else
pg_catalog.pg_stat_get_last_autovacuum_time(
cast(pg_catalog.pg_my_temp_schema() as oid)) end
) as timestamptz) as c1,
pg_catalog.justify_interval(
cast(case when ((select f1 from public.invalid_check_con
limit 1 offset 5)
is not NULL)
or ((((select p from sch20.pnt_fdw limit 1 offset 71)
>^ (select p from sch30.pnt limit 1 offset 4)
)
or ((((select bitv from public.caster limit 1 offset 3)
<= (select bitv from public.caster limit
1 offset 4)
)
or ((false)
and (cast(null as name) ~~ (select t from
others.measurement_y1975_fdw limit 1 offset 5)
)))
or ((select home_base from public.slow_emp4000
limit 1 offset 6)
~= cast(null as box))))
or (((select pg_catalog.min(last_msg_send_time) from
pg_catalog.pg_stat_subscription)
> sch18.dt.t)
or (cast(null as xid) <> (select
pg_catalog.bit_or(i) from public.tstu)
))) then public.tstz_dist(
cast(cast(null as timestamptz) as timestamptz),
cast((select creation_time from pg_catalog.pg_cursors
limit 1 offset 4)
as timestamptz)) else public.tstz_dist(
cast(cast(null as timestamptz) as timestamptz),
cast((select creation_time from pg_catalog.pg_cursors
limit 1 offset 4)
as timestamptz)) end
as "interval")) as c2,
pg_catalog.json_build_object() as c3,
pg_catalog.pg_control_system() as c4,
pg_catalog.json_build_object() as c5,
pg_catalog.xpath(
cast(public.dblink_connect(
cast(pg_catalog.timeofday() as text),
cast(public.connection_parameters() as text)) as text),
cast(pg_catalog.query_to_xml(
cast(public.whoami() as text),
cast(pg_catalog.pg_jit_available() as bool),
cast(pg_catalog.pg_rotate_logfile() as bool),
cast(pg_catalog.pg_current_logfile() as text)) as xml)) as c6
TRAP: FailedAssertion("!(((allPgXact[proc->pgprocno].xid) !=
((TransactionId) 0)))", File: "procarray.c", Line: 430)
2022-11-30 12:27:28.001 UTC [271498] LOG: could not receive data from
client: Connection reset by peer
2022-11-30 12:27:28.905 UTC [221737] LOG: server process (PID 271497)
was terminated by signal 6: Aborted
2022-11-30 12:27:28.905 UTC [221737] DETAIL: Failed process was
running: update sch18.dt set
d = cast(nullif(sch18.dt.d,
sch18.dt.d) as date)
returning
pg_catalog.pg_client_encoding() as c0,
cast(coalesce(pg_catalog.pg_conf_load_time(),
case when pg_catalog.random() is NULL then
pg_catalog.pg_stat_get_last_autovacuum_time(
cast(pg_catalog.pg_my_temp_schema() as oid)) else
pg_catalog.pg_stat_get_last_autovacuum_time(
cast(pg_catalog.pg_my_temp_schema() as oid)) end
) as timestamptz) as c1,
pg_catalog.justify_interval(
cast(case when ((select f1 from public.invalid_check_con
limit 1 offset 5)
is not NULL)
or ((((select p from sch20.pnt_fdw limit 1 offset 71)
>^ (select p from sch30.pnt limit 1 offset 4)
)
or ((((select bitv from public.caster limit 1 offset 3)
<= (select bitv from public.caster limit
1 offset 4)
)
or ((false)
and (cast(null as name) ~~ (select t from
others.measurement_y1975_fd
2022-11-30 12:27:28.905 UTC [221737] LOG: terminating any other
active server processes

################################################

2022-11-22 13:44:04.604 UTC [1508667] LOG: could not send data to
client: Broken pipe
2022-11-22 13:44:04.604 UTC [1508667] FATAL: connection to client lost
2022-11-22 13:44:04.706 UTC [1508668] LOG: could not send data to
client: Broken pipe
2022-11-22 13:44:04.706 UTC [1508668] FATAL: connection to client lost
2022-11-22 13:44:05.374 UTC [1508675] LOG: could not send data to
client: Broken pipe
2022-11-22 13:44:05.374 UTC [1508675] FATAL: connection to client lost
2022-11-22 13:44:05.413 UTC [1508040] LOG: process 1508040 still
waiting for ShareLock on virtual transaction 513/212 after 5000.879 ms
2022-11-22 13:44:05.413 UTC [1508040] DETAIL: Process holding the
lock: 1508053. Wait queue: 1508033, 1508064, 1508038, 1508044,
1508040.
2022-11-22 13:44:05.413 UTC [1508040] STATEMENT: REINDEX INDEX
CONCURRENTLY sch7.chr_c_idx4;
2022-11-22 13:44:07.293 UTC [1508676] LOG: could not send data to
client: Broken pipe
2022-11-22 13:44:07.293 UTC [1508676] FATAL: connection to client lost
TRAP: FailedAssertion("!(((allPgXact[proc->pgprocno].xid) !=
((TransactionId) 0)))", File: "procarray.c", Line: 419)
2022-11-22 13:44:08.486 UTC [1508687] LOG: could not receive data
from client: Connection reset by peer
2022-11-22 13:44:09.724 UTC [1315578] LOG: server process (PID
1456482) was terminated by signal 6: Aborted
2022-11-22 13:44:09.724 UTC [1315578] DETAIL: Failed process was
running: select
subq_0.c2 as c0
from
sch20.dt as sample_0 tablesample system (9.8) ,
lateral (select
public.gen_random_uuid() as c0,
sample_1.r_id as c1,
pg_catalog.pg_notification_queue_usage() as c2,
17 as c3
from
others.measurement_y2021_fdw as ref_0
left join public.timetzcmp as sample_1 tablesample
system (6.5)
on (sample_0.d is NULL)
where (true)
or (case when false then cast(null as polygon) else
cast(null as polygon) end
~= (select polygon from public.tab_core_types
limit 1 offset 6)
)) as subq_0
where public.test_found() is not NULL
limit 109
2022-11-22 13:44:09.724 UTC [1315578] LOG: terminating any other
active server processes

################################################

2022-11-19 13:26:40.056 UTC [1484625] LOG: could not send data to
client: Broken pipe
2022-11-19 13:26:40.056 UTC [1484625] FATAL: connection to client lost
2022-11-19 13:26:40.399 UTC [1484628] LOG: could not send data to
client: Broken pipe
2022-11-19 13:26:40.399 UTC [1484628] FATAL: connection to client lost
2022-11-19 13:26:40.713 UTC [1484629] LOG: could not send data to
client: Broken pipe
2022-11-19 13:26:40.713 UTC [1484629] FATAL: connection to client lost
2022-11-19 13:26:41.601 UTC [1484635] LOG: could not send data to
client: Broken pipe
2022-11-19 13:26:41.601 UTC [1484635] FATAL: connection to client lost
2022-11-19 13:26:41.747 UTC [1484643] LOG: could not send data to
client: Broken pipe
2022-11-19 13:26:41.747 UTC [1484643] FATAL: connection to client lost
2022-11-19 13:26:41.917 UTC [1484644] LOG: could not send data to
client: Broken pipe
2022-11-19 13:26:41.917 UTC [1484644] FATAL: connection to client lost
2022-11-19 13:26:42.325 UTC [1484649] LOG: could not send data to
client: Broken pipe
2022-11-19 13:26:42.325 UTC [1484649] FATAL: connection to client lost
2022-11-19 13:26:42.476 UTC [1484652] LOG: could not send data to
client: Broken pipe
2022-11-19 13:26:42.476 UTC [1484652] FATAL: connection to client lost
TRAP: FailedAssertion("!(((allPgXact[proc->pgprocno].xid) !=
((TransactionId) 0)))", File: "procarray.c", Line: 419)
2022-11-19 13:26:43.703 UTC [1484658] LOG: could not receive data
from client: Connection reset by peer
2022-11-19 13:26:45.109 UTC [1484353] LOG: process 1484353 still
waiting for ShareLock on virtual transaction 760/55 after 5001.582 ms
2022-11-19 13:26:45.109 UTC [1484353] DETAIL: Process holding the
lock: 1484351. Wait queue: 1484353.
2022-11-19 13:26:45.109 UTC [1484353] STATEMENT: REINDEX INDEX
CONCURRENTLY sch11.pnt_p_idx;
2022-11-19 13:26:45.161 UTC [1484627] LOG: process 1484627 still
waiting for ShareLock on virtual transaction 1065/47 after 5002.366 ms
2022-11-19 13:26:45.161 UTC [1484627] DETAIL: Process holding the
lock: 1484357. Wait queue: 1484627, 1484637, 1484651, 1484648.
2022-11-19 13:26:45.161 UTC [1484627] STATEMENT: REINDEX INDEX
CONCURRENTLY sch25.txt_t_idx;
2022-11-19 13:26:45.749 UTC [1379904] LOG: server process (PID
1447940) was terminated by signal 6: Aborted
2022-11-19 13:26:45.749 UTC [1379904] DETAIL: Failed process was
running: select
pg_catalog.getdatabaseencoding() as c0,
case when pg_catalog.statement_timestamp() is not NULL then
pg_catalog.pg_get_function_arguments(
cast(cast(null as oid) as oid)) else
pg_catalog.pg_get_function_arguments(
cast(cast(null as oid) as oid)) end
as c1,
public.test_found() as c2,
ref_0.n as c3,
ref_0.n as c4,

public.myaggn08b(
cast(ref_0.n as "numeric")) over (partition by ref_0.n
order by ref_0.n) as c5,
35 as c6,
pg_catalog.pg_my_temp_schema() as c7,
pg_catalog.json_build_array() as c8,
pg_catalog.pg_my_temp_schema() as c9,
pg_catalog.jsonb_build_object() as c10,
public.make_table() as c11,
91 as c12,
public.int8alias1in(
cast(pg_catalog.timetztypmodout(
cast(
pg_catalog.bit_or(
cast(8 as int4)) over (partition by ref_0.n order by
ref_0.n) as int4)) as cstring)) as c13,
pg_catalog.pg_replication_origin_session_is_setup() as c14
from
sch18.num_fdw as ref_0
where pg_catalog.inet_client_port() ===
2022-11-19 13:26:45.749 UTC [1379904] LOG: terminating any other
active server processes

################################################

2022-10-10 12:46:01.941 UTC [887918] LOG: could not send data to
client: Broken pipe
2022-10-10 12:46:01.941 UTC [887918] FATAL: connection to client lost
2022-10-10 12:46:11.034 UTC [889217] LOG: could not send data to
client: Broken pipe
2022-10-10 12:46:11.034 UTC [889217] FATAL: connection to client lost
2022-10-10 12:46:14.295 UTC [889711] LOG: could not send data to
client: Broken pipe
2022-10-10 12:46:14.295 UTC [889711] FATAL: connection to client lost
2022-10-10 12:46:27.973 UTC [891019] LOG: could not send data to
client: Broken pipe
2022-10-10 12:46:27.973 UTC [891019] FATAL: connection to client lost
2022-10-10 12:46:28.952 UTC [891384] LOG: could not send data to
client: Broken pipe
2022-10-10 12:46:28.952 UTC [891384] FATAL: connection to client lost
TRAP: FailedAssertion("!(((allPgXact[proc->pgprocno].xid) !=
((TransactionId) 0)))", File: "procarray.c", Line: 419)
2022-10-10 12:46:34.414 UTC [889798] LOG: could not receive data from
client: Connection reset by peer
2022-10-10 12:46:35.496 UTC [872082] LOG: server process (PID 889797)
was terminated by signal 6: Aborted
2022-10-10 12:46:35.496 UTC [872082] DETAIL: Failed process was
running: update t set
a = t.a
returning
cast(public.dblink_connect(
cast(pg_catalog.timeofday() as text),
cast('dbname=''postgres'' port=54321' as text)) as text) as c0,
(select b from s_fdw limit 1 offset 6) as c15
;
2022-10-10 12:46:35.496 UTC [872082] LOG: terminating any other
active server processes

################################################

-
robins