COPY process hung in 9.2.x (9.2.4)

Hi all,

I have been seeing some copy jobs hang in a odd way. I am unable to
repoduce on demand, but here is what I have.

my setup is a bit complex, but should be old hat for most people.

client connection runs through bouncer (session pooling) to a 9.2.4
postgres back end. I have had this problem happen without bouncer in the
middle but for this limited debug bouncer is in play. have numerous clinets
copying to the same table throughout the day, often concurrently.

this is not the first time I have a copy job that won't quit/die from a
pg_cancel_backend or a pg_terminate backend.

this copy hang issue only stated since upgrading to 9.2 (from 8.3)

it looks to my untrained eye that it's in some wait state that can't be
interrupted?.

here is what the process looks like in pg_stat_activity;

I took this at about 2013-07-17 14:30-06 so the process had been in a funky
state for a number of hours.

-[ RECORD 1
]----+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
datid | 16384
datname | XXXXXXX
pid | 25896
usesysid | 10
usename | postgres
application_name |
client_addr | 10.1.106.131
client_hostname |
client_port | 60875
backend_start | 2013-07-08 14:36:37.223981-06
xact_start | 2013-07-17 08:36:33.139472-06
query_start | 2013-07-17 08:36:33.139472-06
state_change | 2013-07-17 08:36:33.139474-06
waiting | f
state | active
query | COPY (<snip>)

XXXXXXX=# SELECT pg_cancel_backend(25896);
pg_cancel_backend
-------------------
t
(1 row)

XXXXXXX=# SELECT pg_cancel_backend(25896);
pg_cancel_backend
-------------------
t
(1 row)

XXXXXXX=# SELECT pg_cancel_backend(25896);
pg_cancel_backend
-------------------
t
(1 row)

XXXXXXX=# SELECT pg_cancel_backend(25896);
pg_cancel_backend
-------------------
t
(1 row)

XXXXXXX=#SELECT pg_terminate_backend(25896);
pg_terminate_backend
----------------------
t
(1 row)

XXXXXXX=# SELECT pg_terminate_backend(25896);
pg_terminate_backend
----------------------
t
(1 row)

ok... so it will send the signal, just not get around to acting on it.

gdb looked unhelpul since there are no debugging symbols in the PGDG RPMS.
(which I would not expect there to be)

[root(at)]# gstack 25896
#0 0x000000317ece9672 in recv () from /lib64/libc.so.6
#1 0x00000000005a9676 in secure_read ()
#2 0x00000000005b432b in ?? ()
#3 0x00000000005b465c in pq_getmessage ()
#4 0x000000000052cdaa in ?? ()
#5 0x000000000052d297 in ?? ()
#6 0x000000000052d3c6 in ?? ()
#7 0x000000000052dca7 in NextCopyFromRawFields ()
#8 0x000000000052e5b9 in NextCopyFrom ()
#9 0x0000000000531b59 in DoCopy ()
#10 0x000000000065dbeb in standard_ProcessUtility ()
#11 0x000000000065a607 in ?? ()
#12 0x000000000065b5fd in ?? ()
#13 0x000000000065bd82 in PortalRun ()
#14 0x000000000065834d in ?? ()
#15 0x0000000000659200 in PostgresMain ()
#16 0x0000000000615161 in ?? ()
#17 0x0000000000617dcc in PostmasterMain ()
#18 0x00000000005b6830 in main ()

strace showed that nothing was going on for several minutes:

strace -p 25896
Process 25896 attached - interrupt to quit
recvfrom(10, ^C <unfinished ...>
Process 25896 detached

ps -efl | grep 25896
1 S postgres 25896 24852 0 80 0 - 604148 sk_wai Jul08 ? 00:00:11
postgres: postgres XXXXXXX 10.1.106.131(60875) COPY

lsof -n -p 25896
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
postgres 25896 postgres cwd DIR 8,1 4096 131
/raid/data92
postgres 25896 postgres rtd DIR 253,0 4096 2 /
postgres 25896 postgres txt REG 253,0 5477296 7479113
/usr/pgsql-9.2/bin/postgres
postgres 25896 postgres mem REG 253,0 156872 524290 /lib64/
ld-2.12.so
postgres 25896 postgres mem REG 253,0 1922152 524294 /lib64/
libc-2.12.so
postgres 25896 postgres mem REG 253,0 22536 524317 /lib64/
libdl-2.12.so
postgres 25896 postgres mem REG 253,0 145720 524295 /lib64/
libpthread-2.12.so
postgres 25896 postgres mem REG 253,0 91096 524428
/lib64/libz.so.1.2.3
postgres 25896 postgres mem REG 253,0 598680 524315 /lib64/
libm-2.12.so
postgres 25896 postgres mem REG 253,0 124624 524372
/lib64/libselinux.so.1
postgres 25896 postgres mem REG 253,0 113952 524376 /lib64/
libresolv-2.12.so
postgres 25896 postgres mem REG 253,0 43392 524349 /lib64/
libcrypt-2.12.so
postgres 25896 postgres mem REG 253,0 386040 524342
/lib64/libfreebl3.so
postgres 25896 postgres mem REG 253,0 108728 7475210
/usr/lib64/libsasl2.so.2.0.23
postgres 25896 postgres mem REG 253,0 17256 524441
/lib64/libcom_err.so.2.1
postgres 25896 postgres mem REG 253,0 58480 524352
/lib64/libpam.so.0.82.2
postgres 25896 postgres mem REG 253,0 17096 524419
/lib64/libplds4.so
postgres 25896 postgres mem REG 253,0 1286744 7479035
/usr/lib64/libnss3.so
postgres 25896 postgres mem REG 253,0 177952 7475272
/usr/lib64/libnssutil3.so
postgres 25896 postgres mem REG 253,0 224328 7479710
/usr/lib64/libssl3.so
postgres 25896 postgres mem REG 253,0 12592 524438
/lib64/libkeyutils.so.1.3
postgres 25896 postgres mem REG 253,0 243096 524388
/lib64/libnspr4.so
postgres 25896 postgres mem REG 253,0 1665328 7479326
/usr/lib64/libcrypto.so.1.0.0
postgres 25896 postgres mem REG 253,0 183896 7479978
/usr/lib64/libsmime3.so
postgres 25896 postgres mem REG 253,0 1383336 7474856
/usr/lib64/libxml2.so.2.7.6
postgres 25896 postgres mem REG 253,0 944712 524324
/lib64/libkrb5.so.3.3
postgres 25896 postgres mem REG 253,0 177520 524310
/lib64/libk5crypto.so.3.1
postgres 25896 postgres mem REG 253,0 280520 524330
/lib64/libgssapi_krb5.so.2.2
postgres 25896 postgres mem REG 253,0 46368 524304
/lib64/libkrb5support.so.0.1
postgres 25896 postgres mem REG 253,0 60144 7474481
/usr/lib64/liblber-2.4.so.2.8.5
postgres 25896 postgres mem REG 253,0 383608 7480990
/usr/lib64/libssl.so.1.0.0
postgres 25896 postgres mem REG 253,0 326216 7480986
/usr/lib64/libldap-2.4.so.2.8.5
postgres 25896 postgres mem REG 253,0 65928 524589 /lib64/
libnss_files-2.12.so
postgres 25896 postgres DEL REG 0,4 65536
/SYSV0052e2c1
postgres 25896 postgres mem REG 253,0 99158576 7474747
/usr/lib/locale/locale-archive
postgres 25896 postgres mem REG 253,0 21256 524417
/lib64/libplc4.so
postgres 25896 postgres mem REG 253,0 115536 524329
/lib64/libaudit.so.1.0.0
postgres 25896 postgres 0r CHR 1,3 0t0 3842 /dev/null
postgres 25896 postgres 1w FIFO 0,8 0t0 2353970 pipe
postgres 25896 postgres 2w FIFO 0,8 0t0 2353970 pipe
postgres 25896 postgres 3r FIFO 0,8 0t0 51158748 pipe
postgres 25896 postgres 4w FIFO 0,8 0t0 51158748 pipe
postgres 25896 postgres 5u REG 8,1 16384 2147483827
/raid/data92/global/12619
postgres 25896 postgres 6r FIFO 0,8 0t0 2353969 pipe
postgres 25896 postgres 7u REG 253,0 16777216 7078308
/var/xlogs/pg_xlog92/000000010000280D0000006A (deleted)
postgres 25896 postgres 8u REG 8,1 16384 2147483828
/raid/data92/global/12620
postgres 25896 postgres 9u IPv6 2353978 0t0 UDP
[::1]:34550->[::1]:34550
*postgres 25896 postgres 10u IPv4 51158747 0t0 TCP
10.1.106.131:postgres->10.1.106.131:60875 (ESTABLISHED)*
postgres 25896 postgres 11u REG 8,1 16384 2147483785
/raid/data92/global/12868
postgres 25896 postgres 12u REG 8,1 344064 2147489656
/raid/data92/base/16384/247678053_fsm
postgres 25896 postgres 13u REG 8,1 16384 2147483787
/raid/data92/global/12869
postgres 25896 postgres 14u REG 8,1 1048576 2147483873
/raid/data92/base/16384/12636
postgres 25896 postgres 15u REG 8,1 1073741824 2147488687
/raid/data92/base/16384/247678053
postgres 25896 postgres 16u REG 8,1 180224 2147490208
/raid/data92/base/16384/247666671_fsm
postgres 25896 postgres 17u REG 8,1 255205376 2147490131
/raid/data92/base/16384/247678062
postgres 25896 postgres 18u REG 8,1 24576 2147490213
/raid/data92/base/16384/247678802_fsm
postgres 25896 postgres 19u REG 8,1 16384 2147486050
/raid/data92/base/16384/12675
postgres 25896 postgres 20u REG 8,1 24592384 2147490148
/raid/data92/base/16384/247678802
postgres 25896 postgres 21u REG 8,1 32768 2147483805
/raid/data92/base/16384/12687
postgres 25896 postgres 22u REG 8,1 4784128 2147490223
/raid/data92/base/16384/247678811
postgres 25896 postgres 23u REG 8,1 8192 2147483831
/raid/data92/global/12751
postgres 25896 postgres 24u REG 8,1 16384 2147486062
/raid/data92/base/16384/12731
postgres 25896 postgres 25u REG 8,1 1556480 2147490065
/raid/data92/base/16384/247660214_fsm
postgres 25896 postgres 26u REG 8,1 1073741824 2147486178
/raid/data92/base/16384/247660214
postgres 25896 postgres 27u REG 8,1 1073741824 2147491444
/raid/data92/base/16384/247660214.1
postgres 25896 postgres 28u REG 8,1 1073741824 2147486188
/raid/data92/base/16384/247660219
postgres 25896 postgres 29u REG 8,1 1073741824 2147491455
/raid/data92/base/16384/247660219.1
postgres 25896 postgres 30u REG 8,1 106496 2147490211
/raid/data92/base/16384/247661998_fsm
postgres 25896 postgres 31u REG 8,1 357679104 2147486635
/raid/data92/base/16384/247661998
postgres 25896 postgres 32u REG 8,1 24576 2147490254
/raid/data92/base/16384/247678042_fsm
postgres 25896 postgres 33u REG 8,1 2490368 2147488566
/raid/data92/base/16384/247678042
postgres 25896 postgres 34u REG 8,1 278528 2147490210
/raid/data92/base/16384/247678848_fsm
postgres 25896 postgres 35u REG 8,1 1034158080 2147490224
/raid/data92/base/16384/247678848
postgres 25896 postgres 36u REG 8,1 564789248 2147490231
/raid/data92/base/16384/247678858
postgres 25896 postgres 37u REG 8,1 150011904 2147490130
/raid/data92/base/16384/247678061
postgres 25896 postgres 38u REG 8,1 57344 2147490212
/raid/data92/base/16384/247679246_fsm
postgres 25896 postgres 39u REG 8,1 157581312 2147490232
/raid/data92/base/16384/247679246
postgres 25896 postgres 40u REG 8,1 143163392 2147490235
/raid/data92/base/16384/247679254
postgres 25896 postgres 41u REG 8,1 1998848 2147490055
/raid/data92/base/16384/247664756_fsm
postgres 25896 postgres 42u REG 8,1 1073741824 2147488227
/raid/data92/base/16384/247664756
postgres 25896 postgres 43u REG 8,1 1073741824 2147491477
/raid/data92/base/16384/247664756.1
postgres 25896 postgres 44u REG 8,1 1073741824 2147491456
/raid/data92/base/16384/247664756.2
postgres 25896 postgres 45u REG 8,1 409600 2147490122
/raid/data92/base/16384/247662229_fsm
postgres 25896 postgres 46u REG 8,1 646856704 2147488314
/raid/data92/base/16384/247666671
postgres 25896 postgres 47u REG 8,1 1343488 2147490090
/raid/data92/base/16384/247662566_fsm
postgres 25896 postgres 48u REG 8,1 1073741824 2147486688
/raid/data92/base/16384/247662566
postgres 25896 postgres 49u REG 8,1 1073741824 2147549657
/raid/data92/base/16384/247662566.1
postgres 25896 postgres 50u REG 8,1 262144 2147490172
/raid/data92/base/16384/247664480_fsm
postgres 25896 postgres 51u REG 8,1 972759040 2147488173
/raid/data92/base/16384/247664480
postgres 25896 postgres 52u REG 8,1 441982976 2147488210
/raid/data92/base/16384/247664486
postgres 25896 postgres 53u REG 8,1 722575360 2147488174
/raid/data92/base/16384/247664485
postgres 25896 postgres 54u REG 8,1 712818688 2147486668
/raid/data92/base/16384/247662245
postgres 25896 postgres 55u REG 8,1 1073741824 2147486665
/raid/data92/base/16384/247662229
postgres 25896 postgres 56u REG 8,1 1073741824 2147486201
/raid/data92/base/16384/247660220
postgres 25896 postgres 57u REG 8,1 1073741824 2147549688
/raid/data92/base/16384/247660220.1
postgres 25896 postgres 58u REG 8,1 24576 2147490218
/raid/data92/base/16384/247679258_fsm
postgres 25896 postgres 59u REG 8,1 27156480 2147490268
/raid/data92/base/16384/247679258
postgres 25896 postgres 60u REG 8,1 82960384 2147490264
/raid/data92/base/16384/247679256
postgres 25896 postgres 61u REG 8,1 101376000 2147490259
/raid/data92/base/16384/247679255
postgres 25896 postgres 62u REG 8,1 421117952 2147490127
/raid/data92/base/16384/247678060
postgres 25896 postgres 63u REG 8,1 1073741824 2147486694
/raid/data92/base/16384/247662571
postgres 25896 postgres 64u REG 8,1 944750592 2147490228
/raid/data92/base/16384/247678856
postgres 25896 postgres 65u REG 8,1 665083904 2147490230
/raid/data92/base/16384/247678857
postgres 25896 postgres 66u REG 8,1 1073741824 2147488250
/raid/data92/base/16384/247664761
postgres 25896 postgres 67u REG 8,1 1073741824 2147491453
/raid/data92/base/16384/247664761.1
postgres 25896 postgres 68u REG 8,1 1073741824 2147486702
/raid/data92/base/16384/247662572
postgres 25896 postgres 69u REG 8,1 1073741824 2147549740
/raid/data92/base/16384/247664756.4
postgres 25896 postgres 70u REG 8,1 1073741824 2147549690
/raid/data92/base/16384/247660214.2
postgres 25896 postgres 71u REG 8,1 185663488 2147486647
/raid/data92/base/16384/247662004
postgres 25896 postgres 72u REG 8,1 1073741824 2147549703
/raid/data92/base/16384/247664756.3
postgres 25896 postgres 73u REG 8,1 1073741824 2147488255
/raid/data92/base/16384/247664762
postgres 25896 postgres 74u REG 8,1 1073741824 2147549679
/raid/data92/base/16384/247664762.1
postgres 25896 postgres 75u REG 8,1 483328 2147486037
/raid/data92/base/16384/12639
postgres 25896 postgres 76u REG 8,1 401408 2147483825
/raid/data92/base/16384/12638
postgres 25896 postgres 77u REG 8,1 2203648 2147486033
/raid/data92/base/16384/12628
postgres 25896 postgres 78u REG 8,1 9355264 2147483909
/raid/data92/base/16384/12625
postgres 25896 postgres 79u REG 8,1 188416 2147486040
/raid/data92/base/16384/12645
postgres 25896 postgres 80u REG 8,1 1261568 2147483884
/raid/data92/base/16384/12640
postgres 25896 postgres 81u REG 8,1 204800 2147486047
/raid/data92/base/16384/12662
postgres 25896 postgres 82u REG 8,1 630784 2147483934
/raid/data92/base/16384/12660
postgres 25896 postgres 83u REG 8,1 254623744 2147488319
/raid/data92/base/16384/247666677
postgres 25896 postgres 84u REG 8,1 327983104 2147486642
/raid/data92/base/16384/247662003
postgres 25896 postgres 85u REG 8,1 1073741824 2147549709
/raid/data92/base/16384/247660219.2
postgres 25896 postgres 86u REG 8,1 1073741824 2147549707
/raid/data92/base/16384/247664761.2
postgres 25896 postgres 87u REG 8,1 1073741824 2147549696
/raid/data92/base/16384/247662571.1
postgres 25896 postgres 88u REG 8,1 1073741824 2147549719
/raid/data92/base/16384/247662566.2
postgres 25896 postgres 89u REG 8,1 1073741824 2147549730
/raid/data92/base/16384/247660214.3
postgres 25896 postgres 128u REG 8,1 196608 2147486048
/raid/data92/base/16384/12663
postgres 25896 postgres 138u REG 8,1 16384 2147486057
/raid/data92/base/16384/12705
postgres 25896 postgres 139u REG 8,1 122880 2147483807
/raid/data92/base/16384/12699
postgres 25896 postgres 140u REG 8,1 188416 2147483808
/raid/data92/base/16384/12703
postgres 25896 postgres 141u REG 8,1 688128 2147483892
/raid/data92/base/16384/12701
postgres 25896 postgres 142u REG 8,1 212992 2147486036
/raid/data92/base/16384/12635
postgres 25896 postgres 143u REG 8,1 655360 2147483883
/raid/data92/base/16384/12629
postgres 25896 postgres 144u REG 8,1 73728 2147486035
/raid/data92/base/16384/12634
postgres 25896 postgres 145u REG 8,1 335872 2147486030
/raid/data92/base/16384/12616
postgres 25896 postgres 146u REG 8,1 40960 2147483789
/raid/data92/base/16384/12667
postgres 25896 postgres 147u REG 8,1 32768 2147483902
/raid/data92/base/16384/12666
postgres 25896 postgres 148u REG 8,1 16384 2147483900
/raid/data92/base/16384/12826
postgres 25896 postgres 149u REG 8,1 368640 2147483810
/raid/data92/base/16384/12615
postgres 25896 postgres 150u REG 8,1 16384 2147486016
/raid/data92/base/16384/12723
postgres 25896 postgres 151u REG 8,1 999424 2147483921
/raid/data92/base/16384/12613
postgres 25896 postgres 152u REG 8,1 16384 2147483925
/raid/data92/base/16384/12674
postgres 25896 postgres 153u REG 8,1 32768 2147483806
/raid/data92/base/16384/12682
postgres 25896 postgres 154u REG 8,1 16384 2147486055
/raid/data92/base/16384/12698
postgres 25896 postgres 155u REG 8,1 32768 2147486051
/raid/data92/base/16384/12683
postgres 25896 postgres 156u REG 8,1 32768 2147483888
/raid/data92/base/16384/12680
postgres 25896 postgres 158u REG 8,1 1073152 2147486029
/raid/data92/base/16384/12612
postgres 25896 postgres 161u REG 8,1 1073741824 2147491089
/raid/data92/base/16384/245572574
postgres 25896 postgres 162u REG 8,1 1073741824 2147486571
/raid/data92/base/16384/245572574.1
postgres 25896 postgres 163u REG 8,1 1073741824 2147617642
/raid/data92/base/16384/245572574.2
postgres 25896 postgres 164u REG 8,1 1073741824 2147548355
/raid/data92/base/16384/245572574.3
postgres 25896 postgres 165u REG 8,1 1073741824 2147488185
/raid/data92/base/16384/245572574.4
postgres 25896 postgres 166u REG 8,1 1005936640 2147491104
/raid/data92/base/16384/245572574.5
postgres 25896 postgres 167u REG 8,1 1073741824 2147490062
/raid/data92/base/16384/248233035
postgres 25896 postgres 168u REG 8,1 1073741824 2147490063
/raid/data92/base/16384/248233035.1
postgres 25896 postgres 169u REG 8,1 1073741824 2147490070
/raid/data92/base/16384/248233035.2
postgres 25896 postgres 170u REG 8,1 17883136 2147490072
/raid/data92/base/16384/248233035.3
postgres 25896 postgres 171u REG 8,1 1073741824 2147490080
/raid/data92/base/16384/248233173
postgres 25896 postgres 172u REG 8,1 832438272 2147490087
/raid/data92/base/16384/248233173.1
postgres 25896 postgres 173u REG 8,1 1073741824 2147486172
/raid/data92/base/16384/238131664
postgres 25896 postgres 174u REG 8,1 1073741824 2147486356
/raid/data92/base/16384/238131664.1
postgres 25896 postgres 175u REG 8,1 1073741824 2147503855
/raid/data92/base/16384/238131664.2
postgres 25896 postgres 176u REG 8,1 1073741824 2147548614
/raid/data92/base/16384/238131664.3
postgres 25896 postgres 177u REG 8,1 1073741824 2147490347
/raid/data92/base/16384/238131664.4
postgres 25896 postgres 178u REG 8,1 1073741824 2147489603
/raid/data92/base/16384/238131664.5
postgres 25896 postgres 179u REG 8,1 317358080 2147548647
/raid/data92/base/16384/238131664.6
postgres 25896 postgres 180u REG 8,1 1073741824 2147488447
/raid/data92/base/16384/248218634
postgres 25896 postgres 181u REG 8,1 1073741824 2147488448
/raid/data92/base/16384/248218634.1
postgres 25896 postgres 182u REG 8,1 1073741824 2147488450
/raid/data92/base/16384/248218634.2
postgres 25896 postgres 183u REG 8,1 206184448 2147488453
/raid/data92/base/16384/248218634.3
postgres 25896 postgres 184u REG 8,1 1073741824 2147488454
/raid/data92/base/16384/248218763
postgres 25896 postgres 185u REG 8,1 943489024 2147488456
/raid/data92/base/16384/248218763.1
postgres 25896 postgres 186u REG 8,1 1073741824 2147549775
/raid/data92/base/16384/236347143
postgres 25896 postgres 187u REG 8,1 1073741824 2147503866
/raid/data92/base/16384/236347143.1
postgres 25896 postgres 188u REG 8,1 1073741824 2147486568
/raid/data92/base/16384/236347143.2
postgres 25896 postgres 189u REG 8,1 1073741824 2147747483
/raid/data92/base/16384/236347143.3
postgres 25896 postgres 190u REG 8,1 1073741824 2147486232
/raid/data92/base/16384/236347143.4
postgres 25896 postgres 191u REG 8,1 975126528 2147489633
/raid/data92/base/16384/236347143.5
postgres 25896 postgres 192u REG 8,1 1073741824 2147616724
/raid/data92/base/16384/247489640
postgres 25896 postgres 193u REG 8,1 1073741824 2147616728
/raid/data92/base/16384/247489640.1
postgres 25896 postgres 194u REG 8,1 1071448064 2147616729
/raid/data92/base/16384/247489640.2
postgres 25896 postgres 195u REG 8,1 1073741824 2147617609
/raid/data92/base/16384/247489709
postgres 25896 postgres 196u REG 8,1 818839552 2147617613
/raid/data92/base/16384/247489709.1
postgres 25896 postgres 197u REG 8,1 1073741824 2147499926
/raid/data92/base/16384/242785737
postgres 25896 postgres 198u REG 8,1 1073741824 2147488719
/raid/data92/base/16384/242785737.1
postgres 25896 postgres 199u REG 8,1 1073741824 2147878017
/raid/data92/base/16384/242785737.2
postgres 25896 postgres 200u REG 8,1 1073741824 2147747169
/raid/data92/base/16384/242785737.3
postgres 25896 postgres 201u REG 8,1 1073741824 2147747213
/raid/data92/base/16384/242785737.4
postgres 25896 postgres 202u REG 8,1 1073741824 2147617717
/raid/data92/base/16384/242785737.5
postgres 25896 postgres 203u REG 8,1 555130880 2147548558
/raid/data92/base/16384/242785737.6
postgres 25896 postgres 204u REG 8,1 1073741824 2147488714
/raid/data92/base/16384/248227184
postgres 25896 postgres 205u REG 8,1 1073741824 2147488720
/raid/data92/base/16384/248227184.1
postgres 25896 postgres 206u REG 8,1 1073741824 2147488729
/raid/data92/base/16384/248227184.2
postgres 25896 postgres 207u REG 8,1 330121216 2147488730
/raid/data92/base/16384/248227184.3
postgres 25896 postgres 208u REG 8,1 1073741824 2147489600
/raid/data92/base/16384/248227292
postgres 25896 postgres 209u REG 8,1 1014800384 2147489602
/raid/data92/base/16384/248227292.1
postgres 25896 postgres 210u REG 8,1 1073741824 2147548643
/raid/data92/base/16384/235421304
postgres 25896 postgres 211u REG 8,1 1073741824 2147490125
/raid/data92/base/16384/235421304.1
postgres 25896 postgres 212u REG 8,1 1073741824 2147490308
/raid/data92/base/16384/235421304.2
postgres 25896 postgres 213u REG 8,1 1073741824 2147503923
/raid/data92/base/16384/235421304.3
postgres 25896 postgres 214u REG 8,1 1073741824 2147548318
/raid/data92/base/16384/235421304.4
postgres 25896 postgres 215u REG 8,1 1048870912 2147490124
/raid/data92/base/16384/235421304.5
postgres 25896 postgres 216u REG 8,1 1073741824 2147616713
/raid/data92/base/16384/247489191
postgres 25896 postgres 217u REG 8,1 1073741824 2147616715
/raid/data92/base/16384/247489191.1
postgres 25896 postgres 218u REG 8,1 1073741824 2147616716
/raid/data92/base/16384/247489191.2
postgres 25896 postgres 219u REG 8,1 36339712 2147616717
/raid/data92/base/16384/247489191.3
postgres 25896 postgres 220u REG 8,1 1073741824 2147616719
/raid/data92/base/16384/247489565
postgres 25896 postgres 221u REG 8,1 841646080 2147616722
/raid/data92/base/16384/247489565.1
postgres 25896 postgres 222u REG 8,1 1073741824 2147548533
/raid/data92/base/16384/240106277
postgres 25896 postgres 223u REG 8,1 1073741824 2147503894
/raid/data92/base/16384/240106277.1
postgres 25896 postgres 224u REG 8,1 1073741824 2147489616
/raid/data92/base/16384/240106277.2
postgres 25896 postgres 225u REG 8,1 1073741824 2147488331
/raid/data92/base/16384/240106277.3
postgres 25896 postgres 226u REG 8,1 1073741824 2147486555
/raid/data92/base/16384/240106277.4
postgres 25896 postgres 227u REG 8,1 1073741824 2147490338
/raid/data92/base/16384/240106277.5
postgres 25896 postgres 228u REG 8,1 564928512 2147486229
/raid/data92/base/16384/240106277.6
postgres 25896 postgres 229u REG 8,1 1073741824 2147488460
/raid/data92/base/16384/248222581
postgres 25896 postgres 230u REG 8,1 1073741824 2147488461
/raid/data92/base/16384/248222581.1
postgres 25896 postgres 231u REG 8,1 1073741824 2147488462
/raid/data92/base/16384/248222581.2
postgres 25896 postgres 232u REG 8,1 333545472 2147488468
/raid/data92/base/16384/248222581.3
postgres 25896 postgres 233u REG 8,1 1073741824 2147488535
/raid/data92/base/16384/248222760
postgres 25896 postgres 234u REG 8,1 1017339904 2147488538
/raid/data92/base/16384/248222760.1
postgres 25896 postgres 235u REG 8,1 1073741824 2147503943
/raid/data92/base/16384/248123192
postgres 25896 postgres 236u REG 8,1 1073741824 2147491152
/raid/data92/base/16384/248123192.1
postgres 25896 postgres 237u REG 8,1 1073741824 2147503957
/raid/data92/base/16384/248123197
postgres 25896 postgres 238u REG 8,1 1073741824 2147548263
/raid/data92/base/16384/248123198
postgres 25896 postgres 239u REG 8,1 1073741824 2147486349
/raid/data92/base/16384/241329068
postgres 25896 postgres 240u REG 8,1 1073741824 2147617614
/raid/data92/base/16384/241329068.1
postgres 25896 postgres 241u REG 8,1 1073741824 2147617755
/raid/data92/base/16384/241329068.2
postgres 25896 postgres 242u REG 8,1 1073741824 2147548284
/raid/data92/base/16384/241329068.3
postgres 25896 postgres 243u REG 8,1 1073741824 2147488470
/raid/data92/base/16384/241329068.4
postgres 25896 postgres 244u REG 8,1 1073741824 2147490209
/raid/data92/base/16384/241329068.5
postgres 25896 postgres 245u REG 8,1 541220864 2147499910
/raid/data92/base/16384/241329068.6
postgres 25896 postgres 246u REG 8,1 1073741824 2147488541
/raid/data92/base/16384/248222987
postgres 25896 postgres 247u REG 8,1 1073741824 2147488545
/raid/data92/base/16384/248222987.1
postgres 25896 postgres 248u REG 8,1 1073741824 2147488553
/raid/data92/base/16384/248222987.2
postgres 25896 postgres 249u REG 8,1 324403200 2147488569
/raid/data92/base/16384/248222987.3
postgres 25896 postgres 250u REG 8,1 1073741824 2147488587
/raid/data92/base/16384/248223113
postgres 25896 postgres 251u REG 8,1 1011531776 2147488588
/raid/data92/base/16384/248223113.1
postgres 25896 postgres 252u REG 8,1 1073741824 2147548655
/raid/data92/base/16384/239039321
postgres 25896 postgres 253u REG 8,1 1073741824 2147486176
/raid/data92/base/16384/239039321.1
postgres 25896 postgres 254u REG 8,1 1073741824 2147616667
/raid/data92/base/16384/239039321.2
postgres 25896 postgres 255u REG 8,1 1073741824 2147747219
/raid/data92/base/16384/239039321.3
postgres 25896 postgres 256u REG 8,1 1073741824 2147488686
/raid/data92/base/16384/239039321.4
postgres 25896 postgres 257u REG 8,1 1073741824 2147490049
/raid/data92/base/16384/239039321.5
postgres 25896 postgres 258u REG 8,1 101154816 2147548526
/raid/data92/base/16384/239039321.6
postgres 25896 postgres 259u REG 8,1 1073741824 2147486212
/raid/data92/base/16384/248222328
postgres 25896 postgres 260u REG 8,1 1073741824 2147486221
/raid/data92/base/16384/248222328.1
postgres 25896 postgres 261u REG 8,1 1073741824 2147486231
/raid/data92/base/16384/248222328.2
postgres 25896 postgres 262u REG 8,1 96174080 2147486336
/raid/data92/base/16384/248222328.3
postgres 25896 postgres 263u REG 8,1 1073741824 2147486386
/raid/data92/base/16384/248222469
postgres 25896 postgres 264u REG 8,1 878927872 2147488459
/raid/data92/base/16384/248222469.1
postgres 25896 postgres 265u REG 8,1 1073741824 2147503959
/raid/data92/base/16384/237352048
postgres 25896 postgres 266u REG 8,1 1073741824 2147747466
/raid/data92/base/16384/237352048.1
postgres 25896 postgres 267u REG 8,1 1073741824 2147486549
/raid/data92/base/16384/237352048.2
postgres 25896 postgres 268u REG 8,1 1073741824 2147617729
/raid/data92/base/16384/237352048.3
postgres 25896 postgres 269u REG 8,1 1073741824 2147548639
/raid/data92/base/16384/237352048.4
postgres 25896 postgres 270u REG 8,1 1073741824 2147488691
/raid/data92/base/16384/237352048.5
postgres 25896 postgres 271u REG 8,1 268017664 2147503895
/raid/data92/base/16384/237352048.6
postgres 25896 postgres 272u REG 8,1 1073741824 2147488425
/raid/data92/base/16384/248217976
postgres 25896 postgres 273u REG 8,1 1073741824 2147488427
/raid/data92/base/16384/248217976.1
postgres 25896 postgres 274u REG 8,1 1073741824 2147488429
/raid/data92/base/16384/248217976.2
postgres 25896 postgres 275u REG 8,1 182755328 2147488432
/raid/data92/base/16384/248217976.3
postgres 25896 postgres 276u REG 8,1 1073741824 2147488433
/raid/data92/base/16384/248218465
postgres 25896 postgres 277u REG 8,1 927940608 2147488442
/raid/data92/base/16384/248218465.1
postgres 25896 postgres 278u REG 8,1 1073741824 2147548567
/raid/data92/base/16384/243999874
postgres 25896 postgres 279u REG 8,1 1073741824 2147499891
/raid/data92/base/16384/243999874.1
postgres 25896 postgres 280u REG 8,1 1073741824 2147490243
/raid/data92/base/16384/243999874.2
postgres 25896 postgres 281u REG 8,1 1073741824 2147488163
/raid/data92/base/16384/243999874.3
postgres 25896 postgres 282u REG 8,1 1073741824 2147616765
/raid/data92/base/16384/243999874.4
postgres 25896 postgres 283u REG 8,1 1073741824 2147490293
/raid/data92/base/16384/243999874.5
postgres 25896 postgres 284u REG 8,1 351633408 2147491085
/raid/data92/base/16384/243999874.6
postgres 25896 postgres 285u REG 8,1 1073741824 2147489647
/raid/data92/base/16384/248231396
postgres 25896 postgres 286u REG 8,1 1073741824 2147489662
/raid/data92/base/16384/248231396.1
postgres 25896 postgres 287u REG 8,1 1073741824 2147490053
/raid/data92/base/16384/248231396.2
postgres 25896 postgres 288u REG 8,1 226844672 2147490054
/raid/data92/base/16384/248231396.3
postgres 25896 postgres 289u REG 8,1 1073741824 2147490058
/raid/data92/base/16384/248231672
postgres 25896 postgres 290u REG 8,1 954056704 2147490061
/raid/data92/base/16384/248231672.1
postgres 25896 postgres 291u REG 8,1 1073741824 2147747623
/raid/data92/base/16384/247248188
postgres 25896 postgres 292u REG 8,1 1073741824 2147548454
/raid/data92/base/16384/247248188.1
postgres 25896 postgres 293u REG 8,1 1073741824 2147491162
/raid/data92/base/16384/247248188.2
postgres 25896 postgres 294u REG 8,1 1073741824 2147488165
/raid/data92/base/16384/247248188.3
postgres 25896 postgres 295u REG 8,1 1073741824 2147747482
/raid/data92/base/16384/247248188.4
postgres 25896 postgres 296u REG 8,1 1073741824 2147503918
/raid/data92/base/16384/247248188.5
postgres 25896 postgres 297u REG 8,1 477880320 2147503941
/raid/data92/base/16384/247248188.6
postgres 25896 postgres 298u REG 8,1 1073741824 2147747625
/raid/data92/base/16384/247248193
postgres 25896 postgres 299u REG 8,1 1073741824 2147491149
/raid/data92/base/16384/247248193.1
postgres 25896 postgres 300u REG 8,1 1073741824 2147486710
/raid/data92/base/16384/247248193.2
postgres 25896 postgres 301u REG 8,1 1073741824 2147747480
/raid/data92/base/16384/247248193.3
postgres 25896 postgres 302u REG 8,1 1073741824 2147503922
/raid/data92/base/16384/247248193.4
postgres 25896 postgres 303u REG 8,1 224124928 2147503942
/raid/data92/base/16384/247248193.5
postgres 25896 postgres 304u REG 8,1 1073741824 2147747633
/raid/data92/base/16384/247248194
postgres 25896 postgres 305u REG 8,1 1073741824 2147491438
/raid/data92/base/16384/247248194.1
postgres 25896 postgres 306u REG 8,1 1073741824 2147747481
/raid/data92/base/16384/247248194.2
postgres 25896 postgres 307u REG 8,1 360849408 2147490159
/raid/data92/base/16384/247248194.3
postgres 25896 postgres 308u REG 8,1 72196096 2147486026
/raid/data92/base/16384/12607
postgres 25896 postgres 309u REG 8,1 3497984 2147486028
/raid/data92/base/16384/12611
postgres 25896 postgres 310u REG 8,1 206946304 2147486027
/raid/data92/base/16384/12609
postgres 25896 postgres 311u REG 8,1 16384 2147483822
/raid/data92/global/12754
postgres 25896 postgres 312u REG 8,1 8192 2147483814
/raid/data92/global/12752
postgres 25896 postgres 313u REG 8,1 32768 2147483790
/raid/data92/base/16384/12729
postgres 25896 postgres 314u REG 8,1 16384 2147483829
/raid/data92/base/16384/12720

when I log into bouncer, it does not show 60875 as a client port by
anything. so there was nothing from bouncer to show AFAIK.

10u is the connect glue between bouncer and postgres. I know its odd that I
used TCP between bouncer and postgres even though they are on the same box.

it's almost like keep alives should have kicked in and cleaned this up.

I run somewhat aggressive keep alives:

postgres=# show tcp_keepalives_idle;
tcp_keepalives_idle
---------------------
30
(1 row)

postgres=# show tcp_keepalives_interval;
tcp_keepalives_interval
-------------------------
10
(1 row)

postgres=# show tcp_keepalives_count;
tcp_keepalives_count
----------------------
6
(1 row)

I had to restart -m i postgres to get postgres useable again.

not sure what other information I can provide, but I think there may be
something up with the changes that went in with 9.2 and bulk copy.

-Mark