Possible reasons for copy_from to never terminate?

From: <Daniel(dot)Straight(at)Intralox(dot)com>
To: <psycopg(at)postgresql(dot)org>
Subject: Possible reasons for copy_from to never terminate?
Date: 2015-04-24 14:06:49
Message-ID: 94de2ab338b64d78b537a2b003eab1a6@usdlexch3.prod.laitram.loc
Views: Raw Message | Whole Thread | Download mbox | Resend email
Thread:
Lists: psycopg

The line of code I'm executing is

cursor.copy_from(io.StringIO('\n'.join(list_of_strings)), 'tablename')

Every once in a while the command seems to run forever, causing my server to crash.

I'm running this on Python 3 inside a Flask app being served by Waitress on Heroku. The source of the data is a POST request and consists of tab-separated timestamps and numbers. There is nothing different happening on the client side when this error happens. The format of the data is the same with every POST.

This is the error I get from Heroku when I finally manually restart the dyno:

2015-04-24T12:18:39Z app[postgres.23344]: [BLACK] process 23344 acquired ShareLock on transaction 720433 after 59127607.154 ms

Query: COPY tablename FROM stdin WITH DELIMITER AS ' ' NULL AS '\N'

2015-04-24T12:18:39Z app[postgres.23344]: [BLACK] connection to client lost

2015-04-24T12:18:39Z app[postgres.23344]: [BLACK] duration: 59127623.384 ms statement: COPY tablename FROM stdin WITH DELIMITER AS ' ' NULL AS '\N'

2015-04-24T12:18:39Z app[postgres.23344]: [BLACK] could not send data to client: Connection timed out 2015-04-24T12:18:39Z app[postgres.13451]: [BLACK] unexpected EOF on client connection with an open transaction

So the copy command was running for 59 million milliseconds or 16 and a half hours before I finally killed it by restarting the dyno.

Any suggestions on where I should look for the cause? Alternatively, is it possible to simply put a timeout on the copy?

-Daniel

Responses

Browse psycopg by date

  From Date Subject
Next Message Adrian Klaver 2015-04-24 14:34:28 Re: Possible reasons for copy_from to never terminate?
Previous Message Vamsi Krishna Reddy -T (vamsredd - TATA CONSULTANCY SERVICES LIMITED at Cisco) 2015-04-20 13:03:42 Re: Need help with the installation of psycopg2 module in a python virtual env