Blocking every 20 sec while mass copying.

From: Benjamin Dugast <bdugast(at)excilys(dot)com>
To: pgsql-performance(at)postgresql(dot)org
Subject: Blocking every 20 sec while mass copying.
Date: 2014-07-18 10:52:39
Message-ID: CAB9ggRqtNUngxJjAeD=ZLhkzw5OgkkqVBXkjMXC5j_9V50w-WQ@mail.gmail.com
Views: Raw Message | Whole Thread | Download mbox | Resend email
Thread:
Lists: pgsql-performance

Hello,

I'm working on Postgres 9.3.4 for a project.

We are using Scala, Akka and JDBC to insert data in the database, we have
around 25M insert to do which are basically lines from 5000 files. We issue
a DELETE according to the file (mandatory) and then a COPY each 1000 lines
of that file.

*DELETE request :* DELETE FROM table WHERE field1 = ? AND field2 = ?;
*COPY request :* COPY table FROM STDIN WITH CSV

We have indexes on our database that we can't delete to insert our data.

When we insert the data there is some kind of freezes on the databases
between requests. Freezes occur about every 20 seconds.

Here is a screenshot
<http://tof.canardpc.com/view/c42e69c0-d776-4f93-a8a3-8713794a1a07.jpg>from
yourkit.

We tried different solutions:

- 1 table to 5 tables to reduces lock contention
- fillfactor on indexes
- commit delay
- fsync to off (that helped but we can't do this)

We mainly want to know why this is happening because it slowing the insert
too much for us.

Responses

Browse pgsql-performance by date

  From Date Subject
Next Message Albe Laurenz 2014-07-18 11:11:34 Re: Blocking every 20 sec while mass copying.
Previous Message Claudio Freire 2014-07-17 23:21:42 Re: Building multiple indexes on one table.