Re: Handling small inserts from many connections.

From: Michael Vitale <michaeldba(at)sqlexec(dot)com>
To: 우성민 <dntjdals0513(at)gmail(dot)com>, pgsql-performance(at)postgresql(dot)org
Subject: Re: Handling small inserts from many connections.
Date: 2017-09-04 11:57:09
Message-ID: 1119236489.400430.1504526229815@email.1and1.com
Views: Raw Message | Whole Thread | Download mbox | Resend email
Thread:
Lists: pgsql-performance

<!DOCTYPE html>
<html><head>
<meta charset="UTF-8">
</head><body><p>Without more information, this is my initial guess at your insert slowness problem:</p><p>The symptom of this insert slowness/delayed action is delayed, granted, extend locks (locktype=extend) due to many concurrent connections trying to insert into the same table. Each insert request results in an extend lock (8k extension), which blocks other writers. What normally happens is that these extend locks happen so fast that you hardly seem them in the pg_locks table, except in the case where many concurrent connections are trying to do inserts into the same table. The following query will show if this is the case:<br></p><p>select * from pg_locks where granted = false and locktype = &#39;extend&#39;;</p><p>If this is your problem, then some kind of re-architecture is necessary to reduce the number of connections trying to do the inserts at the same time into the same table. &#160;My first hand problem like this goes back to 9.2, so perhaps some good stuff has happened in the newer versions of PG. &#160; Let&#39;s &#160;see what other good ideas come down the pike for this thread...</p><p>Regards,</p><p>Michael Vitale</p><p><br></p><blockquote type="cite">On September 4, 2017 at 4:14 AM 우성민 &#60;dntjdals0513(at)gmail(dot)com&#62; wrote:<br><br><div dir="ltr"><div>Hi team,</div><div><br></div><div>I&#39;m trying to configure postgres and pgbouncer to handle many inserts from many connections.</div><div><br></div><div>Here&#39;s some details about what i want to achieve :</div><div><br></div><div>&#160; We have more than 3000 client connections, and my server program forks backend process for each client connections.</div><div>&#160; If backend processes send a request to its connected client, the client send some text data(about 3000 bytes) to the backend process and wait for <br></div><div>&#160; next request.<br></div><div>&#160; The backend process execute insert text data using PQexec from libpq lbirary, if PQexec is done, backend process send request to <br></div><div>&#160; client again.</div><div><br></div><div>&#160; All the inserts using one, same table.<br></div><div><br></div><div>The problem is, clients wait too long due to insert process is too slow.</div><div>It seems to working fine at first, but getting slows down after couple of hours,</div><div>each insert query takes 3000+ ms and keep growing.<br></div><div><br></div><div>Need some help to figure out an actual causes of this problem.<br></div><div><br></div><div>System information :</div><div>&#160; PGBouncer 1.7.2.<br></div><div>&#160; PostgreSQL 9.6.3 on x86_64-pc-linux-gnu, compiled by gcc (GCC) 4.4.7 20120313 (Red Hat 4.4.7-18), 64-bit on CentOS release 6.9 (Final).</div><div>&#160; Kernel version 2.6.32-696.10.1.el6.x86_64<br></div><div>&#160; Intel(R) Xeon(R) CPU E5-2650 0 @ 2.00GHz<span class="ox-1ffa921873-gmail-"> processor.</span></div><div><span class="ox-1ffa921873-gmail-">&#160; </span><span class="ox-1ffa921873-gmail-">32GB ECC/REG-Buffered RAM.<br></span></div><div><span class="ox-1ffa921873-gmail-">&#160; 128GB Samsung 840 evo SSD.<br></span></div><div><span class="ox-1ffa921873-gmail-"><br></span></div><div><span class="ox-1ffa921873-gmail-"><br></span></div><div><br></div><div><br></div><div><br></div><div><br></div><br></div><br>-- <br>Sent via pgsql-performance mailing list (pgsql-performance(at)postgresql(dot)org)<br>To make changes to your subscription:<br>http://www.postgresql.org/mailpref/pgsql-performance<br></blockquote></body></html>

Attachment Content-Type Size
unknown_filename text/html 3.5 KB

In response to

Browse pgsql-performance by date

  From Date Subject
Next Message Scott Marlowe 2017-09-04 18:15:42 Re: Handling small inserts from many connections.
Previous Message 우성민 2017-09-04 08:14:39 Handling small inserts from many connections.