From: | Bruce Momjian <bruce(at)momjian(dot)us> |
---|---|
To: | Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us> |
Cc: | aditya desai <admad123(at)gmail(dot)com>, Pgsql Performance <pgsql-performance(at)lists(dot)postgresql(dot)org> |
Subject: | Re: Any way to speed up INSERT INTO |
Date: | 2022-03-04 18:47:35 |
Message-ID: | YiJexxjLXukD8vhf@momjian.us |
Views: | Raw Message | Whole Thread | Download mbox | Resend email |
Thread: | |
Lists: | pgsql-performance |
On Fri, Mar 4, 2022 at 01:42:39PM -0500, Tom Lane wrote:
> aditya desai <admad123(at)gmail(dot)com> writes:
> > One of the service layer app is inserting Millions of records in a table
> > but one row at a time. Although COPY is the fastest way to import a file in
> > a table. Application has a requirement of processing a row and inserting it
> > into a table. Is there any way this INSERT can be tuned by increasing
> > parameters? It is taking almost 10 hours for just 2.2 million rows in a
> > table. Table does not have any indexes or triggers.
>
> Using a prepared statement for the INSERT would help a little bit.
Yeah, I thought about that but it seems it would only minimally help.
> What would help more, if you don't expect any insertion failures,
> is to group multiple inserts per transaction (ie put BEGIN ... COMMIT
> around each batch of 100 or 1000 or so insertions). There's not
> going to be any magic bullet that lets you get away without changing
> the app, though.
Yeah, he/she could insert via multiple rows too:
CREATE TABLE test (x int);
INSERT INTO test VALUES (1), (2), (3);
> It's quite possible that network round trip costs are a big chunk of your
> problem, in which case physically grouping multiple rows into each INSERT
> command (... or COPY ...) is the only way to fix it. But I'd start with
> trying to reduce the transaction commit overhead.
Agreed, turning off synchronous_commit for that those queries would be
my first approach.
--
Bruce Momjian <bruce(at)momjian(dot)us> https://momjian.us
EDB https://enterprisedb.com
If only the physical world exists, free will is an illusion.
From | Date | Subject | |
---|---|---|---|
Next Message | Andres Freund | 2022-03-04 18:52:28 | Re: Any way to speed up INSERT INTO |
Previous Message | aditya desai | 2022-03-04 18:42:40 | Re: Any way to speed up INSERT INTO |