Re: Considerable performance downgrade of v11 and 12 on Windows

I have managed to split the 365GB file into 2GB chunks with the help of
'split' unix utility in mingw shell like so
split -C 2GB ways.txt
Then I imported the files into a clean database with the help of the
following cmd command
for /f %f in ('dir /b') do psql -U postgres -w -d osm -t -c "set
client_encoding TO 'UTF8'; copy ways from 'D:\ways\%f';"
The operation took ~3.5 hour which is the same as v10!

Prior to that I set 'parallel_leader_participation = on' and
'synchronous_commit = off' in the config file and restarted the server.

Then I logged into the psql interactive terminal and ran
ALTER TABLE ONLY ways ADD CONSTRAINT pk_ways PRIMARY KEY (id);
It took 1h 10m which is 30m faster than with the default settings (after
'type' commad if it really matters) but still 3 times slower than in v10.

Regards

пн, 2 дек. 2019 г. в 12:04, Laurenz Albe <laurenz(dot)albe(at)cybertec(dot)at>:

> On Sat, 2019-11-30 at 22:47 +0300, Eugene Podshivalov wrote:
> > It sounds strange but the "type" is indeed impacting the overall
> > performance somehow.
> > I've just tried to execute the following sequence of commands on a
> > fresh new database with PostreSQL v10 and both the copy and primary
> > key commands performed as slow as in v11 and 12.
> >
> > SET synchronous_commit TO OFF;
> > SET client_encoding TO 'UTF8';
> > COPY ways FROM program 'cmd /c "type D:\ways.txt"';
> > ALTER TABLE ONLY ways ADD CONSTRAINT pk_ways PRIMARY KEY (id);
> >
> > Regards
> >
> > пт, 29 нояб. 2019 г. в 15:22, Eugene Podshivalov <yaugenka(at)gmail(dot)com>:
> > > Laurenz,
> > > There is no way to run copy without the "type" on v11. See this thread
> > >
> https://www.postgresql.org/message-id/CAKHmqNCfTMM6%3DPqc6RUMEQ_2BPfo5KGGG-0fzRXZCVooo%3DwdNA%40mail.gmail.com
> > >
> > > My machine is running on NVMe disks, so the I/O subsystem very strong.
> > > The 100% overload is not constant but periodical, as if there are some
> > > kind of dumps for recovery performed in the background.
>
> Is it an option to split the file into parts of less than 2GB in size?
>
> Yours,
> Laurenz Albe
> --
> Cybertec | https://www.cybertec-postgresql.com
>
>