pgbulkload - pgbulkload: Add parallel loader for pg_bulkload 3.0.

From: itagaki(at)pgfoundry(dot)org (User Itagaki)
To: pgsql-committers(at)postgresql(dot)org
Subject: pgbulkload - pgbulkload: Add parallel loader for pg_bulkload 3.0.
Date: 2009-06-04 03:26:11
Message-ID: 20090604032611.3664D17ADF12@pgfoundry.org
Views: Raw Message | Whole Thread | Download mbox | Resend email
Thread:
Lists: pgsql-committers

Log Message:
-----------
Add parallel loader for pg_bulkload 3.0.
If LOADER=PARALLEL is specified, input file parse and data load are executed
in different backends. Parsed tuples are passed through a shared memory queue.
Loading would be improved at best twice as faster as loading with single process.

Modified Files:
--------------
pgbulkload/bin:
Makefile (r1.5 -> r1.6)
(http://cvs.pgfoundry.org/cgi-bin/cvsweb.cgi/pgbulkload/pgbulkload/bin/Makefile?r1=1.5&r2=1.6)
pg_bulkload.c (r1.9 -> r1.10)
(http://cvs.pgfoundry.org/cgi-bin/cvsweb.cgi/pgbulkload/pgbulkload/bin/pg_bulkload.c?r1=1.9&r2=1.10)
postgresql (r1.3 -> r1.4)
(http://cvs.pgfoundry.org/cgi-bin/cvsweb.cgi/pgbulkload/pgbulkload/bin/postgresql?r1=1.3&r2=1.4)
pgbulkload/bin/expected:
init.out (r1.2 -> r1.3)
(http://cvs.pgfoundry.org/cgi-bin/cvsweb.cgi/pgbulkload/pgbulkload/bin/expected/init.out?r1=1.2&r2=1.3)
pgbulkload/bin/sql:
init.sql (r1.2 -> r1.3)
(http://cvs.pgfoundry.org/cgi-bin/cvsweb.cgi/pgbulkload/pgbulkload/bin/sql/init.sql?r1=1.2&r2=1.3)
pgbulkload/include:
pg_btree.h (r1.3 -> r1.4)
(http://cvs.pgfoundry.org/cgi-bin/cvsweb.cgi/pgbulkload/pgbulkload/include/pg_btree.h?r1=1.3&r2=1.4)
pg_bulkload.h (r1.7 -> r1.8)
(http://cvs.pgfoundry.org/cgi-bin/cvsweb.cgi/pgbulkload/pgbulkload/include/pg_bulkload.h?r1=1.7&r2=1.8)
reader.h (r1.2 -> r1.3)
(http://cvs.pgfoundry.org/cgi-bin/cvsweb.cgi/pgbulkload/pgbulkload/include/reader.h?r1=1.2&r2=1.3)
writer.h (r1.3 -> r1.4)
(http://cvs.pgfoundry.org/cgi-bin/cvsweb.cgi/pgbulkload/pgbulkload/include/writer.h?r1=1.3&r2=1.4)
pgbulkload/lib:
Makefile (r1.6 -> r1.7)
(http://cvs.pgfoundry.org/cgi-bin/cvsweb.cgi/pgbulkload/pgbulkload/lib/Makefile?r1=1.6&r2=1.7)
pg_btree.c (r1.8 -> r1.9)
(http://cvs.pgfoundry.org/cgi-bin/cvsweb.cgi/pgbulkload/pgbulkload/lib/pg_btree.c?r1=1.8&r2=1.9)
pg_bulkload.c (r1.10 -> r1.11)
(http://cvs.pgfoundry.org/cgi-bin/cvsweb.cgi/pgbulkload/pgbulkload/lib/pg_bulkload.c?r1=1.10&r2=1.11)
pg_bulkload.sql.in (r1.5 -> r1.6)
(http://cvs.pgfoundry.org/cgi-bin/cvsweb.cgi/pgbulkload/pgbulkload/lib/pg_bulkload.sql.in?r1=1.5&r2=1.6)
pg_strutil.c (r1.3 -> r1.4)
(http://cvs.pgfoundry.org/cgi-bin/cvsweb.cgi/pgbulkload/pgbulkload/lib/pg_strutil.c?r1=1.3&r2=1.4)
reader.c (r1.3 -> r1.4)
(http://cvs.pgfoundry.org/cgi-bin/cvsweb.cgi/pgbulkload/pgbulkload/lib/reader.c?r1=1.3&r2=1.4)
uninstall_pg_bulkload.sql (r1.4 -> r1.5)
(http://cvs.pgfoundry.org/cgi-bin/cvsweb.cgi/pgbulkload/pgbulkload/lib/uninstall_pg_bulkload.sql?r1=1.4&r2=1.5)

Added Files:
-----------
pgbulkload/bin/expected:
load_parallel.out (r1.1)
(http://cvs.pgfoundry.org/cgi-bin/cvsweb.cgi/pgbulkload/pgbulkload/bin/expected/load_parallel.out?rev=1.1&content-type=text/x-cvsweb-markup)
pgbulkload/bin/sql:
load_parallel.sql (r1.1)
(http://cvs.pgfoundry.org/cgi-bin/cvsweb.cgi/pgbulkload/pgbulkload/bin/sql/load_parallel.sql?rev=1.1&content-type=text/x-cvsweb-markup)
pgbulkload/lib:
parser_binary.c (r1.1)
(http://cvs.pgfoundry.org/cgi-bin/cvsweb.cgi/pgbulkload/pgbulkload/lib/parser_binary.c?rev=1.1&content-type=text/x-cvsweb-markup)
parser_csv.c (r1.1)
(http://cvs.pgfoundry.org/cgi-bin/cvsweb.cgi/pgbulkload/pgbulkload/lib/parser_csv.c?rev=1.1&content-type=text/x-cvsweb-markup)
parser_tuple.c (r1.1)
(http://cvs.pgfoundry.org/cgi-bin/cvsweb.cgi/pgbulkload/pgbulkload/lib/parser_tuple.c?rev=1.1&content-type=text/x-cvsweb-markup)
source.c (r1.1)
(http://cvs.pgfoundry.org/cgi-bin/cvsweb.cgi/pgbulkload/pgbulkload/lib/source.c?rev=1.1&content-type=text/x-cvsweb-markup)
writer_buffered.c (r1.1)
(http://cvs.pgfoundry.org/cgi-bin/cvsweb.cgi/pgbulkload/pgbulkload/lib/writer_buffered.c?rev=1.1&content-type=text/x-cvsweb-markup)
writer_direct.c (r1.1)
(http://cvs.pgfoundry.org/cgi-bin/cvsweb.cgi/pgbulkload/pgbulkload/lib/writer_direct.c?rev=1.1&content-type=text/x-cvsweb-markup)
writer_parallel.c (r1.1)
(http://cvs.pgfoundry.org/cgi-bin/cvsweb.cgi/pgbulkload/pgbulkload/lib/writer_parallel.c?rev=1.1&content-type=text/x-cvsweb-markup)
pgbulkload/lib/pgut:
pgut-ipc.c (r1.1)
(http://cvs.pgfoundry.org/cgi-bin/cvsweb.cgi/pgbulkload/pgbulkload/lib/pgut/pgut-ipc.c?rev=1.1&content-type=text/x-cvsweb-markup)
pgut-ipc.h (r1.1)
(http://cvs.pgfoundry.org/cgi-bin/cvsweb.cgi/pgbulkload/pgbulkload/lib/pgut/pgut-ipc.h?rev=1.1&content-type=text/x-cvsweb-markup)

Removed Files:
-------------
pgbulkload/lib:
pg_heap_buffered.c
(http://cvs.pgfoundry.org/cgi-bin/cvsweb.cgi/pgbulkload/pgbulkload/lib/pg_heap_buffered.c)
pg_heap_direct.c
(http://cvs.pgfoundry.org/cgi-bin/cvsweb.cgi/pgbulkload/pgbulkload/lib/pg_heap_direct.c)
pg_input_csv.c
(http://cvs.pgfoundry.org/cgi-bin/cvsweb.cgi/pgbulkload/pgbulkload/lib/pg_input_csv.c)
pg_input_fixed.c
(http://cvs.pgfoundry.org/cgi-bin/cvsweb.cgi/pgbulkload/pgbulkload/lib/pg_input_fixed.c)
writer.c
(http://cvs.pgfoundry.org/cgi-bin/cvsweb.cgi/pgbulkload/pgbulkload/lib/writer.c)

Browse pgsql-committers by date

  From Date Subject
Next Message Simon Riggs 2009-06-04 08:38:12 Re: [COMMITTERS] pgsql: Fix LOCK TABLE to eliminate the race condition that could make it
Previous Message Bruce Momjian 2009-06-03 20:34:29 pgsql: Wording improvement for recent sesssion identifier SQL query.