From: | Ron Johnson <ron(dot)l(dot)johnson(at)cox(dot)net> |
---|---|
To: | PgSQL Novice ML <pgsql-novice(at)postgresql(dot)org> |
Subject: | Some CSV file-import questions |
Date: | 2002-05-19 11:01:26 |
Message-ID: | 1021806087.9171.18.camel@rebel |
Views: | Raw Message | Whole Thread | Download mbox | Resend email |
Thread: | |
Lists: | pgsql-novice |
Hello,
If the csv file generated by the application that I am
importing from contains quotes around each field, must I
write a program to strip these "field-level" quotes before
sending the file to COPY?
As we know, COPY is a single transaction. Therefore, it
would be "unpleasant" if, say, the process that is doing the
importing dies 90% of the way through a 10,000,000 row table.
Is there a checkpoint mechanism, that, would do a COMMIT, for
example, every 10,000 rows. Then, if the process that is doing
the importing does 90% of the way through that 10,000,000 row
table, when you restart the COPY, it skips over the inserted
rows.
Here is an example from the RDBMS that I currently use:
$ bulkload -load -log -commit=10000 -tran=exclusive -db=test \
-table=foo ~/foo.csv
Then, if something happens after inserting 9,000,000 rows,
it can be restarted by:
$ bulkload -load -log -commit=10000 -skip=9000000 -db=test \
-tran=exclusive -table=foo ~/foo.csv
From what I've seen in the documentation, and the mailing
list archives, the solution to both of these questions is
to roll my bulk loader.
Ron
--
+---------------------------------------------------------+
| Ron Johnson, Jr. Home: ron(dot)l(dot)johnson(at)cox(dot)net |
| Jefferson, LA USA http://ronandheather.dhs.org:81 |
| |
| "I have created a government of whirled peas..." |
| Maharishi Mahesh Yogi, 12-May-2002, |
! CNN, Larry King Live |
+---------------------------------------------------------+
From | Date | Subject | |
---|---|---|---|
Next Message | Andrew McMillan | 2002-05-19 11:13:39 | Re: Some CSV file-import questions |
Previous Message | thiemo | 2002-05-19 08:15:06 | Forgotten the master password of db |