| From: | Lutz Steinborn <l(dot)steinborn(at)4c-ag(dot)de> |
|---|---|
| To: | Viktor Bojović <viktor(dot)bojovic(at)gmail(dot)com> |
| Cc: | pgsql-sql(at)postgresql(dot)org |
| Subject: | Re: large xml database |
| Date: | 2010-10-31 06:08:57 |
| Message-ID: | 20101031070857.a9298792.l.steinborn@4c-ag.de |
| Views: | Raw Message | Whole Thread | Download mbox | Resend email |
| Thread: | |
| Lists: | pgsql-sql |
On Sat, 30 Oct 2010 23:49:29 +0200
Viktor Bojović <viktor(dot)bojovic(at)gmail(dot)com> wrote:
>
> many tries have failed because 8GB of ram and 10gb of swap were not enough.
> also sometimes i get that more than 2^32 operations were performed, and
> functions stopped to work.
>
we have a similar problem and we use the Amara xml Toolkit for python. To avoid
the big memory consumption use pushbind. A 30G bme catalog file takes a maximum
up to 20min to import. It might be faster because we are preparing complex
objects with an orm. So the time consumption depends how complex the catalog is.
If you use amara only to perform a conversion from xml to csv the final import
can be done much faster.
regards
--
Lutz
| From | Date | Subject | |
|---|---|---|---|
| Next Message | James Cloos | 2010-10-31 06:59:20 | A more efficient way? |
| Previous Message | James Cloos | 2010-10-31 00:26:45 | Re: large xml database |