Re: OSS database needed for testing

From: Josh Berkus <josh(at)agliodbs(dot)com>
To: "Merlin Moncure" <merlin(dot)moncure(at)rcsonline(dot)com>
Cc: <pgsql-performance(at)postgresql(dot)org>
Subject: Re: OSS database needed for testing
Date: 2003-04-04 18:07:49
Message-ID: 200304041007.49417.josh@agliodbs.com
Views: Raw Message | Whole Thread | Download mbox | Resend email
Thread:
Lists: pgsql-hackers pgsql-performance

Merlin,

> I would suggest downloading the data now. I can help get you started

OK, downloading now.

> with the create table statements and the import scripts. There are not
> very many ways to get the data in a reasonable timeframe: the spi
> functions or the copy command are a good place to start. Do not bother
> with running stuff through insert queries: take my word for it, it just
> won't work. Of course, if you use copy, you have to pre-format. Be
> aware that you will have many gigabytes (like more than 20) of data
> before you are done.

From my perspective, the easiest and fastest way to do this is to create the
table definitions in PostgreSQL, and then to use Perl to convert the data
format to something COPY will recognize. If you can do the create table
statements for the LM* data, I can do the Perl scripts.

Given that the *total* data is 20G, we'll want to use a subset of it. Per
your suggestion, I am downloading the *LM* tables. I may truncate them
further if the resulting database is too large. If some of the other tables
are reference lists or child tables, please tell me and I will download them
as well.

--
-Josh Berkus
Aglio Database Solutions
San Francisco

In response to

Browse pgsql-hackers by date

  From Date Subject
Next Message Ed L. 2003-04-04 18:10:16 Re: more contrib: log rotator
Previous Message Merlin Moncure 2003-04-04 18:00:26 Re: [PERFORM] OSS database needed for testing

Browse pgsql-performance by date

  From Date Subject
Next Message Manfred Koizar 2003-04-04 19:03:08 Re: [SQL] can i make this sql query more efficiant?
Previous Message Merlin Moncure 2003-04-04 18:00:26 Re: [PERFORM] OSS database needed for testing