From: | fabrizio(dot)ermini(at)sysdat(dot)it |
---|---|
To: | pgsql-general(at)postgresql(dot)org |
Subject: | Instability in copying large quantities of data |
Date: | 2000-09-04 09:27:31 |
Message-ID: | 39B38723.800.9D9200@localhost |
Views: | Raw Message | Whole Thread | Download mbox | Resend email |
Thread: | |
Lists: | pgsql-general |
Hi all...
I've a big thorne in my side at the moment.
I'developing a web app based essentially on a set of report. This
reports are generated from queryes on my client's legacy system.
For obviuos security reason, my app doesn't interacts directly with
the main server, but is built around a Postgres DB on a separate
machine (that is also the web server), and I set up a "poor man's
replication" that batch transfer data from legacy server to pgsql
server.
In practice, the legacy server generates ASCII dumps of the data
necessary for the reports and the zips'em and ftp'em to the web
server. Then, a little process sheduled in cron get them up and
COPY them in the pgsql system. I built this process using C and
LibPQ (if necessary, I can post the code, but is a very simple thing
and I assume you can figure up how it works).
I used this schema many times for various web app, and I never
encountered problems (I've got an app built eons ago, based on
Slack 3.5 and PG 6.3.2, that's housed on a far-away provider and
that never stopped a single second in all of this time. Wow!).
Now I was trying it on a brand new RH 6.2 with PG 7.0.2, RPM
version. The problem is that the COPY of the data, apparently,
sometimes leaves a table in an inconsistent state.
The command doesn't throw any error, but when I try to SELECT or
VACUUM that table the backend dumps core. Apparently the only
thing I can do is drop the table and recreate it. This is
EXTREMELY unfortunate, since it all must be automated and if I
can't catch any error condition during the update, than also the web
app start crashing down...
Sadly this happens in a very inconsistent way. However, it seems
that the size of the data file is related to the frequency of the
problem: and since some of the table dumps are more then 20
Meg, this is no good news.
I have not got any log, cause the RPM versions doesn't create
them, however, I'll try to fix this as soon as possible.
In the meantime, anybody can share some hint on how to resolve
this nightmare?
/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/
Fabrizio Ermini Alternate E-mail:
C.so Umberto, 7 faermini(at)tin(dot)it
loc. Meleto Valdarno Mail on GSM: (keep it short!)
52020 Cavriglia (AR) faermini(at)sms(dot)tin(dot)it
From | Date | Subject | |
---|---|---|---|
Next Message | Jan Wieck | 2000-09-04 10:05:51 | Re: Updating cursors |
Previous Message | Hiroshi Inoue | 2000-09-04 09:16:45 | RE: subselect in CHECK constraint? |