ERROR: invalid memory alloc request size

From: Johann Spies <jspies(at)sun(dot)ac(dot)za>
To: pgsql-admin(at)postgresql(dot)org
Subject: ERROR: invalid memory alloc request size
Date: 2007-05-14 13:13:18
Message-ID: 20070514131318.GA26537@sun.ac.za
Views: Raw Message | Whole Thread | Download mbox | Resend email
Thread:
Lists: pgsql-admin

I am using postgresql 8.1 on Debian Stable.

I am trying to build a database on a logserver to enable me to query
firewall en squid-logs more efficiently. I am working with large
amounts of data.

I have so far successfully populated 4 tables with data from one month's
firewall logs. One of them is fw_accepted:

firewall=> select count(*) from fw_accepted;
count
-----------
422790881
(1 row)

firewall=> \d fw_accepted;
Table "public.fw_accepted"
Column | Type | Modifiers
------------+------------------------+-----------
num | bigint | not null
date | date | not null
time | time without time zone | not null
start_date | date |
start_time | time without time zone |
elapsed | interval |
bytes | bigint |
src | inet |
dst | inet |
proto | character varying |
service | bigint |
rule | smallint |
fw_user | character varying |
Indexes:
"bestemming" btree (dst, date, "time")
"bron" btree (src, date, "time")
"fw_tyd" btree (date)
"gebruiker" btree (fw_user, date, "time")
"nommer" btree (num, date, "time")

I did that by using the dropping the indexes, copying the data from a
csv-file and recreating the indexes again.

While trying to do the same on the squidlogs-table the following
happens:

$ psql -d firewall -f laai_squid_logs
SET
DROP INDEX
DROP INDEX
psql:laai_squid_logs:4: ERROR: invalid memory alloc request size 1350958157
CONTEXT: COPY squidlogs, line 238780099
CREATE INDEX
CREATE INDEX

$ cat laai_squid_logs
SET client_encoding = 'LATIN1';
DROP INDEX squidbron;
DROP INDEX squidtyd ;
\copy squidlogs from '/home/log/proxy/nasql/accesspg.csv' using delimiters ' ' csv quote as '"'
CREATE INDEX squidbron
ON squidlogs
USING btree
(bron, datum, tyd);
CREATE INDEX squidtyd
ON squidlogs
USING btree
(datum, tyd);

The particular line in the csv-file looks like this (but I doubt whether
it's contents is the cause of the problem):

"2007-04-21" "14:48:48" "192.168.0.100" "134" "http://sharefsn.com/quicksilver\"><img%20src=\"http://sharefsn.com/images/bnr/ban_freestore_sm.gif\"%20border=0></a>"

The table definition:

Table "public.squidlogs"
Column | Type | Modifiers
--------+------------------------+-----------
datum | date | not null
tyd | time without time zone | not null
bron | inet | not null
bytes | bigint |
url | character varying |
Indexes:
"squidbron" btree (bron, datum, tyd)
"squidtyd" btree (datum, tyd)

The server is an old HP LP 2000 with 1G of RAM. Maybe that is the problem,
but then, how could I copy the data into the other 4 tables without
similar problems?

Any idea on how to solve this?

Regards
Johann
--
Johann Spies Telefoon: 021-808 4036
Informasietegnologie, Universiteit van Stellenbosch

"Yea, though I walk through the valley of the shadow of
death, I will fear no evil, for thou art with me;
thy rod and thy staff they comfort me." Psalms 23:4

Responses

Browse pgsql-admin by date

  From Date Subject
Next Message Tom Lane 2007-05-14 14:34:18 Re: ERROR: invalid memory alloc request size
Previous Message Phillip Smith 2007-05-14 05:19:16 Re: Auto Back up in postgresql