help with a palloc error on FreeBSD

I'll ask this on a FreeBSD list as well, but I was wondering if any of you
might have some specific suggestions. Here's what's happening: I'm running
FreeBSD 2.2.5 and PostgreSQL 6.3.2. I have a table with about 1.3 million
rows and I'm trying to create an index on one of the fields (a box type)
using an rtree. About 45 minutes into it, I'm getting a fatal palloc error:

FATAL 1: palloc failure: memory exhausted

the first thing I did was bump up the swap space and tried it again. Same
result. Then I bumped up the process memory limits in the kernel (MAXDSIZE
and DFLDSIZE) to 256MB - 512MB, same thing happens. Then I actually
decided to investigate what was going on with top and vmstat; both show a
lot less memory being used that any of the initial memory limits.

Anybody have any idea what's going on and what settings I have to change?

Thanks,

jeff