Re: extending relations more efficiently

* Robert Haas (robertmhaas(at)gmail(dot)com) wrote:
> I think the first thing we need here is a good test case, so we're
> clear on what we're trying to solve. I was just hoping to make file
> extension *faster* and what you and Simon are talking about is making
> it scale better in the face of heavy parallelism; obviously it would
> be nice to do both things, but they are different problems. Any old
> bulk-loading test will benefit from a raw performance improvement, but
> to test a scalability improvement we would need some kind of test case
> involving parallel bulk loads, or some other kind of parallel activity
> that causes rapid table growth. That's not something I've frequently
> run into, but I'd be willing to put a bit of time into it if we can
> nail down what we're talking about.

Try loading the TIGER 2011 data into a single table, where you load each
county (or perhaps state) in a separate, parallel, process. That's what
I was doing when I hit this lock full-force and bitched about it on this
list.

I'd be happy to help construct that case, as well as test any changes in
this area which might help address it (on a 10G SSD-backed SAN..).

Thanks,

Stephen