Re: GSoC 2011: Fast GiST index build

On Tue, Apr 26, 2011 at 10:46 AM, Heikki Linnakangas <
heikki(dot)linnakangas(at)enterprisedb(dot)com> wrote:

> Just palloc() the buffers in memory, at least in the first phase. That'll
> work fine for index creation. Dealing with concurrent searches and inserts
> makes it a lot more complicated, it's better to make it work for the index
> creation first, and investigate something like the GIN fastupdate buffers
> later if you have time left.
>
Since algorithm is focused to reduce I/O, we should expect best acceleration
in the case when index doesn't fitting to memory. Size of buffers is
comparable to size of whole index. It means that if we can hold buffers in
memory then we mostly can hold whole index in memory. That's why I think we
should have simple on-disk buffers management for first representative
benchmark.

> The first priority should be to have something that works enough to be
> benchmarked. The paper you referred to in the GSoC application [1] contained
> empirical results on the number of I/O operations needed with the algorithm,
> but it didn't take operating system cache into account at all. That makes
> the empiric results next to worthless; keeping some stuff in in-memory
> buffers is obviously going to reduce I/O if you don't take OS cache into
> account.
>
> So we're going to need benchmark results that show a benefit, or there's no
> point in doing this at all. The sooner we get to benchmarking, even with a
> very limited and buggy version of the patch, the better. If the algorithm
> described in that paper doesn't give much benefit, you might have to switch
> to some other algorithm half-way through the project. Fortunately there's
> plenty of R-tree bulk loading algorithms in the literature, it should be
> possible to adapt some of them to GiST.
>
> [1] http://dx.doi.org/10.1007/s00453-001-0107-6

Yes, these priority seems very reasonable. We should have first
effectiveness confirmation as soon as possible. I'll hold on this priority.

----
With best regards,
Alexander Korotkov.