Re: WIP: Fast GiST index build

Looking at the calculation of levelStep:

> + /*
> + * Calculate levelStep by available amount of memory. We should be able to
> + * load into main memory one page of each underlying node buffer (which
> + * are in levelStep below). That give constraint over
> + * maintenance_work_mem. Also we should be able to have subtree of
> + * levelStep level in cache. That give constraint over
> + * effective_cache_size.
> + *
> + * i'th underlying level of sub-tree can consists of
> + * i^maxIndexTuplesPerPage pages at maximum. So, subtree of levelStep
> + * levels can't be greater then 2 * maxIndexTuplesPerPage ^ levelStep
> + * pages. We use some more reserve due to we probably can't take whole
> + * effective cache and use formula 4 * maxIndexTuplesPerPage ^ levelStep =
> + * effectiveCache. We use similar logic with maintenance_work_mem. We
> + * should be able to store at least last pages of all buffers where we are
> + * emptying current buffer to.
> + */
> + effectiveMemory = Min(maintenance_work_mem * 1024 / BLCKSZ,
> + effective_cache_size);
> + levelStep = (int) log((double) effectiveMemory / 4.0) /
> + log((double) maxIndexTuplesPerPage);
> +

I can see that that's equal to the formula given in the paper,
log_B(M/4B), but I couldn't see any explanation for that formula in the
paper. Your explanation makes sense, but where did it come from?

It seems a bit pessimistic: while it's true that the a subtree can't be
larger than 2 * maxIndexTuplesPerPage ^ levelStep, you can put a tighter
upper bound on it. The max size of a subtree of depth n can be
calculated as the geometric series:

r^0 + r^1 + r^2 + ... + r^n = (1 - r^(n + 1)) / (1 - r)

where r = maxIndexTuplesPerPage. For r=2 those formulas are equal, but
for a large r and small n (which is typical), the 2 *
maxIndexTuplesPerPage^levelStep formula gives a value that's almost
twice as large as the real max size of a subtree.

--
Heikki Linnakangas
EnterpriseDB http://www.enterprisedb.com