Re: A better way than tweaking NTUP_PER_BUCKET

On Sun, Jun 23, 2013 at 2:45 AM, Simon Riggs <simon(at)2ndquadrant(dot)com> wrote:

> On 23 June 2013 03:16, Stephen Frost <sfrost(at)snowman(dot)net> wrote:
>
> > Will think on it more.
>
> Some other thoughts related to this...
>
> * Why are we building a special kind of hash table? Why don't we just
> use the hash table code that we in every other place in the backend.
> If that code is so bad why do we use it everywhere else? That is
> extensible, so we could try just using that. (Has anyone actually
> tried?)
>

Do you mean dynahash.c?

Dynahash has no provision to spill to disk gracefully, which seems like a
huge limit for this purpose, but not one for the purposes it currently
serves. Also, dynahash is built around use of C structs, not tuples, as
the things being stored. Perhaps that is not the mis-match I think it is?

>
> * We're not thinking about cache locality and set correspondence
> either. If the join is expected to hardly ever match, then we should
> be using a bitmap as a bloom filter rather than assuming that a very
> large hash table is easily accessible.
>
> * The skew hash table will be hit frequently and would show good L2
> cache usage. I think I'll try adding the skew table always to see if
> that improves the speed of the hash join.
>

Is there a good description of what the skew table is; and of the hash join
code in general? Last time I dug into nodeHash.c, I realized it really
needs a README file (unless there is already one I missed), or at least
some high level comments. Most of the comments in there seem to just
restate what the code itself already says, without explaining why it is
being done or how it fits into the big picture.

Cheers,

Jeff