Partial index-based load balancing

From: Fabio Ugo Venchiarutti <fabio(at)vuole(dot)me>
To: pgsql-general(at)postgresql(dot)org
Subject: Partial index-based load balancing
Date: 2015-03-31 06:58:55
Message-ID: 551A45AF.6040509@vuole.me
Views: Raw Message | Whole Thread | Download mbox | Resend email
Thread:
Lists: pgsql-general

Greetings

I'm working for a startup and our core DB is growing rather fast.

Our target scale is large enough that we expect some of our core tables'
indexes to grow bigger than the memory on any single node over the next
couple years (our current intended design involves conventional
stream-replication-based, write-on-one-read-from-many load balancing).

We don't fancy the idea of using inheritance through partitioning due to
the maintenance overhead and our reliance on validation
constraints/triggers.

My proposal will be to instead create a number of partial indexes
covering predefined ranges of client IDs, then use a connection-level
routing mechanism that relies on what range the relevant client's data
belongs to in order to address the right node and match the right
partial index.

The idea is to have any given read-only node hold just one of the
partial indexes in its cache and never fetch index pages off its
secondary storage.
Scaling would just be a matter of increasing the partitioning density.

I'm going to assume that I'm not the first one to come up with this
strategy (and that there already is a name for it. If so, what is it?).

Is it a valid setup or am I missing some key aspect of how index
partitioning is meant to work?

TIA

Best regards

Fabio Ugo Venchiarutti

Responses

Browse pgsql-general by date

  From Date Subject
Next Message Mark Morgan Lloyd 2015-03-31 08:50:44 [No subject]
Previous Message Deven Phillips 2015-03-31 02:25:39 Re: Muti-table join and roll-up aggregate data into nested JSON?