Re: Horizontal scalability/sharding

> On Tue, Sep 1, 2015 at 9:48 PM, Robert Haas <robertmhaas(at)gmail(dot)com> wrote:
> >
> > On Tue, Sep 1, 2015 at 4:15 AM, Andres Freund <andres(at)anarazel(dot)de>
wrote:
> > > On 2015-08-31 20:54:51 -0400, Bruce Momjian wrote:
> > >> Uh, we already have a list of things we need to add to FDWs to make
them
> > >> work, and Citus Data has provided a document of more things that are
> > >> needed, https://goo.gl/vJWF85. I am not sure how much bigger a red
flag
> > >> you want to confirm that everyone agrees that major FDW improvements
are
> > >> a requirement for this.
> > >
> > > Several people saying that the FDW infrastructure isn't sufficient
right
> > > now is pretty far from implying that all of them agree that the FDW
API
> > > is the way to go.
> > >
> > > I'm not sure myself. If it works out it's going to save us some work
and
> > > make it more realistic to get there sometime not too far off. But I'm
> > > afraid that the resulting system will feel like our current
partitioning
> > > implemenentation. Yes, it kinda works, but it's hard to get started,
it
> > > doesn't support too many features and you're kind afraid your
relatives
> > > will see what you've done.
> >
> > I'm not averse to making the "connect to the remote nodes" part of
> > this solution use something other than the FDW infrastructure at some
> > point in time if somebody's prepared to build something better. On
> > the other hand, I think it's extremely clear that the FDW
> > infrastructure has a large amount of potential upon which we have
> > thoroughly failed to capitalize. Patches have already been written
> > for UPDATE/DELETE pushdown and for join pushdown.

Will pushing down writes (Update/Delete) sufficient to maintain sane locking
behaviour and deadlock detection that can occur during writes on multiple
shards? For example it could easily be the case where a single Update
statement could effect multiple shards and cause deadlock due to waits
across the nodes. Now unless we have some distributed lock manager or
some other way to know the information of locks that happens across
shards, it could be difficult to detect deadlocks.

With Regards,
Amit Kapila.
EnterpriseDB: http://www.enterprisedb.com