Re: On columnar storage

On Fri, Jun 12, 2015 at 10:58 PM, Alvaro Herrera <alvherre(at)2ndquadrant(dot)com>
wrote:
>
> Amit Kapila wrote:
> > On Fri, Jun 12, 2015 at 4:33 AM, Alvaro Herrera <
alvherre(at)2ndquadrant(dot)com>
> > wrote:
> > > One critical detail is what will be used to identify a heap row when
> > > talking to a CS implementation. There are two main possibilities:
> > >
> > > 1. use CTIDs
> > > 2. use some logical tuple identifier
> > >
> > > Using CTIDs is simpler. One disadvantage is that every UPDATE of a
row
> > > needs to let the CS know about the new location of the tuple, so that
> > > the value is known associated with the new tuple location as well as
the
> > > old. This needs to happen even if the value of the column itself is
not
> > > changed.
> >
> > Isn't this somewhat similar to index segment?
>
> Not sure what you mean with "index segment".

The part similar to index segment is reference to heap for visibility
information and tuple id (TID). Have I misunderstood something?

> > Will the column store obey snapshot model similar to current heap
tuples,
> > if so will it derive the transaction information from heap tuple?
>
> Yes, visibility will be tied to the heap tuple -- a value is accessed
> only when its corresponding heap row has already been determined to be
> visible.

Won't it possible that all columns of a table belong to column-store?
I think for such a case heap will just be used to store transaction
information
(visibility info) for a column store tuple and depending on how the
column-store is organized, the reference to this information needs to be
stored in column-store (the same row reference might need to be stored for
each column value). Also any write operation could lead to much more
I/O because of updation at 2 different locations (one in column-store and
other in heap).

> One interesting point that raises from this is about vacuum:
> when are we able to remove a value from the store?

Yes, that could also be quite tricky to handle, may be one naive way
could be to make list of all TID's from heap that needs to be expired
and then search for references of all those TID's in column-store.

I understand your point for re-using the existing transaction infrastructure
for column-store by keeping that information in heap as it is done now,
but I think that won't be free either.

Another point to consider here is does the column-store needs
transactional consistency, do other commercial/opensource column-store
implementation's are transactional consistent and if yes, then can't we
think of doing it in a way where data could be present both in heap as well
as in column-store (I understand that it could lead to duplicate data,
OTOH, such an implementation anyway eliminates the need for indexes,
so may be worth considering).

With Regards,
Amit Kapila.
EnterpriseDB: http://www.enterprisedb.com