Re: [ANNOUNCE] IMCS: In Memory Columnar Store for PostgreSQL

On 01/04/2014 12:05 PM, David Fetter wrote:
> I'm sorry I misunderstood about the extension you wrote.
>
> Is there some way not to use shared memory for it?

No, IMCS ("In-Memory Columnar Store") is storing data in shared memory.
Certainly I could allocate shared memory myself, but due to portability
and easy maintenance reasons I decided to reuse PostgreSQL mechanism of
shared memory. The only requirement is that IMSC extension (as well as
pg_stat_statements extension) should be included in
"shared_preload_libraries" list in postgresql.conf.

IMCS memory is not somehow interleave with shared memory used for
PostgreSQL shared buffers.
And the only limitation is this 2567Gb limit at Linux, which can be
resolved using the patch included in IMCS distributive.

> Cheers,
> David.
>
> On Sat, Jan 04, 2014 at 11:46:25AM +0400, knizhnik wrote:
>> Hi David,
>>
>> Sorry, but I do not completely understand your suggestions:
>>
>> 1. IMCS really contains single patch file sysv_shmem.patch.
>> Applying this patch is not mandatory for using IMCS: it just solves
>> the problem with support of > 256Gb of shared memory.
>> Right now PostgreSQL is not able to use more than 256Gb shared
>> buffers at Linux with standard 4kb pages.
>> I have found proposal for using MAP_HUGETLB flag in commit fest:
>>
>> http://www.postgresql.org/message-id/20131125032920.GA23793@toroid.org
>>
>> but unfortunately it was rejected. Hugepages are intensively used by
>> Oracle and I think that them will be useful for improving
>> performance of PorstreSQL. So not just IMCS can benefit from this
>> patch. My patch is much more simple - I specially limited scope of
>> this patch to one file. Certainly switch huge tlb on/off should be
>> done through postgresql.conf configuration file.
>>
>> In any case - IMCS can be used without this patch: you just could
>> not use more than 256Gb memory, even if your system has more RAM.
>>
>> 2. I do not understand "The add-on is not formatted as an EXTENSION"
>> IMCS was created as standard extension - I just look at the examples
>> of other PostgreSQL extensions included in PostgreSQL distribution
>> (for example pg_stat_statements). It can be added using "create
>> extension imcs" and removed "drop extension imcs" commands.
>>
>> If there are some violations of PostgreSQL extensions rules, please
>> let me know, I will fix them.
>> But I thought that I have done everything in legal way.
>>
>>
>>
>>
>>
>>
>> On 01/04/2014 03:21 AM, David Fetter wrote:
>>> On Thu, Jan 02, 2014 at 08:48:24PM +0400, knizhnik wrote:
>>>> I want to announce implementation of In-Memory Columnar Store
>>>> extension for PostgreSQL.
>>>> Vertical representation of data is stored in PostgreSQL shared memory.
>>> Thanks for the hard work!
>>>
>>> I noticed a couple of things about this that probably need some
>>> improvement.
>>>
>>> 1. There are unexplained patches against other parts of PostgreSQL,
>>> which means that they may break other parts of PostgreSQL in equally
>>> inexplicable ways. Please rearrange the patch so it doesn't require
>>> this. This leads to:
>>>
>>> 2. The add-on is not formatted as an EXTENSION, which would allow
>>> people to add it or remove it cleanly.
>>>
>>> Would you be so kind as to fix these?
>>>
>>> Cheers,
>>> David.