WIP: Access method extendability

From: Alexander Korotkov <aekorotkov(at)gmail(dot)com>
To: pgsql-hackers <pgsql-hackers(at)postgresql(dot)org>
Subject: WIP: Access method extendability
Date: 2014-10-15 12:08:38
Message-ID: CAPpHfdsXwZmojm6Dx+TJnpYk27kT4o7Ri6X_4OSWcByu1Rm+VA@mail.gmail.com
Views: Raw Message | Whole Thread | Download mbox | Resend email
Thread:
Lists: pgsql-hackers

Hackers,

Postgres was initially designed to support access methods extendability.
This extendability lives to present day. However, this is mostly internal
in-core extendability. One can quite easily add new access method into
PostgreSQL core. But if one try to implement access method as external
module, he will be faced with following difficulties:

1. Need to directly insert into pg_am, because of no "CREATE ACCESS
METHOD" command. And no support of dependencies between am and opclasses
etc.
2. Module can't define xlog records. So, new am would be not WAL-logged.

The first problem is purely mechanical. Nothing prevents us to implement
"CREATE ACCESS METHOD" and "DROP ACCESS METHOD" commands and support all
required dependencies.

Problem of WAL is a bit more complex. According to previous discussions, we
don't want to let extensions declare their own xlog records. If we let them
then recovery process will depend on extensions. That is much violates
reliability. Solution is to implement some generic xlog record which is
able to represent difference between blocks in some general manner.

3 patches are attached:

1. CREATE/DROP ACCESS METHOD commands. With support of dependencies.
2. New generic xlog record type.
3. Bloom contrib module as example of usage of previous two features.
This module was posted few years ago by Teodor Sigaev. Now, it's wrapped as
an extension and WAL-logged.

Patches are in WIP state. No documentation and very little of comments.
However, it believe that it's enough to do some general concept review.

Some notes about generic xlog format. Generic xlog represent difference
between pages as operations set of two kinds:

1. Move memory inside the page. Optional flag is to zero gap on a
previous memory location.
2. Copy memory fragment of memory from xlog record to page. As an option
bitwise logical operations are supported as well as plain copy.

Generic xlog can open page in two modes:

1. Create mode: page is zeroed independent on its lsn.
2. Update mode: page is updated only if it's lsn is lower than record lsn

Usually, xlog record is filled in critical sections when memory allocations
is prohibited. Thus, user have to previously initialize it with knowledge
of pages count, total operations count and total length of data.

------
With best regards,
Alexander Korotkov.

Attachment Content-Type Size
create-am.1.patch.gz application/x-gzip 6.3 KB
generic-xlog.1.patch.gz application/x-gzip 4.1 KB
bloom-contrib.1.patch.gz application/x-gzip 37.4 KB

Responses

Browse pgsql-hackers by date

  From Date Subject
Next Message Simon Riggs 2014-10-15 12:27:40 Re: Buffer Requests Trace
Previous Message MauMau 2014-10-15 12:03:42 Re: [9.4 bug] The database server hangs with write-heavy workload on Windows