Bottled Water: PostgreSQL to Kafka replication

From: Martin Kleppmann <martin(at)kleppmann(dot)com>
To: pgsql-announce(at)postgresql(dot)org
Subject: Bottled Water: PostgreSQL to Kafka replication
Date: 2015-04-23 15:29:12
Message-ID: 797DF957-CE33-407F-99DB-7C7125E37ACE@kleppmann.com
Views: Raw Message | Whole Thread | Download mbox | Resend email
Thread:
Lists: pgsql-announce

Hi PostgreSQL world,

I'd like to announce a new open source project, called "Bottled Water", for getting data from PostgreSQL into Kafka:
http://blog.confluent.io/2015/04/23/bottled-water-real-time-integration-of-postgresql-and-kafka/
https://github.com/confluentinc/bottledwater-pg/

In case you're not aware of Kafka (http://kafka.apache.org/) it's an open source message broker that was originally developed at LinkedIn and is now a lively Apache project. Unlike many other messaging systems (AMQP, JMS etc), it is structured as a commit log, which makes it well suited for replicating data from one system to another.

Bottled Water uses PostgreSQL 9.4's logical decoding feature to extract a consistent snapshot of a database, plus an ongoing stream of logical changes. Data is encoded in Avro (http://avro.apache.org/) a language-independent serialization format, with schemas that are automatically derived from the PostgreSQL table schemas. Once the data is in Kafka, it's easier to import into downstream systems, such as full-text search indexes, caches, data warehouses, stream analytics systems, auditing and monitoring tools, etc.

The blog post above has more detail on the design and the rationale behind it. This is an alpha release that is not yet fit for production use, but it's ready for experimentation. Feedback and contributions welcome!

Martin

Browse pgsql-announce by date

  From Date Subject
Next Message David Fetter 2015-04-26 21:51:42 == PostgreSQL Weekly News - April 26 2015 ==
Previous Message Luis Dosso 2015-04-23 13:04:59 Call for papers - PGDay Campinas 2015 (Brazil)