Bottled Water: PostgreSQL to Kafka replication

Hi PostgreSQL world,

I'd like to announce a new open source project, called "Bottled Water", for getting data from PostgreSQL into Kafka:
http://blog.confluent.io/2015/04/23/bottled-water-real-time-integration-of-postgresql-and-kafka/
https://github.com/confluentinc/bottledwater-pg/

In case you're not aware of Kafka (http://kafka.apache.org/) it's an open source message broker that was originally developed at LinkedIn and is now a lively Apache project. Unlike many other messaging systems (AMQP, JMS etc), it is structured as a commit log, which makes it well suited for replicating data from one system to another.

Bottled Water uses PostgreSQL 9.4's logical decoding feature to extract a consistent snapshot of a database, plus an ongoing stream of logical changes. Data is encoded in Avro (http://avro.apache.org/) a language-independent serialization format, with schemas that are automatically derived from the PostgreSQL table schemas. Once the data is in Kafka, it's easier to import into downstream systems, such as full-text search indexes, caches, data warehouses, stream analytics systems, auditing and monitoring tools, etc.

The blog post above has more detail on the design and the rationale behind it. This is an alpha release that is not yet fit for production use, but it's ready for experimentation. Feedback and contributions welcome!

Martin