Re: BDR Rejoin of failed node, hangs.

Good Morning,

Has anyone had a moment to look at this?
It is a bit of a show stopper.

Thanks Steve

________________________________________
From: pgsql-general-owner(at)postgresql(dot)org <pgsql-general-owner(at)postgresql(dot)org> on behalf of Steve Pribyl <spribyl(at)akunacapital(dot)com>
Sent: Thursday, October 1, 2015 12:27 PM
To: pgsql-general(at)postgresql(dot)org
Subject: [GENERAL] BDR Rejoin of failed node, hangs.

Good Afternoon,

I am in the process of testing out BDR and am having problems rejoining after a simulated loss of one node. The join hangs while waiting to complete and there interesting errors in the logs.

Simulation process
After creating the database on both nodes.

Shutdown postgres and reset db on node 2
rm -rf /var/lib/postgresql/9.4/main/*
/usr/lib/postgresql/9.4/bin/initdb -D /var/lib/postgresql/9.4/main -A trust -U postgre

Clean up node 1.
select bdr.bdr_part_by_node_names('{node2}');
delete from bdr.bdr_nodes where node_status='k';

When I try to re-add node2
Using the NODE 2 Create sql again, it hangs
SELECT bdr.bdr_node_join_wait_for_ready(); hangs.

The log on node2 has errors
http://pastebin.com/8ZsTe5cG
55000: System identification mismatch between connection and slot
00000: worker process: bdr db: bdrdemo (PID 12042) exited with exit code 1

The log on node1 does not have any errors.
http://pastebin.com/njVJ9WX7

Both nodes show up in select * from bdr.bdr_nodes; on node1 and node2;

Database Creation:
NODE 1
create database bdrdemo;
\connect bdrdemo
CREATE EXTENSION btree_gist;
CREATE EXTENSION bdr;
select bdr.bdr_group_create(local_node_name := 'node1', node_external_dsn := 'host=192.168.101.41 port=5432 dbname=bdrdemo');
SELECT bdr.bdr_node_join_wait_for_ready();

NODE 2
create database bdrdemo;
\connect bdrdemo
CREATE EXTENSION btree_gist;
CREATE EXTENSION bdr;
select bdr.bdr_group_join(local_node_name := 'node2', node_external_dsn := 'host=192.168.101.42 port=5432 dbname=bdrdemo', join_using_dsn := 'host=192.168.101.41 port=5432 dbname=bdrdemo');
SELECT bdr.bdr_node_join_wait_for_ready();

Ubuntu 14.04 packages
ii postgresql-bdr-9.4 9.4.4-1trusty amd64 object-relational SQL database, version 9.4 server
ii postgresql-bdr-9.4-bdr-plugin 0.9.2-1trusty amd64 BDR Plugin for PostgreSQL-BDR 9.4
ii postgresql-bdr-client-9.4 9.4.4-1trusty amd64 front-end programs for PostgreSQL-BDR 9.4
ii postgresql-bdr-contrib-9.4 9.4.4-1trusty amd64 additional facilities for PostgreSQL
ii postgresql-client-common 154 all manager for multiple PostgreSQL client versions
ii postgresql-common 154 all PostgreSQL database-cluster manager

TIA,
Steve
________________________________
[http://www.akunacapital.com/images/akuna.png]
Steve Pribyl | Senior Systems Engineer
Akuna Capital LLC
36 S Wabash, Suite 310 Chicago IL 60603 USA | www.akunacapital.com <http://www.akunacapital.com>
p: +1 312 994 4646 | m: | f: +1 312 750 1667 | spribyl(at)akunacapital(dot)com

Please consider the environment, before printing this email.

This electronic message contains information from Akuna Capital LLC that may be confidential, legally privileged or otherwise protected from disclosure. This information is intended for the use of the addressee only and is not offered as investment advice to be relied upon for personal or professional use. Additionally, all electronic messages are recorded and stored in compliance pursuant to applicable SEC rules. If you are not the intended recipient, you are hereby notified that any disclosure, copying, distribution, printing or any other use of, or any action in reliance on, the contents of this electronic message is strictly prohibited. If you have received this communication in error, please notify us by telephone at (312)994-4640 and destroy the original message.

--
Sent via pgsql-general mailing list (pgsql-general(at)postgresql(dot)org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-general
________________________________
[http://www.akunacapital.com/images/akuna.png]
Steve Pribyl | Senior Systems Engineer
Akuna Capital LLC
36 S Wabash, Suite 310 Chicago IL 60603 USA | www.akunacapital.com <http://www.akunacapital.com>
p: +1 312 994 4646 | m: | f: +1 312 750 1667 | spribyl(at)akunacapital(dot)com

Please consider the environment, before printing this email.

This electronic message contains information from Akuna Capital LLC that may be confidential, legally privileged or otherwise protected from disclosure. This information is intended for the use of the addressee only and is not offered as investment advice to be relied upon for personal or professional use. Additionally, all electronic messages are recorded and stored in compliance pursuant to applicable SEC rules. If you are not the intended recipient, you are hereby notified that any disclosure, copying, distribution, printing or any other use of, or any action in reliance on, the contents of this electronic message is strictly prohibited. If you have received this communication in error, please notify us by telephone at (312)994-4640 and destroy the original message.