Re: AWS and postgres issues

On Mon, Apr 08, 2013 at 02:59:56PM -0700, David Kerr wrote:
- On Mon, Apr 08, 2013 at 02:09:42PM -0700, David Kerr wrote:
- - On Mon, Apr 08, 2013 at 02:14:14PM -0600, Quentin Hartman wrote:
- - - What version of pgpool are you using?
- - -
- - - Are there other commands you have a problem with? I would suspect that the
- - - restart is causing the postgres server to go away, pgpool decides to
- - - disconnect, and then it has to be manually added back to the cluster.
- - - Unless of course you've got automatic failback setup, but even then I would
- - - expect that command to do weird things when issued through middleware like
- - - pgpool, regardless of what sort of infrastructure you are running on.
- - -
- - - QH
- -
- - This is actually from the command line, PgPool isn't involved at all. I just
- - mentioned it to give some context.
-
- I had a brief conversation with Quentin offline which indicated that I wasn't
- being nearly clear nor direct enough.
-
- I believe that this probelm is specific to AWS+Postgres. (and possibly specific to
- VPC+Amazon Linux). Non postgres commands run fine, and even psql works fine.
- so far just pg_ctl fails.
-
- I'm running the command directly from an interactive shell, so this is as basic as it gets.
-
- The command runs correctly on the remote server, it just never exits the ssh connection.
- specifically I never get: "debug2: channel 0: rcvd close " as if the packet gets
- dropped every time.
-
- I don't believe it's a network hiccup because I can reproduce it every time.
-
- It's likely something with Amazon's infrastructure that's eating it, but whateever
- it is, it seems to specifically not like pg_ctl.
-
- Here is what I see happen:
- [pgpool(at)ccpgp05 ~]$ ssh -vvv postgres(at)10(dot)0(dot)1(dot)30 '/usr/pgsql-9.2/bin/pg_ctl -D /db/pg -m fast restart'
- OpenSSH_6.1p1, OpenSSL 1.0.1e-fips 11 Feb 2013
- [..snip..]
- debug1: Sending command: /usr/pgsql-9.2/bin/pg_ctl -D /db/pg -m fast restart
- [..snip..]
- waiting for server to shut down.... done
- server stopped
- server starting
- [..snip..]
- debug1: client_input_channel_req: channel 0 rtype exit-status reply 0
- debug1: client_input_channel_req: channel 0 rtype eow(at)openssh(dot)com reply 0
- debug2: channel 0: rcvd eow
- debug2: channel 0: close_read
- debug2: channel 0: input open -> closed
- ^Cdebug1: channel 0: free: client-session, nchannels 1 # <---- This is where I ^C it
- debug3: channel 0: status: The following connections are open:
- #0 client-session (t4 r0 i3/0 o0/0 fd -1/5 cc -1)
-
-
- Notice the input open -> closed is where it basically hangs
-
- Now look at:
- [pgpool(at)ccpgp05 ~]$ ssh -vvv postgres(at)10(dot)0(dot)1(dot)30 'ls -ltr'
- OpenSSH_6.1p1, OpenSSL 1.0.1e-fips 11 Feb 2013
- [..snip..]
- debug1: Sending command: ls -ltr
- debug2: channel 0: output open -> drain
- debug1: channel 0: forcing write
- total 8
- drwx------ 4 postgres postgres 4096 Apr 4 22:50 9.2
- drwx------ 2 postgres postgres 4096 Apr 5 22:58 bin
- [..snip..]
- debug2: channel 0: input open -> closed
- debug2: channel 0: rcvd close
- debug3: channel 0: will not send data after close
- debug2: channel 0: almost dead
- debug2: channel 0: gc: notify user
- debug2: channel 0: gc: user detached
- debug2: channel 0: send close
- debug2: channel 0: is dead
- debug2: channel 0: garbage collecting
- debug1: channel 0: free: client-session, nchannels 1
- debug3: channel 0: status: The following connections are open:
- #0 client-session (t4 r0 i3/0 o3/0 fd -1/-1 cc -1)
-
- Transferred: sent 2456, received 2448 bytes, in 0.0 seconds
- Bytes per second: sent 105666.4, received 105322.3
- debug1: Exit status 0

I've verified that it's not related to the linux flavor. I've tried it with a Server on both Amazon Linux
and Ubuntu. And with a client on Amazon Linux and my own desktop.

I can't be the only person using PG in AWS+VPC, can someone else with a similar test bed give it a shot
and tell me if it works for them? (at least then I'd know if it's likely something I'm doing...)

Thanks