Re: [SOLVED] Re: Hung postmaster (8.3.9)

On Monday 01 March 2010 @ 18:29, Tom Lane wrote:
> "Ed L." <pgsql(at)bluepolka(dot)net> writes:
> > That did the trick. Thank you very much, Sensei.
>
> I'd still like to know about platform etc. I see that we
> shouldn't be allowing a username to trigger @-file expansion,
> but even with that it's not clear how it turned into an
> infinite loop; and as I said, I can't reproduce that here.

The original problem occurred on that CentOS box, but I just
reproduced it on a new cluster on Ubuntu running LVM but no
RAID. Here's the transcript:

ed(at)duke:~/test$ uname -a
Linux duke 2.6.31-15-generic #50-Ubuntu SMP Tue Nov 10 14:53:52 UTC 2009 x86_64 GNU/Linux

ed(at)duke:~/test$ export PGPORT=9000
ed(at)duke:~/test$ w^C
ed(at)duke:~/test$ svi
ed(at)duke:~/test$ export PGDATA=`pwd`/data
ed(at)duke:~/test$ initdb --locale=C -D $PGDATA
The files belonging to this database system will be owned by user "ed".
This user must also own the server process.

The database cluster will be initialized with locale C.
The default database encoding has accordingly been set to SQL_ASCII.
The default text search configuration will be set to "english".

creating directory /home/ed/test/data ... ok
creating subdirectories ... ok
selecting default max_connections ... 100
selecting default shared_buffers/max_fsm_pages ... 32MB/204800
creating configuration files ... ok
creating template1 database in /home/ed/test/data/base/1 ... ok
initializing pg_authid ... ok
initializing dependencies ... ok
creating system views ... ok
loading system objects' descriptions ... ok
creating conversions ... ok
creating dictionaries ... ok
setting privileges on built-in objects ... ok
creating information schema ... ok
vacuuming database template1 ... ok
copying template1 to template0 ... ok
copying template1 to postgres ... ok

WARNING: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the -A option the
next time you run initdb.

Success. You can now start the database server using:

postgres -D /home/ed/test/data
or
pg_ctl -D /home/ed/test/data -l logfile start

ed(at)duke:~/test$ pg_ctl -D /home/ed/test/data -l logfile start
server starting
ed(at)duke:~/test$ psql -c "select version()"
psql: FATAL: database "ed" does not exist
ed(at)duke:~/test$ create^C
ed(at)duke:~/test$ createdb
ed(at)duke:~/test$ psql -c "select version()"
version
-------------------------------------------------------------------------------------------------
PostgreSQL 8.3.9 on x86_64-unknown-linux-gnu, compiled by GCC gcc (Ubuntu 4.4.1-4ubuntu8) 4.4.1
(1 row)

ed(at)duke:~/test$ createuser "@" with login
createuser: too many command-line arguments (first is "with")
Try "createuser --help" for more information.
ed(at)duke:~/test$ psql -c "create user \"@\" with login"
CREATE ROLE
ed(at)duke:~/test$ cat data/global/
1136 1214 1260 2396 2672 2694 2698 2844 2847 pg_database
1137 1232 1261 2397 2676 2695 2842 2845 pg_auth pgstat.stat
1213 1233 1262 2671 2677 2697 2843 2846 pg_control
ed(at)duke:~/test$ cat data/global/pg_auth
"@" "" ""
"ed" "" ""
ed(at)duke:~/test$ pg_ctl -D $PGDATA stop
waiting for server to shut down...........................................^C
ed(at)duke:~/test$ ^C
ed(at)duke:~/test$ ^C
ed(at)duke:~/test$ ^C
ed(at)duke:~/test$ pg_ctl -D $PGDATA -m fast stop
waiting for server to shut down.............^C
ed(at)duke:~/test$ ^C
ed(at)duke:~/test$ ps augxwwwf | grep postma
ed 8419 0.0 0.0 7336 868 pts/4 S+ 18:36 0:00 \_ grep postma
ohsdba 6115 0.0 0.3 503692 28028 pts/0 S 18:03 0:00 postmaster -D /users/ohsdba/dbclusters/ohs/postgresql-8.3.x/data -i -p 9001
ed(at)duke:~/test$ gdb `which postgre^C
ed(at)duke:~/test$ pg_ctl -D /home/ed/test/data -l logfile start
pg_ctl: another server might be running; trying to start server anyway
pg_ctl: could not start server
Examine the log output.
ed(at)duke:~/test$ cat logfile
LOG: database system was shut down at 2010-03-01 18:34:25 MST
LOG: autovacuum launcher started
LOG: database system is ready to accept connections
FATAL: database "ed" does not exist
WARNING: worker took too long to start; cancelled
FATAL: lock file "postmaster.pid" already exists
HINT: Is another postmaster (PID 8203) running in data directory "/home/ed/test/data"?
ed(at)duke:~/test$ ps augxwwwf | grep 8203
ed 8465 0.0 0.0 7336 868 pts/4 S+ 18:37 0:00 \_ grep 8203
ed 8203 73.7 0.0 63080 4248 pts/4 R 18:34 2:07 /opt/pgsql/installs/postgresql-8.3.9/bin/postgres -D /home/ed/test/data
ed(at)duke:~/test$ gdb `which postgres`
GNU gdb (GDB) 7.0-ubuntu
Copyright (C) 2009 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>...
Reading symbols from /opt/pgsql/installs/postgresql-8.3.9/bin/postgres...done.
(gdb) attach 8203
Attaching to program: /opt/pgsql/installs/postgresql-8.3.9/bin/postgres, process 8203
Reading symbols from /lib/libcrypt.so.1...(no debugging symbols found)...done.
Loaded symbols for /lib/libcrypt.so.1
Reading symbols from /lib/libdl.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib/libdl.so.2
Reading symbols from /lib/libm.so.6...(no debugging symbols found)...done.
Loaded symbols for /lib/libm.so.6
Reading symbols from /lib/libc.so.6...(no debugging symbols found)...done.
Loaded symbols for /lib/libc.so.6
Reading symbols from /lib64/ld-linux-x86-64.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib64/ld-linux-x86-64.so.2
Reading symbols from /lib/libnss_compat.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib/libnss_compat.so.2
Reading symbols from /lib/libnsl.so.1...(no debugging symbols found)...done.
Loaded symbols for /lib/libnsl.so.1
Reading symbols from /lib/libnss_nis.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib/libnss_nis.so.2
Reading symbols from /lib/libnss_files.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib/libnss_files.so.2
0x00007f3c58f9ea30 in read () from /lib/libc.so.6
(gdb) bt
#0 0x00007f3c58f9ea30 in read () from /lib/libc.so.6
#1 0x00007f3c58f3ead8 in _IO_file_underflow () from /lib/libc.so.6
#2 0x00007f3c58f4065e in _IO_default_uflow () from /lib/libc.so.6
#3 0x00007f3c58f3741b in getc () from /lib/libc.so.6
#4 0x00000000005460d8 in T.136 ()
#5 0x00000000005466e4 in tokenize_file ()
#6 0x00000000005467ed in tokenize_file ()
#7 0x0000000000546cef in load_role ()
#8 0x00000000005937ed in sigusr1_handler ()
#9 <signal handler called>
#10 0x00007f3c58fa5373 in select () from /lib/libc.so.6
#11 0x0000000000591cc2 in ServerLoop ()
#12 0x0000000000594467 in PostmasterMain ()
#13 0x000000000054a9b0 in main ()
(gdb) fin
Run till exit from #0 0x00007f3c58f9ea30 in read () from /lib/libc.so.6
0x00007f3c58f3ead8 in _IO_file_underflow () from /lib/libc.so.6
(gdb) fin
Run till exit from #0 0x00007f3c58f3ead8 in _IO_file_underflow () from /lib/libc.so.6
0x00007f3c58f4065e in _IO_default_uflow () from /lib/libc.so.6
(gdb) fin
Run till exit from #0 0x00007f3c58f4065e in _IO_default_uflow () from /lib/libc.so.6
0x00007f3c58f3741b in getc () from /lib/libc.so.6
(gdb) fin
Run till exit from #0 0x00007f3c58f3741b in getc () from /lib/libc.so.6
0x00000000005460d8 in T.136 ()
(gdb) fin
Run till exit from #0 0x00000000005460d8 in T.136 ()
0x00000000005466e4 in tokenize_file ()
(gdb) fin
Run till exit from #0 0x00000000005466e4 in tokenize_file ()

(Infinite loop here...)

Program terminated with signal SIGKILL, Killed.
The program no longer exists.
(gdb)
The program is not running.
(gdb)
The program is not running.
(gdb) quit
ed(at)duke:~/test$ uname -a
Linux duke 2.6.31-15-generic #50-Ubuntu SMP Tue Nov 10 14:53:52 UTC 2009 x86_64 GNU/Linux
ed(at)duke:~/test$ uname -a
Linux duke 2.6.31-15-generic #50-Ubuntu SMP Tue Nov 10 14:53:52 UTC 2009 x86_64 GNU/Linux
ed(at)duke:~/test$ df
Filesystem 1K-blocks Used Available Use% Mounted on
/dev/mapper/duke-root
468727296 229717260 215199988 52% /
udev 4096736 320 4096416 1% /dev
none 4096736 1280 4095456 1% /dev/shm
none 4096736 156 4096580 1% /var/run
none 4096736 0 4096736 0% /var/lock
none 4096736 0 4096736 0% /lib/init/rw
/dev/sda5 233335 47309 173578 22% /boot
/dev/sr0 388 388 0 100% /media/cdrw