psql hangs when trying to access a database

Running into an interesting problem when trying to do a psql from the
commandline.

My infrastructure:
Intel Architecture
Redhat 7.1
PostgreSQL 7.0.3 (from stock RPM's of the 7.1 distribution)
320MB RAM

Kernel shared memory increases:
# Up the maximum shared memory values
kernel.shmall = 207618048
kernel.shmmax = 207618048

pg_ctl startup in /etc/rc.d/init.d/postgresql:
su -l postgres -c "/usr/bin/pg_ctl -D $PGDATA -p /usr/bin/postmaster -o
'-d 5 -i -B 24576' start >/tmp/postgres.log 2>&1" < /dev/null

Database Environment
5 databases on this one server. The one in question has about 5 tables
with 60,000 rows per table average. The entire directory under $PGDATA
is only about 160MB

At the command line I type 'psql datawarehouse'. I expect to see the
banner from the psql command but in this case, nothing. The system
seems to just hang. Running a top shows that my process is taking up
99% of the User CPU but according to iostat, there is no activity on the
harddrive.

Here's what the 'ps -awx|grep post' command shows me:
3073 pts/1 S 0:00 /bin/sh /usr/bin/pg_ctl -D /db/pgsql -p
/usr/bin/postmaster -o -d 5 -i -B 24576 start
3075 pts/1 S 0:00 /usr/bin/postmaster -d 5 -i -B 24576
3085 pts/1 R 0:11 /usr/bin/postgres localhost mnickel
datawarehouse startup

generally I would expect the /usr/bin/postgres line to eventually change
to 'idle' instead of startup...

Also, pg_dump and vacuumdb do not work. The postmaster happily spawns
off another /usr/bin/postgres, but the state never switches out of
'idle' for pg_dump or vacuumdb.

Upon inspection of the /tmp/postgres.log file, (included at the end), I
see that the processing is hanging right after the 'InitPostgres'.
Normally there is a bunch of other "stuff" that is there when normal
entry into the psql command completes...

I have no problem using psql on any of the other 5 databases on this
server. Everything runs smoothly. This is the only database that is
giving us grief. This is the first problem that we've come across in
almost a year of PostgreSQL utilization. I'm totally baffled... :)

Thanks so much for looking at this, as I'm getting the "see, I told you
that open source/freeware stuff wasn't any good" pressure. :)

Mark Nickel

Here is the logfile in question:

postmaster successfully started up.
FindExec: found "/usr/bin/postgres" using argv[0]
binding ShmemCreate(key=52e2c1, size=205273088)
011105.20:22:27.451 [3076] DEBUG: Data Base System is starting up at
Mon Nov 5 20:22:27 2001
011105.20:22:27.451 [3076] DEBUG: Data Base System was shut down at
Mon Nov 5 20:22:13 2001
011105.20:22:27.453 [3076] DEBUG: Data Base System is in production
state at Mon Nov 5 20:22:27 2001
011105.20:22:27.453 [3076] proc_exit(0)
011105.20:22:27.454 [3076] shmem_exit(0)
011105.20:22:27.454 [3076] exit(0)
/usr/bin/postmaster: reaping dead processes...
/usr/bin/postmaster: ServerLoop: handling reading 5
/usr/bin/postmaster: ServerLoop: handling reading 5
/usr/bin/postmaster: ServerLoop: handling writing 5
/usr/bin/postmaster: BackendStartup: environ dump:
-----------------------------------------
PWD=/var/lib/pgsql
HOSTNAME=topaz.dunsirn.com
ODBCINI=/etc/odbc.ini
PVM_RSH=/usr/bin/rsh
QTDIR=/usr/lib/qt-2.3.0
LESSOPEN=|/usr/bin/lesspipe.sh %s
XPVM_ROOT=/usr/share/pvm3/xpvm
KDEDIR=/usr
USER=postgres
LS_COLORS=no=00:fi=00:di=01;34:ln=01;36:pi=40;33:so=01;35:bd=40;33;01:cd=40;33;01:or=01;05;37;41:mi=01;05;37;41:ex=01;32:*.cmd=01;32:*.exe=01;32:*.com=01;32:*.btm=01;32:*.bat=01;32:*.sh=01;32:*.csh=01;32:*.tar=01;31:*.tgz=01;31:*.arj=01;31:*.taz=01;31:*.lzh=01;31:*.zip=01;31:*.z=01;31:*.Z=01;31:*.gz=01;31:*.bz2=01;31:*.bz=01;31:*.tz=01;31:*.rpm=01;31:*.cpio=01;31:*.jpg=01;35:*.gif=01;35:*.bmp=01;35:*.xbm=01;35:*.xpm=01;35:*.png=01;35:*.tif=01;35:

MACHTYPE=i386-redhat-linux-gnu
MAIL=/var/spool/mail/postgres
INPUTRC=/etc/inputrc
LANG=en_US
LOGNAME=postgres
SHLVL=2
MQSERVER=OPAL.AMBER/TCP/opal(1414)
SHELL=/bin/bash
PGLIB=/usr/lib/pgsql
HOSTTYPE=i386
OSTYPE=linux-gnu
HISTSIZE=1000
LAMHELPFILE=/etc/lam/lam-helpfile
PVM_ROOT=/usr/share/pvm3
TERM=xterm
HOME=/var/lib/pgsql
SSH_ASKPASS=/usr/libexec/openssh/gnome-ssh-askpass
PGDATA=/db/pgsql
PATH=/usr/kerberos/bin:/bin:/usr/bin:/usr/local/bin:/usr/bin/X11:/usr/X11R6/bin

_=/usr/bin/postmaster
POSTPORT=5432
POSTID=2147483646
IPC_KEY=5432000
PG_USER=mnickel
-----------------------------------------
/usr/bin/postmaster: BackendStartup: pid 3085 user mnickel db
datawarehouse socket 5
/usr/bin/postmaster child[3085]: starting with (/usr/bin/postgres -d5 -B
24576 -v131072 -p datawarehouse )
FindExec: found "/usr/bin/postgres" using argv[0]
011105.20:22:46.186 [3085] started: host=localhost user=mnickel
database=datawarehouse
011105.20:22:46.187 [3085] InitPostgres
011105.20:27:45.645 [3075] pmdie 15
Smart Shutdown request at Mon Nov 5 20:27:45 2001
/usr/bin/pg_ctl: line 1: 3075 Killed
/usr/bin/postmaster -d 5 -i -B 24576