Re: Server hangs on multiple connections

On Thursday, Sep 19, 2002, at 18:33 US/Eastern, Tom Lane wrote:

> David Christian <davidc(at)comtechmobile(dot)com> writes:
>> Happy to. Interestingly, when I build with --enable-debug and
>> --enable-cassert, the server doesn't lock up during 'make check', it
>> just (very quickly) fails all of the tests and exits. I tried several
>> times.
>
> Oh, that's interesting; that says that an Assert() check is failing.
> We should investigate that first.
>
> There should be a core file left in the database subdirectory after
> the assert failure --- would you gdb it and get a stack trace from it?
> Also, you will probably find some useful messages in the postmaster
> log (which should be left in the log/ subdirectory of the regress
> tests)

Unfortunately, I see no core file under the source tree after the
assert failure.

The postmaster.log does show entries for failed assertions. It is 246
lines long, and I am pasting it to the bottom of this message.

>> (gdb) bt
>> #0 0x0fdc297c in __syscall_ipc () at soinit.c:76
>> #1 0x0fdc38c0 in semop (semid=4, sops=0x7fffea18, nsops=1) at
>> ../sysdeps/unix/sysv/linux/semop.c:36
>> #2 0x100e4424 in IpcSemaphoreLock ()
>> #3 0x100eb018 in LWLockAcquire ()
>> #4 0x100e7f3c in LockAcquire ()
>> #5 0x100e7434 in LockRelation ()
>
> Sure enough, it would seem that everyone's stuck waiting for a lock.
> But let's chase the Assert first; that might identify the problem.

Okay, hope this helps. I really appreciate the time you are taking to
look at this.

David

[davidc(at)chef ~/src/PostgreSQL/postgresql-7.2.2]$ find . -name '*core*'
./contrib/retep/uk/org/retep/xml/core
./src/interfaces/jdbc/org/postgresql/core

[davidc(at)chef ~/src/PostgreSQL/postgresql-7.2.2/src/test/regress/log]$
cat postmaster.log
DEBUG: database system was shut down at 2002-09-20 02:46:51 GMT
DEBUG: checkpoint record is at 0/113640
DEBUG: redo record is at 0/113640; undo record is at 0/0; shutdown TRUE
DEBUG: next transaction id: 89; next oid: 16556
DEBUG: database system is ready
ERROR: DROP GROUP: group "regressgroup1" does not exist
TRAP: Failed Assertion("!(lock->shared > 0):", File: "lwlock.c", Line:
434)
!(lock->shared > 0) (0) [Success]
DEBUG: server process (pid 22628) was terminated by signal 6
DEBUG: terminating any other active server processes
NOTICE: Message from PostgreSQL backend:
The Postmaster has informed me that some other backend
died abnormally and possibly corrupted shared memory.
I have rolled back the current transaction and am
going to terminate your database system connection and exit.
Please reconnect to the database system and repeat your query.
NOTICE: Message from PostgreSQL backend:
The Postmaster has informed me that some other backend
died abnormally and possibly corrupted shared memory.
I have rolled back the current transaction and am
going to terminate your database system connection and exit.
Please reconnect to the database system and repeat your query.
NOTICE: Message from PostgreSQL backend:
The Postmaster has informed me that some other backend
died abnormally and possibly corrupted shared memory.
I have rolled back the current transaction and am
going to terminate your database system connection and exit.
Please reconnect to the database system and repeat your query.
FATAL 1: The database system is in recovery mode
NOTICE: Message from PostgreSQL backend:
The Postmaster has informed me that some other backend
died abnormally and possibly corrupted shared memory.
I have rolled back the current transaction and am
going to terminate your database system connection and exit.
Please reconnect to the database system and repeat your query.
NOTICE: Message from PostgreSQL backend:
The Postmaster has informed me that some other backend
died abnormally and possibly corrupted shared memory.
I have rolled back the current transaction and am
going to terminate your database system connection and exit.
Please reconnect to the database system and repeat your query.
NOTICE: Message from PostgreSQL backend:
The Postmaster has informed me that some other backend
died abnormally and possibly corrupted shared memory.
I have rolled back the current transaction and am
going to terminate your database system connection and exit.
Please reconnect to the database system and repeat your query.
DEBUG: all server processes terminated; reinitializing shared memory
and semaphores
DEBUG: database system was interrupted at 2002-09-20 02:46:51 GMT
DEBUG: checkpoint record is at 0/113640
DEBUG: redo record is at 0/113640; undo record is at 0/0; shutdown TRUE
DEBUG: next transaction id: 89; next oid: 16556
DEBUG: database system was not properly shut down; automatic recovery
in progress
DEBUG: redo starts at 0/113680
DEBUG: ReadRecord: record with zero length at 0/138818
DEBUG: redo done at 0/1387F0
FATAL 1: The database system is starting up
FATAL 1: The database system is starting up
FATAL 1: The database system is starting up
FATAL 1: The database system is starting up
FATAL 1: The database system is starting up
FATAL 1: The database system is starting up
FATAL 1: The database system is starting up
FATAL 1: The database system is starting up
FATAL 1: The database system is starting up
FATAL 1: The database system is starting up
FATAL 1: The database system is starting up
FATAL 1: The database system is starting up
FATAL 1: The database system is starting up
FATAL 1: The database system is starting up
FATAL 1: The database system is starting up
FATAL 1: The database system is starting up
FATAL 1: The database system is starting up
FATAL 1: The database system is starting up
FATAL 1: The database system is starting up
FATAL 1: The database system is starting up
FATAL 1: The database system is starting up
FATAL 1: The database system is starting up
FATAL 1: The database system is starting up
FATAL 1: The database system is starting up
FATAL 1: The database system is starting up
FATAL 1: The database system is starting up
FATAL 1: The database system is starting up
FATAL 1: The database system is starting up
FATAL 1: The database system is starting up
FATAL 1: The database system is starting up
FATAL 1: The database system is starting up
FATAL 1: The database system is starting up
FATAL 1: The database system is starting up
FATAL 1: The database system is starting up
FATAL 1: The database system is starting up
FATAL 1: The database system is starting up
FATAL 1: The database system is starting up
FATAL 1: The database system is starting up
FATAL 1: The database system is starting up
FATAL 1: The database system is starting up
FATAL 1: The database system is starting up
FATAL 1: The database system is starting up
FATAL 1: The database system is starting up
FATAL 1: The database system is starting up
FATAL 1: The database system is starting up
FATAL 1: The database system is starting up
FATAL 1: The database system is starting up
FATAL 1: The database system is starting up
FATAL 1: The database system is starting up
FATAL 1: The database system is starting up
FATAL 1: The database system is starting up
FATAL 1: The database system is starting up
FATAL 1: The database system is starting up
FATAL 1: The database system is starting up
FATAL 1: The database system is starting up
FATAL 1: The database system is starting up
FATAL 1: The database system is starting up
FATAL 1: The database system is starting up
FATAL 1: The database system is starting up
FATAL 1: The database system is starting up
FATAL 1: The database system is starting up
DEBUG: database system is ready
ERROR: CREATE USER: user name "regressuser4" already exists
NOTICE: ALTER GROUP: user "regressuser2" is already in group
"regressgroup2"
ERROR: atest2: Permission denied.
ERROR: atest2: Permission denied.
ERROR: atest2: Permission denied.
ERROR: atest2: Permission denied.
ERROR: LOCK TABLE: permission denied
ERROR: atest2: Permission denied.
ERROR: permission denied
ERROR: atest2: Permission denied.
ERROR: atest1: Permission denied.
ERROR: atest2: Permission denied.
ERROR: atest1: Permission denied.
ERROR: atest1: Permission denied.
ERROR: atest2: Permission denied.
ERROR: atest1: Permission denied.
ERROR: atest2: Permission denied.
ERROR: atest2: Permission denied.
ERROR: atest2: Permission denied.
ERROR: atest2: Permission denied.
ERROR: atest2: Permission denied.
ERROR: atest3: Permission denied.
ERROR: has_table_privilege: relation "pg_shad" does not exist
ERROR: user "nosuchuser" does not exist
ERROR: has_table_privilege: invalid privilege type sel
ERROR: pg_aclcheck: invalid user id 4293967297
ERROR: has_table_privilege: invalid relation oid 1
ERROR: Relation "onek" does not exist
ERROR: Relation "onek" does not exist
ERROR: Relation "tmp" does not exist
ERROR: Relation "tmp" does not exist
ERROR: table "tmp" does not exist
ERROR: Relation "onek" does not exist
ERROR: Relation "onek" does not exist
ERROR: Relation "onek" does not exist
ERROR: Relation "onek" does not exist
ERROR: Relation "onek2" does not exist
ERROR: Relation "onek2" does not exist
ERROR: Relation "onek2" does not exist
ERROR: Relation "stud_emp" does not exist
ERROR: Relation "stud_emp" does not exist
ERROR: Relation "stud_emp" does not exist
ERROR: Relation "stud_emp" does not exist
ERROR: Relation "a_star" does not exist
ERROR: Relation "b_star" does not exist
ERROR: Relation "c_star" does not exist
ERROR: Relation "d_star" does not exist
ERROR: Relation "e_star" does not exist
ERROR: Relation "f_star" does not exist
ERROR: Relation "a_star" does not exist
ERROR: Relation "a_star" does not exist
ERROR: Relation "f_star" does not exist
ERROR: Relation "e_star" does not exist
ERROR: Relation "d_star" does not exist
ERROR: Relation "c_star" does not exist
ERROR: Relation "b_star" does not exist
ERROR: Relation "a_star" does not exist
ERROR: Relation "a_star" does not exist
ERROR: Relation "a_star" does not exist
ERROR: Relation "a_star" does not exist
ERROR: Relation "a_star" does not exist
ERROR: Relation "a_star" does not exist
ERROR: Relation "f_star" does not exist
ERROR: Relation "f_star" does not exist
ERROR: Relation "e_star" does not exist
ERROR: Relation "e_star" does not exist
ERROR: Relation "a_star" does not exist
ERROR: Relation "a_star" does not exist
ERROR: Relation "person" does not exist
ERROR: Relation "person" does not exist
ERROR: Relation "hobbies_r" does not exist
ERROR: Relation "hobbies_r" does not exist
ERROR: Relation "person" does not exist
ERROR: Relation "person" does not exist
ERROR: Relation "person" does not exist
ERROR: Relation "person" does not exist
ERROR: Relation "person" does not exist
ERROR: Relation "person" does not exist
ERROR: Function 'user_relns()' does not exist
Unable to identify a function that satisfies the given argument
types
You may need to add explicit typecasts
ERROR: Function 'hobbies_by_name(unknown)' does not exist
Unable to identify a function that satisfies the given argument
types
You may need to add explicit typecasts
ERROR: Function 'oldstyle_length(int4, text)' does not exist
Unable to identify a function that satisfies the given argument
types
You may need to add explicit typecasts
ERROR: Relation "street" does not exist
ERROR: Relation "iexit" does not exist
ERROR: Relation "toyemp" does not exist
TRAP: Failed Assertion("!(lock->shared > 0):", File: "lwlock.c", Line:
434)
!(lock->shared > 0) (0) [Interrupted system call]
DEBUG: server process (pid 23536) was terminated by signal 6
DEBUG: terminating any other active server processes
NOTICE: Message from PostgreSQL backend:
The Postmaster has informed me that some other backend
died abnormally and possibly corrupted shared memory.
I have rolled back the current transaction and am
going to terminate your database system connection and exit.
Please reconnect to the database system and repeat your query.
NOTICE: Message from PostgreSQL backend:
The Postmaster has informed me that some other backend
died abnormally and possibly corrupted shared memory.
I have rolled back the current transaction and am
going to terminate your database system connection and exit.
Please reconnect to the database system and repeat your query.
NOTICE: Message from PostgreSQL backend:
The Postmaster has informed me that some other backend
died abnormally and possibly corrupted shared memory.
I have rolled back the current transaction and am
going to terminate your database system connection and exit.
Please reconnect to the database system and repeat your query.
DEBUG: all server processes terminated; reinitializing shared memory
and semaphores
DEBUG: database system was interrupted at 2002-09-20 02:46:54 GMT
DEBUG: checkpoint record is at 0/138818
DEBUG: redo record is at 0/138818; undo record is at 0/0; shutdown TRUE
DEBUG: next transaction id: 140; next oid: 24748
DEBUG: database system was not properly shut down; automatic recovery
in progress
DEBUG: redo starts at 0/138858
DEBUG: ReadRecord: record with zero length at 0/16D820
DEBUG: redo done at 0/16D7F8
FATAL 1: The database system is starting up
FATAL 1: The database system is starting up
FATAL 1: The database system is starting up
DEBUG: smart shutdown request
DEBUG: database system is ready
DEBUG: shutting down
DEBUG: database system is shut down