Re: Bug in ecpg lib ?

Hi Laurenz,

I have now generate a rather small example where I experience the problem, attached. It is linked with the mudflapth library using the commands below. You may have to change the DBNAME and DBUSER. The delay just before the pthread_cancel(), i.e. sleep(10), is rather critical for the problem to appear and you might have to change it to something less. On some very slow machines I wasn't able to produce the problem.

$ ecpg crashex.pgc
$ /usr/local/Packages/gcc-4.4.0/bin/gcc -O0 -c -fmudflap -fmudflapth -fomit-frame-pointer -B/usr/local/Packages/gcc-4.4.0/bin/ -Wwrite-strings -std=gnu89 -ggdb -fPIC -Wall -I/usr/local/Packages/pgsql/include -I./Modules -I./ -o crashex.o crashex.c
$ /usr/local/Packages/gcc-4.4.0/bin/gcc -O0 -B/usr/local/Packages/gcc-4.4.0/bin/ -Wl -o crashex crashex.o -L/usr/local/Packages/pgsql/lib -lecpg -lpq -lmudflapth -lpthread -ldl

And this is the output from running the program:

leif$ LD_LIBRARY_PATH=/usr/local/Packages/gcc-4.4.0/lib/ ./crashex
Couldn't open somename(at)localhost:5432
2+2=0.
*** glibc detected *** /home/leif/tmp/crashex: free(): invalid pointer: 0x081f3958 ***
======= Backtrace: =========
/lib32/libc.so.6[0xf7c30615]
/lib32/libc.so.6(cfree+0x90)[0xf7c34080]
/usr/local/Packages/gcc-4.4.0/lib/libmudflapth.so.0(__real_free+0x3f1)[0xf7d39061]
/lib32/libecpg.so.6[0xf7e3fb5c]
/lib32/libpthread.so.0[0xf7d1bbb0]
/lib32/libpthread.so.0[0xf7d1c509]
/lib32/libc.so.6(clone+0x5e)[0xf7c9d08e]
======= Memory map: ========
08048000-0804a000 r-xp 00000000 08:0a 1671173 /home/leif/tmp/crashex
0804a000-0804b000 rwxp 00001000 08:0a 1671173 /home/leif/tmp/crashex
0804b000-081f8000 rwxp 0804b000 00:00 0 [heap]
f71eb000-f71ec000 ---p f71eb000 00:00 0
f71ec000-f79ec000 rwxp f71ec000 00:00 0
f79ec000-f79f5000 r-xp 00000000 08:01 97934 /lib32/libnss_files-2.7.so
f79f5000-f79f7000 rwxp 00008000 08:01 97934 /lib32/libnss_files-2.7.so
f79f7000-f79ff000 r-xp 00000000 08:01 97936 /lib32/libnss_nis-2.7.so
f79ff000-f7a01000 rwxp 00007000 08:01 97936 /lib32/libnss_nis-2.7.so
f7a01000-f7a15000 r-xp 00000000 08:01 97931 /lib32/libnsl-2.7.so
f7a15000-f7a17000 rwxp 00013000 08:01 97931 /lib32/libnsl-2.7.so
f7a17000-f7a19000 rwxp f7a17000 00:00 0
f7a19000-f7a20000 r-xp 00000000 08:01 97932 /lib32/libnss_compat-2.7.so
f7a20000-f7a22000 rwxp 00006000 08:01 97932 /lib32/libnss_compat-2.7.so
f7a22000-f7a2c000 r-xp 00000000 08:08 227374 /usr/lib32/libgcc_s.so.1
f7a2c000-f7a2d000 rwxp 0000a000 08:08 227374 /usr/lib32/libgcc_s.so.1
f7a2d000-f7a2f000 rwxp f7a2d000 00:00 0
f7a2f000-f7a38000 r-xp 00000000 08:01 97927 /lib32/libcrypt-2.7.so
f7a38000-f7a3a000 rwxp 00008000 08:01 97927 /lib32/libcrypt-2.7.so
f7a3a000-f7a61000 rwxp f7a3a000 00:00 0
f7a61000-f7b4a000 r-xp 00000000 08:01 98015 /lib32/libcrypto.so.0.9.7
f7b4a000-f7b5c000 rwxp 000e8000 08:01 98015 /lib32/libcrypto.so.0.9.7
f7b5c000-f7b5f000 rwxp f7b5c000 00:00 0
f7b5f000-f7b8d000 r-xp 00000000 08:01 98021 /lib32/libssl.so.0.9.7
f7b8d000-f7b90000 rwxp 0002d000 08:01 98021 /lib32/libssl.so.0.9.7
f7b90000-f7bb3000 r-xp 00000000 08:01 97929 /lib32/libm-2.7.so
f7bb3000-f7bb5000 rwxp 00023000 08:01 97929 /lib32/libm-2.7.so
f7bb5000-f7bb6000 rwxp f7bb5000 00:00 0
f7bb6000-f7bc2000 r-xp 00000000 08:01 97971 /lib32/libpgtypes.so.3.0
f7bc2000-f7bc4000 rwxp 0000c000 08:01 97971 /lib32/libpgtypes.so.3.0
f7bc4000-f7d0d000 r-xp 00000000 08:01 97925 /lib32/libc-2.7.so
f7d0d000-f7d0e000 r-xp 00149000 08:01 97925 /lib32/libc-2.7.so
f7d0e000-f7d10000 rwxp 0014a000 08:01 97925 /lib32/libc-2.7.so
f7d10000-f7d13000 rwxp f7d10000 00:00 0
f7d13000-f7d15000 r-xp 00000000 08:01 97928 /lib32/libdl-2.7.so
f7d15000-f7d17000 rwxp 00001000 08:01 97928 /lib32/libdl-2.7.so
f7d17000-f7d2b000 r-xp 00000000 08:01 97939 /lib32/libpthread-2.7.so
f7d2b000-f7d2d000 rwxp 00013000 08:01 97939 /lib32/libpthread-2.7.so
f7d2d000-f7d2f000 rwxp f7d2d000 00:00 0
f7d2f000-f7d49000 r-xp 00000000 08:09 934128 /usr/local/Packages/gcc-4.4.0/lib/libmudflapth.so.0.0.0
f7d49000-f7d4c000 rwxp 0001a000 08:09 934128 /usr/local/Packages/gcc-4.4.0/lib/libmudflapth.so.0.0.0
f7d4c000-f7e1b000 rwxp f7d4c000 00:00 0
f7e1b000-f7e35000 r-xp 00000000 08:01 98012 /lib32/libpq.so.5.0
f7e35000-f7e36000 rwxp 0001a000 08:01 98012 /lib32/libpq.so.5.0
f7e36000-f7e44000 r-xp 00000000 08:01 97969 /lib32/libecpg.so.6.0
f7e44000-f7f05000 rwxp 0000d000 08:01 97969 /lib32/libecpg.so.6.0
f7f18000-f7f1b000 rwxp f7f18000 00:00 0
f7f1b000-f7f38000 r-xp 00000000 08:01 97922 /lib32/ld-2.7.so
f7f38000-f7f3a000 rwxp 0001c000 08:01 97922 /lib32/ld-2.7.so
fff3f000-fff59000 rwxp 7ffffffe5000 00:00 0 [stack]
ffffe000-fffff000 r-xp ffffe000 00:00 0 [vdso]
Aborted (core dumped)

leif$ gdb ~/tmp/crashex core.30920
GNU gdb 6.8-debian
Copyright (C) 2008 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu"...

warning: Can't read pathname for load map: Input/output error.
Reading symbols from /lib32/libecpg.so.6...done.
Loaded symbols for /lib32/libecpg.so.6
Reading symbols from /lib32/libpq.so.5...done.
Loaded symbols for /lib32/libpq.so.5
Reading symbols from /usr/local/Packages/gcc-4.4.0/lib/libmudflapth.so.0...done.
Loaded symbols for /usr/local/Packages/gcc-4.4.0/lib/libmudflapth.so.0
Reading symbols from /lib32/libpthread.so.0...done.
Loaded symbols for /lib32/libpthread.so.0
Reading symbols from /lib32/libdl.so.2...done.
Loaded symbols for /lib32/libdl.so.2
Reading symbols from /lib32/libc.so.6...done.
Loaded symbols for /lib32/libc.so.6
Reading symbols from /lib32/libpgtypes.so.3...done.
Loaded symbols for /lib32/libpgtypes.so.3
Reading symbols from /lib32/libm.so.6...done.
Loaded symbols for /lib32/libm.so.6
Reading symbols from /lib32/libssl.so.0...done.
Loaded symbols for /lib32/libssl.so.0
Reading symbols from /lib32/libcrypto.so.0...done.
Loaded symbols for /lib32/libcrypto.so.0
Reading symbols from /lib32/libcrypt.so.1...done.
Loaded symbols for /lib32/libcrypt.so.1
Reading symbols from /lib/ld-linux.so.2...done.
Loaded symbols for /lib/ld-linux.so.2
Reading symbols from /usr/lib32/libgcc_s.so.1...done.
Loaded symbols for /usr/lib32/libgcc_s.so.1
Reading symbols from /lib32/libnss_compat.so.2...done.
Loaded symbols for /lib32/libnss_compat.so.2
Reading symbols from /lib32/libnsl.so.1...done.
Loaded symbols for /lib32/libnsl.so.1
Reading symbols from /lib32/libnss_nis.so.2...done.
Loaded symbols for /lib32/libnss_nis.so.2
Reading symbols from /lib32/libnss_files.so.2...done.
Loaded symbols for /lib32/libnss_files.so.2

warning: Lowest section in system-supplied DSO at 0xffffe000 is .hash at ffffe0b4
Program terminated with signal 6, Aborted.
[New process 30922]
[New process 30920]
#0 0xffffe405 in __kernel_vsyscall ()
(gdb) bt
#0 0xffffe405 in __kernel_vsyscall ()
#1 0xf7bef335 in raise () from /lib32/libc.so.6
#2 0xf7bf0cb1 in abort () from /lib32/libc.so.6
#3 0xf7c286ec in ?? () from /lib32/libc.so.6
#4 0xf7c30615 in ?? () from /lib32/libc.so.6
#5 0xf7c34080 in free () from /lib32/libc.so.6
#6 0xf7d39061 in free (buf=0x81f3958) at ../../../libmudflap/mf-hooks1.c:241
#7 0xf7e3fb5c in ecpg_sqlca_key_destructor () from /lib32/libecpg.so.6
#8 0xf7d1bbb0 in __nptl_deallocate_tsd () from /lib32/libpthread.so.0
#9 0xf7d1c509 in start_thread () from /lib32/libpthread.so.0
#10 0xf7c9d08e in clone () from /lib32/libc.so.6
(gdb)

As you might have noticed, this particular run is on a 64bit architecture (Ubuntu 8.04) and the crashex program is generated on a 32bit machine with gcc-4.4.0. I have tried with PostgreSQL version 8.3.5 and 8.3.7. All give the same result, though the specific program addresses of course might differ from system to system.

Please help,

Leif

----- leif(at)crysberg(dot)dk wrote:

> Hi Laurenz,
>
> Thanks for the suggestion. It sure wasn't easy, but I should have
> done that right away. It turned out not to be in the ecpg module, but
> somewhere in my own code (of course ;-) ). At least I haven't been
> able to reproduce it in a simple example and I haven't figured out
> where in my own code yet either.
>
> Leif
>
>
> ----- "Albe Laurenz" <laurenz(dot)albe(at)wien(dot)gv(dot)at> wrote:
>
> > leif(at)crysberg(dot)dk wrote:
> > > I'm using PostgreSQL in a server project that uses many
> > > forks and many threads in each forked process.
> > >
> > > Almost everytime I do a pthread_cancel() I get a SIGSEGV.
> > > I have then linked the libmudflapth into my program to catch
> > > the problem sooner and now that reports either 'invalid
> > > pointer' or 'double free or corruption' when a thread is
> > > cancelled. Typically I have 2 database connection opened
> > > before any of the threads are created. I am pretty sure that
> > > I'm only using 1 connection in any 1 thread, i.e. only 2 of
> > > the threads are doing database access and using each their
> > > allocated connection.
> > >
> > > After the main thread has done a pthread_cancel() I get a
> > > "mudflapth dump" with the following trace back (the abort
> > > comes from the mudflapth lib when detecting the bad pointer):
> > >
> > > #0 0xffffe405 in __kernel_vsyscall ()
> > > #1 0xf7ca2335 in raise () from /lib32/libc.so.6
> > > #2 0xf7ca3cb1 in abort () from /lib32/libc.so.6
> > > #3 0xf7cdb6ec in ?? () from /lib32/libc.so.6
> > > #4 0xf7ce71ab in free () from /lib32/libc.so.6
> > > #5 0xf7dec061 in free (buf=0x87ed138) at
> > ../../../libmudflap/mf-hooks1.c:241
> > > #6 0xf7ef2b5c in ecpg_sqlca_key_destructor () from
> > /lib32/libecpg.so.6
> > > #7 0xf7dcebb0 in __nptl_deallocate_tsd () from
> > /lib32/libpthread.so.0
> > > #8 0xf7dcf509 in start_thread () from /lib32/libpthread.so.0
> > > #9 0xf7d5008e in clone () from /lib32/libc.so.6
> > >
> > > Looking in the ecpg_sqlca_key_destructor(), it seems to me
> > > that the sqlca can be deallocated several times !? (I'm not
> > > too much into the Postgres code including ecpg, so that is a
> > > novice point of view.)
> > >
> > > I have tried both pgsql-8.3.5 and pgsql-8.4rc1, with
> > > exactly the same result and and on many different Linux
> > > systems, mainly Slackware 10.2 and Ubuntu 7. I have on all
> > > systems configured and compiled Postgres with this configure line:
> > >
> > > ./configure --prefix=/usr/local/Packages/pgsql-8.3.5
> > > --with-openssl --enable-thread-safety
> >
> > Could you create a small sample program that reproduces the bug?
> >
> > That would make it easier for me or somebody else to do something
> > about it.
> >
> > Yours,
> > Laurenz Albe