Re: Storage Manager crash at mdwrite()

On Fri, Mar 16, 2012 at 8:34 PM, Greg Stark <stark(at)mit(dot)edu> wrote:

> On Fri, Mar 16, 2012 at 11:29 PM, Jeff Davis <pgsql(at)j-davis(dot)com> wrote:
> > There is a lot of difference between those two. In particular, it looks
> > like the problem you are seeing is coming from the background writer,
> > which is not running during initdb.
>
> The difference that comes to mind is that the postmaster forks. If the
> library opens any connections prior to forking and then uses them
> after forking that would work at first but it would get confused
> quickly once more than one backend tries to use the same connection.
> The data being sent would all be mixed together and they would see
> responses to requests other processes sent.
>

The opened connections thing was the first thing I thought of.. the HDFS
documentation claims the C interface is thread-safe..
However, since I noticed that different processes (having different process
IDs) are calling the mdwrite, then it might be a possibility.
I tried reconnecting to HDFS on every mdwrite request, but it didn't work
out.

> You need to ensure that any network connections are opened up *after*
> the new processes are forked.
>
> There are other things going on that could cause problems. initdb
> probably doesn't deal with many errors so it might not be casing any
> longjmps that happen when Postgres handles errors. I suspect it
> doesn't create any temporary files, etc.
>
> There's also a long history of threads not playing well with Postgres.
> I think that's overblown and I believe they should work fine. Most of
> it was caused by a bug in an old version of libc. But you do have to
> ensure that the other threads don't call any Postgres functions
> because the Postgres code is very much not thread-aware.
>
> --
> greg
>

I'm novice in PG, and if I got this one running then I'll have achieved
what I wanted to do without further digging in PG. So I was thinking if
there was a configuration option (or something similar) that will eliminate
(or reduce) the role the background writer process.. I believe it can be
one of the WAL options but I'm not sure.. Any suggestions?