Re: Two weeks to feature freeze

On Mon, 23 Jun 2003, Dann Corbit wrote:

> > -----Original Message-----
> > From: scott.marlowe [mailto:scott(dot)marlowe(at)ihs(dot)com]
> > Sent: Monday, June 23, 2003 12:25 PM
> > To: Dann Corbit
> > Cc: Bruce Momjian; Tom Lane; Jason Earl; PostgreSQL-development
> > Subject: Re: [HACKERS] Two weeks to feature freeze
> >
> >
> > On Mon, 23 Jun 2003, Dann Corbit wrote:
> >
> > > Vendor A: "We think our tool is pretty solid and our end
> > users hardly
> > > ever turn up any bugs."
> > >
> > > Vendor B:" We think our tool is pretty solid and our 8500 tests
> > > currently show only 3 defects with the released version,
> > and these are
> > > low impact issues. To view our current database of issues,
> > log onto
> > > web form <page>."
> > >
> > > Which tool would you prefer to install?
> >
> > The one I've tested and found to meet my needs, both now and
> > by providing
> > fixes when I needed it.
> >
> > Real world example: We run Crystal Reports Enterprise
> > edition where I
> > work. It's tested thouroughly (supposedly) and has all kinds of QA.
> > However, getting it to work right and stay up is a nightmare.
> > It's taken
> > them almost a year to get around to testing against the OpenLDAP LDAP
> > server we use. The box said "LDAP V3 compliant" and they
> > assured us that
> > it was. Well, it doesn't work with our LDAP V3 compliant
> > LDAP server at
> > all, and the problem is something they can't fix for months
> > because it
> > doesn't fit into their test cycle.
> >
> >
> > Real world example: Postgresql aggregates in subselects.
> > Someone found a bug in subselects in Postgresql with inner
> > references to
> > outter aggregates. The postgresql team delivered a patch in
> > less than a
> > week. User tested it and it works.
> >
> > I'm not against testing and all, but as one of the many beta
> > testers for
> > Postgresql, I do feel a bit insulted by your attitude that only a
> > cohesive, organized testing effort can result in a reliable product.
>
> Let me rephrase it:
> "Only a cohesive, organized testing effort can result in a product that
> is proven reliable."

No, it isn't proven reliable PERIOD, it's proven reliable under the exact
conditions of the testing procedure you've implemented. And no matter how
idiot proof we try to make Postgresql and its testing, someone else will
always make a better idiot. :-) Actually, what I'm saying is that the
corner cases are the ones that are hard to predict, and no amount of
effort up front is going to find a corner case you haven't thought of yet.

> Without such an effort, it is only an educated guess as to whether the
> product is reliable or not. The data is the most valuable software
> component in an organization. It is worth more than the hardware and it
> is worth more than the software. If you are going to trust one billion
> dollars worth of corporate data on a software system, you ought to
> ensure that the system has been carefully tested. I don't think that is
> just an opinion. It's simply common sense.

But if that is true, then Postgresql should cause me no end of problems as
it crashes down around my feet every other week. Oddly, the dbas for the
other systems here at work (Oracle and MSSQL server) have a much higher
workload keeping their factory tested databases up and running. In over
four years of use, we have had exactly ZERO downtime of postgresql.

Carefully testing the system is what I, the DBA of our postgresql servers,
do. Only I know how we use the system, so no matter how you or Bruce or
Tom might test it, I'll always be able to do something you wouldn't think
of, because you're simply not in my shoes.

It's not an educated guess that postgresql works for us, it's proven over
and over again every single day by the throrough testing and use of every
Postgresql user.

> > I take my support of Postgresql seriously, and answer many
> > questions every
> > week in the general, sql, and performance mailing lists. I'm
> > not paid to
> > do it, I stay at work an extra hour or so each day to "pay
> > for it." I
> > test every beta and RC release on our dataset at work, and
> > with a test
> > load to make sure it works for us, and it does.
> >
> > I offered to beta test for Crystal Reports and was told they weren't
> > interested, they can do it in house. Their support, like many big
> > commercial houses, is designed more to make my boss's boss
> > happy, not me,
> > and it shows every day in how they fail to provide timely support for
> > their product while playing CYA to the higherups.
>
> A long test cycle does result in a slower patch. But when you get the
> patch, it is going to work and not introduce new problems.

Nice theory, but it isn't provable by my experience. While I've put .0
releases of postgresql into production many a times, usually with no
issues at all, I never have and never will put a .0 release of Crystal
Reports online. They've taught me well not to trust their initial
release.

> The resistance to testing is typical of programmers. The PostgreSQL
> group is a group of programmers. I don't think I can change anyone's
> mind, since the most significant people on the list don't think it is
> worth the bother.

No, I am NOT A postgresql programmer, I am a Postgresql user. I do
program, but that's in support of my job as a PHP dev / database admin.
I've found myself looking through the postgresql code only an odd half
dozen times. I've never NEEDED to hack it, as it seems to just work.

> Therefore, I am going to stop harping on it.

Good move. You're busily shooting at ghosts right now.

While there's always room for more testing, the best way to do it is to
roll up your sleaves and write your own tests or add to the regression
tests. I've written quite a few load tests in Perl over the years to make
sure that postgresql can hold up to the kind of load we tend to throw at
it. So have hundreds of other Postgresql users you've never met. Just
because no one has a centralized plan and document for testing doesn't
mean it isn't getting done, it just means you can't see it all that well.

I think of your testing methodology as the equivalent of the old Soviet
Union's Five year plans. They were thorough and well documented, but
never worked as well as planned. Postgresql is like the free market
system. no one's completely in charge, and the more you try to control it
the more it tries to be free. Ultimately, the chaos of the free market
won out.

It may be reassuring to think your product is very well tested before it
goes out the door, but it's a false security, proven over and over by
commercial products that simply don't work in the field because of
problems that the original designers never envisioned, and now that they
have a thorough and long drawn out testing cycle, it simply takes longer
and longer to get fixes, while providing little, if any, improvement in
quality.