Re: Duration of beta period

On Sat, 23 Feb 2002, Bruce Momjian wrote:

> > You've totally lost me here ... how do you speed up a beta when there are
> > unknown number of bugs that have to be resolved before a release can
> > happen? A beta's duration is as long as it takes to be stable ...
> >
> > Personally, I think this last beta period was one of our best ones yet ...
> > considering the negligible number of bug reports following the release, we
> > did exactly what had to be done in teh beta to make sure that the end
> > result was as stable as possible for the testing pool we had ...
>
> It was a good period, but we were in beta longer than we were in
> development. That isn't good, and I think we could have shortened it if
> we had pushed people to test beta sooner and harder, avoided some
> sideroads in improving things that could have waited for 7.3, and pushed
> a few people to complete bug fixes quicker than letting it all wait
> until the end of beta. Of course, I may be wrong. :-)

Where this a commercial product, your assessments would be correct ...
but, alas, we aren't ... "pushing ppl" that are doing something because
they enjoy it, and receive no compensation for it, is a very effective way
of driving good ppl away ...

> > Don't start trying to set arbitrary dates on when a beta should start, or
> > end ... you'll just screw with what is already a very effective
> > development model ...
>
> I sensed people were not happy with the beta duration. I think it was
> sort of like being on a baseball team, and someone hits the ball to an
> outfielder, and they drop it, and you feel, "Hey, my team stinks."
> People like to be in groups that can successfully get things done, and
> in a timely manner. No one wants to wait for the last person to get in
> the bus so we can leave. If we don't tell people when the bus is
> leaving, and just say, hey, get on the bus as soon as you can, it
> doesn't seem to work as well as saying, "We are leaving in one hour. Be
> on the bus, or explain why we should wait another hour." Of course,
> just an analogy, but I think it has a point.

Again, great one for a commercial product, totally inappropriate for us
... and even for a commercial product, I dont' think it works unless you
want to have your paying clients go to someone else cause of all the bugs
...

You can't, and won't, force someone to fix bugs just because you've set a
date for them ... and you cna't put out a product without the known bugs
fixed, unless you want to lose marketshare ...

We aren't a 20k line C program where the effects of fixing one thing won't
affect somewhere else ... how many long discussions were there during the
beta period this time where the ramifications of that change wouldn't be
felt in other places?

We are no longer in the "let's fix a bunch of bugs and call it a release"
mode ... there are *alot* of *very* large changes that have gone into each
consecutive release, the code base is getting gradually larger, and the
ramifications of the changes are getting more diverse ...

God, I can remember a time when Oleg's group submitting a patch for the
GiST stuff would have been shoved in during beta cause, hell, it didn't
work anyway ... now we look at it sideways and say it has to wait until
the next dev cycle ...

Do not set 'timeline's, set 'milestones' ... generate a listing of what
ppl are working on for v7.3, follow that development and when its done,
*then* its time for beta ... if that is 6-8's down the road, so be it ...
that is why projects like FreeBSD have a -STABLE vs -CURRENT branch, so
that the big projects have time to get done, but there is a mechanism for
doing more short term releases as appropriate ...

> > If you are so bored that you need to organize something, come up with a
> > list of what we hope to accomplish for v7.3 and use that for a guideline
> > for when the beta will start ...
>
> That is a tough one, and something that seems very hard to organize, but
> I can try if you wish. The TODO list can easily be used for that.
> Another marking stating "Will be done for 7.3" is all it needs. When
> all the marks are turned to dashes (done) we are ready for beta.

In the future, what should be done, is once we've done the release, take
the "coolling down period" that we alawys have afterwards to generate some
of that list ... a simple post asking who is hoping to accomplish what is
all that is needed ... if they don't speak up and get themselves added to
the list, then what they are workign on is obviously not big enough, or
important enough, to be tied into criteria for beta status ...

> > What major items do ppl have planned for v7.3? Once those items are
> > completed, then we go into beta for v7.3 and beta lasts until we've gotten
> > her to the point that she's as bug free as she can be without more
> > extensive/broader testing ...
>
> That is a good question and something we do need to ask. Thanks.
>
> > If those major items are done on the 1st of April, I'll happily declare it
> > the shortest development cycle we've had to date and go beta ... if it
> > takes until the 1st of September, so be it ... but its not going to go
> > beta until its ready, and its not going to be released until its ready ...
> > and setting some arbitrary date is going to do one of two things: a)
> > pressure developers into making mistakes that will make us look bad, or b)
> > stiffle development while one developer figures "no way I'll get it done
> > by then, so why bother" ...
>
> Agreed, don't release until it is ready, but I think we could make it
> ready faster with some dates. I am feeling people are doing beta
> testing only near the end of beta because they realize how long it is
> taking. That is bad. There is always pressure to lengthen the
> schedule. I think we need some pressure to shorten it. That is all I
> am saying.

You can't pressure ppl you aren't paying ... quite frankly, from what I
recall of this past beta, it wasn't testing the code that was a problem,
it was some of the bugs that came up took a bit more then a few minute
fixes ... hell, it says alot when the fix for a beta requests an initdb to
get into the next beta, but we had that here too ...

> > ... and that isn't to throw in the users that are going to say "but you
> > said ... "
>
> Agreed, we certainly want to stay away from that --- users holding us to
> our dates is a recipe for disaster.

As are creating the dates themselves ...

Not sure why you want to change a recipe for development that has worked
for ... what ... 7 years now ... change for the sake of change isn't
necessarily a good idea ...

If you want shorter development times, start making better use of CVS ...
spend the time to back-patch appropriate patches into the REL7_2_STABLE
tree and we can do a release of that every couple of months ... but,
unless I'm mistaken, as we mature more, the enhancements and changes that
are going into each new release is taking longer to implement, with more
far-reaching ramifications, and both *should not* and *can not* be rushed
... I don't know about anyone else, but it doesn't surprise me that beta
testing takes longer then a development cycle anymore ...