Re: vacuum locking

Josh Berkus writes:
> I hope that you'll stay current with PostgreSQL developments so that you can
> do a similarly thourough evaluation for your next project.

Oh, no worries. This project just happens to be a tough one. We're
heavily invested in Postgres. Other projects we maintain that use
Postgres are zoescore.com, colosla.org, and paintedsnapshot.com.

I am currently working on a very large project where the customer is
very committed to Postgres/open source. We're in discussions about
what to do about the scalability problems we saw in the other project.
You can help by addressing a dilema we (my new customer and I) see.
I apologize for the length of what follows, but I'm trying to be as
clear as possible about our situation.

I have had a lot push back from the core Postgres folks on the idea of
planner hints, which would go a long way to solve the performance
problems we are seeing. I presented an alternative approach: have a
"style sheet" (Scribe, LaTex) type of solution in the postmaster,
which can be customized by end users. That got no response so I
assume it wasn't in line with the "Postgres way" (more below).

The vacuum problem is very serious for the problematic database to the
point that one of my customer's customers said:

However, I am having a hard time understanding why the system is so
slow... from my perspective it seems like you have some fundamental
database issues that need to be addressed.

This is simply unacceptable, and that's why we're moving to Oracle.
It's very bad for my business reputation.

I don't have a ready solution to vacuuming, and none on the list have
been effective. We'll be adding more memory, but it seems to be disk
bandwidth problem. I run Oracle on much slower system, and I've never
noticed problems of this kind, even when a database-wide validation is
running. When vacuum is running, it's going through the entire
database, and that pretty much trashes all other queries, especially
DSS queries. As always it is just software, and there's got to be
80/20 solution.

Our new project is large, high-profile, but not as data intensive as
the problematic one. We are willing to commit significant funding and
effort to make Postgres faster. We are "business value" driven. That
means we solve problems practically instead of theoretically. This
seems to be in conflict with "the Postgres way", which seems to be
more theoretical. Our business situation comes ahead of theories.

My customer (who monitors this list) and I believe that our changes
would not be accepted back into the Postgres main branch. That
presents us with a difficult situation, because we don't want to own a
separate branch. (Xemacs helped push emacs, and maybe that's what has
to happen here, yet it's not a pretty situation.)

We'll be meeting next week to discuss the situation, and how we'll go
forward. We have budget in 2003 to spend on this, but only if the
situation can be resolved. Otherwise, we'll have to respect the data
we are seeing, and think about our choice of technologies.

Thanks for the feedback.

Rob