Re: Why we lost Uber as a user

On 03/08/16 02:27, Robert Haas wrote:
>
> Personally, I think that incremental surgery on our current heap
> format to try to fix this is not going to get very far. If you look
> at the history of this, 8.3 was a huge release for timely cleanup of
> dead tuple. There was also significant progress in 8.4 as a result of
> 5da9da71c44f27ba48fdad08ef263bf70e43e689. As far as I can recall, we
> then made no progress at all in 9.0 - 9.4. We made a very small
> improvement in 9.5 with 94028691609f8e148bd4ce72c46163f018832a5b, but
> that's pretty niche. In 9.6, we have "snapshot too old", which I'd
> argue is potentially a large improvement, but it was big and invasive
> and will no doubt pose code maintenance hazards in the years to come;
> also, many people won't be able to use it or won't realize that they
> should use it. I think it is likely that further incremental
> improvements here will be quite hard to find, and the amount of effort
> will be large relative to the amount of benefit. I think we need a
> new storage format where the bloat is cleanly separated from the data
> rather than intermingled with it; every other major RDMS works that
> way. Perhaps this is a case of "the grass is greener on the other
> side of the fence", but I don't think so.
>
Yeah, I think this is a good summary of the state of play.

The only other new db development to use a non-overwriting design like
ours that I know of was Jim Starky's Falcon engine for (ironically)
Mysql 6.0. Not sure if anyone is still progressing that at all now.

I do wonder if Uber could have successfully tamed dead tuple bloat with
aggressive per-table autovacuum settings (and if in fact they tried),
but as I think Robert said earlier, it is pretty easy to come up with a
highly update (or insert + delete) workload that makes for a pretty ugly
bloat component even with real aggressive autovacuuming.

Cheers

Mark