Re: Making the initial and maximum DSA segment sizes configurable

Hello Masahiko-san,

I'm not super-familiar with the DSA/DSM stuff, but I think your proposal
makes sense.

I agree with your observation that DSA is a bit like AllocSet, so if
that allows specifying min/max block size, maybe DSA should allow the
same thing for segments ...

However, does it actually address the problem you've described? If I
understand it correctly, the problem is that with the doubling logic,
we're likely to overshoot the limit. For example with m_w_m=512MB we
first undershoot it a bit (510MB), and then overshoot it a lot (766MB).
Which is not great, I agree (especially the overshooting).

But is the modification you propose much better? I mean, we still
overshoot the limit, right? By a smaller amount (just 62MB instead of
254MB), but it's still more than the limit ...

Moreover, this really depend on the caller using lower init/max segment
size, right? I'd bet most places would just hard-code something, which
means it won't respond to changes in the m_w_m value.

Could instead allow specifying the expected size / memory limit,
calculate the maximum segment size in DSA code, and also modify how the
segment size evolves over time to decrease as we get closer to the
expected size / limit?

For example, let's say we'd create DSA with 512MB limit. Then we could
do this:

1MB, 2MB, 4MB, ..., 128MB, 256MB, 1MB, 1MB, ...

because after 256MB we have 511MB of segments (in total), and we have to
go all the way back to the smallest segment to not exceed the limit (or
to minimize how much we exceed it). If the limit was set to 600MB, we'd
go back to 64MB, then 16MB, etc.

Or maybe we could be smarter and calculate an "optimal point" at which
point to start decreasing the segment size, roughly half-way through. So
we'd end up with something like

1MB, 2MB, 4MB, ..., 128MB, 128MB, 64MB, 32MB, 16MB, ..., 1MB

But maybe that's unnecessarily complicated ... or maybe I'm missing some
details that make this impossible for the DSA/DSM code.

FWIW the aset.c code has the same problem - it's not aware of limits
like work_mem / maintenance_work_mem, and with hard-coded limits we may
easily hit exceed those (if we get to sufficiently large blocks,
although in most cases the max block is 8MB, which limits how much we
overshoot the limit). Not sure if that's an issue in practice, maybe the
virtual memory thing deals with this for us.

If you choose to go with passing the min/max segment size to DSA, maybe
this should do a similar thing to aset.c and define a couple "good"
values (like ALLOCSET_DEFAULT_SIZES, ALLOCSET_SMALL_SIZES, ...) and/or a
macro to calculate good segment sizes for a given limit.

Also, there's a comment:

* See dsa_create() for a note about the tranche arguments.

which should probably reference dsa_create_extended() instead.

regards

--
Tomas Vondra
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company