Re: Fast data, slow data

On 06/26/2014 02:29 AM, Tim Uckun wrote:
> I have a use case in which the most recent data experiences a lot of
> transactions (inserts and updates) and then the churn kind of calms
> down. Eventually the data is relatively static and will only be
> updated in special and sporatic events.
>
> I was thinking about keeping the high churn data in a different table
> so that the vacuums on that table can go faster and the vacuums on the
> rest of the data will rarely be needed.
>
> Ideally the high churn data would be in an in memory table but failing
> that I thought that an unlogged table would be good enough. So now I
> need a way to flush the oldest data in this table to another logged
> table and do it in an efficient manner. I don't think partitioning is
> a good idea in this case because the partitions will be for small time
> periods (5 to 15 minutes).
>
> Anybody have a similar problem? If so how did you solve it?

It's not at all unusual for a database to experience a high portion of
its activity on a small subset of the data so in that sense the
"problem" is already solved by appropriate sizing and tuning to make
appropriate adjustments to the caching and other properties already
provided by the OS and DB.

Having said that, there is far too little information here to provide
specific advice on tuning and other approaches that may be of use for
you. In particular:

Is this currently a hypothetical question or do you have a running
system that is experiencing performance problems?

In either case can you provide some parameters including rows and size
of your large table(s), what you mean by "high churn" (rate of inserts,
deletes, updates and how they are grouped into transactions), the size
of the table that is heavily updated, how the database is used
(transactional, data-mining, ...) and anything else you feel might help
the list understand your use-case.

If you have a live database, any metrics/observations you can supply
might be helpful. For example, are particular queries slow? What speed
is required? Do you have lots of simultaneous connections or is
everything through a single connection.

Finally, confirm that the data is not important or is easily recreated
(implied by your in-memory/unlogged-table comments).

Cheers,
Steve