Webcluster session storage, was vacuum, performance, and MVCC

Verging on offtopic, but ...

Regarding the best place for session data, memcached isn't really the
answer, for opposite reasons as to why it isn't so great to store it
in the central DB for a bug web farm.

Folks on the memcached lists propose this [ "I keep all my session
data in memcached, it works great!" ] from time to time, and always
get smacked down with "Don't do that -- memcached is just for caching
stuff that is retrievable from some stable / slower source.
Memcached's LRU policy could conceivably evict that session data at
any time and you'll loose."

So, just it works fine under low / moderate load [ just as it does in
PG when periodic vacuuming can keep up ], but under higher user
volume than anticipated your memcaches could well decide to evict
otherwise good sessions. Especially if all your session data objects
are about the same size, but you also store things either smaller or
bigger in memcache -- its current slab allocator subdivides total
heap space into slabs for like-sized objects on powers of two I
believe, so even though you gave memcache a half a gig or whatever to
play with, the available size for 1K objects will not be that half a
gig if you're also storing 20 byte and 10K values.

Therefore I would not recommend memcached as the sole container for
session data. It's a well designed champ as a fast cache, but it
oughta be used just as a cache, not as a datastore. Hence the name.

For folks who say that the stuff should be in RAM at the appserver
side -- well -- that does imply sticky session load balancing which I
believe most folks agree ought to be avoided if at all possible. And
in our specific case, our appserver code is fastcgi / multiprocess on
each webserver, so there's no central RAM to store it in, save for
swallowing the shared memory bullet. Some sort of distributed shared
memory [ which seems to be rather how Mohawksoft's MCache seems to be
designed ] probably also performs best with sticky sessions --
otherwise the session pages thrash between appservers on writes. I've
not read the docs with fine-tooth comb to really infer how it is
designed, so please forgive and educate me if MCache has an elegant
solution for write-heavy non-sticky session use.

I'm squarely in the camp of just suck up and live with serialization
-- storing session data in RAM on appservers in appserver processes
really really binds your hands from a deployment and administration
angle. And you don't even really realize it until you've tasted the
freedom of having be outside. It lets you, oh, say, do code updates
in the middle of the day -- not at 5AM. It lets you cluster
horizontally in a much more simple manner -- no needing to deal with
any broadcast-writes voodoo. It lets you not have to have sticky
sessions. It lets you be much more, well, stateless, in your
appserver code. Our site used to be single-JVM JBoss-based with
sessions in RAM, and after moving away to multiprocess fastcgi model
-- the grass is way greener on this side.

In rewriting we're also going completely stateless -- no illusion of
scratch session data store [ we're so read-mostly that it is easily
possible ]. But if we 'had' to have session support, I'd like to use
a system which worked something like:

1) Deployed centrally parallel to memcached and DB servers for the
webcluster.
2) Performs update-in-place.
3) Will not arbitrarily evict data.
4) Stores arbitrary volumes of info -- i.e. spills to disk if it has
to, but doesn't have to worry about fsyncing all day long -- be they
tx logs or datafiles or what have you.
5) Fast as hell for reading / writing 'hot' session data. Does not
have to worry about fast concurrent access to session data -- we're
probably handling at most one single HTTP request per this session at
a time.

6) Clusterable in memcached model -- client-side-code hashes the
session key to the backend session store. If said backend session
server becomes unresponsive, flag him as temporarily dead [ try again
in 30 seconds? ] and remove from possible candidates. rehash and re-
query.

7) Migrateable: If I need to bring down session server N, it stops
answering requests and re-hashes what it has stored and transmits to
its live peers. If hashing algorithm and list of servers were common
across all servers + clients, and if perhaps a server response for a
fetch / store could contain a "redirect this to other server" type
response, then perhaps as the server going down streams the data to
its peers and they ack reception, then this could possibly work w/o
loss of data and would minimize delay at topology change time.

I could live with 'if one of my session servers croaks unplanned, I
loose those sessions'. Wishlist 6 and 7 items remind me more and more
of AFS or NFS4 client / server interaction. Not trivial unfortunately.

I hate to say it, but would mysql / myisam not work well for points
1-5 ? I have not idea about it and 'fast as hell' -- not ever run it
in production for anything. 6 + 7 could possibly be done atop mysql
using a 3-tier model.

----
James Robinson
Socialserve.com