Sunday, September 21, 2008

Sharded Counters Take Too Much Time

The purpose of sharded counters is to reduce contention for a single entity on serialized writes.  The concept is to create multiple sub-counters that are added together to create a single counter with the sub-counter that is incremented selected at random.  This seemed ideal for TCP3 since we need to count pretty much everything.  We count clicks, previews, clicks per day, previews per day, etc. 

The problem is that sharded counters take too long.  I'm still profiling, but i'm getting quota warnings on even the memcached sharded counter implementation suggested in one of the Google I/O briefings.  Updating the sharded counters takes 3.5 seconds in my application.  My problem is that i'm not sure if this is a performance problem for appengine or a bug in my code.

I'm going to have to consider an alternate implementation.  I'll post details as soon as I come up with something that works.

0 comments: