firefoxman1 on Realtime Metrics for 128 Million Users with Redis: 50ms + 16MB RAM

Comment on Realtime Metrics for 128 Million Users with Redis: 50ms + 16MB RAM

This was posted a while ago, and I have since implemented bitmaps myself. One thing I learned from the documentation[1] is that setting an initial bit at a very high-numbered place (like 2^30 - 1) takes a while to allocate (compared to Redis's normal speed) and blocks other operations in the process.

In my case, and it appears to be true for Spool too, I don't know what bit will be set first. It could be 12 or it could be 2938251, so to prevent a slowdown if the initial bit is a high place I use buckets of bitmaps, each holding around 8 million bits.

[1] Read Warning: http://redis.io/commands/setbit

antirez 14y

There is an easy fix for this, that is also the recommended way to use bitmaps in Redis: split your bitmap among multiple keys.

For instance you want to set bit i, but you want k bits per every key, the you do:

    keyname = "bitmap:"+(i/k)
    keybit = i%k

k can be fairly large, like 128k bytes per key. It's still small but big enough for keys overhead to be negligible.

firefoxman1 14y

Wow, that's almost exactly how I implemented it:

   var bucketSize = 8190;

   ...

   var bucketNumber = Math.floor(userId / bucketSize),
       bitInBucket = userId % bucketSize;

...correction on my last comment, looks like I use ~8 thousand bits per bucket, not 8 million.

Comments