it is possible that the MD5 hash of your partition keys isn't evenly distributed
how? i mean, apart from poisson stats / shot noise, obviously (and which is noise, so you can't predict it anyway).
thinking some more, i guess this (splitting and merging partitions in a non-generic way) is to handle when a consumer is slow for some reason. perhaps that partition is backing up because the consumer crashed.
but then why not say that, instead of postulating the people are going to have uneven hashes?
Yes, duplicates, I think. Looks like the partition key can be set to whatever you want, so maybe you log, I dunno, hits sharded by page, and your homepage gets a ton. I'd lean towards sharding randomly to avoid that, but, eh, they're just giving you enough rope to mess up your logging pipe with.
Comments
it is possible that the MD5 hash of your partition keys isn't evenly distributed
how? i mean, apart from poisson stats / shot noise, obviously (and which is noise, so you can't predict it anyway).
thinking some more, i guess this (splitting and merging partitions in a non-generic way) is to handle when a consumer is slow for some reason. perhaps that partition is backing up because the consumer crashed.
but then why not say that, instead of postulating the people are going to have uneven hashes?
[edit:] maybe they allow duplicates?
Yes, duplicates, I think. Looks like the partition key can be set to whatever you want, so maybe you log, I dunno, hits sharded by page, and your homepage gets a ton. I'd lean towards sharding randomly to avoid that, but, eh, they're just giving you enough rope to mess up your logging pipe with.