Also, better insight into partition sizes / what's causing hot spotting. The DB abstracts a lot from the user, which isn't necessarily great, because it's still subject to the normal pitfalls of a NoSQL database.
Not a particularly easy solution, but you can use dynamo streams to achieve this by loading fast into a temporary table, trickle-feeding via a stream into another table. When it’s caught up, stop writes on the import table then swap over to the permanent table.
A way of doing this without expending all that effort is oh my wish list too.
Comments
My wishlist for DynamoDB is now down to:
* Fast one-time data import without permanently creating a lot of shards (important if you are restoring from a backup)
* Better visibility into what causes throttling (e.g. was it a hot shard? Was it a brief but large burst of traffic?)
* Lower p99.9 latency. It occasionally has huge latency spikes.
* Indexes of more than 2 columns
* A solution for streaming out updates that is better than dynamodb streams
Also, better insight into partition sizes / what's causing hot spotting. The DB abstracts a lot from the user, which isn't necessarily great, because it's still subject to the normal pitfalls of a NoSQL database.
Bigtable is a different beast, but it's new "Key Visualizer" is impressive. Has helped us quickly find anomalies https://cloud.google.com/bigtable/docs/keyvis-overview
Wish Dynamo had something similar
Not a particularly easy solution, but you can use dynamo streams to achieve this by loading fast into a temporary table, trickle-feeding via a stream into another table. When it’s caught up, stop writes on the import table then swap over to the permanent table.
A way of doing this without expending all that effort is oh my wish list too.
What bothers you about dynamodb streams specifically?
What kind of p99.9 latency are you looking for?
and would Dax help?