Quick note about client retry, I highly suggest some type of guard on side effecting operations, either rejecting duplicate transaction ids (generated by the client or retrieved from the server before the side effecting operation) or returning the previous result if the transaction id was recently processed.

Obviously being stateful and correctly routing the request across shards becomes harder and can hurt this scale out solution. It also depends on the functionality of the request, for example an update of the same data may not cause any damage, but a double submit of an order could be.

https://en.wikipedia.org/wiki/Idempotence#Computer_science_m...