Thanks for answering, that makes sense. Also - your setup seems like it could greatly benefit from speculative decoding. Have you guys given any thought to how that might work in this system?
P.s. for #2, you can probably do something like RAFT-styled interleaved computation. But this could get tricky unless you commit to a sharding scheme that makes it easier.
Comments
Thanks for answering, that makes sense. Also - your setup seems like it could greatly benefit from speculative decoding. Have you guys given any thought to how that might work in this system?
P.s. for #2, you can probably do something like RAFT-styled interleaved computation. But this could get tricky unless you commit to a sharding scheme that makes it easier.