Speculative pre-positioning: off-path decode for stateful inference sessionsarxiv.org 1 pointlogotype2 months ago1 commentSaveHideCopy link On HNComments−logotypeOP2moWith native support for sessions in an inference engine we can make use of idle GPUs... doing work!
Comments
With native support for sessions in an inference engine we can make use of idle GPUs... doing work!