Skip to content

Comment on LeftoverLocals: Listening to LLM responses through leaked GPU local memory

Comments

At this point, I assume this is the default and don’t expect data recovery to not be provide on the same physical machine (even across virtualization barriers).

If your data is that sensitive, run it on dedicated hardware. Papering over this with mitigation over mitigation is a fool’s errand: both a genuine waste of compute resources and guaranteed to be a game of cat and mouse.

This is certainly the pragmatic approach to GPU memory in 2024, but I don't think it's a fool's errand. It's a solved problem on the CPU side of things, and I don't see any reason why we can't solve it in the GPU domain too.

Notably:

NVIDIA: confirmed that their devices are not currently impacted
ARM: also confirmed that their devices are not currently impacted.
It's a solved problem on the CPU side of things

Is it? gestures at pile of cpu bugs

The reason why arm is not affected is that their "local" memory is non existent and they just spill everything to cache.

TIL that ARM has GPUs

AboutSource Built by g1lg1l

Hackerly is an independent reader for Hacker News, built on the public HN API. Not affiliated with Y Combinator.