Yeah, I had a lot of nVidia GPUs suddenly disappear mid-training when even nvidia-smi couldn't find them; this was on different systems (Linux) and only a reboot fixed it.
You don't want this kind of thing happening when it is running a filesystem.
Strange. I never had any problem with nvidia GPUs, but I only ever used data center GPU like the V100 (and don't set them up myself). There's a lot of things that go wrong, at least my nvidia GPU always works.
Comments
Yeah, I had a lot of nVidia GPUs suddenly disappear mid-training when even nvidia-smi couldn't find them; this was on different systems (Linux) and only a reboot fixed it.
You don't want this kind of thing happening when it is running a filesystem.
Strange. I never had any problem with nvidia GPUs, but I only ever used data center GPU like the V100 (and don't set them up myself). There's a lot of things that go wrong, at least my nvidia GPU always works.