Hmm, it's more like any ionizing particle (charged) creates pairs electron/hole, many of them, so that the charge distribution in the pn junctions are altered. Sometimes gamma photons generate Compton electrons in the material that act as delta rays, to the same effect. Heavy particles can alter the crystal structure, inducing defects that may change its properties. Magnetic storage devices don't have polarised junctions and could withstand all this if they didn't include solid state electronics. Radiation hardening techniques and shielding can help a lot.