"One thing I'd like to comment on is that I think you should revise your use of unsafeThaw/unsafeFreeze: you don't easily get guarantees of determinism when using these unsafe functions."

Unfortunately that seems to be the only way to get a mutable Vector from a immutable one without copying it. As long as the mutable version of the vector isn't used anymore after calling set, it should be safe. But yes, guarantees are always nicer.

Nice to learn about the SPECIALIZE pragma. :)