It's not an issue of "alignment." The reward function was poorly written.

What else do you imagine “alignment” is?