So that's already happened. It's called an algae bloom. They don't necessarily fix their own nitrogen though, we fix it for them and dump it into oceans (we're quite a bit better at it than nature at this point). Biological fixation of nitrogen like this would be overly complicated to engineer (compared to regular symbiosis like between legumes and root bacteria) and isn't as energy efficient (plants would waste a lot of energy maintaining the organelles). We have gotten quite good at pulling nitrogen from the air and turning it into bioavailable nitrogen and flooding ecosystems with it already. In terms of how we keep them at bay, we don't it's kind of a problem.