They actually did change their diet to fish. Toward the end of the colony, the diet was predominantly seafood, and dramatically different from the diet of the early colonists:

https://pdfs.semanticscholar.org/e392/f5fc6683acf7c1f05df7b3...