I just didn't realize that angles could be meaured so accurately by eye at that time

Sextants have a bit of magnification (usually 4x, but sometimes 7x or higher). Higher mag allows for better accuracy at the cost of more shaking of the view.

Basic explanation:

* https://www.youtube.com/watch?v=30_wEda3ltM