It doesn't have to be a one or the other choice. For example, txtai supports a number of different ANN backends including a simple NumPy implementation (https://neuml.github.io/txtai/embeddings/configuration/ann/). There is value in the plumbing to vectorize data, normalize embeddings and find matches.

It's a good idea to find a solution that enables starting simple and scaling up as needed without having to fully rewrite the code.