In order for me to trust a benchmark, it needs to be a lot more transparent than this
- Show the code that runs the bench mark
- Give opportunities for everyone to recreate the benchmark
- Give opportunities for every technology to 'respond' and point out where the benchmark/tech configuration is wrong (ie "PRs welcome")
Otherwise, this just looks like cherry-picked data points, and even those I won't trust. Nor would I show this to any of my clients (whom I help select search engine technology). I dearly hope nobody makes real decisions based on this blog post until the code, and everything is opened up.
Comments
In order for me to trust a benchmark, it needs to be a lot more transparent than this
- Show the code that runs the bench mark
- Give opportunities for everyone to recreate the benchmark
- Give opportunities for every technology to 'respond' and point out where the benchmark/tech configuration is wrong (ie "PRs welcome")
Otherwise, this just looks like cherry-picked data points, and even those I won't trust. Nor would I show this to any of my clients (whom I help select search engine technology). I dearly hope nobody makes real decisions based on this blog post until the code, and everything is opened up.