This is awesome. I was looking at benchmarking speed and quality myself but didn't go this far!
I wonder about Claude Instant and Phi 2?
Modal.com for inference felt crazy fast, but I didn't note the metrics.
Good ones to add?
Replicate.com too maybe?
Thanks! For Claude instant, select the dropdown on the top right of the card where it says '8 Selected' and can add it to the graphs. Thanks for the suggestions for adding Phi 2, Model.com as a host, can look into these!
Comments
This is awesome. I was looking at benchmarking speed and quality myself but didn't go this far! I wonder about Claude Instant and Phi 2? Modal.com for inference felt crazy fast, but I didn't note the metrics. Good ones to add? Replicate.com too maybe?
Thanks! For Claude instant, select the dropdown on the top right of the card where it says '8 Selected' and can add it to the graphs. Thanks for the suggestions for adding Phi 2, Model.com as a host, can look into these!