Comment on GPT-4o mini: advancing cost-efficient intelligenceComments−freediver2yBased on PyLLMs benchmark. [1]Slightly better than Haiku and slightly slower. Much cheaper.OpenAIProvider('gpt-4o-mini') Total Cost: 0.00385 | Aggregated speed: 105.72 tok/sec | Accuracy: 51.85%AnthropicProvider('claude-3-haiku-20240307') Total Cost: 0.00735 | Aggregated speed: 117.53 tok/sec | Accuracy: 48.15%[1] https://github.com/kagisearch/pyllms−sauwan2yHow long before Anthropic releases Claude-3.5-Haiku at the same price with significantly better performance? OpenAI in trouble...
Comments
Based on PyLLMs benchmark. [1]
Slightly better than Haiku and slightly slower. Much cheaper.
OpenAIProvider('gpt-4o-mini') Total Cost: 0.00385 | Aggregated speed: 105.72 tok/sec | Accuracy: 51.85%
AnthropicProvider('claude-3-haiku-20240307') Total Cost: 0.00735 | Aggregated speed: 117.53 tok/sec | Accuracy: 48.15%
[1] https://github.com/kagisearch/pyllms
How long before Anthropic releases Claude-3.5-Haiku at the same price with significantly better performance? OpenAI in trouble...