That's a great question. ChatGPT would probably do better. It's just a matter of cost and speed.
Let’s calculate the price of using GPT-3.5 to classify 10 million tweets. A very typical job.
The price is $0.002 per 1k tokens on GPT-3.5 Turbo. (Really it’s $0.0015 for output, $0.002 for input).
That’s $1 for 500k tokens, or $2 for 1M tokens.
Now lets’s classify 10M tweets. A tweet is 144 characters, so it’s roughly 100 tokens. Let’s also say the instructions are the size of a tweet, and the output is just 1 token (yes or no). That gives us 200 tokens per tweet classification, for a total of 2B tokens to process.
That costs 2B / 500k = $2,000 to run this job. Not so bad if it's mission critical, but starting to get pretty expensive. If I can get comparable performance using a homemade classifier, it makes much more sense to use that instead.
Fundamentally, it's overkill to use a 175B parameter model on many of these tasks. Also, the number of classifications can start to grow very quickly if doing things like classifying pairs of data points.
Comments
Isn't that just asking ChatGPT yes/no questions?
Why use a classifier instead of just having it answer the question directly?
That's a great question. ChatGPT would probably do better. It's just a matter of cost and speed.
Let’s calculate the price of using GPT-3.5 to classify 10 million tweets. A very typical job.
The price is $0.002 per 1k tokens on GPT-3.5 Turbo. (Really it’s $0.0015 for output, $0.002 for input).
That’s $1 for 500k tokens, or $2 for 1M tokens.
Now lets’s classify 10M tweets. A tweet is 144 characters, so it’s roughly 100 tokens. Let’s also say the instructions are the size of a tweet, and the output is just 1 token (yes or no). That gives us 200 tokens per tweet classification, for a total of 2B tokens to process.
That costs 2B / 500k = $2,000 to run this job. Not so bad if it's mission critical, but starting to get pretty expensive. If I can get comparable performance using a homemade classifier, it makes much more sense to use that instead.
Fundamentally, it's overkill to use a 175B parameter model on many of these tasks. Also, the number of classifications can start to grow very quickly if doing things like classifying pairs of data points.