That's a great question. ChatGPT would probably do better. It's just a matter of cost and speed.
Let’s calculate the price of using GPT-3.5 to classify 10 million tweets. A very typical job.
The price is $0.002 per 1k tokens on GPT-3.5 Turbo. (Really it’s $0.0015 for output, $0.002 for input).
That’s $1 for 500k tokens, or $2 for 1M tokens.
Now lets’s classify 10M tweets. A tweet is 144 characters, so it’s roughly 100 tokens. Let’s also say the instructions are the size of a tweet, and the output is just 1 token (yes or no). That gives us 200 tokens per tweet classification, for a total of 2B tokens to process.
That costs 2B / 500k = $2,000 to run this job. Not so bad if it's mission critical, but starting to get pretty expensive. If I can get comparable performance using a homemade classifier, it makes much more sense to use that instead.
Fundamentally, it's overkill to use a 175B parameter model on many of these tasks. Also, the number of classifications can start to grow very quickly if doing things like classifying pairs of data points.
Comments
That's a great question. ChatGPT would probably do better. It's just a matter of cost and speed.
Let’s calculate the price of using GPT-3.5 to classify 10 million tweets. A very typical job.
The price is $0.002 per 1k tokens on GPT-3.5 Turbo. (Really it’s $0.0015 for output, $0.002 for input).
That’s $1 for 500k tokens, or $2 for 1M tokens.
Now lets’s classify 10M tweets. A tweet is 144 characters, so it’s roughly 100 tokens. Let’s also say the instructions are the size of a tweet, and the output is just 1 token (yes or no). That gives us 200 tokens per tweet classification, for a total of 2B tokens to process.
That costs 2B / 500k = $2,000 to run this job. Not so bad if it's mission critical, but starting to get pretty expensive. If I can get comparable performance using a homemade classifier, it makes much more sense to use that instead.
Fundamentally, it's overkill to use a 175B parameter model on many of these tasks. Also, the number of classifications can start to grow very quickly if doing things like classifying pairs of data points.