I hate to be that guy. But this seems like the perfect use case for an LLM. First put content through your script and then through a decently prompted LLM. Anything it catches, put in a queue for manual review.
i'm pretty sure most content moderation strategies operate on a budget below what most LLMs would cost.
for any site hosting user-generated content, how efficiently you can run your moderation system is essentially your unique selling point - LLM is the baseline.
You may be shocked at how inexpensive some of the LLMs are these days.
Google Gemini 1.5 Flash 8B charges $0.04/million input tokens and $0.15/million output tokens.
If a piece of content that needs to be moderated is 1,000 tokens (that's pretty long!) and you expect a 10 token return it will cost you 0.0039 cents - that's not a dollar amount, that's less than a 250th of a single cent.
So 1 cent will moderate 250 items of content. $1 will moderate 25,000 items of content.
you could use some cheaper LLMs for this, even GPT-3.5turbo could excel at this or the moderations API (but that in my experience was more trained to obey the laws of USA, for example guns were okay to be asked about but are pretty much illegal in my country). The simplest way is a blocklist of terms which was what we had previously used containing an insane number of terms beyond the regular abusive terms, but it would need some updating from time to time.
A phisher may insert text for an LLM with a disclaimer that's only an educational example of what not to do, or that they're the PayPal CEO authorizing this page.
Comments
I hate to be that guy. But this seems like the perfect use case for an LLM. First put content through your script and then through a decently prompted LLM. Anything it catches, put in a queue for manual review.
Yep, Azure offers automated content moderation: https://azure.microsoft.com/en-us/products/ai-services/ai-co...
I don't know if they're using a LLM specifically, as opposed to say computer vision models or some other method.
Pricing: https://azure.microsoft.com/en-us/pricing/details/cognitive-...
AWS has something similar: https://aws.amazon.com/rekognition/content-moderation/
i'm pretty sure most content moderation strategies operate on a budget below what most LLMs would cost.
for any site hosting user-generated content, how efficiently you can run your moderation system is essentially your unique selling point - LLM is the baseline.
You may be shocked at how inexpensive some of the LLMs are these days.
Google Gemini 1.5 Flash 8B charges $0.04/million input tokens and $0.15/million output tokens.
If a piece of content that needs to be moderated is 1,000 tokens (that's pretty long!) and you expect a 10 token return it will cost you 0.0039 cents - that's not a dollar amount, that's less than a 250th of a single cent.
So 1 cent will moderate 250 items of content. $1 will moderate 25,000 items of content.
LLMs are dirt cheap. You can play around with pricing across different models using my calculator here: https://tools.simonwillison.net/llm-prices
you could use some cheaper LLMs for this, even GPT-3.5turbo could excel at this or the moderations API (but that in my experience was more trained to obey the laws of USA, for example guns were okay to be asked about but are pretty much illegal in my country). The simplest way is a blocklist of terms which was what we had previously used containing an insane number of terms beyond the regular abusive terms, but it would need some updating from time to time.
The alternative is paying a person…
Definitely considering LLMs. At the day job, we had a team fine tune a model built to detect phishing content and it worked surprisingly well.
I'd be worried that LLMs are incredibly gullible.
A phisher may insert text for an LLM with a disclaimer that's only an educational example of what not to do, or that they're the PayPal CEO authorizing this page.
I agree that LLM could do most of the moderation work. You could use a multi-modal LLM for image moderation.