Follow-up question: what work has been done on client-side moderation? I know this gets dangerously close to the kind of content scanning that eg apple has tried (to very detrimental results), but hear me out: I really think this is a prerequisite to end-to-end encryption on a social network (there has to be some level of protection; even if 100% of users report 100% of bad content, imagine scrolling a feed and stumbling upon CSAM simply because you were the first person to see it). I also also think it's possible to strike a balance that preserves user agency while still protecting them, by simply inserting a manual reporting step. So, for example, potentially problematic content gets put behind an interstitial with a content warning and options to view, hide, report, etc. But again, this requires client-side content classification.
I'm aware of eg NSFWJS, which is a tensorflowJS model [1]. Is there anything else that, say, can also do violence/gore detection?
I don't think such scanning is logically a prerequisite to end-to-end encryption. In the case of a direct message from one person to another, one presumes that the recipient either wishes to receive the content or considers it spam.
Spam filters on the recipient side are already viable; there's no advantage to having them on the sending side where they can be more easily bypassed. If the recipient does intend to receive illegal content, then they will simply arrange to use a communication method which does not have scanning on the sender's side. Whether or not encryption is used is immaterial.
What about group chats? Again, if the group doesn't want to receive illegal content then they can use a bot account for spam detection. End-to-end encryption doesn't change the situation at all.
I'm weary (if not wary) of discussion on this topic because it always comes down to the idea that developers of communication methods have a responsibility to read and potentially censor everything that is sent by that method. Yet that was never the usual expectation for snail mail; indeed many countries have laws against reading other people's letters. It was also never the expectation for the Internet back when it was being designed. So by all means discuss the relative merits of censorship, but the existence of encryption is not a valid argument one way or the other!
I think OP is asking for the opposite. Platform-level moderation practically has to happens server-side because clients can just lie about having done the scanning.
I believe the ask here is for receive-side moderation, similar to an ad blocker. If the messages are encrypted, the platform can't moderate them. Consumers would do their own content moderation locally and discard/hide/clickthrough/etc content they don't want to see, without actually removing it from the platform.
Eg I'm sure the video of the UHC CEO getting shot made the rounds in some group chats, and that would be hard for some people to see. These kinds of filters would let those users hide the video on their messenger without impacting anyone else.
If it's going on in any large scale way on a major site, I've not encountered it (but it might not be something they'd shout about) but similar types of models are being put into apps frequently, as seen in this post recently posted to HN: https://altayakkus.substack.com/p/you-wouldnt-download-an-ai (a mobile app that detects currency client-side)
One problem is legal liability around training and testing. If you use a large foundation vision model, you could describe your moderation criteria in text and cross your fingers, but those aren't going to be running client-side. If you want a compact, efficient model to embed into an app, you need to train it.. but with what? There's a dilemma here where it's easier to build filters for legal things because you can't legally use the illegal things to train a filter for the illegal things! Systems like NSFWJS work because run of the mill porn is legal in most jurisdictions, but an equivalent "CSAMJS" filter would be a legal nightmare to produce.
Re: liability: definitely an issue. Might be the kind of thing that can only really be done in cooperation with eg ICMEC.
That being said, it might be possible to get to "better than nothing" with transfer learning, by training a model to detect both "there are children here" as well as "this is pornography". I have no idea what the success rate would be, but there have been some pretty impressive generalization results in recent years with classification models.
That seems viable. But even if it works, you need to eval/test it.. which isn't a job I'd ever want to be within a million miles of. I hope the organizations tasked with tackling the problem are working on solutions third parties can deploy because they're probably the only people who legally and morally can. New laws like the UK's Online Safety Act make this quite an urgent task, too.
Comments
Follow-up question: what work has been done on client-side moderation? I know this gets dangerously close to the kind of content scanning that eg apple has tried (to very detrimental results), but hear me out: I really think this is a prerequisite to end-to-end encryption on a social network (there has to be some level of protection; even if 100% of users report 100% of bad content, imagine scrolling a feed and stumbling upon CSAM simply because you were the first person to see it). I also also think it's possible to strike a balance that preserves user agency while still protecting them, by simply inserting a manual reporting step. So, for example, potentially problematic content gets put behind an interstitial with a content warning and options to view, hide, report, etc. But again, this requires client-side content classification.
I'm aware of eg NSFWJS, which is a tensorflowJS model [1]. Is there anything else that, say, can also do violence/gore detection?
[1] https://github.com/infinitered/nsfwjs
I don't think such scanning is logically a prerequisite to end-to-end encryption. In the case of a direct message from one person to another, one presumes that the recipient either wishes to receive the content or considers it spam.
Spam filters on the recipient side are already viable; there's no advantage to having them on the sending side where they can be more easily bypassed. If the recipient does intend to receive illegal content, then they will simply arrange to use a communication method which does not have scanning on the sender's side. Whether or not encryption is used is immaterial.
What about group chats? Again, if the group doesn't want to receive illegal content then they can use a bot account for spam detection. End-to-end encryption doesn't change the situation at all.
I'm weary (if not wary) of discussion on this topic because it always comes down to the idea that developers of communication methods have a responsibility to read and potentially censor everything that is sent by that method. Yet that was never the usual expectation for snail mail; indeed many countries have laws against reading other people's letters. It was also never the expectation for the Internet back when it was being designed. So by all means discuss the relative merits of censorship, but the existence of encryption is not a valid argument one way or the other!
I think OP is asking for the opposite. Platform-level moderation practically has to happens server-side because clients can just lie about having done the scanning.
I believe the ask here is for receive-side moderation, similar to an ad blocker. If the messages are encrypted, the platform can't moderate them. Consumers would do their own content moderation locally and discard/hide/clickthrough/etc content they don't want to see, without actually removing it from the platform.
Eg I'm sure the video of the UHC CEO getting shot made the rounds in some group chats, and that would be hard for some people to see. These kinds of filters would let those users hide the video on their messenger without impacting anyone else.
Exactly correct, and excellent real-world example!
If it's going on in any large scale way on a major site, I've not encountered it (but it might not be something they'd shout about) but similar types of models are being put into apps frequently, as seen in this post recently posted to HN: https://altayakkus.substack.com/p/you-wouldnt-download-an-ai (a mobile app that detects currency client-side)
One problem is legal liability around training and testing. If you use a large foundation vision model, you could describe your moderation criteria in text and cross your fingers, but those aren't going to be running client-side. If you want a compact, efficient model to embed into an app, you need to train it.. but with what? There's a dilemma here where it's easier to build filters for legal things because you can't legally use the illegal things to train a filter for the illegal things! Systems like NSFWJS work because run of the mill porn is legal in most jurisdictions, but an equivalent "CSAMJS" filter would be a legal nightmare to produce.
Re: liability: definitely an issue. Might be the kind of thing that can only really be done in cooperation with eg ICMEC.
That being said, it might be possible to get to "better than nothing" with transfer learning, by training a model to detect both "there are children here" as well as "this is pornography". I have no idea what the success rate would be, but there have been some pretty impressive generalization results in recent years with classification models.
That seems viable. But even if it works, you need to eval/test it.. which isn't a job I'd ever want to be within a million miles of. I hope the organizations tasked with tackling the problem are working on solutions third parties can deploy because they're probably the only people who legally and morally can. New laws like the UK's Online Safety Act make this quite an urgent task, too.