Twitch creates new "ban evasion" flag to identify problem users
posted Saturday Dec 4, 2021 by Scott Ertz
The past few years have been a constant up and down for Twitch. The company has seen its user base grow to incredible levels. It's seen new types of users join the platform, bringing artists and musicians, podcasters and IRL streamers, to join the gamers who already inhabited the platform. With new users come new problems, and one of those problems has been massive harassment in the streamers' chats. The company has used traditional methods of identifying and eliminating problems, but a new approach has been developed: ban evasion.
How we got here
Since its acquisition by Amazon in 2014, the platform has grown immensely. It inspired Microsoft to purchase Beam and turn it into Mixer (which it later shut down). It inspired Facebook to focus on gaming, and YouTube created a gaming platform, which it folded into the main offering. But it's all because of Twitch.
However, not everything has been rosy for Twitch. A data breach revealed how much streamers get paid. They suddenly received DMCA takedown requests and didn't know how to properly handle them. However, the platform's biggest issue has been harassment.
The problem
In recent months, the existing harassment issues have gotten significantly worse. The problem has come from a combination of the tactic used to flood chat with messages as with the approach which Twitch took to determine issues. Traditionally, platforms have used a rudimentary content filtering approach, where a dictionary lookup is used to see if someone is misbehaving. This is the same approach that led to a gamer being banned from Xbox Live because he lived in Fort Gay, West Virginia.
On the other side is the actual issue - it's really easy to create a new, free account and continue the attack. The process is called Sock Puppet Account creation, and it's really easy. You create a free Gmail account and use it to create a free Twitch account. If the platform requires a 2FA, you use the email to create a free Google Voice number and activate 2FA. You then spam hate and vitriol until the system detects that you're terrible and bans you. Oh well, just start over and continue. This is the problem that the new flag is designed to address.
The solution
Twitch announced a new flag called "ban evasion" which takes a server-side approach to detecting harassment issues before they happen. The idea is that the system will, upon account creation, attempt to detect if the user in question appears to be attempting to avoid the effects of being banned on the platform or from a chat.
The Suspicious User Detection system, which is now turned on by default, will classify users into "likely" and "possible" ban evaders. For "likely" evaders, the account will not have the ability to publicly chat. The system will hide the messages from the public's view but will show the content to the creator themselves and their mods. For "possible" ban evaders, the user will have chat ability, but will show up in the creator and mod dashboard as a potential issue. Creators and mods will be able to properly handle these users if they believe they are threats.
It's important to note that AI is never perfect. Twitch warns creators, saying,
One thing to prepare for, particularly around launch, is that no machine learning will ever be 100% accurate, which means there is a possibility of false positives and false negatives. That's why Suspicious User Detection doesn't automatically ban all possible or likely evaders. You're the expert when it comes to your community, and you should make the final call on who can participate. The tool will learn from the actions you take and the accuracy of its predictions should improve over time as a result.
So, as a creator, be proactive on moderation so that the system can get better for everyone.
