Discord scoops up harassment-detection AI startup Sentropy

    
18
Oof.

Happy Thursday: Here’s your weekly argument over toxicity, moderation, and AI, courtesy of Discord, which has apparently purchased a company you’ve probably heard of once and then immediately forgot. It’s a security company that was launched publicly last year thanks to funding from everything from Twitch and Riot Games to Twitter and Reddit founders. Its pitch at the time was a product called Sentropy Detect, which purported to offer data-driven moderation tools to fight userbase toxicity at scale.

While existing customers of Sentropy’s products are essentially being given a couple of months to migrate elsewhere, Discord users might welcome the move, as Sentropy’s founders say they’re “joining Discord to continue fighting against hate and abuse on the internet.”

“Together, we can create belonging safely. As we have gotten to know Discord, we have been impressed by their deep commitment to safety — a commitment that extends beyond their own communities to the broader internet. One of the hard-and-fast requirements Sentropy had as we discussed working together was that our future home allows us to keep our focus on improving the internet as a whole, not just improving the Trust and Safety (T&S) capabilities of the company. Our team’s focus will be on helping Discord expand and evolve its T&S capabilities; we are also inspired by Discord’s commitment to knowledge-sharing and capability-building in the content moderation space, as exemplified by their Moderator Academy. We are excited to help Discord decide how we can most effectively share with the rest of the Internet the best practices, technology, and tools that we’ve developed to protect our own communities.”

Discord’s toxicity problem is well-publicized at this point, largely thanks to Discord’s regular transparency reports. The last one issued noted that user reports skyrocketed through the pandemic, with general harassment and cybercrime being the most-reported problems but spammers being deleted the most – on the order of over 3M spam accounts in half a year. The company deleted over 27K servers in the same period, most often for cybercrime and “exploitative content”; it reported almost 7K to the National Center for Missing and Exploited Children; and it removed over 1500 for violent extremism and 334 for QAnon extremism. Needless to say, Discord has a problem – but whether even the combination of Discord’s safety team and Sentropy’s AI-driven detection can stem the tide is an open question.

Source: Medium via Kotaku
Advertisement

No posts to display

18
LEAVE A COMMENT

Please Login to comment
  Subscribe  
newest oldest most liked
Subscribe to:
Reader
Rndomuser

Harassment is bad and should be properly be dealt with, however AI-driven detection is not a solution because it will not be able to detect whether the harassment is serious or just a joke between friends who know each other well enough to know that stuff they type to each other should not be treated seriously and it will not be able to detect harassment by very creative people who will be able to know how to harass others without triggering the automatic detection. It may also lead to situations where creative trolls will be able to force AI-driven detection to mark specific words/emotes as “harassment” even though those words/emotes would not be originally considered as “harassment”.

Reader
Brinto Sfj

Doesn’t matter to me at all. All of this look like 1st world problem to me.

Reader
Utakata

So harassment is a 1st world problem?

EmberStar
Reader
EmberStar

To someone who’s never been affected by it, almost certainly. :(

Reader
Utakata

…or lived under a pig privilege rock all their life. /sigh

Reader
Bruno Brito

Pigs are great. Don’t compare cute pigs to these rampant assholes who think the world revolve around their issues uwu

Reader
Utakata

Off Topic: I like snakes too. But that doesn’t stop me calling out “snake in the grass” or “snake oil” when it’s deserved.

Reader
Kickstarter Donor
Peregrine Falcon

Agreed. In some countries people are starving, dying of easily cured diseases, attacked and imprisoned by horrible dictators, but in the first world nations? Oh no. Here we complain about the “horrors” of online harassment instead of just putting the person on ignore or mute.

Reader
Utakata

Pigtailed Pro Tips:

1) There is no real such thing as here versus there.

2) Online harassment is a very real thing. Don’t be so dismissive.

3) Still don’t believe me then; have you ever been Swatted? That can actually kill you.

Reader
Kickstarter Donor
Peregrine Falcon

1) Yes there is.

2) I agree with you that it’s a problem. It’s also one that’s easily solved by the block/ignore button.

3) I’ve been online since 1996. How come I’ve never been SWATed? Oh yeah, I don’t show my face or give people any info that they can use to get my address. I’ve been commenting on MOP since it started, and on Massively of Old for years more, and you don’t even know which country I’m in. Try SWATing that!

I’m not being dismissive. But what it boils down to is most people in first world nations bring most of their problems on themselves. They cause the drama themselves and then they overreact to it, and they want the rest of us to play along. When the reality is that it’s a problem that you can solve by simply clicking on the ignore/block button.

Reader
Utakata

This is one of the worst ways of dealing with this. That is, using digital algorithms to solve their harassment issues. And when it goes south, to which it likely will, it will only add fuel to the Tucker Carlson types on the internets who resists anything that threatens their privilege. Just saying.

MilitiaMasterV
Reader
MilitiaMasterV

Definition of discord

(Entry 1 of 2)
1a : lack of agreement or harmony (as between persons, things, or ideas) … must we fall into the jabber and babel of discord while victory is still unattained?— Sir Winston Churchill
b : active quarreling or conflict resulting from discord among persons or factions : strife marital discord discord between the two parties
2a music
(1) : a combination of musical sounds that strikes the ear harshly
(2) : dissonance The song ends on a discord.
b : a harsh or unpleasant sound

discord verb
dis·​cord | \ ˈdi-ˌskȯrd
, di-ˈskȯrd \
discorded; discording; discords

Definition of discord (Entry 2 of 2)

intransitive verb
: disagree, clash

Because I’m sure they care about toxicity…

/sarcasm

Reader
Vanquesse V

Anyone leaving discord over this should probably not use any google product (chrome, gmail, youtube, android etc), anything owned facebook or amazon and probably also microsoft.
This fight is one we lost more than a decade ago

EmberStar
Reader
EmberStar

Fisty
Reader
Fisty

Every part of the internet that ever existed has had groups of weird and extreme people. It is the nature of the beast. I don’t like them getting away with shit, but I don’t like the idea of some AI policing that.

Reader
Kickstarter Donor
Peregrine Falcon

Well, I guess it’s time for me and my friends, Deplorables all, to uninstall Discord and find another chat app.

Saluka
Reader
Saluka

I can see a no win no fee lawyer with dollar signs in their eyes from a class action. It’s ok man, you can stop camping the emergency room now.

Seriously though it’s a bad idea because lots of people could get wrongly targeted. Only real solution is to have a white list of words you are only allowed to say and nobody wants to see that.

Reader
Kickstarter Donor
Loyal Patron
Jack Pipsam

It’s tricky, the nature of Discord allows hate groups an easy and accessible place hangout under the radar, there’s no doubt the platform’s success has brought ill-intentioned users enmass with them getting up to horrible things.

But something about AI scanners just really gives me an icky feeling, even if it’s inevitable on larger platforms and Discord I suppose never pitched themselves as a privacy platform such as Signal or Session.