Oh, D., & Downey, J. (2024). Does algorithmic content moderation promote democratic discourse? Radical democratic critique of toxic language AI. Information, Communication & Society, 1–20.
Link: https://doi.org/10.1080/1369118X.2024.2346531
Open access: Yes
Notes: Anti-social behavior on digital platforms is often recognized through multiple terminology, such as online hate, incivility, or toxicity. Instead of simply representing the wide range of words we can use to denote such behaviors, these multiple terms point to the complexity in recognizing what anti-sociality is, what are its implications, and how we should best address it. Taking this issue seriously, Oh and Downey explore in this paper the differences between toxicity and civility, arguing that focusing on toxicity risks loosing sight of how these practices are used for pro-social goals (e.g., resist oppression), while focusing on civility remarks how we come together (or apart) through digital conversations. Analyzing data from Perspective API, the authors explore how automated content moderation approaches fail to recognize the nuances in language that denote civility, thus offering a decontextualized (and thus, inefficient) view of anti-sociality on digital platforms. Here, they note that “What makes it difficult for algorithmic moderation to discern incivility and intolerance and hate speech is that they manifest in language differently. Uncivil rhetoric often includes visible swear words and rude, aggressive language that are more easily detectable with pre-trained AI whereas intolerant and hateful messages can be hidden in embedded nuances without explicit hateful slurs and emotional words” (p. 17). And, while I disagree with their final recommendation that what we need is a way of contextualizing automated approaches (instead, I would argue that we need to de-escalate how we talk to one another), their study provides a critically necessary reading into the nuances of addressing anti-behavior.
Abstract: Algorithmic content moderation is becoming a common practice employed by many social media platforms to regulate ‘toxic’ language and to promote democratic public conversations. This paper provides a normative critique of politically liberal assumption of civility embedded in algorithmic moderation, illustrated by Google’s Perspective API. From a radical democratic standpoint, this paper normatively and empirically distinguishes between incivility and intolerance because they have different implications for democratic discourse. The paper recognises the potential political, expressive, and symbolic values of incivility, especially for the socially marginalised. We, therefore, argue against regulation of incivility using AI. There are, however, good reasons to regulate hate speech but it is incumbent upon the users of AI moderation to show that this can be done reliably. The paper emphasises the importance of detecting diverse forms of hate speech that convey intolerant and exclusionary ideas without using explicitly hateful or extremely emotional wording. The paper then empirically evaluates the performance of the current algorithmic moderation to see whether it can discern incivility and intolerance and whether it can detect diverse forms of intolerance. Empirical findings reveal that the current algorithmic moderation does not promote democratic discourse, but rather deters it by silencing the uncivil but pro-democratic voices of the marginalised as well as by failing to detect intolerant messages whose meanings are embedded in nuances and rhetoric. New algorithmic moderation should focus on the reliable and transparent identification of hate speech and be in line with the feminist, anti-racist, and critical theories of democratic discourse.
