ZiF Research Group

Multimodal Rhetoric in Online Media Communications

May 2020 - September 2020

Convenors: Kay O'Halloran (Perth, AUS), John Bateman (Bremen, GER), John Mohr (Santa Barbara, USA), Mehul Bhatt (Örebro, SWE)

The Research Group will investigate how the proliferation of media channels enables political sub-communities to manage and control the creation and dissemination of alternative rhetorical discourses, including advertisements that are personalized according to user profiles and false news stories which have been found to spread faster and more widely than true news stories in platforms such as Twitter. Given that these discourses are increasingly supplanting traditional consensus-based media frameworks, it is essential to understand the mechanisms through which these discourses operate. This includes the prime sites identified as carriers of these discourses and the multimodal strategies (linguistic, visual, filmic) used for target audiences and the resultant effects. In particular, we will establish the mechanisms of such rhetorical formations with respect to their relationship with mainstream news and the deployment of social media for their amplification and transportation. Particular methods employed will involve the tracking of news articles featured on the homepages of prominent news outlets and responses to those articles across different media platforms (e.g. social media, blogs and other sites).

An integral component of these investigations will be the ideational formulations being supported and resisted in rhetorical formations across mainstream and social media platforms (e.g. nationalism, popularism, humanitarism and racism). The project will highlight key questions of how meanings arising from the integration of language, images and videos can undercut or repoint fragments of discourses grounded in conventional systems of truth and rational argumentation in order to promote more loaded and extremist rhetorical formulations. This is seen as an increasingly critical factor in interpreting and understanding the emergence and proliferation of alternative logics of the social order (social ontologies) and conceptions of justice, morality and social obligations (moral orders) at a time of increased tension, unrest and disillusionment in the West (Mohr and Friedland, 2008; Mohr and White, 2008).

The primary approaches addressing these issues to date are 'big data'-based methodologies such as social network analysis, data mining and other tools for analysing large datasets that are grounded in content- and/or platform-focused analyses of messaging and interactions. Such analyses offer important insights about the meaning potential of political and media reports and channels for their dissemination. Nevertheless, approaches of this kind are still predominantly language based and lack theoretically well-founded methods for addressing those sociallyrelevant meanings that emerge from juxtapositions of visual messages, such as images and videos. Such juxtapositions are increasingly seen as decisive for the uptake for messages. As a consequence, existing big-data approaches are insufficient for understanding the impact of such multimodal media messages and their effects within many contexts critical for shaping public opinion.

This project will deliver a Proof of Concept methodology and functional computational system building on multimodal discourse analysis, sociopolitical models of rhetorical effects, and computational deep semantic processing of language, images and their combinations. These together will, on the one hand, augment results being obtained from computer vision and natural language understanding systems, content-based video retrieval and machine learning and, on the other hand, provide detailed discourse-based tracking of messages and their reinterpretations. For this, multimodal discourse analysis offers a highly developed and finely differentiating account of how meaning arises from the integration of language, images and other resources in texts, interactions and events. The combined approach thus aims to resolve the gap between highly-detailed, contextualised analyses of small samples of multimodal texts on the one hand, with highly-aggregated, decontextualised big data approaches (e.g. reductive content analysis) on the other, by leveraging recently developed and emerging multidisciplinary theories and techniques of multimodal analysis, supported by models of sociopolitical processes, and visual communication and critical discourse analysis perspectives. This in itself will demarcate a new high-water mark for productive interactions between the social sciences, humanities and computational approaches that may stand as a model for similar indepth interactions with respect to other themes and topics of investigation.