Multimodal Rhetoric in Online Media Communications
Multimodal Rhetoric on Online Media Communications
May 2020 - September 2020
The Research Group will investigate how the proliferation of media channels enables political sub-communities to manage and control the creation and dissemination of alternative rhetorical discourses, including advertisements that are personalized according to user profiles and false news stories which have been found to spread faster and more widely than true news stories in platforms such as Twitter. Given that these discourses are increasingly supplanting traditional consensus-based media frameworks, it is essential to understand the mechanisms through which these discourses operate. This includes the prime sites identified as carriers of these discourses and the multimodal strategies (linguistic, visual, filmic) used for target audiences and the resultant effects. In particular, we will establish the mechanisms of such rhetorical formations with respect to their relationship with mainstream news and the deployment of social media for their amplification and transportation. Particular methods employed will involve the tracking of news articles featured on the homepages of prominent news outlets and responses to those articles across different media platforms (e.g. social media, blogs and other sites).
An integral component of these investigations will be the ideational formulations being supported and resisted in rhetorical formations across mainstream and social media platforms (e.g. nationalism, popularism, humanitarism and racism). The project will highlight key questions of how meanings arising from the integration of language, images and videos can undercut or repoint fragments of discourses grounded in conventional systems of truth and rational argumentation in order to promote more loaded and extremist rhetorical formulations. This is seen as an increasingly critical factor in interpreting and understanding the emergence and proliferation of alternative logics of the social order (social ontologies) and conceptions of justice, morality and social obligations (moral orders) at a time of increased tension, unrest and disillusionment in the West (Mohr and Friedland, 2008; Mohr and White, 2008).
The primary approaches addressing these issues to date are 'big data'-based methodologies such as social network analysis, data mining and other tools for analysing large datasets that are grounded in content- and/or platform-focused analyses of messaging and interactions. Such analyses offer important insights about the meaning potential of political and media reports and channels for their dissemination. Nevertheless, approaches of this kind are still predominantly language based and lack theoretically well-founded methods for addressing those sociallyrelevant meanings that emerge from juxtapositions of visual messages, such as images and videos. Such juxtapositions are increasingly seen as decisive for the uptake for messages. As a consequence, existing big-data approaches are insufficient for understanding the impact of such multimodal media messages and their effects within many contexts critical for shaping public opinion.
This project will deliver a Proof of Concept methodology and functional computational system building on multimodal discourse analysis, sociopolitical models of rhetorical effects, and computational deep semantic processing of language, images and their combinations. These together will, on the one hand, augment results being obtained from computer vision and natural language understanding systems, content-based video retrieval and machine learning and, on the other hand, provide detailed discourse-based tracking of messages and their reinterpretations. For this, multimodal discourse analysis offers a highly developed and finely differentiating account of how meaning arises from the integration of language, images and other resources in texts, interactions and events. The combined approach thus aims to resolve the gap between highly-detailed, contextualised analyses of small samples of multimodal texts on the one hand, with highly-aggregated, decontextualised big data approaches (e.g. reductive content analysis) on the other, by leveraging recently developed and emerging multidisciplinary theories and techniques of multimodal analysis, supported by models of sociopolitical processes, and visual communication and critical discourse analysis perspectives. This in itself will demarcate a new high-water mark for productive interactions between the social sciences, humanities and computational approaches that may stand as a model for similar indepth interactions with respect to other themes and topics of investigation.
Members
- Convenors
Prof. Dr. John Bateman
John Bateman, professor of Applied Linguistics, Bremen University, Germany, specializes in functional, computational and multimodal linguistics. His research interests include functional linguistic approaches to multilingual and multimodal document design, semiotics, and theories of discourse.Bremen University (GER)
Prof. Dr. Mehul Bhatt
Mehul Bhatt's research addresses the confluence of Cognition, Artificial Intelligence, Interaction, and Design Science for the development of human-centred cognitive assistive technologies and interaction systems; main focus is on formal, cognitive, and computational foundations for AI technologies with a principal emphasis on knowledge representation, semantics, integration of commonsense reasoning & learning, explainability, spatial representation and reasoning, and cognitive vision.Örebro University (SWE)
Prof. Dr. Kay O'Halloran
Professor Kay O'Halloran is an internationally recognized leading academic in the field of multimodal analysis, involving the study of the interaction of language with other resources in texts, interactions and events. In particular, a key focus of her work is the development of digital tools and techniques for multimodal analysis. Kay is developing mixed methods approaches that combine multimodal analysis, data mining and visualisation for big data analytics.University of Liverpool (UK)
✝ John Mohr
Tribute- Fellows
Ronald L. Breiger
Ronald Breiger works on network theory and methods, networks and culture (with Robin Wagner-Pacifici), analyzing national security language, and multivariate analysis as a network problem.University of Arizona (Tucson, USA)
Kevin Chai
Dr. Chai is currently working as the Lead Data Scientist at the Curtin Institute for Computation. He has a background in data mining, machine learning, natural language processing and computer vision and has experience applying these techniques to research domains such as social sciences, the Humanities, health, medicine, engineering, astronomy and planetary sciences.Curtin University (Bentley, Perth, AUS)
Devin Cornell
Devin Cornell uses computational methods to study cultural processes through which organizations and individuals produce and are shaped by meaning.Duke University (Durham, USA)
Ralph Ewerth
Ralph Ewerth's research focuses on the understanding of multimodal information from a computational perspective; In particular, on computational models that are able to identify cross-modal relationships on different levels, be it on the entity or status level, but especially on an interpretative level.TIB – Leibniz Information Centre for Science and Technology, and Leibniz University of Hannover
Katharina Lobinger
Katharina Lobinger is Senior Assistant Professor for online communication at the Institute of Digital Technologies for Communication (ITDxC). Her main research interests include networked photography, online communication, digital (visual) culture, ethics for the digital age, and creative and visual research methods.Università della Svizzera italiana (USI) (Lugano, SUI)
Robin Wagner-Pacifici
Robin Wagner-Pacifici has developed the framework of political semiosis to analyze the forms and flows of events. A long-term collaboration with John W. Mohr and Ronald L. Breiger analyzing national security language has generated several publications incorporating both close and distant readings.The New School for Social Research (New York, USA)
Publications
- All Publications
John Bateman (2016). Methodological and Theoretical Issues for the Empirical Investigation of Multimodality.
In: N.-M. Klug & H. Stöckl (Eds.), Sprache im Multimodalen Kontext / Language and Multimodality. Berlin: de Gruyter Mouton.
Bhatt, Mehul and Suchan, Jakob (2020). Cognitive Vision and Perception: Deep Semantics Integrating AI and Vision for Reasoning about Space, Motion, and Interaction.,
In ECAI 2020 - 24th European Conference on Artificial Intelligence, 29 August-8 September 2020, Santiago de Compostela, Spain, August 29 - September 8, 2020 - Including 10th Conference on Prestigious Applications of Artificial Intelligence (PAIS 2020), IOS Press, volume 325, 2020. DOI / http://ebooks.iospress.nl/publication/55229
Jakob Suchan and Mehul Bhatt (2018). Semantic Question-Answering with Video and Eye-Tracking Data: AI Foundations for Human Visual Perception Driven Cognitive Film Studies.
In: Proceedings of the Twenty-Fifth International Joint Conference on Artificial Intelligence, IJCAI 2016, New York, NY, USA, 9-15 July 2016 (Subbarao Kambhampati, ed.), ISBN: 9781577357704, IJCAI/AAAI Press, 2016. PDF: https://www.ijcai.org/Abstract/16/374
R. L. Breiger, R. Wagner-Pacifici, and J. W. Mohr (2018). Capturing distinctions while mining text data: Toward low-tech formalization for text analysis.
Poetics, 68:104 – 119, 2018. ISSN 0304-422X. doi: https://doi.org/10.1016/j.poetic.2018.02.005
K. Chai, V. Potdar, and T. Dillon (2009). Content quality assessment related frameworks for social media.
In: O. Gervasi, D. Taniar, B. Murgante, A. Lagana`, Y. Mun, and M. L. Gavrilova (eds), Computational Science and Its Applications – ICCSA 2009, pages 791?805, Berlin, Heidelberg: Springer. ISBN 978-3-642-02457-3.
E. Müller-Budack, J. Theiner, S. Diering, M. Idahl, R. Ewerth (2020). Multimodal Analytics for Real-world News using Measures of Cross-modal Entity Consistency.
In: Proceedings of ACM International Conference on Multimedia Retrieval (ICMR), Dublin, Ireland, ACM, 2020, 16-25.
C. Otto, M. Springstein, A. Anand, & R. Ewerth (2020): Characterization and Classification of Semantic Image-Text Relations.
In: International Journal on Multimedia Information Retrieval (IJMIR), Special Issue (Top papers of ACM ICMR 2019), Vol. 9, Issue 1, 2020, 31-45.
E. Müller-Budack, K. Pustu-Iren, S. Diering, & R. Ewerth (2018). Finding Person Relations in Image Data of News Collections in the Internet Archive.
In Proceedings of International Conference on Theory and Practice of Digital Libraries (TPDL), Porto, Portugal, 2018, 229-240.
C. A. Henning & R. Ewerth (2017): Estimating the Information Gap between Textual and Visual Representations.
In: Proceedings of ACM International Conference on Multimedia Retrieval (ICMR), Bucharest, Romania, ACM, 2017, 14-22.
Venema R., Lobinger K. (2017) And Somehow It Ends up on the Internet.
Agency, Trust and Risks in Photo-Sharing Among Friends and Romantic Partners, First Monday, 22 (7)
Mohr, J.W., Wagner-Pacifici, R., Breiger, R.L., and Bogdanov, P. (2013). "Graphing the Grammar of Motives in U.S. National Security Strategies: Cultural Interpretation, Automated Text Analysis and the Drama of Global Politics."
Poetics: Journal of Empirical Research on Literature, the Media, and the Arts. Vol. 41(6): 670-700.
O'Halloran, K. L. (2015). Multimodal Digital Humanities.
In: P. Trifonas (Ed.), International Handbook of Semiotics (pp. 383-409). Dordrecht: Springer.
R. Wagner-Pacifici, J. W. Mohr, and R. L. Breiger (2015). Ontologies, methodologies, and new uses of big data in the social and cultural sciences.
Big Data & Society, 2(2):2053951715613810, 2015. doi: 10.1177/205391715613810. URL https://doi.org/10.1177/2053951715613810.