Multimodal Prediction of Public Trust in Politicians from Speech and Text

This study investigates whether public trust in politicians can be automatically predicted from their speech or twitter messages, and integration of these two communication modalities into a multimodal prediction procedure. A database of speech samples and twitter messages representing ten USA political figures was generated and labeled based on the outcomes of a publicly available online ranker. Two trust labels were created, i.e., the low-trust (five politicians) and the high-trust (five politicians). The database was first used to test unimodal prediction based either on speech or text, and then it was applied to validate a proposed multimodal approach. The unimodal classification was achieved by training two separate multilayer perceptron neural network (MLP-NN) models, one for speech and one for text. Whereas the proposed multimodal approach concatenated prediction vectors resulting from individual modalities to train a third multimodal decision-making MLP-NN. The text-based prediction achieved the F1score of 83%, the speech-based - 89%, and the combined speech and text-based prediction resulted in the F1-score of 92.5%.