Applications of Pre-Trained Language Models in Sentiment and Authorship Tasks



Journal Title

Journal ISSN

Volume Title



In recent years, transfer learning in the form of pre-trained neural language models have significantly transformed the research and application of NLP. Techniques and models such as ELMo, ULMFit, Transformer and BERT have claimed state-of-the-art results on a wide range of NLP tasks. In this dissertation, we explore utilizing pre-trained language models to address two major areas in Natural Language Processing: Authorship Verification and Sentiment Analysis. Our design focus for these models is not only to achieve state-of-the-art performance in the respective tasks, but also high flexibility and high interpretability. For the problem of Authorship Verification, we propose an unsupervised solution that utilizes pre-trained deep language models to compute a new metric called \emph{DV-Distance}. The proposed metric is a measure of the difference between the two authors comparing against pre-trained language models. Our design addresses the problem of non-comparability in authorship verification, frequently encountered in small or cross-domain corpora. To the best of our knowledge, this work is the first one to introduce a method designed with non-comparability in mind from the ground up, rather than indirectly. It is also one of the first to use Deep Language Models in this setting. The approach is intuitive, and it is easy to understand and interpret through visualization. Experiments on four datasets show our methods matching or surpassing current state-of-the-art and strong baselines in most tasks. For sentiment analysis, we proposed two iterations of a framework called Sentiment-Aspect Attribution Module (SAAM) . SAAM works on top of traditional neural networks and is designed to address the problem of multi-aspect sentiment classification and sentiment regression. The framework works by exploiting the correlations between sentence-level embedding features and variations of document-level aspect rating scores. We first propose several variations of SAAM and demonstrate their effectiveness on top of CNN and RNN based models. Experiments on a hotel review dataset and a beer review dataset have shown SAAM can improve sentiment analysis performance over corresponding base models. Moreover, because of the way our framework intuitively combines sentence-level scores into document-level scores, it is able to provide a deeper insight into data (e.g., semi-supervised sentence aspect labeling). Hence, we also provided a detailed analysis that shows the potential of our models for other applications such as sentiment snippet extraction. Lastly, this dissertation also presents SAAM v2. SAAM v2 is a dramatically improves over the original version, by addressing three of its major shortcomings. We demonstrate SAAM v2's capabilities by combining it with pre-trained language model architectures AWD-LSTM and RoBERTa. Evaluation of SAAM v2 on the hotel and beer review datasets confirms the module can provide better expressiveness and overall performance. Furthermore, the model can also estimate sentence-level aspects at a much higher accuracy. We end our model analysis by showcasing some of the fine-grained latent information discovered by SAAM v2.



Machine Learning, Deep Learning, Sentiment Analysis, Authorship Verification, Transfer Learning


Portions of this document appear in: Zhang, Y., Boumber, D., Hosseinia, M., Yang, F. and Mukherjee, A., 2021. Improving Authorship Veri cation using Linguistic Divergence. In Workshop on Reducing Online Misinformation through Credible Information Retrieval (ROMCIR 2021), held as part of ECIR 2021; and in: Boumber, D., Zhang, Y., Hosseinia, M., Mukherjee, A. and Vilalta, R., 2019. Robust authorship verification with transfer learning. In Proceedings of the 2020 International Conference on Computational Linguistics and Intelligent Text Processing (CICLing 2020); and in: Zhang, Y., Yang, F., Hosseinia, M. and Mukherjee, A., 2020. Multi-Aspect Sentiment Analysis with Latent Sentiment-Aspect Attribution. In The 2020 IEEE/WIC/ACM International Joint Conference on Web Intelligence and Intelligent Agent Technology (WI-IAT 2020).