NLP-enabled Design Assistance For Visual Communication



Journal Title

Journal ISSN

Volume Title



In visual communication, a wide range of design components are typically used to increase the comprehension of content and to convey the author’s intent. Different authoring and graphic design applications perform automatic design assistance that include images and text in different forms and shapes. However, a majority of publicly available tools are mainly driven by some basic heuristics in assisting users during authoring. Prior research has begun to use Artificial Intelligence (AI) to provide users with interface suggestions. Considering a wide range of applications and its unique challenges, this interdisciplinary area has not been fully studied and has little cross-disciplinary collaboration. In this dissertation, we aim at advancing new technology to employ AI-based models to assist users during authoring by recommending appropriate design components based on the content. In particular, the first part of this dissertation focuses on the task of emphasis selection, i.e., choosing candidates for emphasis, where we propose label distribution learning methods to capture the ambiguity of the input. These models are designed to comprehend the most common interpretation of a written text, so the right emphasis can be achieved automatically or interactively. In the second part of this dissertation, we focus on the task of font recommendation, i.e., suggesting fonts based on input text, where we model the associations between visual font attributes and textual context, with the final goal of better font recommendation during text composition. Specifically, we propose different end-to-end models that exploit contextual and emotional representations of the input text to recommend fonts. We introduce a total of three new language resources for both tasks of emphasis selection and font recommendation. We focus on social media data as well as presentation slides, and by examining the challenges of the language resources, we provide different analysis components. Besides a wide range of applications, the methods discussed in this dissertation are meant to benefit relevant tasks such as machine-human interaction, reading comprehension, graphic design, and user experience.



automatic design assistance, NLP-enabled design assistance


Portions of this document appear in: Shirani, A., Dernoncourt, F., Asente, P., Lipka, N., Kim, S., Echevarria, J., and Solorio, T. Learning emphasis selection for written text in visual media from crowd-sourced label distributions. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics (Florence, Italy, July 2019), Association for Computational Linguistics, pp. 1167–1172; and in: Shirani, A., Dernoncourt, F., Echevarria, J., Asente, P., Lipka, N., and Solorio, T. Let me choose: From verbal context to font selection. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics (Online, July 2020), Association for Computational Linguistics, pp. 8607–8613; and in: Shirani, A., Dernoncourt, F., Lipka, N., Asente, P., Echevarria, J., and Solorio, T. SemEval-2020 task 10: Emphasis selection for written text in visual media. In Proceedings of the Fourteenth Workshop on Semantic Evaluation (Barcelona (online), Dec. 2020), International Committee for Computational Linguistics, pp. 1360–1370; and in: Shirani, A., Tran, G., Trinh, H., Dernoncourt, F., Lipka, N., Echevarria, J., Solorio, T. and Asente, P. PSED: A Dataset for Selecting Emphasis in Presentation Slides. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics, Association for Computational Linguistics. (ACL 2021) [Accepted]