- Developed an AI system that recommends hashtags by utilizing text and images of social media postings and human knowledge

- 39.86% better recommendation over existing hashtag recommendation systems based on text images alone

- Selected for an oral presentation at IEEE ICWS 2023, an international conference on computer science

* Image description: Overview of the multimodal hashtag recommendation system (EXTRA) developed by Professor Kim Gang-min's research team at the CUK

A research team led by Professor Kim Gang-min of the Department of Data Science and Artificial Intelligence at the Catholic University of Korea (President Fr. Luke Won Jong-chul) has succeeded in developing a multimodal AI system that thinks based on common sense like a human. The research team developed a transformer-based “multimodal hashtag recommendation system (“EXTRA”)” that recommends appropriate hashtags for posts by utilizing human knowledge acquired through a database in addition to the text and images of posts on social media platforms.

On social media platforms, hashtags (#) are an important marketing tool that compiles and categorizes a large amount of data and creates social issues or trends. However, existing hashtag recommendation models utilized on social media platforms such as Instagram and Facebook have been limited as they generally use only texts and images of the posts, which do not reflect background knowledge beyond the information provided.

For example, in a photo of a certificate of completion of a pet training center, a human can understand that the key subject matters of the image are "pet training" and "completion of training," but existing hashtag recommendation models do not have the background knowledge that "dogs with certificates are usually trained," so they only recommend hashtags that contain simple information such as "pet" and "certificate" and rarely recommend hashtags related to training.

Accordingly, Professor Kim’s research team sought a new approach to the hashtag recommendation system to consider human knowledge. The team's multimodal hashtag system uses a methodology that extracts external knowledge from images to effectively comprehend the content of posts.

By utilizing a classifier trained on the Open Directory Project, a hierarchical structure of the knowledge base of web directories, the research team extracted knowledge embedded in the image and used it to understand the unstated context. In this process, the team utilized Google's deep learning model, transformer, which excels at utilizing the attention mechanism (a way to understand the dependency and relevance of words in a sentence) of the encoder-decoder structure.

The results of the study showed that feeding external knowledge into the multimodal transformer model, along with the text and images of the post, significantly improved hashtag recommendation performance over existing hashtag recommendation systems. The average F1-score, a metric that evaluates hashtag recommendation performance, improved by 39.86%.

"This study confirmed that the human knowledge system expressed in the form of a knowledge base helps the multimodal AI model understand posts on social media," said Professor Kim Gang-min of the Catholic University of Korea's Department of Data Science and Department of Artificial Intelligence. "We hope our study results can be used in various ways, such as for the development of external knowledge extraction methodologies in the field of multimodal AI models and the application of hashtag recommendation systems to actual social media services."

The research was conducted by a team of researchers, including Graduate School students Hyunsik Won and Soomin Noh, led by Professor Kim Gang-min of the Catholic University of Korea's Department of Data Science (also known as the Department of Artificial Intelligence), and it was funded by the National Research Foundation of Korea's Excellent Young Researcher Support Program. Additionally, the National Research Foundation of Korea selected the research for an oral presentation at the IEEE International Conference on Web Services (ICWS) 2023, a renowned international conference in the field of computer science. The first author, Hyunsik Won, attended and gave an oral presentation at the conference, which took place in Chicago in early July. (End of Document)