Keywords

1 Introduction

1.1 Relevance

Emojis are widely used on social media sites like Twitter, Facebook and Instagram to enrich the conversation with emotions and elaborate meaning in fewer words. Their use, however, is not limited to social media: Chat groups, mail and text messages are examples of other areas where emojis are employed. Emojis are a quick and easy to use way to express one’s emotions in online communications. They appeared on Japanese mobile phones in the 1990s. The origin of the word emoji [25] comes from Japanese e ( , “picture”) + moji ( , “character”). Their popularity has increased internationally in the past two decades. Many people prefer texts with emojis and according to an AMEX OPEN Forum infographic [3], emojis can make a big difference to “post” engagement rates. Posts with emojis get 33% more comments, they are shared 33% more often and they get liked 57% more often than posts without them. A study of the traits of highly influential social media profiles by Simo Tchokni et al. [21] showed the use of emoticons was common factor among these powerful users. However, there are specific cases of emoji usage where emoji usage does not give positive outcomes.

1.2 Relevance

It is somewhat ambiguous, whether one should use an emoji at work related emails or not. Jina Yoo [28] tested how people perceive smiley faces in work email as compared to social email. Researchers sent two types of email messages to a group: a flirtatious message, and another one about extending a job interview request. Emoticons were added to some texts of each type. “The usage of emoticons induced stronger relational outcomes in a task-oriented context than in a socioemotional context. Participants in a task-oriented condition liked the sender more if the sender used an emoticon rather than if the sender used no emoticons” and the sender’s credibility wasn’t affected by the emoticons even when they used up to four emoticons. As the possible explanation of the result is given the following: “emoticons are overused already in socio-emotional contexts, and no special value is assigned to using emoticons in email in the same context. However, when the emoticons are used in a task-oriented context, they might function as a positive expectancy violation, which could bring positive relational outcomes.” Contrary to this, Glikson et al. [14] published a paper “The Dark Side of a Smiley, Effects of Smiling Emoticons on Virtual First Impressions” where it is stated that “contrary to actual smiles, smileys do not increase perceptions of warmth and actually decrease perceptions of competence” and “Perceptions of low competence, in turn, undermined information sharing.” (The authors also mentioned that if all the team members were younger, the likelihood of using emoticons in the team’s conversation was higher). We cannot draw a general conclusion as both studies are evaluated in specific scenarios (1 - extending a job interview request, 2 - First impressions over internet). Most of these papers study only emoticons not visual emojis as emojis have only been commonly available since 2010.

Wang et al. [26] showed that emoticons reduced the negativity effect in business-related email messages, that is the same message sounded less negative when paired with a positive (smiley) emoticon. In addition, emojis have been shown to lead to better memorization of content [10]. Kalyanaraman et al. [17] conducted a study that had participants chat online with “health experts” and “film experts” who either used or avoided emoticons, the participants rated the experts in both topics friendlier and more competent when they communicated with emoticons. This study also noted that emoticons might help you remember what you’ve read – “It appears that the presence of emoticons affects cognition as well, because participants” scores on memory for chat content were significantly higher in the “emoticons present” condition than in the “emoticons absent” condition.”

1.3 History

Emojis are often confused with emoticons. An emoticon is a representation of a facial expression using punctuation marks, numbers and letters, usually written to express a person’s feelings or mood, e.g. While emojis are used like emoticons they are small digital images or icons that exist in various genres, including facial expressions, common objects, food, activities, animals, places and types of weather. For example:

figure c

The first emoji was created in 1999 in Japan by Shigetaka Kurita [4]. However, “The development of emoji was predated by text-based emoticons, as well as graphical representations, inside and outside of Japan. Scott Fahlman, a computer scientist at Carnegie Mellon University, was credited with popularizing early text-based emoticons in 1982 when he proposed using the following: -) and: - (sequence of characters to help readers of a school message board distinguish between serious posts and jokes. From 2010 onwards, hundreds of emoji character sets have been incorporated into Unicode, a standard system for indexing characters, which has allowed them to be used outside Japan and to be standardized across different operating systems. After 2010 each update of the Unicode standards introduced new sets of emojis. “Corporate demand for emoji standardization has placed pressures on the Unicode Consortium, with some members complaining that it had overtaken the group’s traditional focus on standardizing characters used for minority languages and transcribing historical records [4].”

2 Difficulties of Building an Emoji Recommender System

The fact that emojis are related to emotions and are becoming means of communication makes emoji prediction an interesting problem for Natural Language Processing (NLP). If we assume that an emoji is a label of the text corresponding to an emotion, then we would face the sentiment analysis problem. However, the classical sentiment analysis problem only analyses whether a text is positive or negative – sentiment polarities of sentences and documents. Advanced models only have several additional emotions like happiness and anger. On the other hand, emoji classification has a larger population of candidates. As of November 2017 - there are 2623 emojis available in unicode [16]. They are much more detailed and complicated to predict, because one emoji corresponds to many emotions based on its use and the same emotion can be expressed with various emojis. Sentiment analysis using emojis as emotional markers would be a tedious task. Emojis not only express emotions, they also describe professions, activities, flags, food and have gender and racial diversity. This diversity and quantity of emojis makes a recommender system necessary.

Furthermore, recommending emojis in a chat application also requires the understanding of a conversation, since an emoji can be used as:

  • An answer to the previous text if it was a question,

  • A reply for the previous text,

  • A next word in the sentence.

If we were to fully cover the task of emoji prediction, we would need to address question-answering, smart reply and next word prediction problems. Next word prediction is covered because it is hard to determine a start and an endpoint of a conversation. People do not necessarily use the same emojis in the same situations, they might have completely different emoji usage and conversation styles. For example, consider the conversation of two texts: text 1 from Bob to Alice and text 2 from Alice to Bob. Text 1 is lengthy and includes emojis that are mostly negative, however, Alice never uses negative emojis.

3 Description of the Task

The problem can be stated as: Predict the top K emojis that Alice would use after writing a short message that may or may not already include emojis. The Facebook sentiment analysis paper by Tial et al. [23] shows that a sentence with an emoji does not necessarily equal to the same sentence without emojis. Thus, both the words in a sentence and the emojis in the text need to be considered. Studying the Twitter dataset shows three main cases of emoji usage with a text:

  • Expressing an emotion about the text it accompanies.

    i.e. “Last couple months have been crazy! ”

  • As a word in the sentence.

    i.e. “I you”, “Damn, I would love this. Or suicide squad, working towards that. Patty was ”

  • Emoji combinations to express…

    • … a phrase – “ ”- what time should we get coffee?. “ ”- Do you need a ride to the airport? “Had an accounting midterm today that I wasn’t expecting until next week. Lots of unfamiliar terms . Here stands for ‘kill me now’.

    • … the strength of emotion – “ ”- very sad. “ ”- bravo. - very funny. etc.

    • … an emotion/entity that has no separate emoji in the unicode yet? With me now: Every damn corner #TacoTuesday? Here stands for a taco food truck. “ ”- life cycle. “Anti-gravity “ - stands for ’city parkour ’.

The given sentence is studied for the above aspects, but there are many other factors besides the text that affect the usage of emoji. The relations between emojis and emotions are ambiguous, given that there are many emojis [12] to express the same emotion and the same emoji can be used to express different emotions [18]. Furthermore, the same emoji can have very different meanings for different people, and it can lead to misunderstandings [24]. This becomes an obvious problem by looking at the different representations of the same unicode emoji across various platforms. Miller et al. [18] shows differences in the way the same person perceives the same emojis. People can change their emoji usage while typing, based on the device they use.

Various studies over the years show that emoji usage can be different based on gender (females are using emojis with tears more frequently [2]) and culture (Different countries have different favorite emojis [1]). Emoji usage can also depend on the location of the person texting. For instance, a person in Hawaii is more likely to use palm tree and pineapple than a person in Siberia.

Emoji usage can also depend on the weather in the area. For example, if Hawaiian residents visit Mauna Kea (a dormant volcano whose peak is the highest point in the state of Hawaii) in winter, they are likely to use winter emojis. The location and nationality of the person can predict the flags that are used. The dataset suggests that people in the same friend circle tend to use similar emojis and vocabulary. People also tend to use more emojis with short texts than with larger texts. In addition, age can be an interesting feature to observe, not only because there are differently aged emojis, but also people in different generations view emojis differently. These differences make it very difficult to make individual predictions from a general dataset, so the decision was made to personalize the training and create a system that recommends emojis. Taking age, gender, location, and culture into account, it is possible to create user profiles and combine similar profiles for training and prediction.

3.1 Contribution

Unlike existent models that limit the existing unicode set of emojis to a smaller subset, we created a recommender system to personalize emoji predictions. This system categorizes the emoji prediction problem as ’sentiment analysis’, ’next word prediction’ and ‘word-emotion mapping’ tasks, and combines these three models with a heuristic to recommend the top k most relevant emojis.

4 Related Work

Related work on emoji predictions focus on emojis as a sentiment marker. A recent study from Barbieri et al. [5] addresses the prediction problem using a multi class LSTM classifier to predict one emoji per tweet. They mentioned that their system predictions are more accurate than human evaluators, since it is not easy for humans to predict emoji from the text without background knowledge about the text and the person. This combined with the fact that individuals have a different understanding of the same emojis [18] suggests that predictions are dependent on individual people and their way of writing and expressing emotions. In addition, the same study [18] shows that because of the different representations of emojis across different platforms people assign various emotions to the same unicode emoji. Therefore, our system to personalize emoji recommendations includes “individual device” as a separate entity. Sentiment analysis is an interesting research topic for NLP, however, there are not many studies about emoji usage and predictions. Barbieri et al. [5] address the problem to predict one emoji per tweet but they only attempt to predict X different emojis. Currently, there is no research that analyses data using all 2623 emojis. Papers concerning emoji predictions [5, 27] limit the number of emojis used to under 65 of the most popular ones. Most of the models with good performance (accuracy >50%) [5, 27] classify under 10 emojis. Since some of the platforms, sort the emoji list putting the most used ones on the top, recommending only one of the top k emojis does not help the users. User satisfaction and emoji popularity could be increased if relevant to the text emojis were recommended even though they were infrequently used and hard to find.

5 Dataset and Methods

For this study, four datasets were collected from Twitter, including 600,000 tweets (Dataset #1) and 50,000 tweets (Dataset #2) for the following 74 emojis:

Fig. 1.
figure 1

74 popular emojis on Twitter that are used to create the datasets #1 and #2.

Full size image

The third dataset (Dataset #3) was collected for a limited amount of emojis and amounts to 50,000 tweets:

Fig. 2.
figure 2

10 popular emojis on Twitter that were used to create dataset #3.

Full size image

Complete datasets gathered from five volunteers were used to study the personalized emoji recommendations.

5.1 Methods

The following formulation of tasks was designed for the purpose of this study:

  • The initial goal is to predict an emoji used in the tweet, in any possible position, based on the general dataset (the tweets are gathered from multiple users in a 5 to 10 h time frame).

  • Considering the complexity and diversity of emoji prediction, the problem was simplified by making it a binary classification problem: For a chosen emoji, the prediction was whether it will be used in the tweet.

  • Finally, predict an emoji that is used after the given text, based on one user’s data.

  • The fact that one emotion can be described with various emojis suggest that for better results we should recommend a set of emojis that correspond to the same emotion. With this modification the binary classification problem changes to predict whether a tweet contains an emoji from the given set and for the general prediction we can recommend top k emojis based on their probabilities.

5.2 Dataset

For not personalized predictions the data was collected using the Twitter Streaming API [11] in the time period of April 29–May 1, 2017. The language used in the dataset is English. The only preprocessing that has modified the dataset was removing the unnecessary information: URLs, #-signs, malformed words containing numbers, etc. However, the data contains a significant amount of spelling errors so using spelling correction might increase the quality of the dataset and lead to a better solution.

The above datasets are labeled in the two different ways:

  • Each tweet labeled with the emoji used in it.

  • Each tweet labeled with 1/0 based on whether a specific emoji/or an emoji from a specific set of emojis is used in the tweet.

The second case also needs balancing the dataset after the labeling, since it can result in a disproportionate amount of data for a binary classification problem. One more thing to mention is that there are many cases when a tweet contains several emojis. In this case, we labeled the tweet with the first occurring emoji. The emoji is by default one of the labels of the tweet and the choice of the first emoji is random; in addition labeling the same tweet with various labels would create a problem that we would have to address later on the learning phase.

For personalized prediction the data for training, testing and evaluation is from the same Twitter user and includes 47000 tweets. For labeling the dataset we use all available unicode emojis. We created a mapping of each emoji to its description and key words that it associates with. Tweets are split into sentences that are followed by an emoji and then labeled with the emoji that was following the text.

figure s

  • E.g. I like that <3 thought I would not participate :/

  • Would produce the following sentences and labels:

  • Sentence 1: I like that

  • Label 1: <3

  • Sentence 2: I like that <3 thought I would not participate

  • Label 2: :/

In addition, we took the combinations of emojis that were used together to express a non-existent emoji or a phrase (in short we call them combojis) into account and added them into the labels’ list. For each label we calculate and update the frequency of how many times it is used. When combojis frequency reaches to certain limit we generate a mapping and keywords for it, based on previous use. The frequency is also used as a feature for training.

5.3 Word Representations

For the word representations, we used one hot encoding and word embeddings.

  • One Hot Encoding. For one hot encoding [8], we calculate the frequency of words in all tweets. Then, we take the k most frequently used words and create a binary vector for each tweet. Each binary vector has 5,000 entries, where an entry corresponds to one of the k most frequent words. Entries are filled with an 1, if the corresponding word is present in the tweet and 0 if it is not.

  • Word Embeddings. For word embeddings [6], we, again, calculate the frequency of words in all tweets. For each tweet, we create a vector with as many entries as there are words in a tweet. Each entry is filled by using a word’s index in the k most frequently used words as the value. If a word does not occur in the k most frequent words, we fill the vector entry with zero. In the end, we train the word embeddings with the neural network. Similar words should be placed close together in the vector space after the learning. Finally, the resulting word representations are split up into training, testing and evaluation sets.

5.4 Personalized Learning

The model for personalized learning combines three solutions for subtasks and a scoring function. Emojis and combojis can be expressing an emotion and a next word in the sentence. Therefore, we combined the approaches of next word suggestion, algorithm and sentiment analysis.

5.5 Next Word Suggestion Algorithm

We need to construct an algorithm that fulfills the following steps: Build a language model using twitter text and then use this language model to predict the next word as the user types.

We need to calculate the frequency of words and n-grams and use a sliding window.

If we assume the training data shows the frequency of “university” is 198, “university student” is 12 and “university professor” is 10. We calculate the maximum likelihood estimate (MLE) as:

  • The probability of “university student”:

    Pmle (entry|data) = 12/198 = 0.06 = 6%

  • The probability of “university professor” is:

    Pmle (streams|data) = 10/198 = 0.05 = 5%

If the user types, “university”, the model predicts that “student” is the most likely next word.

The n-gram model description steps are:

  • Generate 2-grams and 3-grams.

  • Select n-grams that account for 60% of word instances. This reduces the size of the models.

  • Calculate the maximum likelihood estimate (MLE) for words, for each model.

As for the prediction: We use the ngram models on tokenized and preprocessed user input. We implement a Stupid backoff [7] starting on the 3-gram model backing off to the 2-gram model and returning 3 words with the largest MLE.

5.6 Mapping from a Word to Emoji

We created the mapping using 2623 available unicode emojis from unicode.org and their descriptions as names. We generate keywords based on the frequency of the words that they are used with in a sentence. The mapping file is updated to store information for combojis. The file will be used to map the next predicted word to associated emojis.

5.7 Evaluating Emotion

Both in general and personalized tasks we train a Long Short-Term Memory (LSTM) network to predict which emojis are used with the text. LSTM is a special kind of Recurrent Neural Network (RNN) able to learn long term dependencies. This kind of RNNs are good at remembering information for long periods of time.

LSTMs were introduced by Hochreiter and Schmidhuber [15]. Over the course of years they were popularized and developed by various contributors, since they perform well on a large variety of problems. Over the course of years, LSTMs proved to perform well for NLP tasks including sentiment analysis.

5.8 Scoring

Scoring of emojis in the final stage is based on the prediction probability produced by a trained model for labeling, label frequency in the existing dataset that the model is trained on and an additional feature for measuring confidence.

Out of the above two models we get two sets of predictions for each emoji. The next word prediction task with mapping assigns probabilities to the possible labels (emojis) - whether they have been used before or not. The sentiment prediction part uses already used emojis as labels. Therefore, it generates probabilities for a limited set of emojis. We have a confidence score to adjust the two probabilities based on the relative frequency of emojis.

For example: If an emoji has a prediction probability 0.9 and its relative frequency is in the top 10%, we give it a high confidence score. Finally, we calculate the weighted sum of the prediction probability and confidence and recommend the top k emojis for a given text.

5.9 Implementation

While achieving the results of this paper required a substantial amount of scripting, three Python libraries were essential to the analysis.

  • NLTK [19]. The Natural Language ToolKit (NLTK) is a Python library specialized in natural language processing. We made use of its word tokenizing capabilities and used its naive Bayes classifier to create the baseline method.

  • SciKit [13] Scikit is a machine learning library for python that offers tools for data mining and analysis. We have used its implementations of the algorithms - Logistic Regression and Stochastic Gradient Descent.

  • Keras on TensorFlow [20, 22]. Keras is a high-level neural networks API, written in Python and capable of running on top of either TensorFlow or Theano. Tensorflow, an open-source library for numerical computations, was developed by Google Brain Team for the purposes of conducting machine learning and deep neural networks research. This study used it to train word embedding vectors, build an LSTM classifier and optimize it for accuracy.

The NLTK and SciKit classifiers are used with the one hot encoding of the tweets. Each tweet is labeled with the emoji, that is contained in the tweet.

The LSTM neural network is used with word embeddings vectors. The dataset has been labeled in the following ways:

  • For a specific emoji:

    1. 1.

      The label for each tweet equals to 1 if the tweet contains the emoji.

    2. 2.

      Otherwise the label equals to 0.

  • For a specific emotion:

    1. 1.

      Create a set of emojis that correspond to the emotion.

    2. 2.

      The label for each tweet equals to 1 if the tweet contains an emoji from the given set.

    3. 3.

      Otherwise the label equals to 0.

The emojis used for a single emoji classifier are:

figure t

The emoji sets used related to the same emotion are:

Fig. 3.
figure 3

List of the emojis grouped by relation to similar emotions.

Full size image

The example of combojis generated after training on an individual data are:

Fig. 4.
figure 4

Combojis that were generated after a training process on the individual data do-nated by a twitter user.

Full size image

6 Evaluation Methods

For the general task we use the prediction accuracy as a metric.

As for the individual prediction we have two evaluation metrics: Precision at the top k candidates and the mean reciprocal rank that is used to evaluate the ranking of the top predictions.

In the future, it is necessary to use human evaluation - If people did not use emojis in a text that does not mean that they would not use them if a recommender system was available.

7 Analysis

Since the computations used in the implementation are quite time costly, the classifiers for multi-variable problem were trained on the smaller dataset (Dataset #2). For the results please see the Table 1.

Table 1. Accuracy of the naive approach
Full size table

Unfortunately, Stochastic Gradient Descent was not a fit for the problem, as for the Naive Bayes and Logistic Regression they have shown improvements when hyper-parameters were reset. The LSTM neural network was implemented solely for the binary classification problems, using word embeddings. It has been directed to optimize the accuracy over the training phase. This resulted in an average of 71% accuracy for a single emoji classifier, and an average of 70% accuracy for the classifier of a set containing 4 emojis related to the same emotion. The emoji sets related to the same emotion were chosen naively and for the future improvements it is necessary to explore the ways to create the emojis sets that are the most related to each other and interchangeable in the everyday usage.

The following figure summarizes the overall evaluation for the average values of accuracy:

Fig. 5.
figure 5

Accuracy measures for different set of emojis.

Full size image

The binary classifier result demonstrated in the figure is trained, tested and evaluated on 6:2:2 proportions of 365,000 tweets. The original dataset used is Dataset #1, which after preprocessing and balancing is reduced to 365,000 tweets. A binary classifier is used to determine for each emoji whether it should be recommended for a tweet or not. Given the result we can assume that, if recommendation for an emoji is given with 71% of accuracy, a recommender system that uses suggestion of three top results would significantly increase overall accuracy of the suggestion.

Experiments [5] showed that human evaluators on average achieve 80% accuracy on twitter dataset. Sine the accuracy of the recommendation for each emoji equals to 0.71, the system is close to human performance.

To compare the results with existing papers, we created a combination of 5 binary classifiers that predicts the one emoji that has the highest probability out of 5. However we need to take into account that the datasets are not the same. Especially Xie et al. [27] who use Weibo data in Chinese language.

Fig. 6.
figure 6

P@1 evaluation for 5 emoji binary classifier compared to other papers.

Full size image

Accuracy for the next word suggestion algorithm using stupid backoff is 13.5%. After mapping the words with emojis and combining it with LSTM predicted emojis recommender system had the accuracy of 34% on average. While LSTM only achieves 74%.

Fig. 7.
figure 7

P@1 and P@3 evaluations for LSTM recommenders.

Full size image

In fact, incorporating the Next Word Prediction (NWP) algorithm may decrease the performance measure. Since NWP together with mapping does not limit any emojis from the full unicode emoji list [16] it forms recommendations by including a wide range of rarely used emojis. The users in our personalized dataset only chose up to 125 emojis out of the 2623 available. In order to better evaluate the accuracy of our recommender system, we need active Twitter users to train on their dataset and record useful recommendations in real-world setting.

8 Conclusion

Emojis are used in everyday life by millions of people, which makes them a widely used tool for expressing emotions. A vast amount of data is available for experimenting with Machine Learning algorithms for an emoji related problem. It is an interesting NLP task to analyze and explore the ways they relate to emotions. In addition, the existence of emojis gives us the opportunity to research the emotions of online usage of particular events such as analyzing the sentiment towards U.S. Presidential Candidates [9]. Potential applications give software developers the incentive to create systems that make it easier for users to include emojis in their texts [29]. This paper shows that it is possible to create a recommender system for emojis with reasonable accuracy.

8.1 Multi-variable vs. Binary Classifier

The task of creating a multi-variable classifier is easy to formulate. It is also easy to create the necessary dataset. However, it required exploring emoji usage first: Which emojis correspond to the same emotion? How many emojis to select for a single study? Which emojis are used together? It is possible to predict usage of one emoji. Solving the problem for individual cases lets us formulate the solution of the initial problem. Recommending several emojis from the same classifier also increases the accuracy of a recommender system.

Additionally, there are also issues to address regarding the binary classifier. It introduces bias and needs balancing of the dataset.

Another way to resolve all the differences in emoji usage for a multi classifier is personalizing the dataset for a particular user. This helps develop a system that would be highly useful for active users, but can be problematic for the users that do not have enough data for the model to learn. Finally, generalizing task by not restricting the dataset but creating user profiles can have a potential to achieve a good result. Since it would be possible to create a personalization element for ambiguous emojis and benefit from the knowledge acquired from a larger dataset.

8.2 Future Work

As future work, we can do improvements for various steps of the development:

  • Dataset:

    • Training the model on a bigger dataset from individual users to make it possible to profile based on features like location and ethnicity.

    • We need to evaluate accuracy of the above methods compared to human operators.

  • Clustering emojis: Exploring ways to create improved mapping (emoji:words) for the classifiers:

    • Set of emotions that have corresponding emojis without intersection.

    • Set of the emojis that correspond to the same emotion.

The latter would allow us to improve the error function, since recommending an emoji related to the correct label should be weighted as a smaller error. Ultimately the system performance must be tested in a real-world setting.