Essays, Posts & Presentations

Face Emoji Identification

Jade Han

Author

Jade Han

Title

Face Emoji Identification

Description

Face Emoji Identification

Training a Classifier to Learn Various Types of Face Emojis and Identifying Them in an Image

Author/Mentor/Date

Author: Jade Han
Mentor: Faizon Zaman
Date: July 12th, 2019

Introduction

So, what does it mean to train a classifier to learn various types of face emojis and identify them in an image? For example, a person might add a face emoji on his or her image in order to decorate it. When the image is processed, the program should be able to find the face emojis he or she used in the image. After locating them, the program should return what the face emoji is, such as, ‘Crying face’ or ‘Sleepy face.’

Finding Face Emojis in an Image

Locating the emojis is accomplished via SelectComponents and a masking workflow using ColorDistance and Filling Transform. I made a few example images to test my functions.

Example Image

Step One: Color Distance

Given that most of the face emojis have a dominant color of yellow, I ran ColorDistance with yellow on the example image.

Gray-scale image where darker shades are closer to the target color, yellow

In[]:=

stepOne=ColorDistanceexampleImage,



Out[]=

Step Two: Binarize

I then applied the Binarize function to the output of ColorDistance in order to get a black and white image.

Generate the Binarized image where pixel values between 0 and 0.3 become 1 and all others values become 0

In[]:=

stepTwo=Binarize[stepOne,{0,0.3}]

Out[]=

Step Three: FillingTransform

After taking the step of Binarize, I used FillingTransform in order to get perfect circles in the example image.

Simplified features of the image by filling the entire circle with white

In[]:=

stepThree=FillingTransform[stepTwo]

Out[]=

Step Four: SelectComponents

Next, I used SelectComponents to find features with circularity greater than .9 in the example image

Detects the circles in the image

In[]:=

stepFour=SelectComponents[stepThree,#Circularity>.9&]

Out[]=

Step Five: ComponentMeasurements

I then applied ComponentMeasurements which returned the position of those circles.

Position of the circles in the image

In[]:=

stepFive=ComponentMeasurements[stepFour,{"Centroid","EquivalentDiskRadius"},All,"ComponentPropertyAssociation"]

Out[]=

1Centroid{142.672,192.477},EquivalentDiskRadius23.8566,2Centroid{43.022,161.744},EquivalentDiskRadius26.5169,3Centroid{16.5,167.5},EquivalentDiskRadius0.56419

Step Six: Disk

Later, Disk function became useful. It created a disk out of centroid and equivalent disk radius properties, helping me to find the exact coordinates of the circles’ positions.

Disk created out of centroid and equivalent disk radius properties

In[]:=

stepSix=Disk[#Centroid,#EquivalentDiskRadius]&/@stepFive

Out[]=

1Disk[{142.672,192.477},23.8566],2Disk[{43.022,161.744},26.5169],3Disk[{16.5,167.5},0.56419]

Step Seven: Extract & emojiC (Custom)

In order to return the corners of the sub-images that contain circles from these coordinate values, I wrote a function called emojiC.

Corners of the sub-images that contains circles

In[]:=

emojiC[{{x_,y_},r_}]:={{x-r,y-r},{x-r,y+r},{x+r,y-r},{x+r,y+r}}

Corners of the sub-images that contains circles in the image

In[]:=

stepSeven=emojiC[Extract[Part[Values[stepSix],#],{{1},{2}}]]&/@Range[1,Length[Values[stepSix]]]

Out[]=

{{{118.815,168.62},{118.815,216.334},{166.528,168.62},{166.528,216.334}},{{16.505,135.227},{16.505,188.261},{69.5389,135.227},{69.5389,188.261}},{{15.9358,166.936},{15.9358,168.064},{17.0642,166.936},{17.0642,168.064}}}

Step Eight: ImageTrim

Using emojiC, I picked up the coordinate values of the corners of the sub-images and used ImageTrim to trim the input image using the coordinates.

Trimmed sub-images based on the coordinates that I got

In[]:=

stepEight=ImageTrim[exampleImage,#]&/@stepSeven

Out[]=





Step Nine: Select

The issue of this process was that there were some sub-images of a circle that don’t contain face emojis. Given that in most cases the sub-image containing the emoji will have the largest dimensions, I went through the area of each sub-images and drop those that are smaller than 1% of the largest sub-image using Select.

Areas of each sub-images

In[]:=

areasOfPixels[{x_,y_}]:=(x*y)

Sub-images bigger than 1% of the biggest sub-image

In[]:=

stepNine=Select[stepEight,(areasOfPixels[ImageDimensions[#]])>0.01(Last[Sort[Map[areasOfPixels,Map[ImageDimensions,stepEight]]]])&]

Out[]=





Running the Face Emoji Classifier

Step Ten: emojiClassification (Custom)

Lastly, I applied custom face emoji classifier trained on different face emoji images with a method of Image Feature Extractor & Logistic Regression, and ran on the subimages containing the face emojis. Those face emojis with different colors such as red, green, blue, and purple, will also be easily detected by using their dominant colors.

What the face emojis in the image are

In[]:=

stepTen=emojiClassification[stepNine]

Out[]=

{Pleading Face,Face With Rolling Eyes}

How was the face emoji classifier built?

Classifying the emojis is accomplished via the Classify function where the training data is an association of emoji-names and emoji-images from Apple, Google, and Twitter.

Small Part of the Dataset (Demonstration)

Emoji Classifier

Using that dataset, I trained the classifier on different face emoji images with Logistic Regression and the “ImageFeatures” FeatureExtractor.

In[]:=

emojiClassification

Out[]=

ClassifierFunction

Input type: Image

Number of classes:



In[]:=

emojiClassification



Out[]=

Grinning Face

In[]:=

emojiClassification



Out[]=

Pouting Face

In[]:=

emojiClassification

,"TopProbabilities"

Out[]=

{Smiling Face With Hearts0.108849,Face Savoring Food0.0110509,Astonished Face0.0109093,Angry Face With Horns0.0109005}

Extension

Face Emojis with Different Dominant Color (ex. Red, Blue, Green, etc.)

Detecting different color emojis can also be achieved by using color distance with different colors.

Face Emojis with Different Shape (ex. Face emojis with horns, glasses, etc.)

Since I detected the face emojis on the images via their circularity, I experienced difficulty in creating masks for those face emojis that are not circular. In order to expand on my work, I have to find out a way to detect those face emojis with external features, such as horns, glasses, and hearts. I believe this concept would be achieved by refining or finding the properties and values for the function SelectComponents instead of circularity.

Cite this as: Jade Han, "Face Emoji Identification" from the Notebook Archive (2019), https://notebookarchive.org/2019-08-0gjfwwg