[WSC19] Automated Spot the Difference
Author
Stella Maymin
Title
[WSC19] Automated Spot the Difference
Description
A website for users to spot automatically generated differences on curated images.
Category
Essays, Posts & Presentations
Keywords
Image Processing, Spot the Difference, Machine Learning, Game, Website, Wolfram Cloud, WSC19
URL
http://www.notebookarchive.org/2019-07-5jueeyb/
DOI
https://notebookarchive.org/2019-07-5jueeyb
Date Added
2019-07-12
Date Last Modified
2019-07-12
File Size
62.34 megabytes
Supplements
Rights
Redistribution rights reserved
Download
Open in Wolfram Cloud
WOLFRAM SUMMER SCHOOL 2019
Autogenerated Spot the Difference Creation
Autogenerated Spot the Difference Creation
Stella Maymin
Rory Fougler
Rory Fougler
Summary
Summary
Spot the differences have become fairly popular as they are commonly found in magazines and online. Yet they have always been done by hand, be it changing a color or removing an object all together. Until now. I have created a website that auto-generates images by removing their contents or editing them, followed by shoving the edited content back into the original image. This way, once a user chooses a difficulty level: easy, medium, hard, or impossible, whether they would like one randomly chosen image or all of the images in the difficulty range they chose before, and the image they would like to use, the images are shown in a gallery format with the original image on the left and the edited one, the one with a difference, is on the right. Once they have their answer, they click on the blank image to the very right in order to reveal the correct answer.
General Image Functions
General Image Functions
The heart of all this functionality is ImageContents, a function which uses machine learning to find and identify things in a given image. Though ImageContents uses the same idea as ImageIdentify, it can find items in subimages and not only the entire image.
Let’s take the ImageContents of this elephant image:
In[]:=
ImageContents
Out[]=
|
The function boundingboxes returns all the positions where the contents were found.
In[]:=
boundingboxes[image_]:=Quiet[Check[List@@@Flatten@Values@Normal@ImageContents[image,All,"BoundingBox"],{}]]
Here we will find the positions of the contents:
In[]:=
boundingboxes
Out[]=
{{{495.333,60.0667},{789.287,561.619}},{{103.53,17.7209},{432.382,288.452}}}
The function mask solves the problem of not being able to auto-generate masks based solely on their bounding boxes. Mask takes the positions in the order given by boundingboxes, and the dimensions of the image. It also takes an optional parameter called previous which if equaled to none does nothing, and otherwise is applied to rows and columns instead of the default 0. Finally for the rectangle inside the given rows and columns becomes 1.
In[]:=
mask[{{col1_,row1_},{col2_,row2_}},{nrows_,ncols_},previous_:None]:=Table[If[nrows-row2<row≤nrows-row1+1&&col1<col≤col2,1,If[previous===None,0,previous[[row,col]]]],{row,nrows},{col,ncols}]
For example, let’s mask the elephants:
In[]:=
Image@mask#,Reverse@ImageDimensions
&/@boundingboxes
Out[]=
,
The function expunge uses the function mask to create a mask for the background to recognize that the image contents should be masked by their surroundings and not by parts of other contents in the image; this prevents images where Inpaint is applied to, say, the larger elephant and instead of Inpainting the empty space with grass, it would be covered with multiple baby elephant heads. Unlike mask, the expunge function only takes one parameter: the image.
In[]:=
expunge[image_]:=With[{dim=Reverse@ImageDimensions[image],boxes=boundingboxes[image]},With[{background=Fold[mask[#2,dim,#]&,None,boxes]},Inpaint[image,mask[#,dim],Method{"TextureSynthesis",MaskingImage[1-background]}]&/@boxes]]
Let’s try using expunge on the elephants:
In[]:=
expunge
Out[]=
,
Enlist creates a list from any given parameter, and if the parameter is already in the form of a list, Enlist does nothing, in order to keep the parameter as a list.
In[]:=
Enlist[l_List]:=lEnlist[l_]:=List[l]
The function ImageMapAt takes three parameters: the image, a function, and the coordinates of the contents in the forms of x1, x2, etc. ImageTrim is used to trim the contents out of the image and applies the given function to the trimmed contents. ImageMapAt uses ImageCompose to shove the edited contents back into the original image, one at a time, and uses the function Enlist shown above. Also, ImageMapAt uses the parameters {Left,Top} to align the contents correctly, so it doesn’t return this output:
In[]:=
ImageMapAt[image_,f_,{{x1_,x2_},{y1_,y2_}}]:=ImageCompose[image,#,{x1,y2},{Left,Top}]&/@Enlist@f[ImageTrim[image,{{x1,x2},{y1,y2}}]]
Let’s try applying ImageMapAt to the elephants:
In[]:=
ImageMapAt
,aquarecolor,First@boundingboxes
Out[]=
,
,
In the function ImageMapAll, there are two possibilities: either the function given equals None in which case the function expunge would be applied to the image, or there is a function in which case it would be applied to every content’s bounding box.
In[]:=
ImageMapAll[image_,f_:None]:=If[f===None,expunge[image],Table[ImageMapAt[image,f,box],{box,boundingboxes[image]}]]
Let’s try ImageMapAll on the elephant image:
In[]:=
ImageMapAll
,adjust
Image Difference Functions
Image Difference Functions
My next step was to create functions to be applied to the images to automate the differences. The main function being mask, defined above. The following functions were based on built-in functions in the Wolfram Language, but specified to visible yet not too obvious variations and though they may seem very clear when applied to the entire image, when they are used on smaller contents, depending on the difficulty chosen, they are increasingly difficult to spot.
The blurred function blurs an image by 2 units.
In[]:=
blurred[image_]:=Blur[image,2]
Let’s apply blurred to the elephant image now:
In[]:=
blurred
Out[]=
The darker function recolors up to three of the dominant colors in an image to colors 1/2 darker.
In[]:=
darker[image_]:=ImageRecolor[image,#Darker[#,1/2]]&/@Take[DominantColors[image],UpTo[3]]
Let’s apply the function darker to the dominant colors in the elephant image:
In[]:=
darker
Out[]=
,
,
The aquarecolor function, similar to darker, recolors up to three of the dominant colors and replaces it with 0.45 of the gradient “Aquamarine.”
In[]:=
aquarecolor[image_]:=ImageRecolor[image,#ColorData["Aquamarine",0.45]]&/@Reverse@Take[Reverse@DominantColors[image],UpTo[3]]
Let’s try it now:
In[]:=
aquarecolor
Out[]=
,
,
The grain function uses ImageGraphics to create a grain-like texture of .05 units over the given image.
In[]:=
grain[image_]:=ImageGraphics[image,MinColorDistance.05]
Let’s look at what this function does with the elephants:
In[]:=
grain
Out[]=
The quantize function uses ColorQuantize and a randomly chosen integer by creating a texture visually similar to the grain one. Generally, ColorQuantize separates colors by quantizing them.
In[]:=
quantize[image_]:=ColorQuantize[image,RandomInteger[{8,11}]]
Let’s see how the elephant picture looks with this function applied:
In[]:=
quantize
Out[]=
The adjust function takes an image and applies ImageAdjust to it with parameters that make it distinctly brightened in some areas and darkened in others as it adds a tilter over its entirety, while being not overly obvious.
In[]:=
adjust[image_]:=ImageAdjust[image,{0,0.2,1.55}]
Let’s see what happens on the elephants:
In[]:=
adjust
Out[]=
Edits is a list of all the image edit functions explained above to make it easier to call them in later functions. Alone, edits does nothing, for it is merely a list of separate functions, but when used in larger code, it is fairly useful.
In[]:=
edits={blurred,darker,aquarecolor,grain,quantize,adjust,None};
Displaying Image Differences
Displaying Image Differences
The next step was to display the auto-generated differences. In theory, this is fairly simple: merely subtract the original image from image with differences, one by one. The difficult part, though, is how to display them as the default version is placed on a black background, making the differences, or edits, hard to understand.
Let’s look at the differences:
In[]:=
expunge
Out[]=
,
,
,
,
,
In[]:=
%-
Out[]=
,
,
,
,
,
As you can see above, the black background almost hides parts of the differences. To solve this, I applied ColorNegate to the input shown above, using the %, referring to the previous output cell according to the number given.
ColorNegate%29-
Out[]=
,
,
,
,
,
ColorNegate did work, and it is somewhat easier to see the differences, but the white background seems to be swallowing the differences that also include white, showing only part of them. To create a visual that is both bright enough to identify and dark enough to show all the parts, I tried using the function Image Adjust.
In[]:=
ImageAdjust/@%29-
Out[]=
,
,
,
,
,
Image Distance Sorting
Image Distance Sorting
Once I have created and shown the differences, my next step was to sort the edited images into the order of larger changes to smaller ones. I coded this using the function ImageDistance and placed the edited images before the original one.
From the data, I concluded that the larger the distance is, the bigger the change. The smaller changes were harder to spot and the edits were done on smaller contents. I also concluded that the smaller distances in this data had neater and nicer-looking masks.
For example, compare these two distances:
ImageDistance
,
Out[]=
160.055
Clearly, this image has a very large distance from the original, to be exact 160.055 units. Most of the image has been edited. If you haven’t noticed already, the mask function took the texture from the chair on the right and covered the table with it, because the mask recognizes the textures around the chosen image.
ImageDistance
,
17.2903
This one is harder to spot according to the computer-generated distance, because the distance is from the original to the edited image is smaller, exactly 17.2903 units. If you haven’t spotted it yet, the silverware by the bread has been masked, and it seems as it is also using the texture of the chair to hide them, just as the image above did. This distance is about 10 times harder to spot than the previous one, and a human would say so as well.
TakeQuantile
TakeQuantile
The idea of the function TakeQuantile is a function that returns a scaled portion of a given list. It takes the parameters: list and part (the given list, and the part one wants to scale to). I used Take of the list, ranging from the first to the last element of it. Then, I take the quantiles I need and round them, so they are integers, because it is nearly impossible for a computer to take the portion of the list associated with 3.69. Lastly, I take that given part of the list. This is used later to get, for example, the second quartile of images, ranked by difficulty.
In[]:=
TakeQuantile[list_,part_]:=Take[list,Round[Quantile[Range[Length[list]],part]]]
Some examples of this are shown below:
In[]:=
TakeQuantile[Range[20],{1/2,3/4}]
Out[]=
{10,11,12,13,14,15}
In[]:=
TakeQuantile[Range[20],{3/4,1}]
Out[]=
{15,16,17,18,19,20}
Adding Images to the Website
Adding Images to the Website
In General, LibraryAdd is a function that is used to add images into my curated image library, called gallery. I can LibraryAdd a new image to my gallery anytime, though it takes a while. When I reevaluate the website code, the new image will be shown in the image selection area.
Specifically, using With, I define time as the string of the number of seconds elapsed since 1.1.1970. Next, I define counter as 0 using Block, because within Block I can change the value of the variable, unlike With. If adjusted is True, then apply the function ImageAdjust to the differences and CloudExport that output; if not, then just CloudExport the new image by itself. Also, if it is adjusted, I add an “a” at the end. This is CloudExport-ed in the form of a .png image, into my image gallery, joined with time. Prepend puts the original image first in the list of all images and those other images are created by using every possible combination of edit function on every content. I sort with easiest first because the easier images have a larger distance than the more difficult ones.
Specifically, using With, I define time as the string of the number of seconds elapsed since 1.1.1970. Next, I define counter as 0 using Block, because within Block I can change the value of the variable, unlike With. If adjusted is True, then apply the function ImageAdjust to the differences and CloudExport that output; if not, then just CloudExport the new image by itself. Also, if it is adjusted, I add an “a” at the end. This is CloudExport-ed in the form of a .png image, into my image gallery, joined with time. Prepend puts the original image first in the list of all images and those other images are created by using every possible combination of edit function on every content. I sort with easiest first because the easier images have a larger distance than the more difficult ones.
In[]:=
LibraryAdd[image_]:=With[{time=ToString[UnixTime[]]},MapIndexed[Table[CloudExport[If[adjusted,ImageAdjust[#-image],#],"PNG","gallery/"<>time<>"/"<>ToString[First[#2]]<>If[adjusted,"a",""]<>".png",Permissions"Public"],{adjusted,{True,False}}]&,Prepend[ReverseSortBy[Flatten@Table[ImageMapAll[image,e],{e,edits}],ImageDistance[#,image]&],image]]]
Creating Difficulty Settings
Creating Difficulty Settings
In order to breakdown the code for my website, I first tried creating difficulty settings. They would be organized using the ImageDistance could shown and described above. My first step was to just create a drop down menu called “difficulty”, in which the user would choose from the settings: easy, medium, hard, or impossible. If you click on the link, all you will see is a drop down menu from which you choose from easy, medium, hard, or impossible, and the website returns that word.
In[]:=
CloudDeploy[FormPage[{"difficulty"{"easy","medium","hard","impossible"}},#difficulty&]]
Out[]=
Finalizing Website Code
Finalizing Website Code
CloudDeploy creates a website and FormFunction creates a page that asks a user questions, they answer by selecting from a drop down menu and choosing an image. “Difficulty” is the first drop down menu for difficulty settings that a user chooses. Because I don’t always know for certain the amount of contents and total images, “easy” is the bottom quartile, “medium” is the second quartile, and so on and so forth. Lastly, the user decides between having one randomly chosen image to spot the differences from, called “random”, or all the possibilities in the difficulty section they chose previously, referred to as “all”.
The basic idea is that next to the original (left) and the new image (right), there will be a blank, or white image that I refer to as “blank.” When the user clicks on the “empty space” once they think they know the difference, the correct difference is shown. If it is clicked again, the image returns to its original white state.
To code this, I first created a white image that is 500 by 500 and applied CloudExport to it so that it could be easily used in my function later.
Next, I need First of slot because CloudObjects[] returns a CloudObject["filename"] object, and I only need the string, so I take First to drop out the CloudObject[] wrapper. This is StringJoin-ed with "/1" to get the first image from that directory, because inside of the directory gallery are directories based on the timestamps, and inside those are images numbered 1, 2, 3, etc. Next, I define total, original, new, and orderly. Total is the number of images, original is the original image, new is the automatically edited image from 2 until total, and orderly is a Table of HTML and JavaScript code that the browser displays as images, organized in a Grid. I coded the way I want the images to be seen, original then new, and lastly, the click on or off of the solutions defined above. I used my function TakeQuantile as described previously. Lastly, I added finishing touches to the website such as adding a title, a description, and a link to email me.
In[]:=
CloudDeploy[FormFunction[{"difficulty"{"easy"{0,1/4},"medium"{1/4,1/2},"hard"{1/2,3/4},"impossible"{3/4,1}},"selection"{"random","all"},"image"(CloudImport[First@#<>"/1.png"]First[#]&/@Select[CloudObjects["gallery"],!StringContainsQ[First@#,".png"]&])},With[{total=Length[CloudObjects[#image<>"/"]]/2,original=#image<>"/1"},Grid[With[{orderly=Table[With[{new=#image<>"/"<>ToString[i]},"<img height=\"200px\" src=\""<>If[#2,blank,#]<>"\""<>If[#2," onclick=\"javascript:this.src=(this.src=='"<>#<>"' ? '"<>blank<>"' : '"<>#<>"');\"",""]<>">"&@@@{original<>".png"False,new<>".png"False,new<>"a.png"True}],{i,TakeQuantile[Range[2,total],#difficulty]}]},If[#["selection"]==="random",{RandomChoice[orderly]},orderly]]]]&,AppearanceRules<|"Title""Spot the Difference","Description""Choose a difficulty, whether you would like to see the solutions or not, and an image, then hit submit to try to spot the difference between the original image on the left and the edited one on the right. <i><a href=\"mailto:stella@maymin.com\">stella@maymin.com</a></i>"|>],"spot.me",Permissions"Public"]
Out[]=
Future Plans
Future Plans
As an extension of my website, I would like to find a way to create more of a game, perhaps by using dynamics, especially the LocatorPane function. Though I played around with it using various methods, I couldn’t figure out the code to make it work successfully on the web. In the future, I would like to find the differences, locate their positions, and apply that to the LocatorPane, in order to create a working game. Another extension may be to create a timer to show how long it took the user to find the difference, or a counter of how many incorrect clicks on the Locator there were. Using the high scores, I think it might be possible to create a Leaderboard across all users for them to have the opportunity to compete with each other or beat their own high scores in attempt to be at the very top. Lastly, an easier stretch goal is to create the option of allowing the user to upload their own images to create their very own collection and possibly save those uploaded images to my master library, so others can select them as well. Currently, I haven’t found a way to incorporate all these plans into my website, as the Cloud isn’t very supportive of the LocatorPane, which is the basis for most of the others. Coming here has been a once-in-a-lifetime experience for me, and I am truthfully so thankful to have been a part of the Wolfram High School Summer Camp 2019 program.
Acknowledgements
Acknowledgements
I would like to thank my mentor, Rory Fougler, for helping me with my code every step of the way. It really made a huge difference. I would also like to thank Chip Hurst for providing various insights and helpful functions throughout my project. Lastly, I would like to thank Mads Bahrami, Kyle Keane, and Anna Musser for making this entire experience truly unforgettable. Overall, I am so grateful for my knowledge and capabilities now, thanks to my stay here at the Wolfram High School Summer Camp.
Contact Information
Contact Information
Cite this as: Stella Maymin, "[WSC19] Automated Spot the Difference" from the Notebook Archive (2019), https://notebookarchive.org/2019-07-5jueeyb
Download
