Wolfram Summer School 2018: Morphing Using ANNs and Shape Interpolation
Author
Manan Aggarwal
Title
Wolfram Summer School 2018: Morphing Using ANNs and Shape Interpolation
Description
Wolfram Summer School 2018: Morphing Using ANNs and Shape Interpolation
Category
Essays, Posts & Presentations
Keywords
Wolfram Summer School 2018
URL
http://www.notebookarchive.org/2018-12-53xj8vz/
DOI
https://notebookarchive.org/2018-12-53xj8vz
Date Added
2018-12-11
Date Last Modified
2018-12-11
File Size
1.61 megabytes
Supplements
Rights
Redistribution rights reserved
Download
Open in Wolfram Cloud
WOLFRAM SUMMER SCHOOL 2018
Last modified on: Wednesday, July 11, 2018 at 5:17
Author Info
Name:
Manan Aggarwal
Mentor:
Garrett Ducharme
Affiliation:
Johns Hopkins University
Poster Session Content
Title of project:
Morphing Using ANNs and Shape Interpolation
Goal of the project:
Over the past few decades, geometric morphometric methods have become increasingly popular and powerful tools to describe morphological data. Meanwhile, artificial neural networks have had a similar rise in the classification of specimens to preconceived groups. However, there has been little research into how well these two systems operate together. The goal of this project is to create a morph between two images using semantic segmentation and geometric interpolation.
Add the most representative image of your project here. (We recommend just 1 image, if you add more, we will make a collage of the images.)
Image:
Summary of Results:
We succeeded in transforming the input images into the output images. To do this, we first segmented the region of interest, then found a one-to-one mapping between corresponding segments, and finally morphed the 2D space using the thin plate spline. The proposed technique works best for images that have low density of intersection points. This is because the radial basis function interpolation used in the algorithm tears the 2D space for overlapping vectors. This method does not extract features from the images, but rather exploits the curvature of the splines. The quality of the results obtained is also guided by the accuracy of the segmentation algorithm.
Future work:
1. Implement a shape interpolation scheme to produce results with a bounded amount of conformal distortion
2. Extract matching image features for alignment and local warping
3. Develop a morphing sequence from automatically generated sparse correspondence points
2. Extract matching image features for alignment and local warping
3. Develop a morphing sequence from automatically generated sparse correspondence points
Note
: Everything above this bar is your poster.Make sure it fits on a single page. Preview Poster
End of School Presentation Content
In addition to the poster content, include other content to present at the 2 minute presentation for end of school. Use the buttons below to add more sections.
Add Header
Add Text
Add Code or Image
METHODOLOGY
1.SEGMENTATION:createamaskusingneuralnet
2.MAPPING:a.findintersectionpointsb.createalistofcorrespondingpointsforeachsegment
3.INTERPOLATION:applyradialbasisinterpolationonsourcepoints
Note
: Everything above this bar is in your 2 minute presentation. Make sure it fits on 2 slides. Preview Presentation
Detailed Project Notes
Main Results in Detail
Main Results in Detail
Code
Code
MorphImage::usage = "MorphImage[im1, im2] returns a morph of im1 on im2.";MorphImage[im1_, im2_] := Block[ os, cutout, op, pxlist, intersections, tourlist, overlay, sortinter, dist, intersectionpts, segments, numpts, mappts, correspondingpts, objects, sources, targets, median, os = SegmentImage /@ {im1, im2}; cutout = ColorReplace[ImageAdd[im1, First@os], White]; op = MorphologicalPerimeter /@ ColorNegate /@ ConformImages[ImageCrop /@ os]; pxlist = Table[PixelValuePositions[x, 1], {x, op}]; intersections = Intersection[First@pxlist, Last@pxlist]; overlay = Show[ ListPlot[First@pxlist, PlotStyleGreen], ListPlot[Last@pxlist, PlotStyleBlue], ListPlot[intersections, PlotStyleDirective[Red, PointSize[Medium]]]]; tourlist = Table[TourPoints[x, First@intersections], {x, pxlist}]; sortinter = SortBy[intersections, Position[First@tourlist, #]&]; dist = EuclideanDistance[{0,0}, ImageDimensions[overlay]*.02]; intersectionpts = Table[ If[dist < EuclideanDistance[sortinter[[x]], sortinter[[x+1]]], sortinter[[x]], Nothing], {x, Length[sortinter]-1}]; segments = Table[SegmentPoints[x, intersectionpts], {x, tourlist}]; numpts = Join[Round[(IntegerPart[(Length[(First@segments)[[#]]] + Length[(Last@segments)[[#]]])/2] &/@ Range[Length[First@segments]])/20] + 1]; mappts = Table[MapThread[SelectPoints, {x, numpts}], {x, segments}]; correspondingpts = Table[Flatten[x, 1], {x, mappts}]; objects = ConformImages[ImageCrop /@ {cutout, Last@os}]; sources = First@correspondingpts // N; targets = Last@correspondingpts // N; median = Table[MapThread[((1-a) * #1 + a * #2)&, {sources, targets}], {a, 0, 1, 0.1}] // N; PartialTransform[img_, itr_] := ImageTransformation[img, RadialBasisInterpolation[median[[itr+1]], median[[itr]], 1], DataRange All]; Fold[PartialTransform, First@objects, Range[Length[median]-1]]]
In[]:=
ThinPlateSplineNorm::usage = "ThinPlateSplineNorm[n][v] returns the n-th thin plate spline norm of vector v.";ThinPlateSplineNorm[1] = &;ThinPlateSplineNorm[p_?EvenQ] := If#>0, Log[#]2, 0& [#.#]&;
#.#
&;ThinPlateSplineNorm[2] = (If[#>0, # Log[#]/2, 0]& [#.#])&;ThinPlateSplineNorm[p_?OddQ] := p/2
(#.#)
p/2
#
In[]:=
RadialBasisInterpolation::usage = "RadialBasisInterpolation[source, target, λ] creates a compiled thin plate spline interpolation functions that maps the List of source points onto the List of target points. λ is a smoothness parameter.";RadialBasisInterpolation[sources_List, targets_List, λ_?NumberQ,(opts___)?OptionQ] := Module[ {n, dim, Q, ϕ, B, M, wa}, {n, dim} = Dimensions[sources]; Q = PadLeft[sources, {n, dim+1}, 1]; ϕ = ThinPlateSplineNorm[2]; B = Map[ϕ, Outer[Subtract, sources, sources, 1], {2}]; M = ArrayFlatten[{{B-λ IdentityMatrix[n], Q}, {Transpose[Q], ConstantArray[0, {dim,dim}+1]}}]; Off[LinearSolve::luc]; wa = Map[LinearSolve[M, PadRight[#, n+dim+1]]&, Transpose[targets]]; On[LinearSolve::luc]; Compile[{{x, _Real, 1}}, #1+#2.x+#3.Map[#4, Transpose[x-#5]]]& [wa〚All,-(dim+1)〛, Take[wa,All,-dim], Take[wa,All,n], ϕ, Transpose[sources]]] /; Dimensions[sources] === Dimensions[targets]
SelectPoints::usage = "SelectPoints[l, n] returns n equispaced elements from List l.";SelectPoints[l_List, n_] := Block[ {totake}, totake = Round[Range[n]*(Length[l]/(n+1))]; l[[totake]]]
SegmentPoints::usage = "SegmentPoints[tourpts, breakpts] returns a nested List of elements in tourpts that are present between each pair of consecutive breakpts.";SegmentPoints[tourpts_List, breakpts_List] := Block[ {positions, rotatedpos}, positions = Sort[Position[tourpts, #]&/@breakpts]; rotatedpos = Append[Drop[positions,1], {{All}}]; MapThread[tourpts[[#1[[1,1]] ;; #2[[1,1]]]]&, {positions,rotatedpos}]]
TourPoints::usage = "TourPoints[points, firstintersection] returns a sorted List of points starting from firstintersection.";TourPoints[points_List, firstintersection_] := Block[ {startpt, tour}, startpt = Position[points, firstintersection][[1,1]]; tour = FindShortestTour[points, startpt, startpt] // Last; points[[tour]]]
$netModel := NetModel["Ademxapp Model A1 Trained on PASCAL VOC2012 and MS-COCO Data"];SegmentImage[img_] := Block[ {labels, objectshape}, labels = "background", "airplane", "bicycle", "bird", "boat", "bottle", "bus", "car", "cat", "chair", "cow", "table", "dog", "horse", "motorcycle", "person", "plant", "sheep", "sofa", "train", "television"; objectshape = Binarize[Colorize[NetEvaluate[img]]]]
NetEvaluate[img_, device_: "CPU"] := Block[ {net, resized, encData, dec, mean, var, prob}, net = NetModel[ "Ademxapp Model A1 Trained on PASCAL VOC2012 and MS-COCO Data"]; resized = ImageResize[img, {504}]; encData = Normal@NetExtract[net, "Input"]; dec = NetExtract[net, "Output"]; {mean, var} = Lookup[encData, {"MeanImage", "VarianceImage"}]; prob = NetReplacePart[net, {"Input" -> NetEncoder[{"Image", ImageDimensions@resized, "MeanImage" -> mean, "VarianceImage" -> var}], "Output" -> Automatic} ][resized, TargetDevice -> device]; prob = ArrayResample[prob, Append[Reverse@ImageDimensions@img, 21]]; dec[prob]]
Written Content / Lesson Plans
Written Content / Lesson Plans
The
Conclusions in Detail
Conclusions in Detail
All Visualizations
All Visualizations
Data Sources Links/References
Data Sources Links/References
Future Directions
Future Directions
Background Info Links/References
Background Info Links/References
Keywords
Keywords
◼
Image Processing
◼
Morphing
◼
Neural Networks
◼
Segmentation
◼
Interpolation
Other information
Other information
Cite this as: Manan Aggarwal, "Wolfram Summer School 2018: Morphing Using ANNs and Shape Interpolation" from the Notebook Archive (2018), https://notebookarchive.org/2018-12-53xj8vz
Download
