Essays, Posts & Presentations

Neural Networks: An Introduction

Tuseeta Banerjee

Author

Tuseeta Banerjee

Title

Neural Networks: An Introduction

Description

A technical primer on machine learning and neural nets using the Wolfram Language. Learn about components of neural networks--encoders and decoders, layers, containers—and what they do. Access pre-trained nets and architectures from the Neural Net Repository.

Neural Networks: An Introduction

May 2, 2019

Tuseeta Banerjee, Research Scientist, Machine Learning

If you haven’t used machine learning, deep learning and neural networks yourself, you’ve almost certainly heard of them. You may be familiar with their commercial use in self-driving cars, image recognition, automatic text completion, text translation and other complex data analysis, but you can also train your own neural nets to accomplish tasks like identifying objects in images, generating sequences of text or segmenting pixels of an image. With the Wolfram Language, you can get started with machine learning and neural nets faster than you think. Since deep learning and neural networks are everywhere, let’s go ahead and explore what exactly they are and how you can start using them.

What Exactly Are Neural Networks?

Neural networks are a programming approach that is inspired by the neurons in the human brain and that enables computers to learn from observational data, be it images, audio, text, labels, strings or numbers. They try to model some unknown function (for example,

f:PQ

) that maps this data to numbers or classes by recognizing patterns in the data. They are built from these components:

◼

Encoders and decoders (to convert the input data type to numeric tensors)

◼

Layers (to perform operations on these tensors, depending on the applications)

◼

Containers (to hold these operations in a sensible way)

Once the neural network is built from these components, it needs to be trained (in other words, optimized).

As you might have guessed, the optimization (or minimizing the “loss” of the network) is done through stochastic gradient descent in an iterative fashion. The inputs are fed to the net repeatedly; the error/loss is computed each time and is then used to update the model’s parameters using back propagation. Back propagation, or “back error propagation,” involves distributing the error computed during forward propagation back to the network’s layers.

Understanding Encoders and Decoders

The input data provided in any form needs to be converted to numeric tensors. Here are a few examples of tensors and their corresponding ranks (dimensions):

◼

Rank 0 (scalars): 0.0

◼

Rank 1 (vectors): {0.0, 1.0}

◼

Rank 2 (matrices): {{1.,2.,3.} , {3., 2., 1.}}

◼

Rank-n tensors: {... {... {1., 2., 3.}...}...}

These examples provide valuable insight into how we can transform images into the corresponding ranked tensors:

Out[]=

≈

0	0	1	1	1	1	1	0	0
0	1	1	1	1	1	1	1	0
1	1	1	1	1	1	1	1	1
1	1	1	1	1	1	1	1	1
1	1	1	1	1	1	1	1	1
1	1	1	1	1	1	1	1	1
1	1	1	1	1	1	1	1	1
0	1	1	1	1	1	1	1	0
0	0	1	1	1	1	1	0	0

Out[]=

≈

1	0	0
1	0	0
1	1	0

0	0	0
0	1	0
0	0	0

0	1	1
0	0	1
0	0	1

Encoder in Action

Take an image and convert it to tensors using a

NetEncoder

function, applied to images:

In[]:=

image=

;

In[]:=

enc=NetEncoder[{"Image",{64,64},ColorSpace"Grayscale"}]

Out[]=

NetEncoder



Type:	Image
Output:	array (size: 1×64×64)



The variable encoded indeed contains a 64x64 matrix of a single dimension (corresponding to the grayscale ColorSpace). We can confirm that by looking at the dimensions of this array:

In[]:=

encoded=enc[image];Dimensions@encoded

Out[]=

{1,64,64}

NetDecoder, on the other hand, can take a numeric tensor and convert it to the data type of your choice. Typically, one takes the output of the neural net and feeds it to the net decoder. Here, to illustrate the workings of a net decoder, let us simply feed back the “encoded” image to see if the decoder works as expected. Because we get our original image back in the specified 64x64 matrix, we have confirmation that it’s working:

In[]:=

dec=NetDecoder[{"Image",ColorSpace"Grayscale"}]

Out[]=

NetDecoder



Type:	Image
Input:	array (rank: 3)



In[]:=

dec[encoded]

Out[]=

Applying Layers

A neural network consists of “layers” through which information is processed from the input to the output tensor. Each layer is defined by its mathematical operation. So, mathematically, we can define a linear layer is an affine transformation

xAx+b

, where

is the “weight matrix” and the vector

is the “bias vector”:

In[]:=

data={2,10,3};layer=NetInitialize@LinearLayer[2,"Input"3]layer[data]

Out[]=

LinearLayer



Input:	vector (size: 3)
Output:	vector (size: 2)



Out[]=

{-7.85758,-2.81023}

In[]:=

linear[data_,weight_,bias_]:=weight.data+bias

In[]:=

linear[data,Normal@NetExtract[layer,"Weights"],Normal@NetExtract[layer,"Biases"]]

Out[]=

{-7.85758,-2.81023}

Here I’ve listed the various layers in the Wolfram Language. You can start exploring the layers, each of which is defined by its associated mathematical operation, to see what tasks they accomplish and how they do it.

Layer Type	Layers
Basic Layers	LinearLayer \| ElementwiseLayer \| SoftmaxLayer
Elementwise Computation Layers	ElementwiseLayer \| ThreadingLayer \| ConstantTimesLayer \| ConstantPlusLayer
Convolution and Filtering Layers	ConvolutionLayer \| DeconvolutionLayer \| PoolingLayer \| ResizeLayer \| SpatialTransformationLayer
Training Optimization Layers	ImageAugmentationLayer \| NormalizationLayer \| DropoutLayer \| LocalResponseNormalizationLayer
Structure Manipulation Layers	CatenateLayer \| FlattenLayer \| ReshapeLayer \| ReplicateLayer \| PaddingLayer \| PartLayer \| TransposeLayer\| AppendLayer \| PrependLayer\|ExtractLayer
Array Operation Layers	ConstantArrayLayer \| SummationLayer \| TotalLayer \| AggregationLayer \| DotLayer\|OrderingLayer
Recurrent Layers	BasicRecurrentLayer \| GatedRecurrentLayer \| LongShortTermMemoryLayer
Sequence-Handling Layers	EmbeddingLayer \| SequenceLastLayer \| SequenceReverseLayer \| SequenceMostLayer \| SequenceRestLayer \| AttentionLayer \| UnitVectorLayer

Generally, most people use the following rule of thumb: “When you don’t know how to fine-tune your neural network, stack more layers.” So it makes us wonder: how do you stack them? The answer leads us to the third component mentioned earlier: containers.

Using Containers

Once the data is converted into numeric tensors using NetEncoder and the layers are chosen for particular applications, they need to be “stitched” together using the containers.

NetChain

can be used to connect different layers in a chain-like fashion to create a neural net, while

NetGraph

can be used to connect different layers to create a graph, connecting these layers. Let’s try out some ways to use these containers to create networks.

Here we construct a chained network with nonlinear activation, which specifies the vector input of a specified size and produces vector outputs of size 3:

In[]:=

NetChain[{LinearLayer[30],ElementwiseLayer[Tanh],LinearLayer[3],ElementwiseLayer[Tanh],LinearLayer[3],ElementwiseLayer[LogisticSigmoid]},"Input"2]

Out[]=

NetChain



uniniti

aliz

	Input	vector (size: 2)
1	LinearLayer	vector (size: 30)
2	Tanh	vector (size: 30)
3	LinearLayer	vector (size: 3)
4	Tanh	vector (size: 3)
5	LinearLayer	vector (size: 3)
6	LogisticSigmoid	vector (size: 3)
	Output	vector (size: 3)



Combine two layers with NetGraph, using

ThreadingLayer

for the corresponding operation:

In[]:=

net=NetGraph[{ElementwiseLayer[Ramp],ElementwiseLayer[Tanh],ThreadingLayer[Times],ThreadingLayer[Plus]},{123,134,24}]

Out[]=

NetGraph



Inputs				Outputs
Input:	array			Output:	array



The Problem of Overfitting

Stacking more layers brings us to a very serious problem of overfitting. When you keep stacking layers, you increase the number of tunable parameters in a network to thousands and even millions. The “deep” networks that are well suited to perform tasks like image classification, object detection and natural language processing are comprised of lots of stacked layers, and contain millions of parameters. When you have millions of parameters in your network, the network tends to “memorize” your data. In other words, the network will closely fit to your data, learn the eccentricities (or the “noise”) of your data and will not be able to generalize to other data. In an ideal world, where you have access to infinite data, the problem of overfitting would not arise. However, since we do not have access to infinite data (or the infinite training time required for infinite data), we need to make the best of what we have.

We divide our data into training, validation and test sets, keeping good enough percentages for the latter two. Finally, we can perform data augmentation:

ImageAugmentationLayer

automatically helps augment images to your dataset by performing random cropping and other operations. Next, we use various regularization techniques. Apart from the very familiar concepts of L1 and L2 regularization, which can be found as options for

NetTrain

DropoutLayer

can also be used to tackle the problem of overfitting. DropoutLayer sets the input elements to 0 with probability

during training, multiplying the remainder by

1/(1-p)

, and works as an effective and widely used regularization technique. Other methods like early stopping and multi-task learning can also be useful.

Using the Wolfram Neural Net Repository

The current consensus in the neural net community is that building your own net architecture is unnecessary for the majority of neural net applications, and will usually hurt performance. Rather, adapting a pretrained net to your own problem is almost always a better approach in terms of performance. Luckily, this approach has the added benefit of being much easier to work with!

Thus, having a large neural net repository is absolutely key to being productive with the neural net framework, as it allows you to look for a net close to the problem you are solving, do minimal amounts of “surgery” on the net to adapt it to your specific problem and then train it.

The Wolfram Neural Net Repository gives Wolfram Language users easy access to the latest net architectures and pretrained nets, representing thousands of hours of computation time on powerful GPUs. The repository consists of publicly available models converted from other neural net frameworks (such as Caffe, Torch, MXNet, TensorFlow, etc.) into the Wolfram neural network format. In addition, we have trained a number of nets ourselves, which you can find in the Wolfram Neural Net Repository or the introductory blog. All the neural network models in the repository can be programmatically accessed via

NetModel

. Here’s a sample of 10 random models:

In[]:=

RandomSample[NetModel[],10]

Out[]=

{CycleGAN Winter-to-Summer Translation,VGG-16 Trained on ImageNet Competition Data,GPT Transformer Trained on BookCorpus Data,Wolfram Python Character-Level Language Model V1,OpenFace Face Recognition Net Trained on CASIA-WebFace and FaceScrub Data,Ademxapp Model A1 Trained on PASCAL VOC2012 and MS-COCO Data,Pix2pix Street-Map-to-Photo Translation,CycleGAN Summer-to-Winter Translation,Wolfram C Character-Level Language Model V1,YOLO V2 Trained on MS-COCO Data}

Performing Network Surgery

Let’s look at an example of the net surgery process to solve the cat-versus-dog classification problem. First, obtain a net similar to our problem:

In[]:=

net=NetModel["ResNet-50 Trained on ImageNet Competition Data"]

Out[]=

NetChain



Input port:	image
Output port:	class
Number of layers:	25



The last two layers are specialized for the ImageNet classification task, and won’t be needed for our purposes. So we simply remove the last two layers using

NetDrop

In[]:=

netFeature=NetDrop[net,-2]

Out[]=

NetChain



Input port:	image
Output port:	vector (size: 2048)
Number of layers:	23



Note that it is particularly easy to do “net surgery” in the Wolfram Language: nets are symbolic expressions that can be manipulated using a large set of surgery functions, such as

NetTake

, NetDrop,

NetAppend

NetJoin

, etc. Now we simply need to define a new NetChain that will classify an image as “dog” or “cat”:

In[]:=

netNew=NetChain[<|"feature"netFeature,"classifier"LinearLayer[],"probabilities"SoftmaxLayer[]|>,"Output"NetDecoder[{"Class",{"dog","cat"}}]]

Out[]=

NetChain



uniniti

aliz

Input

image

array

(size: 3×224×224)

feature

NetChain (23 nodes)

vector

(size: 2048)

classifier

LinearLayer

vector

(size: 2)

probabilities

SoftmaxLayer

vector

(size: 2)

Output

class



In[]:=

NetTrain[netNew,catdogTrain,"FinalPlots",LearningRateMultipliers{"classifier"1,_0},ValidationSetcatdogTest,MaxTrainingRounds10,Method"SGD"]

Out[]=

Loss

	rounds
loss

	validation
	training

,ErrorRate

	rounds
error rate

	validation
	training



Try Your Own Neural Network

Neural networks are data-driven algorithms, so the first step is to investigate your data thoroughly. Various statistical and visualization techniques can be used to see patterns and variations in the data. Once you have a better understanding of your data, decide on your network. The best bet is to start from networks that have been trained and validated by established researchers, or at least take inspiration from the various “building units” in them. A great place to start is the Wolfram Neural Net Repository, where you can play with various network surgery functions. Once you have created the architecture, start experimenting with various parameters, initializations and losses. It is absolutely okay to overfit at this stage! Finally, you can use regularization techniques in the original model or the ones discussed to generalize your model.

Visit the Wolfram Data Repository and Neural Net Repository for a combination of immediately useful resources for getting started.

Want to Go Further?

Wolfram U class: Exploring the Neural Net Framework from Building to Training
Wolfram Blog: "Launching the Wolfram Neural Net Repository"
Get started: Wolfram Neural Net Repository
Need data? Wolfram Data Repository

Cite this as: Tuseeta Banerjee, "Neural Networks: An Introduction" from the Notebook Archive (2019), https://notebookarchive.org/2019-05-ddta3fg

0	0	1	1	1	1	1	0	0
0	1	1	1	1	1	1	1	0
1	1	1	1	1	1	1	1	1
1	1	1	1	1	1	1	1	1
1	1	1	1	1	1	1	1	1
1	1	1	1	1	1	1	1	1
1	1	1	1	1	1	1	1	1
0	1	1	1	1	1	1	1	0
0	0	1	1	1	1	1	0	0

0	0	1	1	1	1	1	0	0
0	1	1	1	1	1	1	1	0
1	1	1	1	1	1	1	1	1
1	1	1	1	1	1	1	1	1
1	1	1	1	1	1	1	1	1
1	1	1	1	1	1	1	1	1
1	1	1	1	1	1	1	1	1
0	1	1	1	1	1	1	1	0
0	0	1	1	1	1	1	0	0

0	0	1	1	1	1	1	0	0
0	1	1	1	1	1	1	1	0
1	1	1	1	1	1	1	1	1
1	1	1	1	1	1	1	1	1
1	1	1	1	1	1	1	1	1
1	1	1	1	1	1	1	1	1
1	1	1	1	1	1	1	1	1
0	1	1	1	1	1	1	1	0
0	0	1	1	1	1	1	0	0