4 The fundamentals of artificial neural networks

4.1 Basic definitions

In this book, we focus on deep learning, which is a type of machine learning based on artificial neural networks (ANNs), which involve several layers of so-called neuron functions. This kind of architecture was loosely inspired by the biological neural networks in animals’ brains.

The key building block of ANNs is what we call here a neuron function. A neuron function is a very simplified mathematical representation of a biological neuron dating back to [34]. This naturally leads to various ways of organizing those neuron functions into interacting networks that are capable of complex tasks.

The perceptron, introduced in [43] (see also [35]), is one of the earliest one layer ANNs. Nowadays, multiple layers are typically used instead because neurons in the same layer are not connected, but neurons can interact with neurons in other layers.

We present in this chapter a rather common architecture for classifiers, which is based on so-called feedforward networks, which are networks with no cycles. A “cycle” means that the output of one layer becomes the input of a previous layer. More complex networks can also be used with some cycles between layers (so-called recurrent neural networks; see for example [11], [17]).

Finally as one readily sees, ANNs typically require a very large number of parameters. This makes identifying the best choice for those coefficients delicate and leads to the important question of learning the parameters, which is discussed later in this book.

A typical example of an activation function is ReLU (Rectified Linear Unit), defined as follows (see Fig. 4.2):

ReLU is a very simple non-linear function composed of two linear functions that model the threshold between an “inactive” and an “active” state of the neuron.

Figure 4.1 A neuron function input x∈Rn, vector of weights α=(α1,…,αn), bias β, and output scalar f(x)=λ(α·x+β).

Figure 4.2 ReLU function. β, “bias” constant.

ReLU is a simple way to model a neuron in a brain which has two states: resting (resting potential) and active (firing). Incoming impulses from other neurons can change the state from resting to active, but the impulse needs to reach a certain threshold first. Thus, ReLU’s change from constant 0 to a linear function at xthresh=β reflects the change from resting to active at this threshold.

Figure 4.3 Layer of neurons fI, 1≤i≤4, with vector of input data xj. Weights αij are assigned to each edge connecting xj and fi, n=3, m=4.

Since the output of a neuron function is a single number, we can combine several neurons to create a vector-valued function called a layer function.

Thus, the layer function is determined by a matrix of parameters,

and a vector of biases,

Hence, (4.3) may be written

where λ¯:Rm→Rm is the vectorial activation function defined as

for a scalar activation function λ as in Definition 4.1.1.

See Fig. 4.3 for a diagram of a layer function. This figure shows a layer as a graph with two columns of nodes. The right column depicts neuron functions in the layer, while the left column depicts three real numbers (data) which are input to the layer. The edge connecting the ith input with the jth neuron is multiplied by the parameter value αij from the matrix of parameters.

Fig. 4.4 shows an ANN composed of two layers. The layer function in Fig. 4.4 between input and output is called a hidden layer (hidden from the user) because its output is passed to another layer, not directly to the user. The number of neurons ni in the ith layer is called the layer’s width, while the total number M of layers in an ANN is the depth of the ANN. The numbers n1,…,nM and M comprise the architecture of this network. ANNs with more than one layer are referred to as deep neural networks (DNNs).

Figure 4.4 Simple network with input layer and two layer functions, n=n0=4, n1=6, m=n2=3.

This network is called fully connected because for all but the last layer, each neuron provides input to each neuron in the next layer. That is, each node in each column of the graph in Fig. 4.4 is connected to each node in the...