Basis of Neural Networks in Visual Basic .NET

Scope

In this article (hopefully, the first of a small series), we'll see how to implement a neural network in Visual Basic .NET, i.e., a model capable of processing input data and adjusting its internal mechanics to learn how to produce a desired result. We'll see more on this later. The present article will focus on generic definitions about neural networks and their behaviours, offering a simple implementation for the reader to test. In the final paragraph, we will code a small network capable of swapping two variables.

Introduction

A First Definition

The term "neural network" is typically used as a reference to a network or circuit constituted by neurons. We can differentiate two types of neural networks: a) biological and b) artificial. Obviously, speaking about software development, we are here referring to artificial ones, but those kind of implementations get their basic model and inspiration from their natural counterparts, so it can be useful to briefly consider the functioning of what we intend when we speak of biological neural networks.

Natural Neural Networks

Those are networks constituted by biological neurons, and they are typical of living creatures. The neurons/cells are interconnected into the peripheral nervous system or in the central one. In neurosciences, groups of neurons are identified by the physiological function they perform.

Artificial Neural Networks

Artificial networks are mathematical models, which could be implemented through an electronic medium, which mime the functioning of a biological network. Simply speaking, we will have a set of artificial neurons apt to solve a particular problem in the field of artificial intelligence. Like a natural one, an artificial network could "learn", through time and trial, the nature of a problem, becoming more and more efficient in solving it.

Neurons

After this simple premise, it should be obvious that in a network, being it natural or artificial, the entity known as "neuron" has a paramount importance, because it receives inputs, and is somewhat responsible of a correct data processing, which ends in a result. Think about our brain: it's a wonderful supercomputer composed by 86*10^9 neurons (more or less). An amazing number of entities which constantly exchange and store information, running on 10^14 synapses. Like we've said, artificial models are trying to capture and reproduce the basic functioning of a neuron, which is based on three main parts:

1. Soma, or cellular body
2. Axon, the neuron output line
3. Dendrite, the neuron input line, which receives the data from other axons through synapses

The soma executes a weighted sum of the input signals, checking if they exceed a certain limit. If they do, the neuron activates itself (resulting in a potential action), staying in a state of quiet otherwise. An artificial model tries to mimic those subparts, with the target of creating an array of interconnected entities capable of adjusting itself on the basis of received inputs, constantly checking the produced results against an expected situation.

How a Network Learns

Typically, the neural network theory identifies three main methods through which a network can learn (where, with "learn", we intend - from now on - the process through which a neural network modifies itself to being able to produce a certain result with a given input). Regarding the Visual Basic implementation, we will focus only on one of them, but it's useful to introduce all the paradigms, in order to have a better overview. For a NN (neural network) to learn, it must be "trained". The training can be supervisioned, if we possess a set of data constituted by input and output values. Through them, a network could learn to infer the relation which binds a neuron with another one. Another method is the unsupervisioned one, which is based on training algorithms which modifies the network's weights relying only on input data, resulting in networks that will learn to group received informations with probabilistic methods. The last method is the reinforced learning, which doesn't rely on presented data, but on exploration algorithms which produce inputs that will be tested through an agent, which will check their impact on the network, trying to determine the NN performance on a given problem. In this article, when we'll come to code, we will see the first presented case, or the supervisioned training.

Supervisioned Training

So, let's consider the method from a closer perspective. What it means to train a NN with supervision? As we've said, it deals primarily with presenting a set of input and output data. Let's suppose we want to teach our network to sum two numbers. In that case, following the supervisioned training paradigm, we must feed our network with input data (lets say [1;5]) but also tell it what we expect as a result (in our case, [6]). Then, a particular algorithm must be applied in order to evaluate the current status of the network, adjusting it by processing our input and output data. The algorithm we will use in our example is called backpropagation.

Backpropagation

The backpropagation of errors is a technique in which we first initialize our network (typically with random values regarding the weights on single neurons), proceeding in forwarding our input data, matching results with output data we expect. Then we calculate the deviation of the obtained values from the desired one, obtaining a delta factor which must be backpropagated to our neurons, to adjust their initial state according to the entity of error we've calculated. Through trials and repetitions, several sets of input and output data are presented to the network, each time repeating the match between the real and the ideal value. In a certain time, that kind of operation will produce increasingly precise outputs, calibrating the weight of each network's components, and ultimately refining its ability to process the received data. For an extensive explanation about backpropagation, please refer to the Bibliography section.

Creating a Neural Network

Now that we saw some preliminary concepts about neural networks, we should be able to develop a model which responds to the discussed paradigms. Without exceeding in mathematical over-explanations (which aren't really needed, unless you want to understand better what we will see), we will proceed step by step in coding a simple yet functional neural network, that we'll test when completed. The first thing we need to consider is the structure of a neural network: we know it's organized in neurons, and the neurons themselves are interconnected. But we don't know how.

Layers

And that's where the layers comes into scene. A layer is a group of neurons which share a more or less common function. For example, think about the entry point of input data. That will be the input layer, in which resides a group of neurons which share a common functionality, in that case limited to receiving and forward informations. Then we'll surely have an output layer, which will group those neurons that receive the result of previous processing. Among those layers could reside many layers, typically called "hidden", because a user won't have direct access to them. The number of those hidden layers, and the number of neurons each of them will contain, depends heavily on the natural and complexity of the problem we want to solve. To sum up, each network will be constituted of layers, each of which will contain a certain predetermined number of neurons.

Neurons and Dendrites

"Artificially" speaking, we could conceive a neuron like an entity which exposes a certain value, adjusted by trial iterations, and binded to other neurons through dendrites, which in our case will be represented by sub-entities possessing an initial random weight. The training process will consist of feeding the input layer neurons, which transmit their value through dendrites to an upper layer, which will do the same thing until the output layer is reached. Finally, we calculate a delta between the current output and the desired one, recurring through the network and adjusting the dendrites weights, the neuron values and each deviation value, to correct the network itself. Then we start again with another training round.

Preparing Network Classes

Having seen how a network is structured, we could sketch down some classes to manage the various entities of a network. In the following snippet, I will outline Dendrite, Neuron and Layer classes, which we will use together in the implementation of a NeuralNetwork class.

Dendrite Class

Public Class Dendrite
    Dim _weight As Double
 
    Property Weight As Double
        Get
            Return _weight
        End Get
        Set(value As Double)
            _weight = value
        End Set
    End Property
 
    Public Sub New()
        Me.Weight = r.NextDouble()
    End Sub
End Class

First, the Dendrite class: as you can see, it's constituted by a sole property, named Weight. In initializing a Dendrite, a random Weight is attributed to our dendrite. The type of the Weight property is Double, because our input values will be between zero and one, so we need a strong precision when it comes to decimal places. More on this later. No other properties nor functions are required for that class.

Neuron Class

Public Class Neuron
    Dim _dendrites As New List(Of Dendrite)
    Dim _dendriteCount As Integer
    Dim _bias As Double
    Dim _value As Double
    Dim _delta As Double
 
    Public Property Dendrites As List(Of Dendrite)
        Get
            Return _dendrites
        End Get
        Set(value As List(Of Dendrite))
            _dendrites = value
        End Set
    End Property
 
    Public Property Bias As Double
        Get
            Return _bias
        End Get
        Set(value As Double)
            _bias = value
        End Set
    End Property
 
    Public Property Value As Double
        Get
            Return _value
        End Get
        Set(value As Double)
            _value = value
        End Set
    End Property
 
    Public Property Delta As Double
        Get
            Return _delta
        End Get
        Set(value As Double)
            _delta = value
        End Set
    End Property
 
    Public ReadOnly Property DendriteCount As Integer
        Get
            Return _dendrites.Count
        End Get
    End Property
 
    Public Sub New()
        Me.Bias = r.NextDouble()
    End Sub
End Class

Next, the Neuron class. As one can imagine, it will expose a Value property (of type Double, for the same motivations saw above), and a series of potential Dendrites, the number of which will depend to the neuron number of the layer our current neuron will be connected to. So we have a Dendrite property, a DendriteCount (which returns the number of Dendrites), and two properties which will serve in the process of recalibration, namely Bias and Delta.

Layer Class

Public Class Layer
    Dim _neurons As New List(Of Neuron)
    Dim _neuronCount As Integer
 
    Public Property Neurons As List(Of Neuron)
        Get
            Return _neurons
        End Get
        Set(value As List(Of Neuron))
            _neurons = value
        End Set
    End Property
 
    Public ReadOnly Property NeuronCount As Integer
        Get
            Return _neurons.Count
        End Get
    End Property
 
    Public Sub New(neuronNum As Integer)
        _neuronCount = neuronNum
    End Sub
End Class

Finally, a Layer class, which is simply a container for an array of neurons. In calling upon the New method, the user must indicate how many neurons the layer is required to have. We'll see in the next section how those classes can interact in the contest of a full-fledged neural network.

NeuralNetwork Class

Our NeuralNetwork can be seen as a list of layers (each of which will inherit the underlying layer properties, i.e., neurons and dendrites). A neural network must be launched (or put in running state) and trained, so we will likely have two methods which can be used for that means. The network initialization must specify, among other properties, a parameter that we will call "learning rate". That will be a variable we'll use in weights recalculation. As the name implies, the learning rate is a factor which determines how fast a network will learn. Since it's a corrective factor, the learning rate must be chosen accurately: if its value is too large, but there is a large multitude of possible inputs, a network may not learn too well, or at all. Generally speaking, a good practice is to set the learning rate to a relatively small value, increasing it if the effective recalibration of our network becomes too slow.

Let's see an almost complete NeuralNetwork class:

Public Class NeuralNetwork
    Dim _layers As New List(Of Layer)
    Dim _learningRate As Double
 
    Public Property Layers As List(Of Layer)
        Get
            Return _layers
        End Get
        Set(value As List(Of Layer))
            _layers = value
        End Set
    End Property
 
    Public Property LearningRate As Double
        Get
            Return _learningRate
        End Get
        Set(value As Double)
            _learningRate = value
        End Set
    End Property
 
    Public ReadOnly Property LayerCount As Integer
        Get
            Return _layers.Count
        End Get
    End Property
 
    Sub New(LearningRate As Double, nLayers As List(Of Integer))
        If nLayers.Count < 2 Then Exit Sub
 
        Me.LearningRate = LearningRate
 
        For ii As Integer = 0 To nLayers.Count - 1
 
            Dim l As Layer = New Layer(nLayers(ii) - 1)
            Me.Layers.Add(l)
 
            For jj As Integer = 0 To nLayers(ii) - 1
                l.Neurons.Add(New Neuron())
            Next
 
            For Each n As Neuron In l.Neurons
                If ii = 0 Then n.Bias = 0
 
                If ii > 0 Then
                    For k As Integer = 0 To nLayers(ii - 1) - 1
                        n.Dendrites.Add(New Dendrite)
                    Next
                End If
 
            Next
 
        Next
    End Sub
 
    Function Execute(inputs As List(Of Double)) As List(Of Double)
        If inputs.Count <> Me.Layers(0).NeuronCount Then
            Return Nothing
        End If
 
        For ii As Integer = 0 To Me.LayerCount - 1
            Dim curLayer As Layer = Me.Layers(ii)
 
            For jj As Integer = 0 To curLayer.NeuronCount - 1
                Dim curNeuron As Neuron = curLayer.Neurons(jj)
 
                If ii = 0 Then
                    curNeuron.Value = inputs(jj)
                Else
                    curNeuron.Value = 0
                    For k = 0 To Me.Layers(ii - 1).NeuronCount - 1
                        curNeuron.Value = curNeuron.Value + _
                        Me.Layers(ii - 1).Neurons(k).Value * curNeuron.Dendrites(k).Weight
                    Next k
 
                    curNeuron.Value = Sigmoid(curNeuron.Value + curNeuron.Bias)
                End If
 
            Next
        Next
 
        Dim outputs As New List(Of Double)
        Dim la As Layer = Me.Layers(Me.LayerCount - 1)
        For ii As Integer = 0 To la.NeuronCount - 1
            outputs.Add(la.Neurons(ii).Value)
        Next
 
        Return outputs
    End Function
 
    Public Function Train(inputs As List(Of Double), outputs As List(Of Double)) As Boolean
        If inputs.Count <> Me.Layers(0).NeuronCount Or _
           outputs.Count <> Me.Layers(Me.LayerCount - 1).NeuronCount Then
            Return False
        End If
 
        Execute(inputs)
 
        For ii = 0 To Me.Layers(Me.LayerCount - 1).NeuronCount - 1
            Dim curNeuron As Neuron = Me.Layers(Me.LayerCount - 1).Neurons(ii)
 
            curNeuron.Delta = curNeuron.Value * (1 - curNeuron.Value) * _
                              (outputs(ii) - curNeuron.Value)
 
            For jj = Me.LayerCount - 2 To 1 Step -1
                For kk = 0 To Me.Layers(jj).NeuronCount - 1
                    Dim iNeuron As Neuron = Me.Layers(jj).Neurons(kk)
 
                    iNeuron.Delta = iNeuron.Value *
                                    (1 - iNeuron.Value) * _
                                    Me.Layers(jj + 1).Neurons(ii).Dendrites(kk).Weight *
                                    Me.Layers(jj + 1).Neurons(ii).Delta
                Next kk
            Next jj
        Next ii
 
 
        For ii = Me.LayerCount - 1 To 0 Step -1
            For jj = 0 To Me.Layers(ii).NeuronCount - 1
                Dim iNeuron As Neuron = Me.Layers(ii).Neurons(jj)
                iNeuron.Bias = iNeuron.Bias + (Me.LearningRate * iNeuron.Delta)
 
                For kk = 0 To iNeuron.DendriteCount - 1
                    iNeuron.Dendrites(kk).Weight = iNeuron.Dendrites(kk).Weight + _
                    (Me.LearningRate * Me.Layers(ii - 1).Neurons(kk).Value * iNeuron.Delta)
                Next kk
            Next jj
        Next ii
 
        Return True
    End Function
 
End Class

New() Method

When our network is initialized, it requires a learning rate parameter, and a list of Layers. Processing that list, you can see how each layer will result in a generation of neurons and dendrites, which are assigned to their respective parents. Calling the New() method of neurons and dendrites, will result in a random assignation of their initial values and weights. If the passed layers are less than two, the subroutine will exit, because a neural network must have at least two layers, input and output.

Sub New(LearningRate As Double, nLayers As List(Of Integer))
    If nLayers.Count < 2 Then Exit Sub
    Me.LearningRate = LearningRate
 
    For ii As Integer = 0 To nLayers.Count - 1
        Dim l As Layer = New Layer(nLayers(ii) - 1)
        Me.Layers.Add(l)
        For jj As Integer = 0 To nLayers(ii) - 1
            l.Neurons.Add(New Neuron())
        Next
        For Each n As Neuron In l.Neurons
            If ii = 0 Then n.Bias = 0
            If ii > 0 Then
                For k As Integer = 0 To nLayers(ii - 1) - 1
                    n.Dendrites.Add(New Dendrite)
                Next
            End If
        Next
    Next
End Sub

Execute() Function

As we've said, our network must possess a function through which we process input data, making them move into our network, and gathering the final results. The following function does this. First, we'll check for input correctness: if the number of inputs are different from the input layer neurons, the function cannot be executed. Each neuron must be initialized. For the first layer, the input one, we simply assign to the neuron Value property our input. For other layers, we calculate a weighted sum, given by the current neuron Value, plus the Value of the previous layer neuron multiplied by the weight of the dendrite. Finally, we execute on the calculated Value a Sigmoid function, which we'll analyze below. Processing all layers, our output layer neurons will receive a result, which is the parameter our function will return, in the form of a List(Of Double).

Function Execute(inputs As List(Of Double)) As List(Of Double)
    If inputs.Count <> Me.Layers(0).NeuronCount Then
        Return Nothing
    End If
    For ii As Integer = 0 To Me.LayerCount - 1
        Dim curLayer As Layer = Me.Layers(ii)
        For jj As Integer = 0 To curLayer.NeuronCount - 1
            Dim curNeuron As Neuron = curLayer.Neurons(jj)
            If ii = 0 Then
                curNeuron.Value = inputs(jj)
            Else
                curNeuron.Value = 0
                For k = 0 To Me.Layers(ii - 1).NeuronCount - 1
                    curNeuron.Value = curNeuron.Value + _
                    Me.Layers(ii - 1).Neurons(k).Value * curNeuron.Dendrites(k).Weight
                Next k
                curNeuron.Value = Sigmoid(curNeuron.Value + curNeuron.Bias)
            End If
        Next
    Next
 
    Dim outputs As New List(Of Double)
    Dim la As Layer = Me.Layers(Me.LayerCount - 1)
    For ii As Integer = 0 To la.NeuronCount - 1
        outputs.Add(la.Neurons(ii).Value)
    Next
    Return outputs
End Function

Sigmoid() Function

The sigmoid is a mathematical function known for its typical "S" shape. It is defined for each real input value. We use such a function in neural network programming, because its differentiability - which is a requisite for backpropagation - and because it introduces non-linearity into our network (or, it makes out network able to learn the correlations among inputs that don't produce linear combinations). Plus, for each real value, the sigmoid function return a value between zero and one (excluding upper limit). That function has peculiarities which make it really apt when it comes to backpropagation.

Train() Function

A network is initialized with random values, so - without recalibration - the results they return are pretty random themselves, more or less. Without a training procedure, a freshly started network can be almost useless. We define, with the word training, the process through which a neural network keeps running on a given set of inputs, and its results are constantly matched against expected output sets. Spotting differences on our outputs from the returned values of the net, we then proceed in recalibrating each weight and value on the net itself, stepping forward a closer resemblance of what we want and what we get from the network.
In VB code, something like this:

Public Function Train(inputs As List(Of Double), outputs As List(Of Double)) As Boolean
    If inputs.Count <> Me.Layers(0).NeuronCount Or outputs.Count <> _
                       Me.Layers(Me.LayerCount - 1).NeuronCount Then
        Return False
    End If
 
    Execute(inputs)
 
    For ii = 0 To Me.Layers(Me.LayerCount - 1).NeuronCount - 1
        Dim curNeuron As Neuron = Me.Layers(Me.LayerCount - 1).Neurons(ii)
        curNeuron.Delta = curNeuron.Value * (1 - curNeuron.Value) * _
                          (outputs(ii) - curNeuron.Value)
        For jj = Me.LayerCount - 2 To 1 Step -1
            For kk = 0 To Me.Layers(jj).NeuronCount - 1
                Dim iNeuron As Neuron = Me.Layers(jj).Neurons(kk)
                iNeuron.Delta = iNeuron.Value *
                                (1 - iNeuron.Value) * _
                                Me.Layers(jj + 1).Neurons(ii).Dendrites(kk).Weight *
                                Me.Layers(jj + 1).Neurons(ii).Delta
            Next kk
        Next jj
    Next ii
 
    For ii = Me.LayerCount - 1 To 0 Step -1
        For jj = 0 To Me.Layers(ii).NeuronCount - 1
            Dim iNeuron As Neuron = Me.Layers(ii).Neurons(jj)
            iNeuron.Bias = iNeuron.Bias + (Me.LearningRate * iNeuron.Delta)
            For kk = 0 To iNeuron.DendriteCount - 1
                iNeuron.Dendrites(kk).Weight = iNeuron.Dendrites(kk).Weight + _
                (Me.LearningRate * Me.Layers(ii - 1).Neurons(kk).Value * iNeuron.Delta)
            Next kk
        Next jj
    Next ii
 
    Return True
End Function

As usual, we check for correctness of inputs, then start our network by calling upon the Execute() method. Then, starting from the last layer, we process every neuron and dendrite, correcting each value by applying the difference of outputs. The same thing will be done on dendrites weights, introducing the learning rate of the network, as saw above. At the end of a training round (or, more realistically, on the completion of several hundreds of rounds), we'll start to observe that the outputs coming from the network will become more and more precise.

A Test Application

In the present section, we will see a simple test application, which we'll use to create a pretty trivial network, used to swap two variables. In the following, we will consider a network with three layers: an input layer constituted by two neurons, an hidden layer with four neurons, and an output layer with two neurons. What we expect from our network, giving enough supervisioned training, will be the ability to swap the values we present it or, in other words, to being able to shift the value of the input neuron #1 on the output neuron #2, and viceversa.

In the downloadable NeuralNetwork class, you'll find a couple of methods which we haven't analyzed in the article, which refers to the graphical and textual network rendering. We will use them in our example, and you can consult them by downloading the source code.

Initializing the Network

Using our classes, a neural network can be simply initialized by declaring it, and feeding it its learning rate and start layers. For example, in the test application, you can see a snippet like this:

Dim network As NeuralNetwork
 
Private Sub Form1_Load(sender As Object, e As EventArgs) Handles MyBase.Load
    Dim layerList As New List(Of Integer)
    With layerList
        .Add(2)
        .Add(4)
        .Add(2)
    End With
 
    network = New NeuralNetwork(21.5, layerList)
End Sub

I've defined my network as globally to a Form, then - in the Load() event - I've created the layers I need, passing them to the network initializator. That will result in having our layers correctly populated with the number of neurons specified, each element starting with random values, deltas, biases and weights.

Running a Network

Running and training a network are very simple procedures as well. All they require is to call upon the proper method, passing a set of input and/or output parameters. In case of the Execute() method, for example, we could write:

Dim inputs As New List(Of Double)
inputs.Add(txtIn01.Text)
inputs.Add(txtIn02.Text)
 
Dim ots As List(Of Double) = network.Execute(inputs)
txtOt01.Text = ots(0)
txtOt02.Text = ots(1)

Where txtIn01, txtIn02, txtOt01, txtOt02 are Form's TextBoxes. What we will do with the above code, is simply to get inputs from two TextBoxes, using them as inputs for the network, and writing on another couple of TextBoxes the returned values. The same procedure applies in case of training.

Swapping Two Variables

In the following video, you could see a sample training session. The running program is the same you'll find by downloading the source code at the end of the article. As you can see, by starting with random values, the network will learn to swap the signals it receives. The values of 0 and 1 must not be intended as precise numbers, but rather as "signals of a given intensity". By observing the results of training, you could note how the intensity of the values will increase toward 1, or decrease to zero, never really reaching the extremes, but representing them anyway. So, at the end of an "intense" training to teach our network that, in case of an input couple of [0,1] we desire an output of [1,0], we can observe that we receive something similar to [0.9999998,0.000000015], where the starting signals are represented, by their intensity, to the closer conducive value, just like a neuron behave himself, speaking about activation energy levels.

Demo video on YouTube:

https://www.youtube.com/embed/zymAdf_zMtQ

Source Code

The source code we've discussed so far can be download at the following link:

• https://code.msdn.microsoft.com/Basis-of-Neural-Networks-a379549c

Bibliography

• Artificial neural network
• Sigmoid function
• Neuron
• Dendrite
• List(Of T) class

History

• 4^th September, 2015: Initial version