As an introduction to artificial neural networks and applications, this overview traces the whole path from one of the building blocks of artificial neural networks and then goes towards LSTMs. Check the video below.

It is interesting to see the discussions taking place along this talk with researchers from different areas/perspectives on this topic. I’m from Computer Science field, while the members of the lab are engineers. It was fun.

By the way, if you have any questions or comments, feel free to share in the comments section. If you find this video useful, please give a thumbs up, share with more people and subscribe to the channel, so I can continue posting more material here :)

For study purposes, I developed this tool as an additional material to be used along with contents of the literature, enabling to recreate a particular state of the network in order to inspect its behavior as a function, as well as its convergence during training, and (almost) everything else that can be studied. See in the video below an overview of this project:

I took the idea of a playground from Tensorflow’s playground, since you are free to play with multilayer perceptrons from this tool. In fact, I encourage you to make your own! It is quite a good way to explore your expertise from different perspectives. Have fun!

This project was developed in JavaScript, HTML and CSS and encompasses the implementation of the classic backpropagation training algorithm. It is entirely available at GitHub. In case you feel like adding new features and adjusting it to your needs, fork me on GitHub :).

It is important to point out that this post is a full guide of the tool. I would recommend to use it for query purposes, because a one-time reading may be a bit boring and worthless.

Open the tool in a new window or use the embedded version below for testing everything described in this guide. Below you can find a full guide of this tool divided into the following sections:

• Topology tab
• Command tab
• Train tab
• Visualize tab
• Sharing
• Closing remarks

Topology tab

Everything regarding structural configuration of a model is done in this tab (you can also use the shortcut Shift+1). Changes are performed through the interface at the bottom area, and inspected at the top area.

The number of inputs, hidden nodes and outputs are adjustable through and buttons, or just change the current number and input a new one from keyboard. Layers of neurons can be removed by button, or added by button.

The activation function used in each layer is indicated by \( g(u) \) or \( u \) (click to toggle between them), being respectively a sigmoid or a linear function:

• \( g(u) = \dfrac{1}{1+e^{-u}} \)
• \( u = u \)

The configurations can be swapped between layers using the button. In case of swapping with the input layer, only the number of nodes does swap and everything else remains in the neural layer.

The Verbose area will be covered in the next section, since both topology and command tabs share the same interface.

Command tab

This tool was initially developed to generate commands to run the C implementation of multilayer perceptrons. So this tab (Shift+2) encompasses the commands to perform training and operation of models configured in topology tab - even though the same task can be done in this tab without visual inspection.

Right below the generated commands, there is a text area which is related to the state of the current model and will be covered in next section.

In verbose area, there are three options to be set regarding whether to verbose when running the generated command for training:

• Total epochs the number of epochs after training
• mean square error at the end of each epoch (useful for plotting)
• synaptic weights the adjusted weights after training

Train tab

Given a model previously configured at topology tab, its weights - initially random - can be changed in this tab (Shift+3) either via training, or manually.

Interactive structure

The structure displayed in this tab is interactive, so one can inspect the entire model, as well as perform changes manually.

As indicated in the image above, nodes - or neurons - are clickable, displaying (on click) its weights, which make up the connection to the previous layer. When the model is idle (i.e., when training iterations are not being performed), weights can be manually changed by modifying the value in the due text field and then pressing Return.

At any moment, regardless of the idleness, the network can be fed. Once all text fields at the left of the structure (see image above) are filled - tip: paste the input entirely in the first text field in comma-separated format, and everything is distributed to the other fields - with real-valued input, the structure is fed and the output appears at the right side.

Dataset

Before training, it is necessary to provide a dataset for the network to learn. Click button to toggle to a text area to provide data - click button to toggle back. Instances must be in CSV format or a data function can be called:

Instances must be compatible to the model IO, i.e., the number of values per instance (input and expected output) equals the total of inputs and outputs of the model. Once a new dataset is inserted, hit update parameters button.

Data functions

For didactic purposes, there are some functions to generate data points which can be used in place of explicit instances, such as ring(), spiral() and sin(), detailed below:

ring() function generates data points for binary classification in \( \mathbb{R}^3 \). Spatially, each ring of points from one class is surrounded by a bigger one from another class. Its usage is defined below:

ring();
//or
ring(points, pairs, noise, flipLabels);

where:

• points:integer number of points to be generated (both classes)
• pairs:integer number of pairs of rings (one for each class)
• noise:decimal the amount of noise to be added
• flipLabels:boolean whether to invert labels of data points

spiral() function generates data points for binary classification in \( \mathbb{R}^3 \). Spatially, points of both classes are arranged in a spiral pattern. Its usage is defined below:

spiral();
//or
spiral(points, twirl, noise, flipLabels);

where:

• points:integer number of points to be generated (both classes)
• twirl:decimal the intensity of the twirl effect
• noise:decimal the amount of noise to be added
• flipLabels:boolean whether to invert labels of data points

sin() function generates data points for regression in \( \mathbb{R}^2 \). Spatially, points describe the behavior of the sine function. Its usage is defined below:

sin();
//or
sin(points, periods, noise);

where:

• points:integer number of points to be generated (both classes)
• periods:integer how many times the period is repeated
• noise:decimal the amount of noise to be added

Training interface

At the bottom area, the adjustable parameters of the classic backpropagation algorithm are learning rate ( \( \eta \) ) and precision ( \( \epsilon \) ) as stopping criterion. Once parameters are changed, hit update parameters button.

To start training, hit iterate button. everything in the structure is updated at each iteration, and the behavior of the mean square error (MSE) along epochs can be inspected by the vertical gauge at the right. As MSE approaches \( \epsilon \), if the convergence slows down and the difference is too small to inspect visually, click the gauge area to zoom in.

Training iterations can be interrupted any time by marking interrupt option, going back to idle state. Changes of weights can be inspected between epochs, once interrupt option is marked and iterate is hit.

It is also possible to inspect changes of weights inside one epoch, as instances are presented to the algorithm, by hitting next instance_c/t button, where c indicates the next i-th instance to be presented out of t instances. After iterating over all instances, an epoch is counted; if iterate button is hit when \( 1 < c < t \), the remaining instances are presented, and an epoch is counted.

In case of restarting training, hit random button, so small random values in \( [0,1] \) are attributed to all weights of the network.

Load and save states

Once a network is trained and ready to use, hit save button. It takes to the command tab and the text area below the generated commands displays the weights that make up the entire model, so it can be used in another piece of software - the color sensor project is a good example that benefits from these weights. The order of weights in relation to the nodes are defined as follows:

The other way around is also possible. A list of weights obtained elsewhere - a MATLAB toolbox, Tensorflow playground, or even my implementation in C language - can be applied to the network structure by pasting it in the same text area from command tab and hitting load button at train tab. Make sure to supply the exact number of weights, in the order illustrated above, necessary to make up a model with the configured structure.

The feed forward button is used when the output or activations of hidden neurons are not refreshed.

Depending of the topological configuration, the button becomes available. More details of this feature are covered in next section.

Visualize tab

When the topology defines a function in \( \mathbb{R}^2 \) or \( \mathbb{R}^3 \), this tab (Shift+4) is enabled and ready to use. It shares the same training interface as in training tab - so the result of iterations can be observed spatially - along with visualization parameters to customize what is shown in the top area.

The size of points (from dataset) to be plotted is defined by PT SIZE and ASPECT defines the aspect ratio of the plotting area. show points button flags whether to show the points along with the output of the model. Once a parameter is changed, plot button will be marked, meaning that there are changes to be done in the visualization. Hit plot button to refresh the plotting area with the new parameters.

The default node to be visualized is the model output (i.e., the first node of the output layer). To select another node for visualization, hit button (also available in training tab, or Shift+V, regardless of the selected tab). It will take to the training tab, prompting to select another node to visualize its output in space - pass the mouse cursor over a node to see a preview of the visualization, or click to be taken to the visualize tab.

A quick note: the image of button is licensed under creative commons license. So let me give the due credits to Amelia Wattenberger.

It is important to point out that whenever a weight is changed at training tab (via training or manually), the visualization, when available, is refreshed. That said, this tool is useful to study the role of each weight in the activation of nodes - performing a manual approximation of the sin function is a good exercise - once it is possible to visualize and change the network at training tab.

Sharing

It is possible to embed this tool in an HTML page with a trained network, so the same model can be tested and used by others. Click at </> button and copy the generated HTML code. Feel free to resize the frame according to your layout, but keep in mind that minimum and maximum widths are 455 and 700 pixels.

Everything shareable is stored within the URL hash. Only the dataset is not shareable - I chose to prioritize the weights in the URL, because large datasets might get truncated.

There is also the possibility of not to embed the weights. To do so, uncheck the synaptic weights option under verbose area in topology tab, as shown below:

URL parameters

There is also the possibility to share the URL (since network weights are carried in hash) - just ignore the HTML code around the URL. The hash contains comma-separated parameters in the following syntax:

#I, N1, ..., Nm [, s | l, ..., s | l] [, eE] [, rR] [, wLIST]

where:

• UPPERCASE are numbers and lowercase are static characters
• I is the number of inputs
• N1, ..., Nm are the numbers of nodes (from layer 1 to layer m (output))
• [, s | l, ..., s | l] are the activation functions to use per neural layer: s for sigmoid and l for linear (Optional. Default is s).
• [, eE] is the precision \( \epsilon \). (Optional. Default is E \( = 0.01 \) ).
• [, rR] is the learning rate \( \eta \). (Optional. Default is R \( = 0.1 \) ).
• [, wLIST] is the list of weights following the order described here. (Optional. Default is a random list).
  • LIST separates weights with separator |

Closing remarks

The purpose of this tool is to study how this architecture of neural networks and the classic backpropagation algorithm work, as well as to share a trained model. It has a lot of limitations and therefore plenty of room for improvements. Again, fork me on GitHub if you are keen on neural networks and this tool.

This is not a data science tool - Cross-validation and preprocessing tools were intentionally not implemented, because they do not concern the main purpose. For real-life applications, Tensorflow library, Python + Scikit-learn and R are the most suitable tools.

Everything was developed in Java and the projects were created using Eclipse. As A* algorithm can be used not only with this Game environment, I separated both projects in GitHub. If you are using Eclipse as well, don’t forget to add A* project to the Java Build Path.

I would say the ability to manipulate the environment is the main feature which justifies - aside from the fun - the development from scratch of this game engine. From appropriate adaptations, there are plenty of areas to develop and apply complementary projects to gaming scenarios, such as artificial intelligence (applicable to several parts of a game, not only giving intelligent behavior to characters), network communication (multiplaying), external user interfaces - from adaptations (e.g., robotic devices for motor rehabilitation) to some new concept of user interfaces - and so on, considering also the combination of each of these and other subjects.

Another benefit from this approach, as mentioned earlier, is the avoidance to write low-level softwares for virtual interfacing purposes. Such an effort can get very complicated and sometimes infeasible for simpler projects. SethBling did an outstanding work in MarI/O applying evolutionary mechanisms to obtain a topology of an artificial neural network that controls Mario properly, being able to cross an entire level. However, it was necessary to deal with an interface in order to exchange IO with an emulator. As far as feasibility is concerned, getting familiar with a particular programming language, as well as studying the achitecture of an emulator are not trivial tasks to be performed and sometimes become more difficult than the project itself. On the other hand it could be a nice excuse to learn more, if you have time :)

It is important to point out that this approach (using this game environment) is useful for recreating 2D games only. In case of a 3D game, Unity 3D does the job very well - it also works with 2D, but I particularly prefer running away from the way they do.

This article encompasses some implementation details. The full UML class diagram can be seen here, but there are some information necessary to assimilate first in order to easily understand the diagram and the code itself. Such information are elucidated as an intuitive notion in the following sections.

• GUI
• Level
• Animation
• Characters
• Game loop

GUI

The GUI was developed using Java’s Swing. The figure below gives a good understanding of the structure of graphic components.

It actually has only two GUI components (a JFrame and a JPanel). Everything else is about drawing graphic primitives, images and calculating positions on canvas. Swing toolkit also provides a good support to listeners, which were also used.

Most of the graphical activity occurs in the double-buffered JPanel, which receives a tiled image and sprite frames of characters. Each iteration of the game loop calulates the position of each animated elements and updates the buffer to be displayed.

The mouse listener is used to click on a point of the map either to go to, or to perform an action in a character (in case the clicked place contains a character). The key listener is used to walk on the map (arrow keys) and to choose actions to perform: follow (F), stun (T), slow (S) and kill (K) - run the code and try it yourself. More about actions is covered ahead.

Level

Given a tileset image and a CSV file with comma-separated tileset coordinates, an image of the map can be built. As illustrates the figure below, each coordinate consists of two values and a special character (between coordinate values) to indicate whether the player can walk on that tile (floor) or not (wall).

Tile map

From the CSV file where a level is designed, TileMap class builds the tilemap image. For each coordinate, the due referenced area of the tileset is copied and drawn on the tilemap image. The arrangement of tiles is obviously the same as the organization of coordinates in the level file.

Pathfinder algorithm

The TileMap class also performs a conversion from the level file structure to a 0-1 matrix. Such matrix composes the input (along with origin and destination locations) of the A* pathfinder algorithm used to obtain shortest paths set as routes of characters. Each location of this matrix and the tilemap is called a cell.

Animation

Every element shown in MapArea that smoothly moves or changes its visual aspect undergoes an animation effect. Motion transition between current and next positions in the map, as well as changes in states of sprite sequences are covered below.

From game mechanics perspective, a walking character, for instance, consists of indicating current and next positions until the path to walk is completely traversed. Positioning characters, graphically, is about a rough change from one cell to another in the map. That is when animation takes place to make these rough transitions more smooth, at a frame update rate (FPS).

Spriter

Still regarding walking character example, it is necessary to show creatures’ movements coupled with motion along the map. These movements are represented in frame-by-frame animations in a spritesheet, or simply sprite, where each frame is an image of a character in a particular pose, composing the whole movement animation, which I am going to call a State.

As can be seen above - Let me give the due credits to those who did a good job gathering GBC Pokemon tiles of the character sprites and the map tileset - Ash walking down, Gary walking down, Ash turning around and Ash walking right are examples of states in a spritesheet, which are respectively indicated by A, B, C and D. Note the arrangement of frames in the sheet, as well as the possibilities to combine frames and come up with animations (the states).

Animator

Transitioning from current to next position, Animator class produces a value within \( [0,1] \) in order to weight the length of a line segment \( l \) that connects origin to destination points in canvas. It basically interpolates values over time (frames, in this case) so values ranging in \( l \) creates motion.

Motion animation, in this project, is used only for walking characters; with the purpose of keeping a constant motion between the ending of one animation (when a cell is reached) and the beginning of the next one (when walking towards another cell) only a linear behavior was considered for interpolating values:

As shown above, given total duration of the animation (i.e., number of total frames \( f_{total} \)), linear interpolation is about having \( f_{total} \) equal segments (\( \dfrac{1}{f_{total}} \)) and as frames run over time, these segments increase uniformly being multiplied by current frame \( f_{current} \) (elapsed time).

I took this oportunity also to implement - because it’s fun - some simplified easing in interpolation behavior, giving a velocity boost either at the beginning or at the end:

Easing in is about giving a boost at the end of interpolation (obviously because it begins with ease), whose behavior is given by linear behavior raised to some power to define the \( order \) of the polynomial and hence the boost intensity. As an example, above we consider \( order=2 \), so we have a quadratic function, or simply a quadratic ease in. Easing out plays the opposite role, whose behavior is the inverse function of the easing in polynomial.

For more about easing, check out Robert Penner’s website. There - different from here, where we deal with the highest-degree term of the polynomial only - can be found a deeper background on easing functions, as well as manipulation of polynomial coefficients and combination of easing in and out.

Characters

Every character (player and non-player) we see on the map is essentialy a GameChar. Looking at the class diagram we can see that no specific behavior (functionality) exists in their actual implementations GamePlayer (human players) and GameComputer (non-human players, a.k.a. NPCs). GameChar assumes an abstract role here, even though technically it is not. I left it like that on purpose for further implementations, because such an arrangement of entities would easily allow to separate different ways to perform the same behavior - polymorphism rocks - and then GameChar would truly become an abstract class.

Actions

Interaction between game characters can be seen as actions being executed to each other. Every action involves two people: the one who executes and the undergone one, people I am respectively calling actor and actee - of course actee is a made-up word, it was inspired in a funny line of Chandler’s in a Friends episode where he said “… the messers become the messees!” - who are instances of GameChar.

Having an actor and an actee, an action behavior can be expressed through GameChar (or even through GamePlayer or GameComputer) methods. I could drain health points (HP) from a character, for example, by simply implementing a total of HP in characters; then public methods could remove some HP from actee and increment it in actor’s, and so on. Currently, as stated in GUI section, the implemented actions are to follow, to stun, to slow and to kill.

Given two characters on a map and melee actions, actor follows actee until they are close enough, and then the action itself is executed. Closeness here is defined by euclidean and manhattan distances (\( d_{euclidean}(\cdot , \cdot ) \) and \( d_{manhattan}(\cdot , \cdot ) \)) - see their usage at Game loop section.

Another interesting detail is that I just mentioned the actor follows the actee before executing an action, and then the action is executed; I also mentioned the existence of a Follow action. If you check the code, you will notice there is nothing to do at FollowAction class, because following is, say, a primitive function in the game, so when actor reaches actee, if action is an instance of FollowAction, actor must not stop following actee - I just translated code to text, check it yourself :).

ActionChar also has a duration (again, in frames), so temporary effects can also exist. When this duration finishes, a method to revert the effect to initial conditions is triggered.

Game loop

If you read until here - I tried to be brief, sorry! - I believe understanding the game loop will not be a problem. Below we have some definitions and the game loop algorithm.

• \( players \) is a collection of GameChar instances (GamePlayer or GameComputer)
• \( p_{target} \) is a GameChar \( p \) has to follow
• \( p_{waypoint} \) a destination cell (the location on the map) to go to
• \( p_{position} \) is the current location of \( p \)
• \( p_{anim} \) is the Animator instance for animating motion of \( p \)
• \( p_{path} \) is a collection of cells, which creates a path from \( p_{position} \) to \( p_{waypoint} \)
• \( p_{busy} \) indicates whether \( p \) is busy traversing a path
• \( p_{spriter} \) is the Spriter instance for frame-by-frame animations of \( p \)
• \( p_{direction} \) is the current direction \( p \) is walking towards

It is a simplified version of the logic running in a separated Thread to update the screen, state of characters and everything else in the game. Click here to open the original image (if everything is too short to read).

• Project overview
• Implementation details
• Tests

This article is divided in the sections listed above. If you wanna jump to the technical details, check implementation details or download the code at the GitHub repository. Before continuing your reading, check the project video to see what this project is really about :)

Project overview

Similar to color vision of the human eye, as well as based in light, the RGB model comprises more than 16 million colors, which are arranged in a 3d space, where integer values of components R (Red), G (Green) and B (Blue), ranging from 0 to 255, constitute coordinates of this space. From this model, color detection and recognition were performed with light-related electronic components and machine learning mechanisms; it is essentially the combination of an RGB LED and a CdS Cell (light sensor, or LDR), both isolated from ambient light. Such components, respectively, emit and sense the intensity of each light (red, green and blue) which was reflected from an object of a particular color.

Color recognition can be performed with machine learning algorithms, such as Multi-Layer Perceptron (MLP) - an architecture of Artificial Neural Networks (ANN). It allows classification and recognition of spatially separable patterns - very useful in this case.

It is important to consider situations when the human eye unsuccessfully attempts to recognize colors due to poor ambient conditions, or even difficulties in distinguishing particular shades of colors. This is comparable to outliers or distorted patterns, which may affect precision in the recognizing task; regarding feature (RGB) space, a misclassified color should be interpreted as a coordinate located at spatial regions associated to another color (sometimes due to poor generalization, overfitting, and many other possibilites).

An MLP, during training, should perform mapping of regions in the RGB color space illustrated below. Each region isolated by hyperplanes represent a color, so every new color pattern located in a particular region is classified as its respective color.

Multi-Layer Perceptron

Multi-Layer Perceptron is a feedforward architecture of ANNs, having an input (non-neural) layer, hidden layers and an output layer. This network is trained by backpropagation algorithm, performing supervised learning (learning by examples).

Topology configuration (i.e., size of each layer) is defined according to a specific problem to be worked on. For this color sensor, as indicated in the MLP Topology Workbench below, the neural network receives 3 inputs (RGB values), having one hidden layer with 6 neurons and an output layer with 10 neurons - just recalling: the output layer must have the same number of classes (colors, in this case), for a binarized output. Hidden-layer sizes are empirically obtained, establishing ranges of values for each topological parameter, so that approaches for Hyperparameter optimization may find potentially good results (this can be a bit difficult and slow sometimes). The network below is already trained and interactive, so it is possible to input values at training tab and check the outputs.

Color recognition

With the purpose of obtaining generalization with an MLP for a good recognition of RGB patterns, a training set (examples of colors with the desired output) must be presented to the network for the training step (download the implemented architecture to perform training). The training set used in this project is available at the project’s GitHub repository.

Generalization will happen within the domain that the training set comprises, so it is worth to pay attention to minimum and maximum values of each component of the space! Do not feed the network with patterns outside this domain, otherwise it would output an unexpected behavior.

The dataset (all examples) contains 75 instances of color patterns ranging from 0 to 1 (logistic activation function was used). Initially ranging from 0 to 255, these instances were rescaled by simply dividing each value by 255, such that \( 0 \leq x_1, x_2, x_3 \leq 1 \). It is important to point out that only one neuron at the output layer must output 1, whereas the remaining ones must output zero. Of course this is not possible using a sigmoid activation function; that’s when post-processing takes place:

where \( y_i \) is the output of the \( i^{th} \) neuron and \( \max(y) \) is the is the greatest output value. In practical terms, the neuron with the greatest output gives 1 as output and the remaining ones give 0. Simple as that.

A trained MLP should create regions in the color space, separating color patterns, as shown below the visualization of the instances of the dataset. Gray color instances are also included for another version of the trained network - but pretend gray instances are not there.

Implementation details

This section is divided in three parts: electronic circuit, color theory and programming. Click the links if you want to jump to a specific part.

Electronic circuit

Arising from objects, all the detection procedure happens in the electronic circuit, encompassing computational activity running in an Atmega328, which is hooked up in an Arduino Uno. Check the scheme below to see the wiring.

The code follows the scheme that uses a common anode RGB LED, so check whether your RGB LED is also a common anode, otherwise just invert the logic in the code.

Another important detail is that I am using only one resistor with the RGB LED. Since one color at a time will be lit, I put the resistor in the common anode, with an average resistance of the resistors that should have been with the cathodes - it is lazy, I know and I am sorry! When I went to buy project parts, they didn’t have everything I needed - but it is however very important to use the correct resistors with the cathodes in order to have fidelity in the collected RGB values in relation to RGB values in the computer. The way I did is not that bad, since the patterns are not distorted; they are just not the same colors we see in a computer screen.

It can be observed from the scheme an adjacency between the RGB LED and the CdS Cell. That is because they must be isolated from ambient light (an oldie black film tube is the perfect piece), so calibration (explained in Programming) and recognition can be performed.

Color theory

Color perception performed by the electronic circuit is based in color theory concepts. Since there are no lenses (yet) involved, only objects with opaque (and matte) material should be considered, avoiding to deal with specular reflection of the LED. Diffuse reflection on the other hand is the key to perform color detection with lights. From an incident light, it is reflected in irregular surfaces, not creating that glowish effect that ruins the function of the CdS Cell.

Back to actual color theory, when light (of a certain color) reaches an object, it is reflected according to properties of that object’s color. For example, a red light reaching a yellow object will be reflected according to how much red exists in the composition of that yellow color - remember, we are talking about lights! - so it is expected to have a lot of red light being reflected, what makes sense when we think of the RGB composition of yellow (essentially red and green). However, when a blue light reaches the yellow object, no strong reflection is expected due to low presence of blue in the color composition.

Considering an additive color system, in which white and black are respectively presence and absence of every colors (more details here), there can be measured (with the CdS Cell) maximum and minimum reflections of each light from the RGB LED which will reach colored objects. That said, it is possible to perform the calibration in electronic components involved in the circuit. This is another key to get fidelity in detection, as well as to ensure a stable detection of patterns (avoiding outliers) - here is a golden tip: after calibrating, try (hard!) not to move or touch neither the electronic components (specially when they are placed in a breadboard), nor the piece you are using (you must use) to isolate components from ambient light.

Programming

For calibration and recognition, the color sensor executes three iterations, once a colored object is exposed to the RGB LED and the CdS Cell. In the first iteration, red light hits the object and the program waits CdS cell to stabilize its sensing; the analog input is then read and the reflection of the red light is stored. The program iterates twice more for green and blue colors. The figure shown in Color theory gives a good visual explanation of this iterative process.

Concerning calibration, the iterative process mentioned above is performed twice: once for black color and once for white color. As explained in Color theory, this is for the detection of maximum and minimum - initially from near zero to near 1024, according to the reading resolution - reflections of red, green and blue lights, obtaining a true range to properly rescale to intervals \( [0,255] \) (for informative purpose) and \( [0,1] \) (the actual input to feed the neural network).

The waiting time to establish reading of the light sensor can vary according to each electronic component, so it is good to give a good delay to ensure a steady sensing. In my case, I gave a 500-millisecond delay, but it is worth to initially use a bigger value and then decreasing it until the verge of a non steady behavior.

In detection, the collected RGB values - ranging from 0 to 1 - feed an MLP, performing the actual color recognition. For the MLP running in Arduino, I am using Neurona - a library I wrote to easily use ANNs in arduino. Check also this post for more details; for training MLPs, I am using an implementation in C language (The MLP Topology Workbench also does the job), giving the adjusted weights to use with Neurona. Still regarding training, it was used \( \alpha=0.8 \), \( \eta=0.1 \) and \( \epsilon=10^{-7} \).

Given topologic configuration and the dataset, five trainings were performed (with cross-validation), each one with a random initial state of the synaptic weights:

From a balanced dataset, accuracy was considered to compare training results. Training #2 presented best generalization, being the one to have its weights embedded along with Neurona and the Arduino program.

Tests

Bringing a more practical perspective, some colors were extracted from the dataset to perform some recognition tests:

Numbers outside the figure are used for identification and numbers inside the figure indicate misclassifications, referencing which colors were classified instead. These colors were printed in sulfite paper with an inkjet printer - check the tiny paper squares in the video in the beginning of this post - so the objects consist of opaque material, proper for color detection.

• Multi-Layer Perceptron (MLP)
• Learning Vector Quantization (LVQ-1) (to be implemented)

Since only the operation mode of these architectures are deployable in microcontrollers, you should also check the respective full implementations (training and operation modes) of the involved architectures; the links in the list above bring more details and the codes available to download. From these programs, it is possible to train topologies of an archictecture, so its output (an array of adjusted weights) constitutes the trained network to be embedded along with an Arduino program.

Download and installation

If you want to install Neurona, you can either search for Neurona in Arduino’s Library Manager or download by clicking the button below:

Check also Neurona documentation for code reference. Click here if you need aditional help in the installation step.

Projects

An usage example of this library can be seen in the Color sensor prototype project; Neurona performs classification of color patterns, given RGB values as input, retrieving the name of the recognized color.

Divided in three sections (implementation details, usage and improvements), this article has the purpose of sharing an implementation of the backpropagation algorithm of the Multi-Layer Perceptron (MLP) architecture in C language as a complement to the therory available in the literature. Such implementation is available at GitHub.

Before approaching details of the implemented ANN architecture, it is important to point out that basic knowledge regarding MLP and backpropagation algorithm is needed. If you are new to ANNs and MLP, I would recommend you to check Freeman and Haykin references.

Implementation details

The implementation was based in this book (which is also a good reference, but only available in portuguese), coded in ANSI-C and should be compiled by GCC.

Among several variations of the backpropagation algorithm, this implementation encompasses the generalized delta-rule with the momentum term in the adjustment of weights. Both training and operation modes are implemented in the same file (check Usage section to see how to trigger each mode). Therefore, this algorithm has the following adjustable parameters:

• \( \eta \) - Learning rate
• \( \epsilon \) - Precision (stopping criterion)
• \( \alpha \) - Momentum rate

The program verboses two types of outputs (when flagged): adjusted weights and the mean square errors (MSEs) of each training epoch. The adjusted weights should be used as input of the operation mode in order to constitute the trained MLP. The weights are ordered according to the appearance of neurons in the topology (i.e., from the first neuron of the first hidden layer to the last neuron of the output layer), as indicated below:

The mean square error at each epoch should be used in visualization, in order to inspect the behavior of the gradient-descent along the iterations from backpropagation algorithm, as can illustrates the plot below:

Usage

As mentioned earlier, the program has two modes: training and operation. Here is how to trigger the training mode:

cat inputFile | ./mlp -i INPUTS -o OUTPUTS -l LAYERS n1 n2 n3 [-[e | E | W]]

where:

• inputFile is the dataset to be used in training
• mlp is the compiled program
• -i indicates the number of INPUTS
• -o indicates the number of OUTPUTS
• -l indicates the number of LAYERS and its sizes
• -[e | E | W]
  • e verboses the number of epochs
  • E verboses the MSE at each epoch
  • W suppresses verbose of adjusted weights

Here is an example of a dataset to use as input in training mode, holding 6 instances; each instance consists of 2 inputs and 3 desired outputs:

6
2.0000 1.0000 1 0 0
7.0050 0.7500 1 0 0
2.0001 0.3240 0 1 0
0.0040 0.2380 0 1 0
7.0050 0.7500 0 0 1
2.0003 2.0001 0 0 1

It is important to point out in this example that numbers without a floating point represent desired outputs for classification. However, it is also possible to use floating points in desired outputs with the purpose of performing regressions or time-series predictions.

Having the output of the training mode (the adjusted weights), here is how to trigger the operation mode:

./mlp -i INPUTS -o OUTPUTS -l LAYERS n1 n2 nLAYERS -w

where:

• -w indicates the insertion of the adjusted weights

Runtime verbose may guide the user for inserting weights and network inputs in order to obtain the due output.

You can also use the MLP Topology Workbench either to generate commands to run the compiled program in training and/or operation modes (command tab), or as an alternative tool for training and/or operation modes:

Improvements

This implementation was focused only in algorithmic fidelity (didactic purposes), and therefore there are several improvements (from which I enumerated a few) to be done, towards an optimized and easy-to-use tool:

• Better memory management: occurrences of memory dynamically allocated could be reduced (or even replaced with static allocation approaches). No allocated memory was freed in this program (sorry, I had a tight deadline to finish this implementation)
  • Operation mode (the forward step) was optimized with statically allocated memory in Neurona – a project involving MLP for AVR microcontrollers
• Parameterize (at program execution) algorithm parameters such as \( \eta \), \( \epsilon \), \( \alpha \) and and the activation function to be used.
• Implement stopping criterion by epochs (also parameterizable).
• Append a new parameter to set a random seed, so outputs/outcomes can become reproductable.
• Input buffer reading improvements, enabling batch-like feed for several instance in operation mode.

Fork me on GitHub if you’re keen on MLPs and ANNs and liked this project :)