The World's Second Computing Architecture:

The Brain


Why Neurons?


Neruons evolved in response to the need for faster inter-cellular communication in mobile large multi-cellular organisms. With chemical diffusion signaling and round cells, communication times between cells depends greatly on distance, and is simply too slow in large critters. Here's an (entirely hypothetical) example of the type of trouble you can get into if you lack neurons:





Signal Processing in a Neuron[1]


The neuron fixes the communication problem with cell body attached to long dendrites (inputs) and a very long axon (output). This allows signals to propagate as a electric potential wave along the membrane of a single neuron, rather than jumping from cell to cell via diffusion.


Anatomy of a Neuron:[2]



Signal Activation


A signal (neuron activation spike) originates at the base of the axon in response to activation of the dendrites. The inside of a neuron is held at a constant rest potential (usually around -70 mV) by potassium and sodium ion pumps in the membrane (sodium(NA+) is kept outside, potassium inside(K+)). This potential can be altered by nearby electric fields (if other sections of the membrane are spiking) or by transmitter compounds binding to the ion pumps. When the membrane potential rises above a certain threshold potential, voltage-controlled sodium channels open in the membrane and sodium (positively charged) diffuses into the cell, further raising the voltage. When the sodium gradient has disappeared, the peak voltage has been reached and the sodium channels close while the potassium channels open. Positively charged potassium rushes out of the cell, and the potential returns to rest potential, at which point sodium and potassium gradients are re-established for the next pulse. See a nifty animation of the whole process here.


Signal Propagation


Once an activation spike has begun, it travels down the neuron as a wave. However, this wave relies on ions moving across the membrane, so is still relatively slow (~1 m/s). Neurons with any significant length to their axon have a Myelin sheath around sections of the axon. This sheath makes the membrane impermeable and raises its resistance, allowing the action potential to propagate electrically. The myelin is punctuated by sections of open membrane to regenerate the signal. Signals in myelinated axons can travel between 18 and 120 m/s, depending on the width of the axon. There's a cool animation of signal propagation here.


Signal Transmittance


When an activation spike reaches the end of the axon, it is transmitted to other neurons by chemical signals that cross the synaptic gap. The action potential stimulates the release of vesicles filled with neuro-transmitting chemicals, which then bind to the ion pumps on the neighboring neuron. These transmitters can be excitatory or inhibitory, and the amount of transmitters released can be varied to vary the strength of the signal. The process can be seen in another cool animation.


Varying Signal Strength


Neuron peak activation voltage is always the same, so signal strength has to be encoded in some other manner. As the activation of a neuron increases, the frequency of spike activation pulses increases. How activation affects spike frequency depends on the type of neuron: burst neuron respond with high frequency bursts followed by a pause, whereas single spike neurons produce spikes at regular intervals.




Advantages and Disadvantages of Biological Neural Networks


What this all boils down to is that the brain is an exceedingly complicated structure. With roughly 100 billion non-deterministic neurons (of multiple types!) in the human brain, connected through perhaps 100 trillion synapses,[3] the task of fully modeling brain computation is beyond modern mathematics.[4] This holds true even for much smaller networks!


Advantages


Disadvantages




Biological Neural Networks – Uses in Computer Science


The two properties of biological neural networks we most often want to use in CS are the massively parallel processing and learning. To use these we can either try to use biological neurons themselves, or simulate them in hardware or software.



Harnessing Biological Neural Networks


Cultured Neural Networks


In recent years many research groups have focused on culturing neurons around electrodes. By reading and applying voltage from and to the electrodes, scientists can allow the neurons to interact with some environment and train them to particular tasks.


Rat Flight Control


One such study headed by Thomas Demarse at the University of Florida, involved 25,000 rat neurons cultured around 60 electrodes and trained to act as an autopilot in a flight simulator. Two of the electrodes allowed the network to control the pitch and yaw of the plane, while others allowed the network to sense the current trajectory and orientation of the plane. Network responses were trained to particular inputs by applying high and low frequency stimulation to network's output nodes, which previous studies had shown respectively lower or raise the network activation at that particular site. A pdf version of the paper can be found here.


Rat Controller for Virtual Avatar


In 2001, Demarse's group also hooked up a similar rat neuron contraption to a simulated environment, and allowed it to control an avatar within a virtual room. Feedback was provided to the network about collisions with its virtual environment and its movement through the environment, but no method for explicitly training the network from the feedback was known at the time. Read about it here.


Enter the Leech-ulator


Rat brains aren't the only option for electronically interface biological neural networks! Demarse and Bill Ditto (also of the University of Florida) recently announced results using leech neurons for performing logic operations. Leech neurons are large enough that they can be controlled and linked individually. By taking two neurons and individually controlling the resting potential, Demarse and Ditto were able to create NOR gates, which they then used to composite simple adders, multipliers, and other operations using digital design techniques. Paper is here.



Hurdles for Cultured Neural Networks




Disadvantages of Cultured Neural Networks




Simulating Neural Networks – Artificial Neural Networks


Rather than rely on biological neural systems, we may also simulate networks in software or electronic hardware using a simplified version of the neuron:[5]





A node in an artificial neural network simply computes the weighted sum of its input activations, and runs the result through a sigmoid function (a non-linear function is required here, or we loose the ability to easily match non-linear situations). The threshold of the sigmoid refers to its point of odd symmetry. Networks are set up as a graph of connected nodes. Typically feed-forward networks are used, meaning that signals move through successive layers of nodes from the inputs to the output without any feedback loops:[6]



For a more complete description of how the ANN abstraction works, check out this site.



Training ANNs


Despite being simpler than biological networks, artificial neural networks are still extremely non-linear and would be an enormous pain to program directly (and that would defeat the whole purpose!). Fortunately their exist a multitude of algorithms to train ANNs (by adjusting the weights and maybe thresholds).


Supervised Learning is possible when the correct responses are known for a certain set of data, and we wish to train the network to match the data. Supervised Learning is achieved by altering weights in response to the networks output for a given training datum, via some algorithm such as backpropagation. This approach is often used in pattern matching problems, such as image processing and character recognition.


Reinforcement Learning is useful when we know the results of a networks action (generally the effect on its environment) and we wish to encourage certain results. A good method for reinforcement learning is to evolve populations of networks, keeping and randomly modifying networks that produce better results. Reinforcement learning is useful for development networks for environmental interaction or gameplay.


For more information on network training techniques, check out wikipedia.



How Much Can ANNs Compute?


It turns out that ANNs are actually Turing complete! Here's a simple NAND gate using a single node ANN:



If A and B are both 1, then the activation on the node is -8, so a 0 is output. Otherwise, the activation is either -4 or 0, both of which are above the threshold, and so the output is 1. Since NAND gates can be used to construct Turing complete machines, so too can ANNs! Note that this does not extend to biological neural networks, since the biological networks are non-deterministic.



Implementing ANNs in Hardware


Since the late 80s scads of companies have attempted to implement artificial neural in hardware (and many have succeeded). These implementations are rarely of the generic ANN abstraction described above, but rather usually some variation designed with a specific application (or type of training) in mind.


Among the different variations implemented are:


Typically, implementations are severely limited in network size, since hardware requirements grow at least with the square and by as much as the cube of the network dimensions.


Since precision is often not an issue in intermediate steps of ANN computations, analog circuitry has also been used for some implementations. Analog circuitry allows space and energy costs to be reduced and speed to be increased. Unfortunately, manufacturing variations mean that weights discovered on one chip may not work on a different chip. Just as in biological networks, the ANN has to be retrained for each chip on which it runs. Generally these chips are used in applications such as pattern matching in particle colliders, so multiple copies of the network are not required.



Examples of Hardware ANNs


ZISC (IBM's Zero Instruction Set Computer)


PRAM 256


LCNN (Analog Local Cluster Neural Network Hardware)


The Future of Hardware ANNs


Hardware implementations of artificial neural networks have proved useful in limited fields, but because of the ease of use of software implementations and the specialized nature of hardware implementations, we have yet to see widespread use. As we begin to see more embedded systems with complex processing functions such as speech recognition, face recognition, or complex control, it is possible hardware ANNs will enjoy wider use.




Footnotes


[1] Unless otherwise referenced, material in this section is from The Central Nervous System: Structure and Function (Third Edition) by Per Brodal (2004).


[2] Shamelessly yoinked from http://www.mindcreators.com/NeuronBasics.htm


[3] http://www.nervenet.org/papers/NUMBER_REV_1988.html


[4] Though understanding the neural pathways are beyond modern science, we can still predict something about the behavior of neural networks. This is the job of psychologist and behavioral biologists!


[5] Picture borrowed from http://homepages.gold.ac.uk/nikolaev/311perc.htm


[6] Picture from http://www.dtreg.com/mlfn.htm