Introduction To Neural Networks - California State University, Northridge

Introduction to Neural Networks for Senior Design August 9 - 12, 2004 Intro-1

Neural Networks: The Big Picture Artificial Intelligence Neural Networks not ruleoriented August 9 - 12, 2004 Expert Systems Machine Learning ruleoriented Intro-2

Types of Neural Networks Architecture Recurrent Feedforward Supervised Learning No Feedback, Training Data Available Learning Rule Unsupervised Learning August 9 - 12, 2004 Intro-3

What Is a Neural Network? (Artificial) neural network, or (A)NN: Information processing system loosely based on the model of biological neural networks Implemented in software or electronic circuits Defining properties Consists of simple building blocks (neurons) Connectivity determines functionality Must be able to learn Must be able to generalize August 9 - 12, 2004 Intro-4

Biological Inspiration for Neural Networks Human Brain: 1011 neurons (or nerve cells) Dendrites: incoming extensions, carry signals in Axons: outgoing extensions, carry signals out Synapse: connection between 2 neurons Learning: Forming new connections between the neurons, or Modifying existing connections axon dendrite synapse Neuron 1 August 9 - 12, 2004 Neuron 2 Intro-5

From Biology to the Artificial Neuron, 1 axon dendrite synapse Neuron 1 Neuron 2 w2,k Neuron 1 Σ, f1 w2,1 Σ, f2 Neuron 2 w2,n b August 9 - 12, 2004 Intro-6

From Biology to the Artificial Neuron, 2 w2,k Neuron 1 Σ, f1 w2,1 w2,n Σ, f2 Neuron 2 b The weight w models the synapse between two biological neurons. Each neuron has a threshold that must be met to activate the neuron, causing it to “fire.” The threshold is modeled with the transfer function, f. Neurons can be excitatory, causing other neurons to fire when they are stimulated; or inhibitory, preventing other neurons from firing when they are stimulated. August 9 - 12, 2004 Intro-7

Applications of Neural Networks Aerospace: aircraft autopilots, flight path simulations, aircraft control systems, autopilot enhancements, aircraft component simulations Banking: credit application evaluators Defense: guidance and control, target detection and tracking, object discrimination, sonar, radar and image signal processing including data compression, feature extraction and noise suppression, signal/image identification Financial: real estate appraisal, loan advisor, mortgage screening, stock market analysis, stock trading advisory systems Manufacturing: process control, process and machine diagnosis, visual quality inspection systems, computer chip quality analysis August 9 - 12, 2004 Intro-8

Applications of Neural Networks, cont.’ Medical: cancer cell detection and analysis, EEG and ECG analysis, disease pathway analysis Communications: adaptive echo cancellation, image and data compression, speech synthesis, signal filtering Robotics: Trajectory control, manipulator controllers, vision systems Pattern Recognition: character recognition, speech recognition, voice recognition, facial recognition August 9 - 12, 2004 Intro-9

Problems Suitable for Solution by NN’s Problems for which there is no clear-cut rule or algorithm e.g., image interpretation Problems that humans do better than traditional computers e.g., facial recognition Problems that are intractable due to size e.g., weather forecasting August 9 - 12, 2004 Intro-10

Math Notation/Conventions Scalars: small italic letters e.g., p, a, w Vectors: small, bold, non-italic letters e.g., p, a, w Matrices: capital, bold, non-italic letters e.g., P, A, W Assume vectors are column vectors (unless stated otherwise) 1 e.g., p 2 August 9 - 12, 2004 Intro-11

Network Architecture and Notation Single-Input Neuron Neuron Scalar Input n w p Σ f a Scalar Output b 1 Network Parameters (weight: w, bias: b) a f(n) f(wp b) Adjusted via learning rule Net Input: n Transfer Function, f (design choice) August 9 - 12, 2004 Intro-12

Transfer Functions – Hard Limiter a 1 0 n 0 a f(n) 0, n 0 1, n 0 MATLAB: a hardlim(n) (often used for binary classification problems) August 9 - 12, 2004 Intro-13

Transfer Functions – Symmetric Hard Limiter a 1 n 0 -1 a f(n) -1, n 0 1, n 0 MATLAB: a hardlims(n) (often used for binary classification problems) August 9 - 12, 2004 Intro-14

Transfer Functions - Linear a slope: 1 y-intercept: 0 0 n a f(n) n MATLAB: a purelin(n) (often used in network training for classification problems) August 9 - 12, 2004 Intro-15

Transfer Functions – Log-Sigmoid a 1 0.8 0.6 0.4 0.2 0 -4 -2 0 2 4 n a f(n) 1 1 e n MATLAB: a logsig(n) (often used for training multi-layer networks with backpropagation) August 9 - 12, 2004 Intro-16

Transfer Function Summary Function Equation Output Range MATLAB Hard limiter Discrete: 0 or 1 hardlim Symmetric Hard Limiter a 0, 1, a -1, 1, Discrete: 1, -1 hardlims Linear a n Continuous: range of n purelin Log-Sigmoid a 1 1 e n Continuous: (0, 1) logsig Continuous: (-1, 1) tansig Discrete: 0, 1 compet Hyperbolic Tangent Sigmoid Competitive August 9 - 12, 2004 n 0 n 0 n 0 n 0 ( en e n ) a (en e n ) a 1, neuron w/ max n (0 else) Intro-17

Multiple-Input Neurons Consider a network with R scalar inputs: p1, p2, , pR : Scalar Inputs p1 w1,1 p2 w1,2 . . . w1,R pR Multiple-Input Neuron n Σ f a Scalar Output b 1 a f(n) f(Wp b) where W is the weight matrix with one row: W [w1,1 w1,2 w1,R] p1 and p is the column vector: p M pR August 9 - 12, 2004 Intro-18

Multiple-Input Neurons: Matrix Form Multiple-Input Neuron Input p Rx1 1 W 1xR b a n f 1x1 1x1 1x1 R August 9 - 12, 2004 a f(n) f(Wp b) Intro-19

Binary Classification Example [Ref. #5] Consider the patterns: Features: # of vertices*, # of holes, symmetry (vertical or horizontal) yes(1) or no (0) Noise-free observation vectors or feature vectors 4 2 A 1 , F 0 0 1 * Here a vertex is an intersection of 2 or more lines. August 9 - 12, 2004 Intro-20

Binary Classification Example, continued Targets (desired outputs for each input): t1 -1 (for “A”) t2 1 (for “F”) Neural Network (Given): p1 .42 p2 -1.3 p3 -1.3 Σ f hardlims a hardlims(Wp) W [.42 -1.3 -1.3] August 9 - 12, 2004 Intro-21

Binary Classification: Sample Calculation p1 .42 p2 -1.3 p3 -1.3 Σ f hardlims a hardlims(Wp) W [.42 -1.3 -1.3] Calculating the neuron output if the input pattern is the letter “A”: Wp [.42 -1.3 -1.3] 4 -.92 1 1 a hardlims(Wp) -1 Find Wp and the neuron output a if the input pattern is “F”? August 9 - 12, 2004 Intro-22

A Layer of 2 Neurons (Single-Layer Network) Sometimes we need more than one neuron to solve a problem. Example consider a problem with 3 inputs and 2 neurons: w1,1 w2,1 p1 n1 Σ f a1 b1 w1,2 p2 p1 p p2 p 3 1 w2,2 p3 w1,3 w2,3 Σ n2 b2 1 August 9 - 12, 2004 w1,1 w1,2 w1,3 W w 2,1 w 2,2 w 2,3 f a2 a f( W : row(1) p b1 ) a 1 a2 f( W : row(2) p b2 ) f( Wp b) Intro-23

Weight Matrix Notation Recall for our single neuron with multiple inputs, we used weight matrix W with one row: W [w1,1 w1,2 w1,R] General Case (multiple neurons): components of W are weights connecting some input element to the summer of some neuron Convention (as used in Hagan), for component wi,j of W First index (i) indicates the neuron # the input is entering the “to” index Second index (j) indicates the element # of input vector p that will be entering the neuron the “from” index” wi,j wto,from August 9 - 12, 2004 Intro-24

Single Layer of S Neurons: Matrix Form Layer of S Multiple-Input Neurons Input p Rx1 1 W SxR b Sx1 R August 9 - 12, 2004 a n f Sx1 Sx1 S a f(Wp b) Intro-25

Multiple Layers of Neurons Allow each layer to have its own weight matrix (W), bias vector (b), net input vector (n), output vector (a), and # of neurons (S) Notation: superscript on the variable name indicates the layer #: 1 2 e.g., W : weight matrix for layer #1, b : indicates bias vector for layer #2, a3: output vector for layer #3 4 th layer e.g., S : # of neurons in the 4 Output of layer 1 is input to layer 2, etc. The last (right-most) layer of the network is called the output layer; the inputs are not counted as a layer at all (per Hagan); layers between the input and output are called hidden layers. August 9 - 12, 2004 Intro-26

2-Layer Network: Matrix Form Layer 1 (Hidden) Layer 2 (Output) Input p Rx1 1 W1 S1xR b1 S1x1 n1 a1 f1 S1x1 S1x1 S1 1 S2xS1 b2 S2x1 R a1 f1(W1p b1) August 9 - 12, 2004 W2 n2 f2 S2x1 a2 S2x1 S2 a2 f2(W2a1 b2) Intro-27

Introduction: Practice Problem p1 6 n1 Σ f a1 1) For the neural network shown, find the weight matrix W and the bias vector b. f a2 2) Find the output if f “compet” and the input vector is p p1 1 . p2 2 2 3 1 5 p2 2 n2 Σ -2 1 a compet(Wp b) where compet(n) 1, neuron w/max n 0, else August 9 - 12, 2004 Intro-28

Project Description August 9 - 12, 2004 Intro-29

Is that a tank or a tree?

Computer Neural Network

Project Overview August 9 - 12, 2004 Intro-32

RC Tank/platform/ clutter Image Capture Software (Image Acquisition Toolbox – MATLAB) Data Processing to obtain NN Inputs Neural Network A B Movement direction for camera Image from Camera Camera to Computer Interface Video Camera Tilt/Pan Servos Servo Controller Computer Interface to Servo Controller These components may be combined in one or more physical units Phase 1: How do we get from A to B? Phase 2 (main project): How to we get from B to A?

What Is a Neural Network? (Artificial) neural network, or (A)NN: Information processing system loosely based on the model of biological neural networks Implemented in software or electronic circuits Defining properties Consists of simple building blocks (neurons) Connectivity determines functionality Must be able to learn