Neuromorphic Learning Systems For Supervised And Unsupervised . - CORE
Syracuse UniversitySURFACEDissertations - ALLSURFACEDecember 2016Neuromorphic Learning Systems for Supervised andUnsupervised ApplicationsQiuwen ChenSyracuse UniversityFollow this and additional works at: https://surface.syr.edu/etdPart of the Engineering CommonsRecommended CitationChen, Qiuwen, "Neuromorphic Learning Systems for Supervised and Unsupervised Applications" (2016).Dissertations - ALL. 567.https://surface.syr.edu/etd/567This Dissertation is brought to you for free and open access by the SURFACE at SURFACE. It has been accepted forinclusion in Dissertations - ALL by an authorized administrator of SURFACE. For more information, please contactsurface@syr.edu.
A BSTRACTThe advancements in high performance computing (HPC) have enabled the large-scaleimplementation of neuromorphic learning models and pushed the research on computational intelligence into a new era. Those bio-inspired models are constructed on top ofunified building blocks, i.e. neurons, and have revealed potentials for learning of complexinformation. Two major challenges remain in neuromorphic computing. Firstly, sophisticated structuring methods are needed to determine the connectivity of the neurons inorder to model various problems accurately. Secondly, the models need to adapt to nontraditional architectures for improved computation speed and energy efficiency. In thisthesis, we address these two problems and apply our techniques to different cognitive applications.This thesis first presents the self-structured confabulation network for anomaly detection. Among the machine learning applications, unsupervised detection of the anomalousstreams is especially challenging because it requires both detection accuracy and real-timeperformance. Designing a computing framework that harnesses the growing computingpower of the multicore systems while maintaining high sensitivity and specificity to theanomalies is an urgent research need. We present AnRAD (Anomaly Recognition AndDetection), a bio-inspired detection framework that performs probabilistic inferences. Weleverage the mutual information between the features and develop a self-structuring procedure that learns a succinct confabulation network from the unlabeled data. This networkis capable of fast incremental learning, which continuously refines the knowledge basefrom the data streams. Compared to several existing anomaly detection methods, the proposed approach provides competitive detection accuracy as well as the insight to reasonthe decision making. Furthermore, we exploit the massive parallel structure of the AnRAD
framework. Our implementation of the recall algorithms on the graphic processing unit(GPU) and the Xeon Phi co-processor both obtain substantial speedups over the sequentialimplementation on general-purpose microprocessor (GPP). The implementation enablesreal-time service to concurrent data streams with diversified contexts, and can be applied tolarge problems with multiple local patterns. Experimental results demonstrate high computing performance and memory efficiency. For vehicle abnormal behavior detection, theframework is able to monitor up to 16000 vehicles and their interactions in real-time witha single commodity co-processor, and uses less than 0.2ms for each testing subject.While adapting our streaming anomaly detection model to mobile devices or unmannedsystems, the key challenge is to deliver required performance under the stringent powerconstraint. To address the paradox between performance and power consumption, braininspired hardware, such as the IBM Neurosynaptic System, has been developed to enablelow power implementation of neural models. As a follow-up to the AnRAD framework,we proposed to port the detection network to the TrueNorth architecture. Implementinginference based anomaly detection on a neurosynaptic processor is not straightforward dueto hardware limitations. A design flow and the supporting component library are developed to flexibly map the learned detection networks to the neurosynaptic cores. Insteadof the popular rate code, burst code is adopted in the design, which represents numericalvalue using the phase of a burst of spike trains. This does not only reduce the hardwarecomplexity, but also increases the result’s accuracy. A Corelet library, NeoInfer-TN, is implemented for basic operations in burst code and two-phase pipelines are constructed basedon the library components. The design can be configured for different tradeoffs betweendetection accuracy, hardware resource consumptions, throughput and energy. We evaluatethe system using network intrusion detection data streams. The results show higher detection rate than some conventional approaches and real-time performance, with only 50mWpower consumption. Overall, it achieves 108 operations per Joule.In addition to the modeling and implementation of unsupervised anomaly detection, we
also investigate a supervised learning model based on neural networks and deep fragmentembedding and apply it to text-image retrieval. The study aims at bridging the gap betweenimage and natural language. It continues to improve the bidirectional retrieval performanceacross the modalities. Unlike existing works that target at single sentence densely describing the image objects, we elevate the topic to associating deep image representations withnoisy texts that are only loosely correlated. Based on text-image fragment embedding, ourmodel employs a sequential configuration, connects two embedding stages together. Thefirst stage learns the relevancy of the text fragments, and the second stage uses the filteredoutput from the first one to improve the matching results. The model also integrates multiple convolutional neural networks (CNN) to construct the image fragments, in which richcontext information such as human faces can be extracted to increase the alignment accuracy. The proposed method is evaluated with both synthetic dataset and real-world datasetcollected from picture news website. The results show up to 50% ranking performanceimprovement over the comparison models.
NEUROMORPHIC LEARNING SYSTEMS FORSUPERVISED AND UNSUPERVISED APPLICATIONSByQiuwen ChenB.Sc. Beijing University of Posts and Telecommunications, 2009M.Sc. Beijing University of Posts and Telecommunications, 2012DISSERTATIONSubmitted in partial fulfillment of the requirements for the degree ofDoctor of Philosophy in Electrical and Computer EngineeringSyracuse UniversityDecember 2016
Copyright c 2016 Qiuwen ChenAll rights reserved
A CKNOWLEDGMENTSMany people have helped during my graduate career, without whom this dissertation could not have been finished.First of all, I would like to express my most sincere gratitude to my advisor,Dr. Qinru Qiu, who trusted me and offered me the chance of doctorate study onMarch 20, 2012 when I needed such recognition most. From since, she providedpatient guidance and support all the way through the completion of this degree.Without her, I would have never experienced and known so much more. I wouldlike to thank my committee members, Dr. Roger Chen, Dr. Peng Gao, Dr. YingbinLiang, Dr. Jian Tang and Dr. Yanzhi Wang, for their valuable feedback. I wouldalso like to thank Dr. Qing Wu for his advices.I would like to thank all my labmates: Dr. Yang Ge, Dr. Hao Shen, Wei Liu,Jianwei Cui, Yukan Zhang, Khadeer Ahmed, Ryan Luley, Zhe Li, Amar Shrestha,Yilan Li and Ziyi Zhao. Your collaboration, company and academic insights areprecious during my research journey.My final but everlasting appreciation goes to my family members, who have always supported me in my choices, and encouraged me when I was down. Withouttheir trusts and constant love, I would never be able to come this far. This thesis isdedicated to my parents, Zixi Chen and Shuqing Lian, and my fiance Zhiruo Zhao.vi
TABLE OF C ONTENTSAcknowledgmentsviList of TablesxiList of Figuresxii12Introduction11.1Motivation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .11.2Neuromorphic Computing Model . . . . . . . . . . . . . . . . . . . . . . .31.3Sample Complexity . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .71.4Cogent Confabulation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 101.5Applications . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 131.6Contributions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14Self-structured Confabulation Network for Anomaly Detection172.1Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 172.2Related Works . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 192.3Confabulation-based Anomaly Detection . . . . . . . . . . . . . . . . . . . 202.42.3.1Scoring Algorithm . . . . . . . . . . . . . . . . . . . . . . . . . . 202.3.2Algorithm Analysis . . . . . . . . . . . . . . . . . . . . . . . . . . 22Case Study: Manually Configured Network for Vehicle behavior . . . . . . 252.4.1Problem and Preprocessing . . . . . . . . . . . . . . . . . . . . . . 252.4.2Confabulation Network Structure . . . . . . . . . . . . . . . . . . 26vii
2.4.32.52.62.73Detection Results . . . . . . . . . . . . . . . . . . . . . . . . . . . 29Network Construction and Learning . . . . . . . . . . . . . . . . . . . . . 302.5.1Key Node Hierarchy . . . . . . . . . . . . . . . . . . . . . . . . . 322.5.2Feature Combination Pooling . . . . . . . . . . . . . . . . . . . . 332.5.3k-NN Node Reduction . . . . . . . . . . . . . . . . . . . . . . . . 342.5.4Link Selection . . . . . . . . . . . . . . . . . . . . . . . . . . . . 362.5.5Incremental Learning . . . . . . . . . . . . . . . . . . . . . . . . . 37Evaluations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 382.6.1Datasets . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 382.6.2Comparison Methods . . . . . . . . . . . . . . . . . . . . . . . . . 392.6.3Vehicle Behavior Detection . . . . . . . . . . . . . . . . . . . . . 402.6.4Comparative Evaluations . . . . . . . . . . . . . . . . . . . . . . . 432.6.5Effects of Self-structuring . . . . . . . . . . . . . . . . . . . . . . 46Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 47Parallel Optimization for Inferencing Concurrent Anomalous Data Streams483.1Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 483.2Related Works . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 503.3Complexity Analysis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 503.4Acceleration with CPU Multi-threading . . . . . . . . . . . . . . . . . . . 523.5Fine-grained Parallelization on GPU and Xeon Phi . . . . . . . . . . . . . 533.63.5.1Inefficient Implementation . . . . . . . . . . . . . . . . . . . . . . 533.5.2In-memory Knowledge Base . . . . . . . . . . . . . . . . . . . . . 543.5.3Workload Mapping and Anomaly Score Computation . . . . . . . . 563.5.4Implementation on Xeon Phi . . . . . . . . . . . . . . . . . . . . . 59Evaluations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 593.6.1Single Data Streams . . . . . . . . . . . . . . . . . . . . . . . . . 593.6.2Power and performance tradeoff . . . . . . . . . . . . . . . . . . . 60viii
3.6.33.74Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 64Low-Power Realtime Detection using Burst Code on a Neurosynaptic Processor654.1Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 654.2Spike Burst Coding . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 684.34.44.54.65Multi-stream Extension on Wide-area Monitoring . . . . . . . . . . 614.2.1Encoding Mechanism . . . . . . . . . . . . . . . . . . . . . . . . . 684.2.2Detection Error Analysis . . . . . . . . . . . . . . . . . . . . . . . 69System Design . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 714.3.1Network Mapping . . . . . . . . . . . . . . . . . . . . . . . . . . 724.3.2Divider and Key Lexicon Burst Scorer . . . . . . . . . . . . . . . . 734.3.3Corelet Library and Architecture . . . . . . . . . . . . . . . . . . . 74Inference Pipeline . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 754.4.1Timing for Real-time Processing . . . . . . . . . . . . . . . . . . . 754.4.2Accuracy Factor . . . . . . . . . . . . . . . . . . . . . . . . . . . 76Evaluation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 774.5.1Experiment Setups . . . . . . . . . . . . . . . . . . . . . . . . . . 774.5.2Burst Code vs. Rate Code . . . . . . . . . . . . . . . . . . . . . . 774.5.3Network Construction . . . . . . . . . . . . . . . . . . . . . . . . 794.5.4Detection Quality . . . . . . . . . . . . . . . . . . . . . . . . . . . 794.5.5Throughput and Accuracy Tradeoff . . . . . . . . . . . . . . . . . 804.5.6Power and Performance . . . . . . . . . . . . . . . . . . . . . . . 81Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 82Bi-directional Association between Deep Image Representations and LooselyCoupled Texts5.184Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 84ix
5.2Related Work . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 875.3Visual-Semantic Embedding . . . . . . . . . . . . . . . . . . . . . . . . . 885.465.3.1Text and Image Representations . . . . . . . . . . . . . . . . . . . 885.3.2Selection of Objectives . . . . . . . . . . . . . . . . . . . . . . . . 895.3.3Speed-up with Fragment Padding . . . . . . . . . . . . . . . . . . 91Text Fragment Filtering . . . . . . . . . . . . . . . . . . . . . . . . . . . . 925.4.1Fragment Importance Measure . . . . . . . . . . . . . . . . . . . . 925.4.2Cascade Embedding Stages . . . . . . . . . . . . . . . . . . . . . 935.5Image Fragment Enrichment . . . . . . . . . . . . . . . . . . . . . . . . . 955.6Evaluations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 965.6.1Datasets . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 965.6.2Comparison Methods . . . . . . . . . . . . . . . . . . . . . . . . . 965.6.3Experiment Setup . . . . . . . . . . . . . . . . . . . . . . . . . . . 975.6.4Improvement in Computation Speed . . . . . . . . . . . . . . . . . 985.6.5Results of Text-Image Retrievals . . . . . . . . . . . . . . . . . . . 995.6.6Qualitative Example of Article Search . . . . . . . . . . . . . . . . 1035.7Limitations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1045.8Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 104Conclusion and Future Work1056.1Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1056.2Future Directions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1066.2.1Spiking Confabulation Network: Optimization of Hardware Resource Costs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1066.2.2Text-Image Modeling: Generate Novel Sentence and Paragraphs . . 107References108x
L IST OF TABLES2.1Correlation between anomaly types and outstanding nodes . . . . . . . . . 412.2AUC scores for local knowledge bases . . . . . . . . . . . . . . . . . . . . 422.3AUC scores for different detectors . . . . . . . . . . . . . . . . . . . . . . 442.4AUC scores for different network structures . . . . . . . . . . . . . . . . . 472.5Network complexity impact of self-structuring . . . . . . . . . . . . . . . . 473.1Complexity Analysis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 513.2Single stream per-frame runtimes . . . . . . . . . . . . . . . . . . . . . . . 603.3Power and performance . . . . . . . . . . . . . . . . . . . . . . . . . . . . 614.1Network complexity impacts of constraint . . . . . . . . . . . . . . . . . . 794.2Detection Qualities of Comparison Models . . . . . . . . . . . . . . . . . 804.3Power and Performance of Different Platforms . . . . . . . . . . . . . . . . 815.1Text-image retrieval results on Pascal1k . . . . . . . . . . . . . . . . . . . 985.2Text-image retrieval results on noisy datasets . . . . . . . . . . . . . . . . 101xi
L IST OF F IGURES1.1Neuron Model . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .41.2Integrate and Fire Unit . . . . . . . . . . . . . . . . . . . . . . . . . . . .51.3Confabulation Network Example . . . . . . . . . . . . . . . . . . . . . . . 112.1A zone partition example . . . . . . . . . . . . . . . . . . . . . . . . . . . 262.2Vehicle record and lexicons . . . . . . . . . . . . . . . . . . . . . . . . . . 272.3Vehicle network structure . . . . . . . . . . . . . . . . . . . . . . . . . . . 282.4Anomaly score trace for vehicles . . . . . . . . . . . . . . . . . . . . . . . 292.5AnRAD workflow . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 312.6Hierarchical Structure Example . . . . . . . . . . . . . . . . . . . . . . . . 332.7Relevant Feature Example . . . . . . . . . . . . . . . . . . . . . . . . . . 342.8Vehicle Detection Result . . . . . . . . . . . . . . . . . . . . . . . . . . . 412.9Comparison between local and single knowledge bases . . . . . . . . . . . 432.10 Results on DARPA dataset . . . . . . . . . . . . . . . . . . . . . . . . . . 442.11 Mutual Info between ADFA-LD system call and its previous calls . . . . . 452.12 Results on ADFA-LD using clean training data . . . . . . . . . . . . . . . 452.13 Results on ADFA-LD using tainted training data . . . . . . . . . . . . . . . 463.1Thread pool for CPU multi-threading . . . . . . . . . . . . . . . . . . . . . 523.2In-memory knowledge base layout . . . . . . . . . . . . . . . . . . . . . . 543.3Memory usage of individual models . . . . . . . . . . . . . . . . . . . . . 563.4Anomaly Score Computation . . . . . . . . . . . . . . . . . . . . . . . . . 58xii
3.52D workload mapping . . . . . . . . . . . . . . . . . . . . . . . . . . . . 623.6Memory consumption of multiple zones . . . . . . . . . . . . . . . . . . . 633.7Vehicle detection throughputs . . . . . . . . . . . . . . . . . . . . . . . . . 644.1Burst Code Neuron Dynamics . . . . . . . . . . . . . . . . . . . . . . . . 684.2System Workflow . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 714.3Network Mapping . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 724.4Key Lexicon Anomaly Scorer . . . . . . . . . . . . . . . . . . . . . . . . 734.5Corelet Architecture . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 744.6Detection Pipeline Timing . . . . . . . . . . . . . . . . . . . . . . . . . . 764.7Excitation Correlation between Spike Code and Reference Program . . . . 774.8Precision of lexicons on anomalies . . . . . . . . . . . . . . . . . . . . . . 784.9Tradeoff between Quality and Speed . . . . . . . . . . . . . . . . . . . . . 804.10 Power/Energy Consumptions for Different Window Lengths . . . . . . . . 825.1Comparison between descriptive text-image pair and picture news . . . . . 855.2Computation of Alignment Matrix . . . . . . . . . . . . . . . . . . . . . . 915.3Configuration of fragment filtering and fragment enrichment . . . . . . . . 945.4Computation Speed Comparisons . . . . . . . . . . . . . . . . . . . . . . . 995.5Weight output from filter embedding on Pascal1k with noises . . . . . . . . 1005.6Top 10 fragments with the highest dot products to the detected face . . . . . 1005.7Training time for 800 samples on People1k . . . . . . . . . . . . . . . . . 1025.8Text Search Results using Example Image Queries . . . . . . . . . . . . . 103xiii
1C HAPTER 1I NTRODUCTIONNeuromorphic learning systems leverage bio-inspired computation to model different applications. Generally, they employ integrate-and-fire architectures that mimics the humandecision-making processes. The challenges of building such systems include fitting datapatterns with such knowledge models using supervised or unsupervised learning, constructing appropriate network structures, as well as implementing the system on paralleland brain-inspired hardware architectures.In this chapter, we discuss the motivation of the study. We introduce the general neuralcomputing model, and discuss the sample complexity and strategies to train such networks.Then, applications of different learning systems are proposed. Finally, the contributions ofthe thesis are reviewed.1.1MotivationIn recent years, studies on machine learning, especially neural networks have received wideattentions. This is due to the advancements in computation devices and the availabilityof the training data, i.e.“Big Data". Sophisticated models can be built to capture verycomplex patterns in various applications. For example, Krizhevsky et. al. [53] developed
2deep convolutional neural network that trained on ImageNet [30] using Graphic ProcessingUnits (GPU), which achieved close to human accuracy in image classification; Hochreiteret. al. [39] proposed Long-short term memory (LSTM) recurrent neural network (RNN)that can effectively process sequential information such as the voice and natural languageas a human does. One groundbreaking feat of the artificial intelligence research is thevictory of Google’s Alpha Go [84] system over the human Go champion on March 2016,which also revealed three major aspects of the machine learning systems. The importance of effective modeling. The solution space of the Go game is in theorder of 10170 possible paths. No ordinary learning model is capable of capturingso many patterns, because fitting the solution space will require a huge amount oflearnt parameters, and thus an infeasible number of training samples. In the AlphaGo case, Monte Carlo tree search is used to mitigate the problem, in that the programplays with itself to improve the learning. But the method is not always availablefor other applications. Therefore, many techniques are studied to prevent overfitting.Typical examples include transfer learning, parameter sharing and regularization. Inthis study, more interests are drawn to use the network structure and learning rule toresolve the sample complexity. The demand of computation power. With all those delicate learning strategies,Alpha Go would not be possible if it cannot perform the computations quickly. According to the reports [31] about the Go game challenge, Alpha Go uses 1920 CPUsand 280 GPUs to keep on the pace of an interactive game. The time consumptionis always a concern as the model being more and more complex, and the solutionusually turns to parallelization. However, parallelization is not simply stacking thehardware. Effective utilization of the computing device depends on the partition ofthe problems and the workload distribution of the particular models. Therefore, another target of this study is to investigate the concurrent structure of the learningmodel that improve the throughput and responsiveness, so that real-time processing
3can be achieved. The challenge of power reduction. The Alpha Go system consumes 106 Watts ofpower, while the human brain works with around 20 Watts. Although the powerconsumption may not be the first priority for a server or data center environment, itis critical when implementing the learning models on mobile devices and embeddedplatforms. Different from traditional signal processing, a learning system’s computation does not always require high precision, and it has intensive memory accesses. Aswe are developing and parallelizing neuromorphic algorithms, the study also looksinto implementation on top of neuromorphic hardware to achieve ultra-high energyefficiency.This thesis addresses the above three major research problems. Firstly, the neuromorphic learning models are explored. We improve the quality of the model from the algorithmpoint of view, and try to solve different problems by modeling the supervised and unsupervised applications with proper network architectures. In this way, different applications canbe adapted with unified computing units that ease the implementation. Secondly, we implement the networks using parallel devices. The concurrent structure of the neural modelsis exploited to accelerate the inference computation. Thirdly, the models are further implemented with non-traditional architectures of spiking neural network. We investigatemapping and signal processing using such hardware to enable the network operate in anenergy efficient manner.1.2Neuromorphic Computing ModelFig. 1.1 shows how neurons communicate with each other. Neuron is the basic buildingblock of the brain. It contains a set of dendrites that receive impulses from other neurons,and a cell body that process the signals. When a neuron accumulates certain amount of
4Fig. 1.1: Neuron Modelimpulses, it fires and passes the electrical signal to the others through its axons. The junction between two neuron cells are called synapse, whose strength decide the influence ofone neuron on the other. The emitter and the receiver of the signal are referred to as thepresynaptic neuron and the postsynaptic neuron respectively. The learning is achieved byforming the neuron connections and the synaptic strength, and the complex inference taskcan be achieved by chain reaction in large neuron networks. Although neurons fire at a lowfrequency (1 - 200Hz), they are massive in the amount and operate in parallel. Therefore,the brain is capable of fast cognitive tasks.By mimicking the biological nerve system, the neuromorphic model is simplified as inFig. 1.2, which shows a single neuron cell and its input/output. The computation referredto as integrate-and-fire is described by Equation (1.1).y f(NXwi xi b)(1.1)i 0Here, this neuron receives signals from another N presynaptic neurons. xi is the signal received from the ith neuron’s axon and wi denotes the synaptic weight, which is a multiplierdetermining the influence of xi . The neuron integrates the incoming excitations and a biasterm b, and applies the sum to f (.), the activation function to add non-linearity. And y isthe output of this neuron that can be passed to other neurons or used as the final output.In this function, wi and b are learnt parameters that are tuned using training samples bydifferent learning methods. The main idea is to have wi and b reach values such that the
5Fig. 1.2: Integrate and Fire Unitgenerated y approaches the desired result.In the case of artificial neural network (ANN), a popular learning method is stochastic gradient descent (SGD). A loss function loss(y) is defined reflecting the discrepancybetween the expected output and the network-generated output. Then the error is backpropagated to update the model by finding the derivatives of the loss with respect to theparameters.wi In Equation (1.2),X @loss(y) @y@y @wibatchj(1.2)wi is the gradient. It is used to adjust wi by a certain updating strat-egy. For a fixed learning rate , the parameter is updated by wi0 wi wi , whilethere are other more sophisticated updating rules to improve the learning quality. Training is performed by iterating the data, drawing random sample mini-batches and updatingthe parameters in multiple epochs. Take logistic regression for example, the activation isPcalculated by sigmoid function f (x) 1/[1 exp( Ni 0 wi xi b)], and the parameteris trained to directly optimize the conditional probability p(y x). In a supervised binaryclassification task, we can use the 0-1 prediction error to tune the parameters. Such methodis categorized as a Discriminative Model [70]. Other than discriminative model and ANN,there are also neuromorhpic methods using Generative Model that optimize the joint probability p(y, x). They offer different asymptotic error and sample complexity in obtainingthe model parameters. We will discuss the model selections in Section 1.3 and further in
6Section 2.3.2.The neuromorphic learning models draw wide attention partly due to their capacity. It isproved that neural network is capable of approximate any function given sufficient neurons[5] and training samples. However, another important reason is that the neuromorphiccomputing architecture facilitates efficient implementation of the learning systems on nontraditional hardware platforms.Traditional computing systems are built based on the Von Neumann architecture, inwhich the computation unit (CPUs) and the data/code storage are separated and interactwith each other through a bus. While such setup is general enough to handle most of thelogic and arithmetic tasks, it also suffers from inefficiency that the data have to be movedback and forth through the bus. This is referred to as the Von Neumann bottleneck [6]. Thislimitation is especially significant with large-scale learning systems because the modelsusually include some form of knowledge base and the computation is data-intensive. Theinefficiency in the traditional hardware results in both slow computation and high powerconsumption.Human brain performs learning tasks in such efficient way, so does the bio-inspiredcomputation model. Firstly, the neuromorphic models feature tightly coupled memory(synapses) and computation (neuron cell body). Therefore, they can easily adapt to nontraditional architectures that eliminate the memory bottleneck. Secondly, the computationof neuromorphic models don’t usually require high precision. So approximate computationsuch as spike codes can be adopted to reduce the power consumption. Thirdly, the inferencecomputation of neurons are massively concurrent. It can be exploited using parallel devicesto improve the performance and provide real-time services.
71.3Sample ComplexityWhile the neuromorphic systems provide benefits such as modeling capacity and efficientimplementation, a key challenge is how to obtain the set of parameters, i.e. wi and b inEquation (1.1). If we have infinite amount of data, we can simply apply SGD and themodel would eventually arrive at a good quality, but this is obviously an unreasonableassumption, as real world data, especially labe
implementation of neuromorphic learning models and pushed the research on computa-tional intelligence into a new era. Those bio-inspired models are constructed on top of unified building blocks, i.e. neurons, and have revealed potentials for learning of complex information. Two major challenges remain in neuromorphic computing. Firstly, sophis-
Introduction to Neuromorphic Computing Insights and Challenges What is a neuromorphic computer? Why is neuromorphic computing confusing? What about building a brain? Todd’s Top 10 list of challenges Alternative ways of thinking about a building a brain Closing thoughts Outline . What is a Neuromorphic Computer? A neuromorphic computer is a machine comprising many .
Bruksanvisning för bilstereo . Bruksanvisning for bilstereo . Instrukcja obsługi samochodowego odtwarzacza stereo . Operating Instructions for Car Stereo . 610-104 . SV . Bruksanvisning i original
neuromorphic vision datasets [ , ] have been released in recent years, which facilitate the neuromorphic vision application forobject detectionand tracking. Recent years also witness the various applications for detection and tracking tasks with neuromorphic vision sensor such as featuretracking[ , ],linetracking[],andmicroparticle tracking[ ].
Neuromorphic computing spans a broad range of scientific disciplines from materials science to devices, to computer science, to neuroscience, all of which are required to solve the neuromorphic computing grand challenge. In our workshop we focus on the computer science aspects, specifically from a neuromorphic device through an application. .
10 tips och tricks för att lyckas med ert sap-projekt 20 SAPSANYTT 2/2015 De flesta projektledare känner säkert till Cobb’s paradox. Martin Cobb verkade som CIO för sekretariatet för Treasury Board of Canada 1995 då han ställde frågan
service i Norge och Finland drivs inom ramen för ett enskilt företag (NRK. 1 och Yleisradio), fin ns det i Sverige tre: Ett för tv (Sveriges Television , SVT ), ett för radio (Sveriges Radio , SR ) och ett för utbildnings program (Sveriges Utbildningsradio, UR, vilket till följd av sin begränsade storlek inte återfinns bland de 25 största
Hotell För hotell anges de tre klasserna A/B, C och D. Det betyder att den "normala" standarden C är acceptabel men att motiven för en högre standard är starka. Ljudklass C motsvarar de tidigare normkraven för hotell, ljudklass A/B motsvarar kraven för moderna hotell med hög standard och ljudklass D kan användas vid
Am I my Brother’s Keeper? Acts 15:19-35 Introduction: Since the beginning of time when the first man and woman rebelled against God, mankind has been separated from God. Every person since that time has been born into that rebellion and sin. Because of sin, people are separated from God and are unable to have a right relationship with Him or each other. Ill. of evil and suffering Inside of .
