Digit Recognizer By Convolutional Neural Network (CNN)

Digit Recognizer by Convolutional Neural Network (CNN)Ding Li2018.06online store: costumejewelry1.com

MNIST database(Modified National Institute of Standards and Technology database)60,000 training images; 10,000 testing imagesKaggle Challenge42,000 training images; 28,000 testing imagesInputHandwritten DigitResultPredict784 (28*28) PixelsPixel color coding [0,255]Input ImagePixel ValueLabel[0,9]https://en.wikipedia.org/wiki/MNIST database2

Conv 13*3Conv 23*316 filters16 filters24*24*1626*26*16Input28*28Conv 33*3Conv 43*332 filters32 filters51212*12*16TrainableParameters:814,778Max pool 22*28*8*3210*10*32FlattenMax pool 12*24*4*32Dense 1ReluDense 2ReluDense 4512https://en.wikipedia.org/wiki/Convolutional neural networkLeNet - dLabel[0,9]10VGG 16https://en.wikipedia.org/wiki/Yann LeCun3

101001010010100*10-110-110-110 10 00 -110 10 010 -110 10 010 -11 100010001000 30Detected edge10 10 1000010 10 1000010 10 1000010 10 1000010 10 1000010 10 100001*0-110-110-1030 300030 300030 300030 300No lanation-convnets/4

*10-110-110-1111000-1-1-1Multiple convolutional filters (kernels) can capture different changes of neighborhood ion-kernels-with-online-demo5

38217911Max Pool924523Size: 2 x 2635612Pool after convolutional layer, reduce noise and data size.Local information are concentrated into higher level information.Besides max pool, there is also average pool.6

Inputx0, x1, x2OutputaParametersw0, w1, w2, b(to be optimized)𝑧 w 0 x0 w 1 x1 w 2 x2 ba f(z)activation functionNonlinearincrease certaintyononoffoffReLUf(z) max(0, d𝑓(𝑧) 11 𝑒 𝑧7

All the input from the previous layer arecombined at each 3]𝑎5[1]11111[1]11111𝑎0 𝑓(𝑤0,0 𝑥0 𝑤1,0 𝑥1 𝑤2,0 𝑥2 𝑤3,0 𝑥3 𝑏0 )𝑎1 𝑓(𝑤0,1 𝑥0 𝑤1,1 𝑥1 𝑤2,1 𝑥2 𝑤3,1 𝑥3 𝑏1 ) .All the local features extracted in previouslayers are fully connected with differentweights to construct global features.Complicated relationship between input canbe revealed by deep networks.Fully Connected at-is-deep-learning/8

[𝐿 1]𝑎0[𝐿 1]𝑎1[𝐿 2]𝑎2[𝐿 1]𝑎1023L-1 layer𝐿𝑤0,0𝐿𝐿 1𝑧0 𝑤0,0 𝑎0𝑦ො 00𝑦ො 11𝑦ො 22 . . 𝐿𝑤1023,0Linear combination of inputs from previous layer:𝐿𝐿 1 𝑤1,0 𝑎1𝐿𝐿 1 𝑤2,0 𝑎2L layer[𝐿 1]𝐿Softmax, normalize the result:𝑦ො 0 𝑒 𝑧0𝑒 𝑧0 𝑒 𝑧1 𝑒 𝑧2 𝑒 𝑧9Probability that 𝑦ො 0𝑦ො 0 𝑦ො 1 𝑦ො 2 𝑦ො 9 1E.g. y 0Prediction:𝑦ො [ 𝑦ො 0, 𝑦ො 1, 𝑦ො 2, 𝑦ො 9 ]𝑦ො [ 𝟎. 𝟗, 0.02, 0.01, 0.02, 0.04]True value:y [ y0 , y 1 , y 2 , y 9 ]y [ 1, 0, 0, 0, 0, 0, 0, 0, 0, 0]9ො 𝑦 𝑦𝑖 log(𝑦ෝ𝑖 )Loss function: 𝐿 𝑦,𝑦ො 9𝐿 𝑤1023,0 𝑎1023 𝑏0𝐿 𝑦,ො 𝑦 1 log 0.9 0.046𝑖 09L 0,ො 𝑦 1 log 1 0𝑎𝑡 𝑏𝑒𝑠𝑡 𝑚𝑎𝑡𝑐ℎ 𝑦ො y, 𝐿 𝑦,1ොCost function: 𝐽(𝑤, 𝑏) 𝑚 σ𝑚 𝐿 𝑦,𝑦𝑡𝑜𝑡𝑎𝑙 𝑙𝑜𝑠𝑠 𝑜𝑓 𝑚 𝑠𝑎𝑚𝑝𝑙𝑒𝑠𝐺𝑜𝑎𝑙: 𝑚𝑖𝑛𝑖𝑚𝑖𝑧𝑒 𝑡ℎ𝑒 𝑐𝑜𝑠𝑡 𝑓𝑢𝑛𝑐𝑡𝑖𝑜𝑛.A Friendly Introduction to Cross-Entropy Loss9

𝐴[𝐿 2][𝐿 2]𝑎0[𝐿 2]𝑎1[𝐿 2]𝑎2𝑊 [𝐿 1]𝑏 [𝐿 1]𝐴[𝐿 1][𝐿 1]𝑎0[𝐿 1]𝑎1[𝐿 2]𝑎21. With the initial parameters W and b, predict the label 𝑦ො withforward propagation, calculate the cost.𝑊 [𝐿] 𝑏 [𝐿]𝑦ො 0𝑦ො 101𝑦ො 2L-2 layer[𝐿 1]𝑎1023L-1 layer2 . . . . 𝑦ො 9L layer[𝐿 2]𝑎1023𝐺𝑜𝑎𝑙: 𝑚𝑖𝑛𝑖𝑚𝑖𝑧𝑒 𝑡ℎ𝑒 𝑑𝑖𝑓𝑓𝑒𝑟𝑒𝑛𝑐𝑒 𝑏𝑒𝑡𝑤𝑒𝑒𝑛 𝑝𝑟𝑜𝑗𝑒𝑐𝑡𝑒𝑑 𝑦ො 𝑎𝑛𝑑 𝑡𝑟𝑢𝑒 𝑦.Forward PropagationBackpropagation2. 𝑂𝑝𝑡𝑖𝑚𝑖𝑧𝑒 𝑡ℎ𝑒 𝑝𝑎𝑟𝑎𝑚𝑒𝑡𝑒𝑟𝑠 𝑜𝑓 𝐿 𝑙𝑎𝑦𝑒𝑟, 𝑊 [𝐿] & 𝑏 [𝐿] ,assuming inputs from L-1 layer, 𝐴[𝐿 1] do not change.3. 𝐶𝑎𝑙𝑐𝑢𝑙𝑎𝑡𝑒 𝑡ℎ𝑒 𝑐ℎ𝑎𝑛𝑔𝑒 𝑜𝑓 𝐿– 1 𝑙𝑎𝑦𝑒𝑟 𝑖𝑛𝑝𝑢𝑡 , 𝐴[𝐿 1] ,which is needed to minimize the cost funtion,assuming parameters 𝑊 [𝐿] & 𝑏 [𝐿] do not change.4. 𝑂𝑝𝑡𝑖𝑚𝑖𝑧𝑒 𝑡ℎ𝑒 𝑝𝑎𝑟𝑎𝑚𝑒𝑡𝑒𝑟𝑠 𝑜𝑓 𝐿– 1 𝑙𝑎𝑦𝑒𝑟, 𝑊 [𝐿 1] & 𝑏 [𝐿 1] ,𝑓𝑟𝑜𝑚 𝑡ℎ𝑒 𝑑𝑒𝑠𝑖𝑟𝑒𝑑 𝑐ℎ𝑎𝑛𝑔𝑒𝑠 of 𝐴[𝐿 1] .95. Proceed like this all the way to the first layer,optimize the parameters W and b of all layers.6. Running forward propagation and backpropagation once is calledone epoch, run multiple epochs until cost is near minimum value.10

A proportion of nodes in thelayer are randomly ignored foreach training sample.This technique can force the network to learn features in a distributed way and reduces the overfitting.Dropout applied after the two pool layers and first two full connected layers.11

After 1 hour, 30 epochs’ training,achieved 99.67% accuracy.Some images are even hard for human torecognize, more samples like these can help.12

Convolution Matrix, 16 filtersGenerated from trainingInputOutput13

More abstract14

lightLogicalsignalssignalsThe machine will combined the 1024 final values to judge the label [0,9].15

Can we understand machine’s ‘mind’?16

Convolutional Neural Network is very powerful for analyzing visual image. The convolutional layers can capture the local features. The pooling layers can concentrate the local changes, as well as reduce the noise and data size. The full connected layers can combine all the local features to generate global features . The global features are combined to make the final judgement, here the probability of label [0,9]. Can human understand Artificial Neural Networks? Is there any similarity between brain and CNN to process the visual information? What is the meaning of local and global features generated by machines? Can human understand machines’ logic?Python code of the project at kaggle: ith-cnn-keras17

Convolutional Neural Network is very powerful for analyzing visual image. The convolutional layers can capture the local features. The pooling layers can concentrate the local changes, as well as reduce the noise and data size. The full connected layers can combine all the local features to generate global features .