Food Image Recognition By Deep Learning - Nvidia
Food Image Recognition byDeep LearningAssoc. Prof. Steven HOISchool of Information SystemsSingapore Management University
National Day Rally 2017:Singapore's War on Diabeteswww.moh.gov.sg/budget2016“Four simple ways to fight diabetes: Go for regular medical check-ups;Exercise more; Watch your diet; and Cut down on soft drinks.”- PM Lee Hsien Loong
Traditional Food les/images/food-journal-1 0.jpg
Smart Food LoggingHealthy 365Powered by
RoadmapProblemApproachResearchCases
Food Image Recognition Visual RecognitionLaksa?Machine Learning
Food Image Recognition Could be very challenging Singapore Tea or Teh Teh, tea with milk and sugar Teh-C, tea with evaporated milk Teh-C-kosong, tea with evaporated milk and no sugar Teh-O, tea with sugar only Teh-O-kosong, plain tea without milk or sugar Teh tarik, the Malay tea Teh-halia, tea with ginger water Teh-bing, tea with ice, aka Teh-ice Teh-siu-dai, tea with less sugar Teh-gah-dai, tea with extra sweetened milk /madnesskopiteh.jpg
Food Name HierarchyFood ItemVisual FoodFood CategoryTeh OTeh O siu daiTeh OTeh O kosong Green teaGreen tea ( no sugar)Green teaTea, no milk Iced lemon teaIced lemon tea
RoadmapProblemApproachResearchCases
Visual Recognition Classical Computer Vision ksaMee siamMee Goreng Deep Learning ApproachTrainableFeatureClassifierDeep NN Deep LearningDeepNN.Extraction(ML)LaksaMee siamMee Goreng
Deep Convolutional Neural Networks (CNN) Convolutional Neural Networks (CNN)Low-levelMid-levelHigh-levelLeNet [LeCun et a. 1998]Photos taken form neural-network.html
Deep CNN for Visual Recognition Revolution of Depth From AlexNet (8-layers) in 2012[ Krizhevsky et al. 2012 ]
Why Deep Learning?AccuracyDeep LearningMachineLearningHPC(GPU)DataTraditional LearningProductSmall dataData SizeBig data13
GPU for High Performance Computing Deep Learning on GPU Clusters DGX-1: NVIDIA Pascal -powered Tesla P100 Performance equal to 250 conventional servers.NVIDIA DGX-1AI SupercomputerSingapore 1st DGX-1 Deep Learning Supercomputer (with P100 GPUs)
SG-FOOD
SGFOOD Data StatisticsSGFood724 DatasetTrainingValidationTest# total images361,6767,24036,200 5001050# Image per class#Food Items:1038#Visual Food:724#Food Category: 158Histogram of #visual foods (724 visual food classes)
FoodAI: Open API Serviceshttp://www.foodai.org
FoodAI System EENGINEAPIServiceMODELTRAININGEXTERNAL DATACOLLECTIONWebDATABASEANNOTATIONSYSTEM
RoadmapProblemApproachResearchCases
Research Challenges How to train a good CNN model? How to deal with new food? How the labeled data size affects the accuracy?
Model Training A Family of CNN models for visual recognitionImageNet 1000 classes, 1.2 million images for training“An Analysis of Deep Neural Network Models for Practical Applications”Alfredo Canziani, Adam Paszke, Eugenio Culurciello Published 2016 in ArXiv
Experimental Setups CNN Models GoogleNet ResNet: 18, 50, 101, 152 Settings Toolbox: Caffe & TensorFelowFinetuned from ImageNet pretrained modelsBatch Size: From 16 to 128Optimizer: SGD with momentum/RMS Prop/AdamLearning rate: Fixed/multi-step/exponential decayDropout/Batch Normalizations
Benchmark of FoodAI724 visual food classes, 361,676 images for training, 500 images per classModels (SGFOOD)Top-1 Accuracy (%)Top-5 Accuracy 93.3ResNet-10173.291.9ResNet-15274.792.71000 object classes, 1.2 million images for training, 1200 images per classModels (IMAGENET)Top-1 Accuracy (%)Top-5 Accuracy .694.3
Food Saliency Map
How to handle NEW food? Too many possible food items in the market Only consider popular food for majority of usersNew foodDiscoveryNew foodimageannotationModelRe-trainingwith new food New food has few images available at the beginningUpdate FoodAIInferenceEngine
What if only 10x less amount of labeled data isavailable to train an CNN model?
Training on 10x less labeled dataResNet-50(10%) augmentationResNet-50 (100%)83.682.760.058.076.193.3ResNet-50 (10%)TOP-1 ACCURACYTOP-5 ACCURACY
RoadmapProblemApproachResearchCases
Case Studies: Food logging photos from usersWebMobile AppPowered by
Case Studies: Easy Cases
Case Studies: Hard CasesLarge inter-class similarity (e.g., drinks)Kopi OAmericano
Case Studies: Hard CasesLarge inter-class similarity (e.g., drinks)Instant CoffeeTeh C / TehPlain PorridgeSoya milk
Case Studies: Hard CasesLarge inter-class similarity (e.g., drinks)Instant CoffeeTeh OTeh / Teh C
Case Studies: Hard CasesLarge intra-class diversity(e.g., Economy rice)
Case Studies: Hard CasesIncomplete Food
Case Studies: Hard CasesNon Food
Case Studies: Hard CasesPoorly taken photos (illumination,rotation, occlusion, etc)
Case Studies: Hard CasesMultiple food items
Case Studies: Hard CasesUnknown food / food not in our list
How to build a more sustainable solution?Better LearningGo beyond supervised CNNCrowdsourcingCombined with human wisdom
Thank w.larc.smu.edu.sg
Food Image Recognition Could be very challenging Singapore Tea or Teh Teh, tea with milk and sugar Teh-C, tea with evaporated milk Teh-C-kosong, tea with evaporated milk and no sugar Teh-O, tea with sugar only Teh-O-kosong, plain tea without milk or sugar Teh tarik, the Malay tea Teh-halia, tea with ginger water Teh-bing, tea with ice, aka Teh-ice
L2: x 0, image of L3: y 2, image of L4: y 3, image of L5: y x, image of L6: y x 1 b. image of L1: x 0, image of L2: x 0, image of L3: (0, 2), image of L4: (0, 3), image of L5: x 0, image of L6: x 0 c. image of L1– 6: y x 4. a. Q1 3, 1R b. ( 10, 0) c. (8, 6) 5. a x y b] a 21 50 ba x b a 2 1 b 4 2 O 46 2 4 2 2 4 y x A 1X2 A 1X1 A 1X 3 X1 X2 X3
18-794 Pattern Recognition Theory! Speech recognition! Optical character recognition (OCR)! Fingerprint recognition! Face recognition! Automatic target recognition! Biomedical image analysis Objective: To provide the background and techniques needed for pattern classification For advanced UG and starting graduate students Example Applications:
The Ultimate Guide to Employee Rewards & Recognition v1.0. Table of contents INTRODUCTION 3 EVOLVING ROLE OF HR 5 REWARDS VS RECOGNITION 8 BENEFITS OF REWARDS AND RECOGNITION 10 TECHNOLOGY IN REWARDS AND RECOGNITION 15 A CULTURE OF PEER TO PEER RECOGNITION 26 SELECTING A REWARDS AND RECOGNITION PLATFORM 30
Types of food environments Community food environment Geographic food access, which refers to the location and accessibility of food outlets Consumer food environment Food availability, food affordability, food quality, and other aspects influencing food choices in retail outlets Organizational food environment Access to food in settings
Deep Convolutional Neural Networks have been shown to be very useful for visual recognition tasks. AlexNet [17] won the ImageNet Large Scale Visual Recognition Chal-lenge [22] in 2012, spurring a lot of interest in using deep learning to solve challenging problems. Since then, deep learning
Actual Image Actual Image Actual Image Actual Image Actual Image Actual Image Actual Image Actual Image Actual Image 1. The Imperial – Mumbai 2. World Trade Center – Mumbai 3. Palace of the Sultan of Oman – Oman 4. Fairmont Bab Al Bahr – Abu Dhabi 5. Barakhamba Underground Metro Station – New Delhi 6. Cybercity – Gurugram 7.
2.3 Deep Reinforcement Learning: Deep Q-Network 7 that the output computed is consistent with the training labels in the training set for a given image. [1] 2.3 Deep Reinforcement Learning: Deep Q-Network Deep Reinforcement Learning are implementations of Reinforcement Learning methods that use Deep Neural Networks to calculate the optimal policy.
Visual recognition is one of the hottest topics in the fields of computer vision and machine learning. In re-cent years, many deep learning models have been built to set the new state-of-the-art results in image classification, face recognition, and many other visual recognition tasks
