An Empirical Comparison Of CNNs And Other Methods For .
An empirical comparison of CNNs and other methodsfor classification of protein subcellular localization withmicroscopy imagesMengli Xiao, Wei PanDivision of BiostatisticsUniversity of MinnesotaJune 5, 2018Mengli Xiao, Wei Pan (UMN)CNNs and microscopy imagesJune 5, 20181 / 20
Outline12345BackgroundProtein subcellular localizationData descriptionImplementationA Convolutional Neural Network: DeepYeastCNN model structureResultResidual Neural NetworkResNet model structuresResultFeature extraction and transfer learningDefinitionResultSummaryComparison of different methodsDiscussionMengli Xiao, Wei Pan (UMN)CNNs and microscopy imagesJune 5, 20182 / 20
Background1BackgroundProtein subcellular localizationData descriptionImplementation2A Convolutional Neural Network: DeepYeastCNN model structureResult3Residual Neural NetworkResNet model structuresResult4Feature extraction and transfer learningDefinitionResult5SummaryComparison of different methodsDiscussionMengli Xiao, Wei Pan (UMN)CNNs and microscopy imagesJune 5, 20182 / 20
BackgroundProtein subcellular localizationProtein subcellular localizationA protein’s subcellular localization functionSpatial temporal variation of a protein’s location results from geneticand environmental perturbationsHigh-throughput imaging: image classificationproteins are fluorescently labeled to track their locations within a cell;Why automating?Manual: labor-intensive and error-prone.Large, but not so large, amounts of data: deep learning? others?Mengli Xiao, Wei Pan (UMN)CNNs and microscopy imagesJune 5, 20183 / 20
BackgroundData descriptionData descriptionData: Pärnamaa and Parts (2017);Each image contains a single cell.Figure 1: DeepYeast dataset overview with 4 images per category (Pärnamaa andParts, 2017)Mengli Xiao, Wei Pan (UMN)CNNs and microscopy imagesJune 5, 20184 / 20
BackgroundData descriptionData descriptionTable 1: Data: sample sizesSubcellular categoriesCell ar talMengli Xiao, Wei Pan 211471312297151793612500CNNs and microscopy 58712500June 5, 20185 / 20
BackgroundImplementationImplementationKeras in Tensorflow - CNNsPython sklearn - RF, XGBoostR - CATCH (Pan et al., 2018a,b)Mengli Xiao, Wei Pan (UMN)CNNs and microscopy imagesJune 5, 20186 / 20
A Convolutional Neural Network: DeepYeast1BackgroundProtein subcellular localizationData descriptionImplementation2A Convolutional Neural Network: DeepYeastCNN model structureResult3Residual Neural NetworkResNet model structuresResult4Feature extraction and transfer learningDefinitionResult5SummaryComparison of different methodsDiscussionMengli Xiao, Wei Pan (UMN)CNNs and microscopy imagesJune 5, 20186 / 20
A Convolutional Neural Network: DeepYeastCNN model structureDeepYeast (11-layer CNN) Model structureA 11-layered CNN; similar to the first few layers of VGG-19;VGG-19 was trained on the (ImageNet) ILSVRC dataset consisting ofnatural objects, aircraft, etc.Several papers: similar CNNs for the current problem.Mengli Xiao, Wei Pan (UMN)CNNs and microscopy imagesJune 5, 20187 / 20
A Convolutional Neural Network: DeepYeastCNN model structureVGG-19 and DeepYeast (11-layered) model structureTable 2: VGG-19 and DeepYeast model strcutureVGG-1919 weight layersInput: 224 224 3conv3-64conv3-64maxpool 2 2conv3-128conv3-128maxpool 2 2conv3-256conv3-256conv3-256conv3-256maxpool 2 2conv3-512conv3-512conv3-512conv3-512maxpool 2 2conv3-512conv3-512conv3-512conv3-512maxpool 2 2Fully-connected layer-4096Dropout-0.5Fully-connected layer-4096Dropout-0.5Fully-connected layer-1000 (softmax)# of parameters is 144,000,000Mengli Xiao, Wei Pan (UMN)DeepYeast11 weight layersInput: 64 64 3conv3-64conv3-64maxpool 2 2conv3-128conv3-128maxpool 2 2conv3-256conv3-256conv3-256conv3-256maxpool 2 2Fully-connected layer-512Dropout-0.5Fully-connected layer-512Dropout-0.5Fully-connected layer-12 (softmax)(BN added except for the last FC layer)# of parameters is 3,128,908CNNs and microscopy imagesJune 5, 20188 / 20
A Convolutional Neural Network: DeepYeastCNN model structureCNN Model structureInput image, 64 64 3Conv layer with 64 3 3 filters, padding 1, stride 1Output dimension: 64 64 64Conv layer with 64 3 3 filters, padding 1, stride 1Output dimension: 64 64 642 2 MaxpoolingOutput dimension: 32 32 64Conv layer with 128 3 3 filters, padding 1, stride 1Output dimension: 32 32 128Conv layer with 128 3 3 filters, padding 1, stride 1Output dimension: 32 32 1282 2 MaxpoolingOutput dimension: 16 16 128Conv layer with 256 3 3 filters, padding 1, stride 1Output dimension: 16 16 256Conv layer with 256 3 3 filters, padding 1, stride 1Output dimension: 16 16 256Conv layer with 256 3 3 filters, padding 1, stride 1Output dimension: 16 16 256Conv layer with 256 3 3 filters, padding 1, stride 1Output dimension: 16 16 2562 2 MaxpoolingOutput dimension: 8 8 256Fully connected with 512 neuronsOutput dimension: 512 1Fully connected with 512 neuronsOutput dimension: 512 1Fully connected with 12 neuronsOutput dimension: 12 1Mengli Xiao, Wei Pan (UMN)CNNs and microscopy imagesJune 5, 20189 / 20
A Convolutional Neural Network: DeepYeastResultResultBase CNN (DeepYeast) performance on the subcellular localization datasetThe test accuracy is 0.8512 (vs 0.8671 in the paper).Mengli Xiao, Wei Pan (UMN)CNNs and microscopy imagesJune 5, 201810 / 20
Residual Neural Network1BackgroundProtein subcellular localizationData descriptionImplementation2A Convolutional Neural Network: DeepYeastCNN model structureResult3Residual Neural NetworkResNet model structuresResult4Feature extraction and transfer learningDefinitionResult5SummaryComparison of different methodsDiscussionMengli Xiao, Wei Pan (UMN)CNNs and microscopy imagesJune 5, 201810 / 20
Residual Neural NetworkResNet model structuresMotivationFigure 2: Poorer performance withdeeper layers (He et al., 2016)Mengli Xiao, Wei Pan (UMN)Figure 3: Convolution layer learns theresidual features left by the identity skipconnection/shortcutCNNs and microscopy imagesJune 5, 201811 / 20
Residual Neural NetworkResNet model structuresResidual neural networksConvolutional layer blocks; no fully-connected layers;Identity shortcut was shown to perform well.We tried 18- and 50-layered ResNet, Res18 and Res50.Mengli Xiao, Wei Pan (UMN)CNNs and microscopy imagesJune 5, 201812 / 20
Residual Neural NetworkResNet model structuresResNetwork model structuresTable 3: Model structureBlock nameDeepYeastRes18 (ours)ResNet 50Res50 (ours)W40-4conv1 xhhhihhi h1 1, 64"conv2 x3 3, 16 4 63 3, 16 4conv3 xi3 3, 64 2i"i7 7, 64#3 3, 64 23 3, 643 3, 128 2hi7 7, 64"#3 3, 64 23 3, 643 3, 256 4"#3 3, 64 23 3, 64conv4 x"#3 3, 64 23 3, 64conv5 xmax pooling[512-d fc] 212-d fc (softmax)Mengli Xiao, Wei Pan (UMN)i7 7, 64 1 1, 64 3 3, 64 3 3 3, 64 3 1 1, 2561 1, 128 1 1, 64 3 3, 64 2 1 1, 5121 1, 256 1 1, 64 1 1, 64 3 3, 256 6 3 3, 64 2 1 1, 10241 1, 512 "#"#"#"#"#3 3, 32 4 63 3, 32 43 3, 64 4 63 3, 64 43 3, 16 2 63 3, 16 23 3, 32 2 63 3, 32 23 3, 64 2 63 3, 64 2 1 1, 64 3 3, 512 31 1, 20481 1, 64i3 3, 16#1 1, 64 3 3, 128 4W40-2h3 3, 16 3 3, 64 31 1, 64average pooling, 12-d fc (softmax)CNNs and microscopy imagesJune 5, 201813 / 20
Residual Neural NetworkResultTest accuracy of residual neural networksRes18 and Res50 performed better than their plain versions;Plain 50 worse than plain 18; but Res50 better than Res18;More benefits with 50 layers.Table 4: Comparison of accuracy among different methodsNetworkplain 18Res 18plain 50Res 50Mengli Xiao, Wei Pan (UMN)Training time1.75 h1.75 h13 h12.75 hTest accuracy0.84320.87080.81900.8856CNNs and microscopy imagesJune 5, 201814 / 20
Feature extraction and transfer learning1BackgroundProtein subcellular localizationData descriptionImplementation2A Convolutional Neural Network: DeepYeastCNN model structureResult3Residual Neural NetworkResNet model structuresResult4Feature extraction and transfer learningDefinitionResult5SummaryComparison of different methodsDiscussionMengli Xiao, Wei Pan (UMN)CNNs and microscopy imagesJune 5, 201814 / 20
Feature extraction and transfer learningDefinitionDefinition and advantagesThe last one or few layers of a pretrained neural network are replacedby new classifiers.More accurate and faster (vs. without feature extraction).Mengli Xiao, Wei Pan (UMN)CNNs and microscopy imagesJune 5, 201815 / 20
Feature extraction and transfer learningResultUse trained network as a feature extractorReplace the last fully-connected layer of the base CNN model(DeepYeast) with a random forest and an XGBoost:Compared to using vectorizing-image input, the test accuracy isimproved (0.85 vs 0.6)Faster: with 512 extracted features vs the original 12288( 64 64 3) featuresReplace all the fully-connected layers of the VGG-19 model with newfully-connected layers, a random forest and an XGBoost respectively,Very quick compared to training a neural network from scratch; decenttest accuracy (0.73, 0.66 , 0.72)Mengli Xiao, Wei Pan (UMN)CNNs and microscopy imagesJune 5, 201816 / 20
Summary1BackgroundProtein subcellular localizationData descriptionImplementation2A Convolutional Neural Network: DeepYeastCNN model structureResult3Residual Neural NetworkResNet model structuresResult4Feature extraction and transfer learningDefinitionResult5SummaryComparison of different methodsDiscussionMengli Xiao, Wei Pan (UMN)CNNs and microscopy imagesJune 5, 201816 / 20
SummaryComparison of different methodsSummaryTable 5: Comparison of accuracy between different methodsNetworkDeepYeast (11-layer CNN)CNN (18-layer)Res18ResNet 18 (He et al., 2016)CNN (50-layer)ResNet 50 (He et al.2016)Wide ResNet (widening factor 2)Random Forest (v-images; 1000 trees)XGBoost (v-images 800 trees)Feature extraction by DeepYeast:Random ForestXGBoostFeature extraction by VGG-19 (transfer learning):FC layersRandom Forest (800 trees)XGBoost (1000 trees)Mengli Xiao, Wei Pan (UMN)CNNs and microscopy imagesTraining time6h1.75 h1.75 h2.45 h13 h12.75 h46 h1.68 h10 hTest accuracy0.8510.8430.871 – 0.8910.8530.8190.8860.8530.6000.610 min1h0.8500.8403 min12 min14 h0.7300.6600.722June 5, 201817 / 20
SummaryDiscussionDiscussionCNNs performed best, though not a thorough evaluation!Why? images, a large dataset, .Other statistical methods: RF, Boosting, SVM good, but not tailoredto images.BUT, some new stat methods: could not even run .Mengli Xiao, Wei Pan (UMN)CNNs and microscopy imagesJune 5, 201818 / 20
SummaryDiscussionAcknowledgmentFunded by NIH, NSF.Thank you!Mengli Xiao, Wei Pan (UMN)CNNs and microscopy imagesJune 5, 201819 / 20
SummaryDiscussionReferencesHe, K., Zhang, X., Ren, S., and Sun, J. (2016). Deep residual learning forimage recognition. In Proceedings of the IEEE conference on computervision and pattern recognition, pages 770–778.Pan, Y., Mai, Q., and Zhang, X. (2018a). catch: Covariate-AdjustedTensor Classification in High-Dimensions. R package version 1.0.Pan, Y., Mai, Q., and Zhang, X. (2018b). Covariate-adjusted tensorclassification in high-dimensions. arXiv preprint arXiv:1805.04421.Pärnamaa, T. and Parts, L. (2017). Accurate classification of proteinsubcellular localization from high-throughput microscopy images usingdeep learning. G3: Genes, Genomes, Genetics, 7(5):1385–1392.Mengli Xiao, Wei Pan (UMN)CNNs and microscopy imagesJune 5, 201820 / 20
Summary Comparison of different methods Summary Table 5:Comparison of accuracy between different methods Network Training time Test accuracy DeepYeast (11-layer CNN) 6 h 0.851 CNN (18-layer) 1.75 h 0.843 Res18 1.75 h 0.871 – 0.891 ResNet 18 (He et al., 2016) 2.45 h 0.853 CNN (50-layer) 13 h 0.819 ResNet 50 (He et al.2016) 12.75 h 0.886
Empirical & Molecular Formulas I. Empirical Vs. Molecular Formulas Molecular Formula actual/exact # of atoms in a compound (ex: Glucose C 6 H 12 O 6) Empirical Formula lowest whole # ratio of atoms in a compound (ex: Glucose CH 2 O) II. Determining Empirical Formulas You can determine the empirical formula
the empirical formula of a compound. Classic chemistry: finding the empirical formula The simplest type of formula – called the empirical formula – shows just the ratio of different atoms. For example, while the molecular formula for glucose is C 6 H 12 O 6, its empirical formula
Convolutional Neural Networks While in fully-connected deep neural networks, the activa-tion of each hidden unit is computed by multiplying the entire in-put by the correspondent weights for each neuron in that layer, in CNNs, the activation of each hidden unit is computed for a small input area. CNNs are composed of convolutional layers which
2 Convolutional neural networks CNNs are hierarchical neural networks whose convolutional layers alternate with subsampling layers, reminiscent of sim-ple and complex cells in the primary visual cortex [Wiesel and Hubel, 1959]. CNNs vary in how convolutional and sub-sampling layers are realized and how the nets are trained. 2.1 Image processing .
pled into a regular 2D grid akin to an image. Geometry images are advantageous for learning shapes using CNNs over free boundary or disc parameterizations as every pixel encodes desired shape information. This reduces memory and learn-ing complexity in CNNs as the need to abstra
for image understanding and search tasks [14, 17]. These pre-trained CNNs are also gaining considerable research interest as a feature extractor for a task of interest, e.g. object or scene recognition [6, 29]. It is argued that CNNs, through their layered c
CNNs for Interference Mitigation and Denoising in Automotive Radar Using Real-World Data Johanna Rock 1, Mate Toth; 2, Paul Meissner , Franz Pernkopf 1Graz University of Technology, Austri
1-4 AutoCAD 2016 Tutorial: 2D Fundamentals Note that AutoCAD automatically assigns generic names, Drawing X, as new drawings are created. In our example, AutoCAD opened the graphics window using the default system units and assigned the drawing name Drawing1. 2. If necessary, click on the down-arrow in the Quick Access bar and select Show Menu Bar to display the AutoCAD Menu Bar. The Menu Bar .
