Stacked Denoising Autoencoders: Learning Useful Representations In A .

1y ago
9 Views
2 Downloads
1.33 MB
38 Pages
Last View : 1m ago
Last Download : 2m ago
Upload by : Nadine Tse
Transcription

Journal of Machine Learning Research 11 (2010) 3371-3408Submitted 5/10; Published 12/10Stacked Denoising Autoencoders: Learning Useful Representations ina Deep Network with a Local Denoising CriterionPascal VincentPASCAL . VINCENT @ UMONTREAL . CADépartement d’informatique et de recherche opérationnelleUniversité de Montréal2920, chemin de la TourMontréal, Québec, H3T 1J8, CanadaHugo LarochelleLAROCHEH @ CS . TORONTO . EDUDepartment of Computer ScienceUniversity of Toronto10 King’s College RoadToronto, Ontario, M5S 3G4, CanadaIsabelle LajoieYoshua BengioPierre-Antoine ManzagolISABELLE . LAJOIE .1@ UMONTREAL . CAYOSHUA . BENGIO @ UMONTREAL . CAPIERRE - ANTOINE . MANZAGOL @ UMONTREAL . CADépartement d’informatique et de recherche opérationnelleUniversité de Montréal2920, chemin de la TourMontréal, Québec, H3T 1J8, CanadaEditor: Léon BottouAbstractWe explore an original strategy for building deep networks, based on stacking layers of denoisingautoencoders which are trained locally to denoise corrupted versions of their inputs. The resultingalgorithm is a straightforward variation on the stacking of ordinary autoencoders. It is howevershown on a benchmark of classification problems to yield significantly lower classification error,thus bridging the performance gap with deep belief networks (DBN), and in several cases surpassing it. Higher level representations learnt in this purely unsupervised fashion also help boost theperformance of subsequent SVM classifiers. Qualitative experiments show that, contrary to ordinary autoencoders, denoising autoencoders are able to learn Gabor-like edge detectors from naturalimage patches and larger stroke detectors from digit images. This work clearly establishes the valueof using a denoising criterion as a tractable unsupervised objective to guide the learning of usefulhigher level representations.Keywords: deep learning, unsupervised feature learning, deep belief networks, autoencoders,denoising1. IntroductionIt has been a long held belief in the field of neural network research that the composition of severallevels of nonlinearity would be key to efficiently model complex relationships between variablesand to achieve better generalization performance on difficult recognition tasks (McClelland et al.,1986; Hinton, 1989; Utgoff and Stracuzzi, 2002). This viewpoint is motivated in part by knowledgec 2010 Pascal Vincent, Hugo Larochelle, Isabelle Lajoie, Yoshua Bengio and Pierre-Antoine Manzagol.

V INCENT, L AROCHELLE , L AJOIE , B ENGIO AND M ANZAGOLof the layered architecture of regions of the human brain such as the visual cortex, and in part by abody of theoretical arguments in its favor (Håstad, 1986; Håstad and Goldmann, 1991; Bengio andLeCun, 2007; Bengio, 2009). Yet, looking back at the history of multi-layer neural networks, theirproblematic non-convex optimization has for a long time prevented reaping the expected benefits(Bengio et al., 2007; Bengio, 2009) of going beyond one or two hidden layers.1 Consequentlymuch of machine learning research has seen progress in shallow architectures allowing for convexoptimization, while the difficult problem of learning in deep networks was left dormant.The recent revival of interest in such deep architectures is due to the discovery of novel approaches (Hinton et al., 2006; Hinton and Salakhutdinov, 2006; Bengio et al., 2007; Ranzato et al.,2007; Lee et al., 2008) that proved successful at learning their parameters. Several alternative techniques and refinements have been suggested since the seminal work on deep belief networks (DBN)by Hinton et al. (2006) and Hinton and Salakhutdinov (2006). All appear however to build on thesame principle that we may summarize as follows: Training a deep network to directly optimize only the supervised objective of interest (for example the log probability of correct classification) by gradient descent, starting from randominitialized parameters, does not work very well. What works much better is to initially use a local unsupervised criterion to (pre)train eachlayer in turn, with the goal of learning to produce a useful higher-level representation from thelower-level representation output by the previous layer. From this starting point on, gradientdescent on the supervised objective leads to much better solutions in terms of generalizationperformance.Deep layered networks trained in this fashion have been shown empirically to avoid gettingstuck in the kind of poor solutions one typically reaches with only random initializations. SeeErhan et al. (2010) for an in depth empirical study and discussion regarding possible explanationsfor the phenomenon.In addition to the supervised criterion relevant to the task, what appears to be key is using anadditional unsupervised criterion to guide the learning at each layer. In this sense, these techniquesbear much in common with the semi-supervised learning approach, except that they are useful evenin the scenario where all examples are labeled, exploiting the input part of the data to regularize,thus approaching better minima of generalization error (Erhan et al., 2010).There is yet no clear understanding of what constitutes “good” representations for initializingdeep architectures or what explicit unsupervised criteria may best guide their learning. We knowbut a few algorithms that work well for this purpose, beginning with restricted Boltzmann machines(RBMs) (Hinton et al., 2006; Hinton and Salakhutdinov, 2006; Lee et al., 2008), and autoencoders(Bengio et al., 2007; Ranzato et al., 2007), but also semi-supervised embedding (Weston et al.,2008) and kernel PCA (Cho and Saul, 2010).It is worth mentioning here that RBMs (Hinton, 2002; Smolensky, 1986) and basic classicalautoencoders are very similar in their functional form, although their interpretation and the procedures used for training them are quite different. More specifically, the deterministic functionthat maps from input to mean hidden representation, detailed below in Section 2.2, is the same forboth models. One important difference is that deterministic autoencoders consider that real valued1. There is a notable exception to this in the more specialized convolutional network architecture of LeCun et al. (1989).3372

S TACKED D ENOISING AUTOENCODERSmean as their hidden representation whereas stochastic RBMs sample a binary hidden representation from that mean. However, after their initial pretraining, the way layers of RBMs are typicallyused in practice when stacked in a deep neural network is by propagating these real-valued means(Hinton et al., 2006; Hinton and Salakhutdinov, 2006). This is more in line with the deterministicautoencoder interpretation. Note also that reconstruction error of an autoencoder can be seen as anapproximation of the log-likelihood gradient in an RBM, in a way that is similar to the approximation made by using the Contrastive Divergence updates for RBMs (Bengio and Delalleau, 2009).It is thus not surprising that initializing a deep network by stacking autoencoders yields almost asgood a classification performance as when stacking RBMs (Bengio et al., 2007; Larochelle et al.,2009a). But why is it only almost as good? An initial motivation of the research presented here wasto find a way to bridge that performance gap.With the autoencoder paradigm in mind, we began an inquiry into the question of what canshape a good, useful representation. We were looking for unsupervised learning principles likely tolead to the learning of feature detectors that detect important structure in the input patterns.Section 2 walks the reader along the lines of our reasoning. Starting from the simple intuitivenotion of preserving information, we present a generalized formulation of the classical autoencoder,before highlighting its limitations. This leads us in Section 3 to motivate an alternative denoisingcriterion, and derive the denoising autoencoder model, for which we also give a possible intuitivegeometric interpretation. A closer look at the considered noise types will then allow us to derive afurther extension of the base model. Section 4 discusses related preexisting works and approaches.Section 5 presents experiments that qualitatively study the feature detectors learnt by a single-layerdenoising autoencoder under various conditions. Section 6 describes experiments with multi-layerarchitectures obtained by stacking denoising autoencoders and compares their classification performance with other state-of-the-art models. Section 7 is an attempt at turning stacked (denoising)autoencoders into practical generative models, to allow for a qualitative comparison of generatedsamples with DBNs. Section 8 summarizes our findings and concludes our work.1.1 NotationWe will be using the following notation throughout the article: Random variables are written in upper case, for example, X. If X is a random vector, then its jth component will be noted X j . Ordinary vectors are written in lowercase bold. For example, a realization of a random vectorX may be written x. Vectors are considered column vectors. Matrices are written in uppercase bold (e.g., W). I denotes the identity matrix. The transpose of a vector x or a matrix W is written xT or WT (not x′ or W′ which may beused to refer to an entirely different vector or matrix). We use lower case p and q to denote both probability density functions or probability massfunctions according to context. Let X and Y two random variables with marginal probability p(X) and p(Y ). Their jointprobability is written p(X,Y ) and the conditional p(X Y ). We may use the following common shorthands when unambiguous: p(x) for p(X x);p(X y) for p(X Y y) (denoting a conditional distribution) and p(x y) for p(X x Y y).3373

V INCENT, L AROCHELLE , L AJOIE , B ENGIO AND M ANZAGOL f , g, h, will be used for ordinary functions. Expectation (discrete case, p is probability mass): E p(X) [ f (X)] x p(X x) f (x). Expectation (continuous case, p is probability density): E p(X) [ f (X)] Rp(x) f (x)dx. Entropy or differential entropy: IH(X) IH(p) E p(X) [ log p(X)]. Conditional entropy: IH(X Y ) E p(X,Y ) [ log p(X Y )].p(X)]. Kullback-Leibler divergence: IDKL (pkq) E p(X) [log q(X) Cross-entropy: IH(pkq) E p(X) [ log q(X)] IH(p) IDKL (pkq). Mutual information: I(X;Y ) IH(X) IH(X Y ). Sigmoid: s(x) 11 e xand s(x) (s(x1 ), . . . , s(xd ))T . Bernoulli distribution with mean µ: B (µ). By extension for vector variables: X B (µ) means i, Xi B (µi ).1.2 General setupWe consider the typical supervised learning setup with a training set of n (input, target) pairs Dn {(x(1) ,t (1) ) . . . , (x(n) ,t (n) )}, that we suppose to be an i.i.d. sample from an unknown distributionq(X, T ) with corresponding marginals q(X) and q(T ). We denote q0 (X, T ) and q0 (X) the empiricaldistributions defined by the samples in Dn . X is a d-dimensional random vector (typically in IRd orin [0, 1]d ).In this work we are primarily concerned with finding a new, higher-level representation Y of X.′′Y is a d ′ -dimensional random vector (typically in IRd or in [0, 1]d ). If d ′ d we will talk of anover-complete representation, whereas it will be termed an under-complete representation if d ′ d.Y may be linked to X by a deterministic or stochastic mapping q(Y X; θ) parameterized by a vectorof parameters θ.2. What Makes a Good Representation? From Mutual Information to AutoencodersFrom the outset we can give an operational definition of a “good” representation as one that willeventually be useful for addressing tasks of interest, in the sense that it will help the system quicklyachieve higher performance on those tasks than if it hadn’t first learned to form the representation. Based on the objective measure typically used to assess algorithm performance, this might bephrased as “A good representation is one that will yield a better performing classifier”. Final classification performance will indeed typically be used to objectively compare algorithms. However, ifa lesson is to be learnt from the recent breakthroughs in deep network training techniques, it is thatthe error signal from a single narrowly defined classification task should not be the only nor primarycriterion used to guide the learning of representations. First because it has been shown experimentally that beginning by optimizing an unsupervised criterion, oblivious of the specific classificationproblem, can actually greatly help in eventually achieving superior performance for that classification problem. Second it can be argued that the capacity of humans to quickly become proficient innew tasks builds on much of what they have learnt prior to being faced with that task.In this section, we begin with the simple notion of retaining information and progress to formallyintroduce the traditional autoencoder paradigm from this more general vantage point.3374

S TACKED D ENOISING AUTOENCODERS2.1 Retaining Information about the InputWe are interested in learning a (possibly stochastic) mapping from input X to a novel representation Y . To make this more precise, let us restrict ourselves to parameterized mappings q(Y X) q(Y X; θ) with parameters θ that we want to learn.One natural criterion that we may expect any good representation to meet, at least to somedegree, is to retain a significant amount of information about the input. It can be expressed ininformation-theoretic terms as maximizing the mutual information I(X;Y ) between an input randomvariable X and its higher level representation Y . This is the infomax principle put forward by Linsker(1989).Mutual information can be decomposed into an entropy and a conditional entropy term in twodifferent ways. A first possible decomposition is I(X;Y ) IH(Y ) IH(Y X) which lead Bell andSejnowski (1995) to their infomax approach to Independent Component Analysis. Here we willstart from another decomposition: I(X;Y ) IH(X) IH(X Y ). Since observed input X comes froman unknown distribution q(X) on which θ has no influence, this makes IH(X) an unknown constant.Thus the infomax principle reduces to:arg max I(X;Y ) arg max IH(X Y )θθ arg max Eq(X,Y ) [log q(X Y )].θNow for any distribution p(X Y ) we will haveEq(X,Y ) [log p(X Y )] Eq(X,Y ) [log q(X Y )], {z}(1) IH(X Y )as can easily be shown starting from the property that for any two distributions p and q we haveIDKL (qkp) 0, and in particular IDKL (q(X Y y)kp(X Y y)) 0.Let us consider a parametric distribution p(X Y ; θ′ ), parameterized by θ′ , and the followingoptimization:max′ Eq(X,Y ;θ) [log p(X Y ; θ′ )].θ,θFrom Equation 1, we see that this corresponds to maximizing a lower bound on IH(X Y ) and thuson the mutual information. We would end up maximizing the exact mutual information provided θ′ s.t. q(X Y ) p(X Y ; θ′ ).If, as is done in infomax ICA, we further restrict ourselves to a deterministic mapping from X toY , that is, representation Y is to be computed by a parameterized function Y fθ (X) or equivalentlyq(Y X; θ) δ(Y fθ (X)) (where δ denotes Dirac-delta), then this optimization can be written:max′ Eq(X) [log p(X Y fθ (X); θ′ )].θ,θThis again corresponds to maximizing a lower bound on the mutual information.Since q(X) is unknown, but we have samples from it, the empirical average over the trainingsamples can be used instead as an unbiased estimate (i.e., replacing Eq(X) by Eq0 (X) ):max′ Eq0 (X) [log p(X Y fθ (X); θ′ )].θ,θ3375(2)

V INCENT, L AROCHELLE , L AJOIE , B ENGIO AND M ANZAGOLWe will see in the next section that this equation corresponds to the reconstruction error criterionused to train autoencoders.2.2 Traditional Autoencoders (AE)Here we briefly specify the traditional autoencoder (AE)2 framework and its terminology, based onfθ and p(X Y ; θ′ ) introduced above.Encoder: The deterministic mapping fθ that transforms an input vector x into hidden representation y is called the encoder. Its typical form is an affine mapping followed by a nonlinearity:fθ (x) s(Wx b).Its parameter set is θ {W, b}, where W is a d ′ d weight matrix and b is an offset vector ofdimensionality d ′ .Decoder: The resulting hidden representation y is then mapped back to a reconstructed ddimensional vector z in input space, z gθ′ (y). This mapping gθ′ is called the decoder. Its typicalform is again an affine mapping optionally followed by a squashing non-linearity, that is, eithergθ′ (y) W′ y b′ orgθ′ (y) s(W′ y b′ ),(3)with appropriately sized parameters θ′ {W′ , b′ }.In general z is not to be interpreted as an exact reconstruction of x, but rather in probabilisticterms as the parameters (typically the mean) of a distribution p(X Z z) that may generate x withhigh probability. We have thus completed the specification of p(X Y ; θ′ ) from the previous sectionas p(X Y y) p(X Z gθ′ (y)). This yields an associated reconstruction error to be optimized:L(x, z) log p(x z).(4)Common choices for p(x z) and associated loss function L(x, z) include: For real-valued x, that is, x IRd : X z N (z, σ2 I), that is, X j z N (z j , σ2 ).This yields L(x, z) L2 (x, z) C(σ2 )kx zk2 where C(σ2 ) denotes a constant that dependsonly on σ2 and that can be ignored for the optimization. This is the squared error objectivefound in most traditional autoencoders. In this setting, due to the Gaussian interpretation, itis more natural not to use a squashing nonlinearity in the decoder. For binary x, that is, x {0, 1}d : X z B (z), that is, X j z B (z j ).In this case, the decoder needs to produce a z [0, 1]d . So a squashing nonlinearity such as asigmoid s will typically be used in the decoder. This yields L(x, z) LIH (x, z) j [x j log z j (1 x j ) log(1 z j )] IH(B (x)kB (z)) which is termed the cross-entropy lossbecause it is seen as the cross-entropy between two independent multivariate Bernoullis, thefirst with mean x and the other with mean z. This loss can also be used when x is not strictlybinary but rather x [0, 1]d .2. Note: AutoEncoders (AE) are also often called AutoAssociators (AA) in the literature. The shorter autoencoder termwas preferred in this work, as we believe encoding better conveys the idea of producing a novel useful representation.Similarly, what we call Stacked Auto Encoders (SAE) has also been called Stacked AutoAssociators (SAA).3376

S TACKED D ENOISING AUTOENCODERSNote that in the general autoencoder framework, we may use other forms of parameterized functions for the encoder or decoder, and other suitable choices of the loss function (corresponding to adifferent p(X z)). In particular, we investigated the usefulness of a more complex encoding functionin Larochelle, Erhan, and Vincent (2009b). For the experiments in the present work however, wewill restrict ourselves to the two usual forms detailed above, that is, an affine sigmoid encoderand either affine decoder with squared error loss or affine sigmoid decoder with cross-entropyloss. A further constraint that can optionally be imposed, and that further parallels the workings ofRBMs, is having tied weights between W and W ′ , in effect defining W ′ as W ′ W T .Autoencoder training consists in minimizing the reconstruction error, that is, carrying the following optimization:arg min Eq0 (X) [L(X, Z(X))],θ,θ′where we wrote Z(X) to emphasize the fact that Z is a deterministic function of X, since Z isobtained by composition of deterministic encoding and decoding.Making this explicit and using our definition of loss L from Equation 4 this can be rewritten as:arg max Eq0 (X) [log p(X Z gθ′ ( fθ (X)))],θ,θ′or equivalentlyarg max Eq0 (X) [log p(X Y fθ (X); θ′ )].θ,θ′We see that this last line corresponds to Equation 2, that is, the maximization of a lower bound onthe mutual information between X and Y .It can thus be said that training an autoencoder to minimize reconstruction error amountsto maximizing a lower bound on the mutual information between input X and learnt representation Y . Intuitively, if a representation allows a good reconstruction of its input, it means thatit has retained much of the information that was present in that input.2.3 Merely Retaining Information is Not EnoughThe criterion that representation Y should retain information about input X is not by itself sufficientto yield a useful representation. Indeed mutual information can be trivially maximized by settingY X. Similarly, an ordinary autoencoder where Y is of the same dimensionality as X (or larger)can achieve perfect reconstruction simply by learning an identity mapping.3 Without any otherconstraints, this criterion alone is unlikely to lead to the discovery of a more useful representationthan the input.Thus further constraints need to be applied to attempt to separate useful information (to be retained) from noise (to be discarded). This will naturally translate to non-zero reconstruction error.The traditional approach to autoencoders uses a bottleneck to produce an under-complete representation where d ′ d. The resulting lower-dimensional Y can thus be seen as a lossy compressedrepresentation of X. When using affine encoder and decoder without any nonlinearity and a squarederror loss, the autoencoder essentially performs principal component analysis (PCA) as showed by3. More precisely, it suffices that g f be the identity to obtain zero reconstruction error. For d d ′ if we had a linearencoder and decoder this would be achieved for any invertible matrix W by setting W′ W 1 . Now there is asigmoid nonlinearity in the encoder, but it is possible to stay in the linear part of the sigmoid with small enough W.3377

V INCENT, L AROCHELLE , L AJOIE , B ENGIO AND M ANZAGOLBaldi and Hornik (1989).4 When a nonlinearity such as a sigmoid is used in the encoder, thingsbecome a little more complicated: obtaining the PCA subspace is a likely possibility (Bourlard andKamp, 1988) since it is possible to stay in the linear regime of the sigmoid, but arguably not the onlyone (Japkowicz et al., 2000). Also when using a cross-entropy loss rather than a squared error theoptimization objective is no longer the same as that of PCA and will likely learn different features.The use of “tied weights” can also change the solution: forcing encoder and decoder matrices tobe symmetric and thus have the same scale can make it harder for the encoder to stay in the linearregime of its nonlinearity without paying a high price in reconstruction error.Alternatively it is also conceivable to impose on Y different constraints than that of a lowerdimensionality. In particular the possibility of using over-complete (i.e., higher dimensional thanthe input) but sparse representations has received much attention lately. Interest in sparse representations is inspired in part by evidence that neural activity in the brain seems to be sparse andhas burgeoned following the seminal work of Olshausen and Field (1996) on sparse coding. Othermotivations for sparse representations include the ability to handle effectively variable-size representations (counting only the non-zeros), and the fact that dense compressed representations tendto entangle information (i.e., changing a single aspect of the input yields significant changes in allcomponents of the representation) whereas sparse ones can be expected to be easier to interpret andto use for a subsequent classifier. Various modifications of the traditional autoencoder frameworkhave been proposed in order to learn sparse representations (Ranzato et al., 2007, 2008). Thesewere shown to extract very useful representations, from which it is possible to build top performingdeep neural network classifiers. A sparse over-complete representations can be viewed as an alternative “compressed” representation: it has implicit straightforward compressibility due to the largenumber of zeros rather than an explicit lower dimensionality.3. Using a Denoising CriterionWe have seen that the reconstruction criterion alone is unable to guarantee the extraction of usefulfeatures as it can lead to the obvious solution “simply copy the input” or similarly uninteresting onesthat trivially maximizes mutual information. One strategy to avoid this phenomenon is to constrainthe representation: the traditional bottleneck and the more recent interest on sparse representationsboth follow this strategy.Here we propose and explore a very different strategy. Rather than constrain the representation,we change the reconstruction criterion for a both more challenging and more interesting objective: cleaning partially corrupted input, or in short denoising. In doing so we modify the implicitdefinition of a good representation into the following: “a good representation is one that can beobtained robustly from a corrupted input and that will be useful for recovering the correspondingclean input”. Two underlying ideas are implicit in this approach: First it is expected that a higher level representation should be rather stable and robust undercorruptions of the input. Second, it is expected that performing the denoising task well requires extracting features thatcapture useful structure in the input distribution.4. More specifically it will find the same subspace as PCA, but the specific projection directions found will in generalnot correspond to the actual principal directions and need not be orthonormal.3378

S TACKED D ENOISING AUTOENCODERSWe emphasize here that our goal is not the task of denoising per se. Rather denoising is advocated and investigated as a training criterion for learning to extract useful features that willconstitute better higher level representation. The usefulness of a learnt representation can then beassessed objectively by measuring the accuracy of a classifier that uses it as input.3.1 The Denoising Autoencoder AlgorithmThis approach leads to a very simple variant of the basic autoencoder described above. A denoisingautoencoder (DAE) is trained to reconstruct a clean “repaired” input from a corrupted version ofit (the specific types of corruptions we consider will be discussed below). This is done by firstcorrupting the initial input x into x̃ by means of a stochastic mapping x̃ qD (x̃ x).Corrupted input x̃ is then mapped, as with the basic autoencoder, to a hidden representationy fθ (x̃) s(Wx̃ b) from which we reconstruct a z gθ′ (y). See Figure 1 for a schematicrepresentation of the procedure. Parameters θ and θ′ are trained to minimize the average reconstruction error over a training set, that is, to have z as close as possible to the uncorrupted input x.The key difference is that z is now a deterministic function of x̃ rather than x. As previously, theconsidered reconstruction error is either the cross-entropy loss LIH (x, z) IH(B (x)kB (z)), with anaffine sigmoid decoder, or the squared error loss L2 (x, z) kx zk2 , with an affine decoder. Parameters are initialized at random and then optimized by stochastic gradient descent. Note that eachtime a training example x is presented, a different corrupted version x̃ of it is generated accordingto qD (x̃ x).Note that denoising autoencoders are still minimizing the same reconstruction loss between aclean X and its reconstruction from Y . So this still amounts to maximizing a lower bound on themutual information between clean input X and representation Y . The difference is that Y is nowobtained by applying deterministic mapping fθ to a corrupted input. It thus forces the learning of afar more clever mapping than the identity: one that extracts features useful for denoising.ygθ′fθLH (x, z)qDx̃xzFigure 1: The denoising autoencoder architecture. An example x is stochastically corrupted (viaqD ) to x̃. The autoencoder then maps it to y (via encoder fθ ) and attempts to reconstructx via decoder gθ′ , producing reconstruction z. Reconstruction error is measured by lossLH (x, z).3379

V INCENT, L AROCHELLE , L AJOIE , B ENGIO AND M ANZAGOL3.2 Geometric InterpretationThe process of denoising, that is, mapping a corrupted example back to an uncorrupted one, canbe given an intuitive geometric interpretation under the so-called manifold assumption (Chapelleet al., 2006), which states that natural high dimensional data concentrates close to a non-linearlow-dimensional manifold. This is illustrated in Figure 2. During denoising training, we learn ae that maps a corrupted Xe back to its uncorrupted X, for example, in thestochastic operator p(X X)case of binary data,eX Xe B (gθ′ ( fθ (X))).Corrupted examples are much more likely to be outside and farther from the manifold than thee learns a map that tends to go from lower probuncorrupted ones. Thus stochastic operator p(X X)ability points Xe to nearby high probability points X, on or near the manifold. Note that when Xe ise should learn to make bigger steps, to reach the manifold. Sucfarther from the manifold, p(X X)cessful denoising implies that the operator maps even far away points to a small region close to themanifold.The denoising autoencoder can thus be seen as a way to define and learn a manifold. In particular, if we constrain the dimension of Y to be smaller than the dimension of X, then the intermediaterepresentation Y f (X) may be interpreted as a coordinate system for points on the manifold. Moregenerally, one can think of Y f (X) as a representation of X which is well suited to capture themain variations in the data, that is, those along the manifold.3.3 Types of Corruption ConsideredThe above principle and technique can potentially be used with any type of corruption process. Alsothe corruption process is an obvious place where prior knowledge, if available, could be easily incorporated. But in the present study we set to investigate a technique that is generally applicable. Inx̃g!′( f!(x̃))qD (x̃ x)xx̃xFigure 2: Manifold learning perspective. Suppose training data ( ) concentrate near a lowdimens

of using a denoising criterion as a tractable unsupervised objective to guide the learning of useful higher level representations. Keywords: deep learning, unsupervised feature learning, deep belief networks, autoencoders, denoising 1. Introduction It has been a long held belief in the field of neural network research that the composition of .

Related Documents:

one for image denoising. In the course of the project, we also aimed to use wavelet denoising as a means of compression and were successfully able to implement a compression technique based on a unified denoising and compression principle. 1.2 The concept of denoising A more precise explanation of the wavelet denoising procedure can be given .

In the recent years there has been a fair amount of research on wavelet based image denoising, because wavelet provides an appropriate basis for image denoising. But this single tree wavelet based image denoising has poor directionality, loss of phase information and shift sensitivity [11] as

4 Image Denoising In image processing, wavelets are used for instance for edges detection, watermarking, texture detection, compression, denoising, and coding of interesting features for subsequent classifica-tion [2]. Image denoising by thresholding of the DWT coefficients is discussed in the following subsections. 4.1 Principles

age denoising based on minimization of total variation (TV) has gained certain popularity in the literature (e.g., [4]), and the TV approach is initially suggested for denoising 2-D images (e.g. [12]). MATLAB pro-grams for 3-D image denoising using anisotropic dif-fusion have also been developed (e.g., [6]). Other

2.2 Image Denoising. A typical application area for image reconstruction is image denoising, where the task is to remove noise to restore the original image. Here, we focus on image denoising tech-niques based on deep neural networks; for more detailed information about image denoising research, please refer to the following survey papers [9,11].

Image denoising and inpainting are common image restoration problems that are both useful by themselves and important preprocessing steps of many other applications. Image denoising problems arise when an image is corrupted by additive white Gaussian

on stacked (often denoising) autoencoders or restricted Boltzmann machines [10-13]. Another intriguing line of work consists of the ladder network [14], which has achieved spectacular results on a semi-supervised variant of the MNIST dataset. More recently, a model based on the VAE has achieved even better semi-supervised results on MNIST [15].

automotive sector to the West Midlands’ economy, the commission identified the need for a clear automotive skills plan that describes the current and future skills needs of the West Midlands automotive sector; the strengths and weaknesses of the region’s further and higher education system in addressing these needs; and a clear road-map for developing new co-designed skills solutions. The .