Learning dynamics of linear denoising autoencoders
Date
2018
Journal Title
Journal ISSN
Volume Title
Publisher
PMLR
Abstract
Denoising autoencoders (DAEs) have proven useful
for unsupervised representation learning, but
a thorough theoretical understanding is still lacking
of how the input noise influences learning.
Here we develop theory for how noise influences
learning in DAEs. By focusing on linear DAEs,
we are able to derive analytic expressions that
exactly describe their learning dynamics. We verify
our theoretical predictions with simulations
as well as experiments on MNIST and CIFAR-10.
The theory illustrates how, when tuned correctly,
noise allows DAEs to ignore low variance directions
in the inputs while learning to reconstruct
them. Furthermore, in a comparison of the learning
dynamics of DAEs to standard regularised
autoencoders, we show that noise has a similar
regularisation effect to weight decay, but with
faster training dynamics. We also show that our
theoretical predictions approximate learning dynamics
on real-world data and qualitatively match
observed dynamics in nonlinear DAEs.
Description
CITATION: Pretorius, A., Kroon, S. & Kamper, H. 2018. Learning dynamics of linear denoising autoencoders. In Proceedings of the 35 th International Conference on Machine
Learning, PMLR 80:4141-4150, 10-15 July 2018, Stockholm, Sweden.
The original publication is available at http://proceedings.mlr.press/
The original publication is available at http://proceedings.mlr.press/
Keywords
Computer science -- Mathematics -- Congresses, Denoising Autoencoders, Computer Science -- Research
Citation
Pretorius, A., Kroon, S. & Kamper, H. 2018. Learning dynamics of linear denoising autoencoders. In Proceedings of the 35 th International Conference on Machine
Learning, PMLR 80:4141-4150, 10-15 July 2018, Stockholm, Sweden