ITEM VIEW

Learning dynamics of linear denoising autoencoders

dc.contributor.authorPretorius, Arnuen_ZA
dc.contributor.authorKroon, Steveen_ZA
dc.contributor.authorKamper, Hermanen_ZA
dc.date.accessioned2019-03-12T08:08:26Z
dc.date.available2019-03-12T08:08:26Z
dc.date.issued2018
dc.identifier.citationPretorius, A., Kroon, S. & Kamper, H. 2018. Learning dynamics of linear denoising autoencoders. In Proceedings of the 35 th International Conference on Machine Learning, PMLR 80:4141-4150, 10-15 July 2018, Stockholm, Swedenen_ZA
dc.identifier.issn2640-3498 (online)
dc.identifier.urihttp://hdl.handle.net/10019.1/105547
dc.descriptionCITATION: Pretorius, A., Kroon, S. & Kamper, H. 2018. Learning dynamics of linear denoising autoencoders. In Proceedings of the 35 th International Conference on Machine Learning, PMLR 80:4141-4150, 10-15 July 2018, Stockholm, Sweden.en_ZA
dc.descriptionThe original publication is available at http://proceedings.mlr.press/en_ZA
dc.description.abstractDenoising autoencoders (DAEs) have proven useful for unsupervised representation learning, but a thorough theoretical understanding is still lacking of how the input noise influences learning. Here we develop theory for how noise influences learning in DAEs. By focusing on linear DAEs, we are able to derive analytic expressions that exactly describe their learning dynamics. We verify our theoretical predictions with simulations as well as experiments on MNIST and CIFAR-10. The theory illustrates how, when tuned correctly, noise allows DAEs to ignore low variance directions in the inputs while learning to reconstruct them. Furthermore, in a comparison of the learning dynamics of DAEs to standard regularised autoencoders, we show that noise has a similar regularisation effect to weight decay, but with faster training dynamics. We also show that our theoretical predictions approximate learning dynamics on real-world data and qualitatively match observed dynamics in nonlinear DAEs.en_ZA
dc.format.extent13 pages : illustrations (some colour)en_ZA
dc.language.isoen_ZAen_ZA
dc.publisherPMLRen_ZA
dc.subjectComputer science -- Mathematics -- Congressesen_ZA
dc.subjectDenoising Autoencodersen_ZA
dc.subjectComputer Science -- Researchen_ZA
dc.titleLearning dynamics of linear denoising autoencodersen_ZA
dc.typeConference Paperen_ZA
dc.description.versionPublisher's versionen_ZA
dc.rights.holderAuthors retain copyrighten_ZA


Files in this item

Thumbnail
Thumbnail

This item appears in the following Collection(s)

ITEM VIEW