Initialisation of noise-regularised neural networks
| dc.contributor.advisor | Kroon, R. S. (Steve) | en_ZA |
| dc.contributor.author | Van Biljon, Elan | en_ZA |
| dc.contributor.other | Stellenbosch University. Faculty of Science. Dept. of Mathematical Sciences. Division Computer Science. | en_ZA |
| dc.date.accessioned | 2021-09-09T14:02:23Z | |
| dc.date.accessioned | 2021-12-22T14:14:36Z | |
| dc.date.available | 2021-09-09T14:02:23Z | |
| dc.date.available | 2021-12-22T14:14:36Z | |
| dc.date.issued | 2021-12 | |
| dc.description | Thesis (MSc)--Stellenbosch University, 2021. | en_ZA |
| dc.description.abstract | ENGLISH ABSTRACT: Recently, proper initialisation and stochastic regularisation techniques have greatly improved the performance and ease of training of neural networks. Some research has gone into how the magnitude of the initial weights impact optimisation, while others have focused on how initialisation affects signal propagation. In terms of noise regularisation, dropout has allowed networks to train relatively quickly and reduced overfitting. Much research has gone towards understanding why dropout improves the generalisation of networks. Two major theories are (i) that it prevents neurons from becoming too dependent on the output of other neurons and (ii) that dropout leads a network to optimise a smoother loss landscape. Despite this, our theoretical understanding of the interaction between regularisation and initialisation is sparse. Thus, the aim of this work was to broaden our knowledge of how initialisation and stochastic regularisation interact and what impact this has on network training and performance. Because rectifier activation functions are widely used, we extended new network signal propagation theory to rectifier networks that may use stochastic regularisation. Our theory predicted a critical initialisation that allows for stable pre-activation variance signal propagation. However, our theory also indicated that stochastic regularisation reduces the depth to which correlation information can propagate in ReLU networks. We validated this theory and showed that it accurately predicts a boundary across which networks do not train effectively. We then extended the investigation by conducting a large-scale randomised control trial to search for initialisations in a region that conserves input signal around the critical initialisation in the hopes of finding initialisations that provide advantages to training or generalisation. We compare the critical initialisation to 10 other initialisation schemes in a trial that consisted of over 12000 networks. We found that initialisations much larger than the critical initialisation provide extremely poor performance, while network initialisations close to the critical initialisation provide similar performance. No initialisations clearly outperformed the critical initialisation. Thus, we recommend it as a safe default for practitioners. | en_ZA |
| dc.description.abstract | AFRIKAANSE OPSOMMING: Geen opsomming beskikbaar. | af_ZA |
| dc.description.sponsorship | The financial assistance of the Council for Scientific and Industrial Research (CSIR) towards this research is hereby acknowledged. Opinions expressed and conclusions arrived at, are those of the author and are not necessarily to be attributed to the CSIR. | en_ZA |
| dc.description.version | Masters | en_ZA |
| dc.format.extent | xiii, 146 pages : illustrations | en_ZA |
| dc.identifier.uri | http://hdl.handle.net/10019.1/123661 | |
| dc.language.iso | en_ZA | en_ZA |
| dc.publisher | Stellenbosch : Stellenbosch University | en_ZA |
| dc.rights.holder | Stellenbosch University | en_ZA |
| dc.subject | Deep learning (Machine learning) | en_ZA |
| dc.subject | Neural networks (Computer science) -- Noise | en_ZA |
| dc.subject | Stochastic regularisation | en_ZA |
| dc.subject | Critical initialisation | en_ZA |
| dc.subject | Signal propagation | en_ZA |
| dc.subject | Neural network initialisation | en_ZA |
| dc.subject | Noise (Computer science) | en_ZA |
| dc.subject | UCTD | |
| dc.title | Initialisation of noise-regularised neural networks | en_ZA |
| dc.type | Thesis | en_ZA |