Browsing by Author "Taylor, Rebecca"

Now showing 1 - 1 of 1
  • Thumbnail Image
    Item
    Approximate message passing algorithms for latent Dirichlet allocation
    (2023-03) Taylor, Rebecca; Du Preez, Johan; Stellenbosch University. Faculty of Engineering. Dept. of Electrical and Electronic Engineering.
    ENGLISH ABSTRACT: Latent Dirichlet allocation (LDA) is a hierarchical Bayesian model that is most well known for its ability to extract latent semantic topics from text corpora. Since exact inference of the posterior distribution is intractable, a variety of approximate inference techniques can be employed for LDA. The traditional variational Bayesian inference (VB) approach is sufficient for most text topic modelling problems. VB, however, does not perform topic extraction well where the corpora of documents are small, where document sizes are limited, or where topic overlap is large. In these cases, collapsed Gibbs sampling, a slower, computationally more expensive inference method, is typically preferred. In this dissertation, we present two variational message passing algorithms for inference in LDA. Our first algorithm, approximate LBU (designated by the acronym ALBU), is derived by applying loopy belief update (LBU) (also known as the Lauritzen-Spiegelhalter algorithm), where possible, and using ideas from hierarchical sampling to derive messages where conjugacy does not apply. The second algorithm, which we designate variational message passing plus (VMP+), is based on variational message passing (VMP), the message passing variant of VB. In VMP+, instead of following the VMP algorithm exactly, we relax the mean-field assumption between parent and child nodes that are in the same cluster of the cluster graph representation of LDA. This results in an algorithm that is similar to VB, but performs better at topic extraction. To evaluate ALBU and VMP+, we use four text corpora (20 Newsgroups, 6 Newsgroups, Covid tweets, and Bible verses), all of which contain short texts, to compare the performance of our two algorithms with VB and collapsed Gibbs sampling using topic coherence scores. Because VB typically performs well on large data sets, we apply these two new variants specifically to smaller data sets where topic extraction is expected to be difficult. In addition to the real-life text documents described above, we perform topic extraction on a variety of simulated corpora which are obtained by using different hyperparameter settings, to enable comparisons among the estimated results and the true distributions. Here, Kullback-Leibler divergence (KLD) is used to evaluate and compare inference algorithm performance. Based on results obtained for both the real-life and synthetic data, we show that both algorithms outperform VB, and that the performance of ALBU is similar to that of collapsed Gibbs sampling only when the sampler is given an extended period of time to converge. ALBU appears to be an efficient approach to inference in LDA in certain situations where topic extraction is expected to be difficult. VMP+ is an alternative inference algorithm for LDA that is simpler than VB and also typically extracts more coherent topics than VB. On the basis of our experiments, we summarise the differences between the variational algorithms, and propose reasons for these differences. We conclude by providing recommendations regarding the methods that we anticipate would be most suitable for inference on particular corpus types.