Topic models are a class of probabilistic models for text analysis, widely used in many research areas that use textual data as their research material, e.g. literature studies, history of ideas, social and political science, and requirements engineering. The most widely used topic model, Latent Dirichlet Analysis (LDA) [Blei et al., 2003] is a hierarchical Bayesian model that is typically implemented using MCMC or variational inference methods.
In recent years, a number of methods have been devised to formulate hierarchical Bayesian models using objective functions to be minimized using gradient-based methods [Kingma and Welling, 2014]. This essentially brings Bayesian models such as LDA into the neural network world [Srivastava and Sutton, 2017; Miao et al., 2017]. This allows the parameters of the models to be estimated using efficient hardware and software commonly used to train neural models, but perhaps more interestingly it allows us to formulate more interesting models that are less simplistic than LDA or that can work with new types of data, not exclusively text.
We can see several directions for a Master's thesis project around neural network-based topic models and we are open to discussion about the research question the project would investigate. For instance, the project could be a careful investigation of the pros and cons of using neural inference compared to traditional inference methods, investigating aspects such as training speed, stability, topic quality and interpretability. Or you could define a new topic model for a new type of data, such as images or graph-structured data, by combining the neural topic model with a neural network for your preferred type of data.
Your first task will be to reimplement a previously published model, and this will then be your starting point for further extensions and evaluations.
Keywords: Machine learning, statistical models, natural language processing
Contact: Richard Johansson, Department of Computer Science and Engineering, richard.johansson@gu.se
Blei, D., Ng., A, and Jordan, M. (2003). Latent Dirichlet Allocation. Journal of Machine Learning Research, 3:993–1022.
Kingma, D. and Welling, M. (2014). Auto-encoding Variational Bayes. International Conference on Learning Representations.
Miao, Y., Grefenstette, E., and Blunsom, P. (2017). Discovering Discrete Latent Topics with Neural Variational Inference. International Conference on Machine Learning.
Srivastava, A. and Sutton, C. (2017). Autoencoding Variational Inference for Topic Models. International Conference on Learning Representations.