Tuning the Learning Rate for Stochastic Variational Inference

• 作 者: Xi-Ming Li Ji-Hong Ouyang?
• 机 构: College of Computer Science and Technology, Jilin University, Changchun 130000, China Key Laboratory
• 出 处: 计算机科学技术学报（英文版）
• 发表日期: 2016年2期
• 关键词: stochastic variational inference online learning adaptive learning rate topic mo

Stochastic variational inference (SVI) can le topic models with very big corpora. It optimizes the variational objective by using the stochastic natural gradient algorithm with a decreasing leing rate. This rate is crucial for SVI;however, it is often tuned by hand in real applications. To address this, we develop a novel algorithm, which tunes the leing rate of each iteration adaptively. The proposed algorithm uses the Kullback-Leibler (KL) divergence to measure the similarity between the variational distribution with noisy update and that with batch update, and then optimizes the leing rates by minimizing the KL divergence. We apply our algorithm to two representative topic models: latent Dirichlet allocation and hierarchical Dirichlet process. Experimental results indicate that our algorithm performs better and converges faster than commonly used leing rates.