MI-NEE: Mutual Information Neural Entropic Estimation
COM2 Level 4
Executive Classroom, COM2-04-02
Abstract:
We point out a limitation of the mutual information neural estimation (MINE) where the network fails to learn at the initial training phase, leading to slow convergence in the number of training iterations. To solve this problem, we propose a faster method called the mutual information neural entropic estimation (MI-NEE). Our solution first generalizes MINE to estimate the entropy using a custom reference distribution. The entropy estimate can then be used to estimate the mutual information. We argue that the seemingly redundant intermediate step of entropy estimation allows one to improve the convergence by an appropriate reference distribution. In particular, we show that MI-NEE reduces to MINE in the special case when the reference distribution is the product of marginal distributions, but faster convergence is possible by choosing the uniform distribution as the reference distribution instead. Compared to the product of marginals, the uniform distribution introduces more samples in low-density regions and fewer samples in high-density regions, which appear to lead to an overall larger gradient for faster convergence.
For more details: https://github.com/ccha23/MI-NEE [2]
Biodata:
Chung Chan received the B.Sc., M.Eng. and Ph.D. from the EECS Department at MIT in 2004, 2005 and 2010 respectively. He was a Research Assistant Professor at the Institute of Network Coding, the Chinese University of Hong Kong from 2013 to 2017. He is currently an Assistant Professor at the Department of Computer Science, City University of Hong Kong. His research interest is to develop general information measures and flow models from network information theory that are applicable to practical problems. His research topics include the development of network link models using matroids, the derivation of theoretical limits and optimal strategies for the problems of multiterminal source coding, data exchange, and secret generation. His most significant work is the extension of Shannon's mutual information to the multivariate case, and the discovery of its connections to various problems in information theory and machine learning.
Links:
------
[1] https://aka.ms/ghei36
[2] https://github.com/ccha23/MI-NEE