Efficient Algorithms with Biases for Optimization under
Uncertainty
Department of Computer Science and Automation,
Indian Institute of Science,
Bangalore, India
COM1 Level 2
Video Conference Room, COM1-02-13
Abstract:
I will first present a formulation for stochastic optimization and then present some recent algorithms of the stochastic approximation type that are based on simultaneous perturbations. These algorithms however involve certain bias terms that depend on a certain sensitivity parameter. I will then discuss our results for the stability and convergence of such schemes involving certain limiting differential inclusions.
I will then discuss a popular algorithm in reinforcement learning that goes under the name Q-learning. Under function approximation, it is well known that Q-learning diverges many times. I will present a reformulation of this algorithm using two-timescale stochastic approximation wherein the policy is updated on a faster timescale and value updates are performed on a slower scale. The reformulated scheme is seen to be convergent. I will finally discuss an application in a problem of sleep-wake scheduling for intruder detection in wireless sensor networks.
Biodata:
Shalabh Bhatnagar received a Ph.D in Electrical Engineering from the Indian Institute of Science, Banmgalore in 1997. During 1997 to 2000, he was a Research Associate at the Institute for Systems Research, University of Maryland, College Park, USA, and during 2000-2001, he was a Postdoc at the Free University, Amsterdam, Netherlands. He joined the IISc in December 2001, where he is now working as a Professor since June 2011.
His research interests are in stochastic approximation algorithms, stochastic optimization, control and reinforcement learning, as well as applications in communication, wireless, and vehicular traffic networks.