Finite state Markov chain (discrete-time) and Markov decision processes (MDP) (controlled Markov chain) and their applications. Discrete-time stochastic control. Finite horizon stochastic control problem, principle of optimality (Bellman equation)

Dynamic Programming and its solutions, Closed form solution for the Linear Quadratic Gaussian (LQG) control problem.

Infinite horizon stochastic control problems (discounted and average cost with finite state and action space), Bellman optimality equation, existence of stationary control policy.

Solution methodologies – value iteration and policy iteration and related algorithms.

Curse of dimensionality in solving Dynamic Programming algorithms, Approximate Dynamic Programming algorithms – approximation in policy space and value space, contraction properties and error bounds, simulation-based implementation.

Curse of dimensionality in solving Dynamic Programming algorithms, Approximate Dynamic Programming algorithms – approximation in policy space and value space, contraction properties and error bounds, simulation-based implementation.

Intro to reinforcement learning in the setting of MDP. Temporal difference methods (TD(0), TD()), convergence properties. On-policy TD control (SARSA), Off-policy TD control such as Qlearning and its convergence properties, Applications

Intro to reinforcement learning in the setting of MDP. Temporal difference methods (TD(0), TD()), convergence properties. On-policy TD control (SARSA), Off-policy TD control such as Qlearning and its convergence properties, Applications

Advanced reinforcement learning. Value function approximation with Linear methods and function approximation. Deep reinforcement learning.

policy gradient methods, actor-critic based reinforcement learning and their applications to continuous control (such as LQG) problems