An Expectation Maximization Algorithm for Continuous Markov Decision Processes with Arbitrary Reward

We derive a new expectation maximization algorithm for policy optimization in linear Gaussian Markov decision processes, where the reward function is parameterized in terms of a flexible mixture of Gaussians. This ap- proach exploits both analytical tractability and numerical optimization. Consequently, on the one hand, it is more flexible and gen- eral than closed-form solutions, such as the widely used linear quadratic Gaussian (LQG) controllers. On the other hand, it is more ac- curate and faster than optimization methods that rely on approximation and simulation. Partial analytical solutions (though costly) eliminate the need for simulation and, hence, avoid approximation error. The experiments will show that for the same cost of computa- tion, policy optimization methods that rely on analytical tractability have higher value than the ones that rely on simulation.