Reinforcement Learning Coach v0.9

Since the release of Coach a couple of months ago, we have been working hard to push it into new frontiers that will improve its usability for real world applications. In this release, we are introducing several new features that will move Coach forward in this direction.

Imitation Learning

First, we added several convenient tools for imitation, along with the basic behavioral cloning imitation algorithm.  Imitation learning can often be very efficient for achieving very good behavior fast, and is an important addition to Coach’s toolbox. Coach now allows users to interact with the simulation environments and collect data from human examples. Additionally, it supports loading a previously collected dataset of experience and training an agent to imitate the behavior in the given dataset. As a starting point, we added a few presets and datasets for several environments in Doom and Gym.

 

A Doom agent and a Montezuma Revenge agent trained using Behavioral Cloning

 

CARLA

The second addition is a built-in support for the recently released CARLA[1] simulator. CARLA is an open-source urban driving simulator developed as a collaboration between Intel Labs and the Computer Vision Center (CVC) that includes realistic urban environments. CARLA enables the training of autonomous driving agents and is now integrated with Coach. We also added several presets for training both reinforcement learning and imitation learning agents for simple driving behaviors.

 

A CARLA agent trained using reinforcement learning

 

QR-DQN

Finally, to keep up with the state-of-the-art in the field of reinforcement learning, we recently added the Quantile Regression DQN[2] algorithm, which was shown to achieve superior results over the Categorical DQN[3] algorithm on the Atari benchmark.

 

To conclude, we believe that the CARLA simulator, along with tools for imitation learning, open a new world of possibilities for users that are interested in applying reinforcement learning to real world applications. Go ahead and try it out by following the instructions on our GitHub repository.

 

References

[1] CARLA: An Open Urban Driving Simulator Alexey Dosovitskiy, German Ros, Felipe Codevilla, Antonio Lopez and Vladlen Koltun. CoRR, abs/1711.03938, 2017.

[2] Distributional Reinforcement Learning with Quantile Regression Will Dabney, Mark Rowland, Marc G. Bellemare and Rémi Munos. CoRR, abs/1710.10044, 2017.

[3] A Distributional Perspective on Reinforcement Learning Marc G. Bellemare, Will Dabney and Rémi Munos. CoRR, abs/1707.06887, 2017.