Our code is based on OpenAI Baselines, which is a set of high-quality implementations of reinforcement learning algorithms. Our code is aimed to provide more algorithms which is not included by OpenAI baselines, such as C51 and rainbow, as well as improvements. These algorithms will make it easier for the research community to replicate, refine, and identify new ideas, and will create good baselines to build research on top of. Our DQN implementation and its variants are roughly on par with the scores in published papers. We expect they will be used as a base around which new ideas can be added, and as a tool for comparing a new approach against existing ones.