Mutation Testing in Python with Cosmic Ray

Abstract

Mutation testing is a technique for systematically mutating source code in order to validate test suites. It operates by making small changes to a program’s source code and then running a test suite; if the test suite ever succeeds on mutated code then a flag is raised. The goal is to check that a system’s test suite is sufficiently powerful to detect a large class of functionality-affecting changes, thereby helping ensure that the system functions as expected. While not in widespread use, mutation testing is a fascinating topic with great potential that has valuable lessons for the broader software development community.

In this talk we’ll look at Cosmic Ray, an open-source mutation testing tool for Python. Mutation testing presents some difficult and fascinating challenges - both conceptually and from an implementation point of view - so we’ll look at how Cosmic Ray addresses (or plans to address) these complexities. While some of these details will necessarily be Python-specific, there are lessons in Cosmic Ray for the development of mutation in any language.

Mutation testing is still a rather exotic testing technique, but it can produce genuinely useful and surprising results. To show this, we’ll look at a number of cases where Cosmic Ray has helped developers improve their test suites and tighten up their implementations.