Skip to main content

Mutation Testing in Python with Cosmic Ray

45 minutes


Mutation testing is a technique for systematically mutating source code in order to validate test suites. It operates by making small changes to a program’s source code and then running a test suite; if the test suite ever succeeds on mutated code then a flag is raised. The goal is to check that a system’s test suite is sufficiently powerful to detect a large class of functionality-affecting changes, thereby helping ensure that the system functions as expected. While not in widespread use, mutation testing is a fascinating topic with great potential that has valuable lessons for the broader software development community.

In this talk we’ll look at Cosmic Ray, an open-source mutation testing tool for Python. Mutation testing presents some difficult and fascinating challenges - both conceptually and from an implementation point of view - so we’ll look at how Cosmic Ray addresses (or plans to address) these complexities. While some of these details will necessarily be Python-specific, there are lessons in Cosmic Ray for the development of mutation in any language.

Mutation testing is still a rather exotic testing technique, but it can produce genuinely useful and surprising results. To show this, we’ll look at a number of cases where Cosmic Ray has helped developers improve their test suites and tighten up their implementations.

The speaker

Austin Bingham

Austin Bingham

Austin is a founding director of Sixty North, a software consulting, training, and application development company. A native of Texas, in 2008 Austin moved to Stavanger, Norway where he helped develop industry-leading oil reservoir modeling software in C++ and Python. Prior to that he worked at National Instruments developing LabVIEW, at Applied Research Labs (Univ. of Texas at Austin) developing sonar systems for the U.S. Navy, and at a number of telecommunications companies. He is an experienced presenter, teacher, and author, as well as an active member of the open source community. Austin holds a Master of Science in Computer Engineering from the University of Texas at Austin.