Mutation Testing in JavaScript Using Grunt Mutation Testing
Disclaimer: This content reflects my personal opinions, not those of any organizations I am or have been affiliated with. Code samples are provided for illustration purposes only, use with caution and test thoroughly before deployment.
Update: Since the development on grunt-mutation-testing has moved to Stryker, I’ve rewritten this post using the new Stryker framework. Please checkout the updated post here.
- Slide show version: link
- Code example: github link
This November I attended the EuroStar Software Testing Conference, and was introduced to a interesting idea called mutation testing. Ask yourself: “How do I ensure my (automated) unit test suite is good enough?”. Did you miss any important test? Is your test always passing so it didn’t catch anything? Is there anything un-testable in your code such that your test suite can never catch it?
Introduction to Mutation Testing
Mutation testing tries to “test your tests” by deliberately inject faults (called “mutants”)into your program under test. If we re-run the tests on the crippled program, our test suite should catch it (i.e. some test should fail.) If you missed some test, the error might slip through and the test will pass. Borrowing terms from Genetics, if the test fails, we say the mutant is “killed” by the test suite; on the opposite, if the tests passes, we say the mutant survived.
The goal of mutation testing is to kill all mutants by enhancing the test suite. Although 100% kill is usually impossible for even small programs, any progress on increasing the number can still benefit your test a lot.
The concept of mutation testing has been around for quite a while, but it didn’t get very popular because of the following reasons: first, it is slow. The number of possible mutations are just too much, re-compiling (e.g. C++) and re-run the test will take too long. Various methods has been proposed to lower the number of tests we need to run without sacrifice the chance of finding problems. The second is the “equivalent mutation” problem, which we’ll discuss in more detail in the examples.
Mutation Testing from JavaScript
There are many existing mutation testing framework. But most of them are for languages like Java, C++ or C#. Since my job is mainly about JavaScript (both in browser and Node.js), I wanted to run mutation testing in JavaScript.
I have found a few mutation testing frameworks for JavaScript, but they are either non-open-source or very academic. The only one I can get started without effort is the grunt-mutatoin-testing on npm. Therefore we will show you some example using it.
This framework supports mutants like changing the math operators, remote elements from arrays or changing comparison operators. You can find a full list of mutations with examples here.
Setting up the environment
You can follow along by cloning this repository – JS-mutation-testing-example, you’ll also need node and npm installed. (I recommended you to use nvm).
There are a few Node packages you need to install,
sudo npm install -g mocha
sudo npm install -g grunt-cli
npm install . #This installs the dependencies in pakcages.json
The npm install . command will install the following packages from packages.json
//===== packages.json =====
{
...
"devDependencies": {
"grunt-mutation-testing": "~1.3.0",
"grunt": "~0.4.5",
"karma": "~0.13.15"
}
}
The grunt-mutation-testing is self-explanatory. As the name implies, it runs in the Grunt task runner. Grunt requires you to install a global grunt-cli package and a local grunt package. In order for the mutation testing framework to run the test, we need a test runner. grunt-mutation-testing’s default is karma, but I’ll use mocha instead. (You still need to install karma otherwise the grunt-mutation-testing will complain.)
A simple test suite
I created a simple program in src/calculator.js, which has two functions:
// ===== calculator.js =====
function substractPositive(num1, num2){
if (num1 > 0){
return num1 - num2;
}
else {
return 0
}
}
function add(num1, num2){
if (num1 == num2){
return num1 + num2;
}
else if (num1 > num2){
return num1 + num2;
}
else {
return num1 + num2;
}
}
module.exports.substractPositive = substractPositive
module.exports.add = add;
The first is substractPositive, it substract num2 from num1 if num1 is a positive number. If num1 is not positive, it will return 0 instead. It doesn’t make much sense, but it’s for demostration purpose.
The second is a simple add function that adds two numbers. It has a useless if...else statement, which is also used to demostrate the power of mutation testing.
The two functions are tested using test/test_calculator.js:
var assert = require('assert');
var cal = require('../src/calculator.js')
describe('Calculator', function(){
it('substractPositive', function(){
assert.equal('2', cal.substractPositive(1, -1));
});
it('add', function(){
assert.equal('2', cal.add(1, 1));
});
})
This is a test file running using mocha, The first verifies substractPositive(1, -1) returns 2. The second tests add(1,1) produces 2. If you run mocha in your commandline, you’ll see both the test passes.
So this test suite looks pretty good, it exerices both function and verifies its output, but is it good enough? Let’s verify this by some mutation testing.
Running the mutation testing
To run the mutation testing, we need to setup a grunt task called mutationTest by creating a Gruntfile.js
//===== Gruntfile.js =====
module.exports = function(grunt) {
grunt.initConfig({
mutationTest: {
options: {
testFramework: 'mocha'
},
target: {
options:{
code: ['src/*.js'],
specs: 'test/test_*.js',
mutate: 'src/*.js'
}
}
}
});
grunt.loadNpmTasks('grunt-mutation-testing');
grunt.registerTask('default', ['mutationTest']);
};
You can see wee choose mocha as the testFramework, and we tell the grunt-mutation-testing framework to run the test files specified in specs on the source files code. The code files may include third-party libraries, which we may not want to mutate, so we need to specify the exact files we want to mutate using mutate field. The three fields can either be a single pattern string or an array of strings.
We register the mutationTest as our default task, so when we run grunt in the commandline, it will run mutationTest directly.
#Finding Missing Tests
Now if we run grunt, you’ll see the following test result:
% grunt
Running "mutationTest:target" (mutationTest) task
(17:34:23.955) INFO [mutation-testing]: Mutating file: /tmp/mutation-testing1151016-11404-alnz38/src/calculator.js
(17:34:23.957) INFO [mutation-testing]: Mutating line 1, 1/22 (5%)
(17:34:23.959) INFO [mutation-testing]: Mutating line 10, 2/22 (9%)
(17:34:24.677) INFO [mutation-testing]: Mutating line 23, 3/22 (14%)
(17:34:24.679) INFO [mutation-testing]: Mutating line 24, 4/22 (18%)
(17:34:24.680) INFO [mutation-testing]: Mutating line 2, 5/22 (23%)
(17:34:24.682) INFO [mutation-testing]: Mutating line 2, 6/22 (27%)
/src/calculator.js:2:11 Replaced > with >= -> SURVIVED
(17:34:24.684) INFO [mutation-testing]: Mutating line 2, 7/22 (32%)
(17:34:24.686) INFO [mutation-testing]: Mutating line 2, 8/22 (36%)
(17:34:24.688) INFO [mutation-testing]: Mutating line 3, 9/22 (41%)
(17:34:24.689) INFO [mutation-testing]: Mutating line 3, 10/22 (45%)
(17:34:24.690) INFO [mutation-testing]: Mutating line 6, 11/22 (50%)
/src/calculator.js:6:5 Removed return 0 -> SURVIVED
...
12 of 22 unignored mutations are tested (54%).
Done, without errors.
This line is a killed mutant,
(17:34:23.959) INFO [mutation-testing]: Mutating line 10, 2/22 (9%)
and this is a survived mutant,
/src/calculator.js:2:11 Replaced > with >= -> SURVIVED
It tells us that by replacing the > with >= in line 2, column 11 of our calculator.js file, the test will pass.
If you look the line, the problem is pretty clear
// ===== calculator.js =====
function substractPositive(num1, num2){
if (num1 > 0){ # This line
return num1 - num2;
}
else {
return 0
}
}
We didn’t test the boundary values, if num1 = 0, then the program should go to the else branch and returns 0. By changing the > to >=, the program will go into the num1 >= 0 branch and returns 0 - num2!
This is one of the power of mutation testing, it tells you which condition you are missing. The solution is very simple, we can add a test like this:
it('substractPositive', function(){
assert.equal('0', cal.substractPositive(0, -1));
});
If you run mutation testing again, the problem with the substractPositive function should go away.
Equivalent Mutation and Dead Code
Sometimes the mutation will not change the behavior of the program, so no matter what test you write, you can never make it fail. So example, a mutation may disable caching in your program, the program will run slower but the behavior will be exactly the same, so you’ll have a mutation you can never kill. This kind of mutation is called “equivalent mutation”.
Equivalent mutation will make you overestimate your mutation survival rate. And they take your time to debug, but may not reveal useful information about your test suite. However, some equivalent mutations do reveal issues about your program under test.
Let look at the mutation results again:
/src/calculator.js:11:11 Replaced == with != -> SURVIVED
/src/calculator.js:14:16 Replaced > with >= -> SURVIVED
/src/calculator.js:14:16 Replaced > with <= -> SURVIVED
/src/calculator.js:15:5 Removed return num1 + num2; -> SURVIVED
/src/calculator.js:15:16 Replaced + with - -> SURVIVED
/src/calculator.js:18:5 Removed return num1 + num2; -> SURVIVED
/src/calculator.js:18:16 Replaced + with - -> SURVIVED
If you look at the code, you’ll find that all the branches of the if...else statement returns the same thing. So no matter how you mutate the if...else conditions, the function will always return the correct result.
10 function add(num1, num2){
11 if (num1 == num2){
12 return num1 + num2;
13 }
14 else if (num1 > num2){
15 return num1 + num2;
16 }
17 else {
18 return num1 + num2;
19 }
20 }
But the if...else is useless, you can simplify the function to only three lines:
function add(num1, num2){
return num1 + num2;
}
If you run the mutation test again, you can see all the mutations being killed.
This is one of the side benefit of equivalent mutations, although your test suite is fine, it tells you that your program code has dead code or untestable code.
Next Steps
By now you should have a rough idea about how mutation testing works, and how to actually apply them in your JavaScript project. If you are interested in mutation testing, there are more interesting question you can dive into, for example, how to use code coverage data to reduce the test you need to run? How to avoid equivalent mutations? I hope you’ll find many interesting methods you can apply to your testing work.
