Kill the mutants of your code

I recently was investigating about best practices for my testing on Android. You can easily make Unit, Instrumentation and UI testing, using tools like JUnit and Espresso. But the question that I always have when writing my test is the next:

How many tests are enough for this feature?

There are many approaches for how correctly measure your tests. The fundamentals of Testing in Android show us the two cycles associated with iterative, test-driven development. They recommend to use the testing pyramid, that shows the three categories of tests that you should include in your app’s test suite.

It is easy to follow this approach, write a lot of unit tests, medium of integration tests and a few ui tests (e2e tests). But I’m still asking the same question.

Code coverage?

Yeah! I found the answer!… I’m just kidding.

Code coverage is very useful for determine if your test are passing for each line of code. In theory, this means that if you change some line of code that your tests are covering, some tests will crash.

A very common issue of this is that when you’re writing your tests, you are finding to cover 100% of your lines of code, but you’re not covering posibles changes and edge cases. So that, you can be confident that your code is 100% covered (Or the percentage you or your team have accepted) but, when someone change some line of code, your tests still pass! This is a problem! you forget to test that case.

Don’t cover your code — Equip it for kill mutants

Yeah! I found the answer!… It’s not a joke.

The problem with code coverage is that you’re alway trying to change your red lines for green lines, but it is very common to forget to cover edge cases. But with mutation testing we prepare our tests for those scenarios that are not very easy to identify.

See the next example:

// This is a simple room query function
@Query("select * from alerts where date between :from and :to")
fun findAlertsByDateRange(from: Date, to: Date)

How do you test this?

// You will maybe do next test
fun `findAlertsByDateRange should return a list of alerts between date range`() {
    // Given
    val fromDate = "2019-12-05"
    val toDate = "2019-12-10"  
    val expectedAlert = Alert(1L, "some alert", "2019-12-07")
    alertDao.insertAlert(expectedAlert)

    // When
    val alerts = findAlertsByDateRange(fromDate, toDate)   

    // Then
    assertThat(alerts[0], `is`(expectedAlert))
}

With this test you can be sure you test your line of code, and that’s correct. But think about edge cases.

The between statement is doing all we need in our function findAlertsByDateRange. Now, imagine someone change your query because he think it is not needed to use between here.

He refactor your code as next:

// This is mutation of your code
@Query("select * from alerts where date > :from and date < :to")
fun findAlertsByDateRange(from: Date, to: Date)

Are your test passing? Yes.

The question now is: What is the requirement here? Are you expecting to find alerts inside or between the range?

A correct approach could be to write your tests using TDD (Test driven development):

• Write a test that crash.

• Write the simplest code to pass your test.

• Refactor your code.

Following this methodology, you can write a test where the date of the alert inserted:

1. Is equal to fromDte.

2. Is equal to toDate.

3. Is bigger than fromDate.

4. Is smaller than toDate.

With that, you’re covering not only your line of code, but you are killing all the posible mutation for your code. Next time someone remove your between sentence, there is some tests that will crash.

With those tests, you can even write your query as next:

@Query("select * from alerts where date >= :from and date <= :to")
fun findAlertsByDateRange(from: Date, to: Date)

// both works correctly
@Query("select * from alerts where date between :from and :to")
fun findAlertsByDateRange(from: Date, to: Date)

And your function is respecting the requirement during your tests guaranteeing no mutants will appear unnoticed.

Conclusion — Mutation testing

Now, every time you write a code:

1. Write a test for the code you would add. Test must crash.

2. Write only what is needed to pass your tests. Test must pass.

3. Write a code that could be there, but should not be. Mutate it.

4. If any test crash, you have a mutant living in your code.

5. Write a tests that crash. Kill the mutant.