Gamifying Test Coverage on a Project

The Problem

At software startups in particular, companies tend to shy from bureaucracy and hard and fast policies. Employees generally formed together in the first place to avoid that sort of work environment. In turn, in order to motivate people on a project given a proverbial carrot and stick, the carrot is almost always the tool of choice.

Work is generally fun and rewarding at companies like this, but eventually you get to a point where work is legitimately work, and it’s not all that fun. Superb test coverage is probably one of those things that most people consider “work.” Tests are great and everyone loves them – as long as someone else writes them. They’re some of the most tedious and sometimes boring tasks associated with software development.

So how do you get people to write tests? Do you force them to? Do you require layers of approval to get code merged? NAY! You want to establish a culture of superb test coverage, and my way to encourage such a culture is to gamify the test coverage stats on a repository.

Gamification

“Gamification” is simply the process of adding gamelike elements to something to encourage participation. If a task is no longer a “task” but instead a “game,” it changes the dynamic of the said task.

The “gamelike elements” you can associate with tasks can be anything, but typically all you might need to do is establish incentives, competition, and notoriety.

My idea is simple: For all of the authors in a repo, publicly display the top contributors with respect to number of lines written and test coverage percent. In this way, you’re not embarassing anyone by displaying the top 3rd, and you’re incentivizing people with as much interest as they have in the game itself. Writing more tests would then have the added benefit of advancing in “the game.”

How to Tabulate Results Conceptually

The general concept behind tabulating results is fairly straightforward: take existing modules that allow you to find out test coverage percent by file and combine that with the results of git blame. Then associate test coverage by author instead of the corresponding lines.

The Code

The example code I ran for the repository I’m working in is on Github, but at the time of this writing you’ll need to modify the file and massage some of the values in order to make it correspond to your setup. The assumption is that this is for python files only (no Javascript tests). You might be best served by taking some of the code snippets below and combining them on your own.

Get the Files to Score in the Repo

Before getting started, it’s necessary to be able to list the files that you want in your dataset. In my case, I only wanted Python files and I didn’t want this to include test files or library files. So for starters, we’ll be using a few different shell commands to get some results, and that will require piping from one command to another. So I wrote this simple function that will take in a shell command inclusives of pipes and output the textual result:

def _get_output_from_pipe_command(command_with_pipes):
    piped_commands = [command.strip() for command in command_with_pipes.split("|")]
    previous_process = None
    for command in piped_commands:
        process = Popen(command.split(), stdin=previous_process and previous_process.stdout, stdout=PIPE)
        previous_process = process
    output, err = previous_process.communicate()
    return output

From there you can use that helper function to collect your list of files:

def get_python_files():
    full_command = "find . | grep py | grep -v pyc | grep -v \.un | grep -v test | grep -v virtualenv | grep -v \.swp"
    output = _get_output_from_pipe_command(full_command)
    filenames = [filename for filename in output.split("\n") if filename]
    return filenames

We’ll be using the file list both to get the test coverage and to get the authors on from Git Blame.

Install and use the Coverage Module

Install the coverage module to use in conjunction with your tests:

pip install coverage

From then you can run coverage. Instead of:

python manage.py test

You can do:

coverage run manage.py test

Once run, you can collect stats on a file. For example:

coverage report -m path/to/python_file.py

And you’ll get the output:

Name                                 Stmts   Miss  Cover   Missing
------------------------------------------------------------------
path/to/python_file.py      29     15    48%   17-23, 26-39, 42, 45-49

Parse the Coverage File

For our purposes, we care about a file and the line numbers that are missing coverage. For that I just put together a class to parse results and output the missing lines:

class ExcludeLineParser(object):

    @classmethod
    def get_missing_lines_for_filename(cls, filename):
        command = "../.virtualenv/bin/coverage report -m %s" % filename
        process = Popen(command.split(), stdout=PIPE)
        output, _ = process.communicate()
        return cls._get_excluded_lines(output)

    @classmethod
    def _get_excluded_lines(cls, coverage_output):
        excluded_line_numbers = []
        ignore_line_count = 2
        ignore_column_count = 4
        lines = [line for line in coverage_output.split("\n")[ignore_line_count:] if line]
        for line in lines:
            exclude_line_strings = line.split()[ignore_column_count:]
            for exclude_line_string in exclude_line_strings:
                exclude_line_string = exclude_line_string.replace(",", "").replace(" ", "")
                exclude_lines = cls._convert_exclude_line_string_to_ints(exclude_line_string)
                excluded_line_numbers.extend(exclude_lines)

        return excluded_line_numbers

    @classmethod
    def _convert_exclude_line_string_to_ints(cls, exclude_line_string):
        if "-" in exclude_line_string:
            line_start, line_end = exclude_line_string.split("-")
            return range(int(line_start), int(line_end) + 1)
        else:
            try:
                line_number = int(exclude_line_string)
            except ValueError:
                print "Error for values (%s)" % exclude_line_string
                return []
            return [line_number]

So now if I just run this:

ExcludeLineParser.get_missing_lines_for_filename("python_file.py")

I’ll get some simple output of a list of numbers:

[1, 2, 3, 4, 5]

Now we need to take those lines and associate them with Git authors.

Getting Authors from Git Blame

There’s probably multiple ways to do this, but this was my method.

If I run this command:

filename = "some_file.py"
full_command = "git blame --line-porcelain %s | grep author | grep -v author-" % filename
output = _get_output_from_pipe_command(full_command)
print output

I’ll get results that look something like this:

author Scott Lobdell
author Scott Lobdell
author Scott Lobdell
author Scott Lobdell
author John Doe
author John Doe
author John Doe

So now I basically want to associate each line number with its author so that I can take the line numbers excluded and associate it with a person. I will also use this information to tabulate the total line counts for each author so I can express test coverage as a percent. The code to write that is relatively straightforward and should be relatively easy for you to express if you’re reading this blog. So now, you can simply tabulate the results and then output them as you please. In this example I have the columns for author, test coverage percent, and total (non-test) lines written.

anon-results

I smudged all the names for anonymity in this example. But you should get the idea.

Next Steps

I’d like to see about packaging this whole process and uploading the results to a web application on Heroku. We’ll see where it goes.

The End

  • Awesome script, thanks!