Judy - Java mutation tester

Documentation

Table of contents

Quick start

  • Download and unzip to a selected folder (referred to as INSTALL_DIR) the newest version of Judy
  • At minimum, you need this parameters to run Judy (hover for info, click for more):
    INSTALL_DIR/bin/judy.sh --workspace /home/user/projects --classes project/target/classes --test-classes project/target/test-classes
  • Depending on the size of your project, the number of unit tests, etc. mutation testing can take a while - go make yourself a green tea/coffee.
  • Judy prints out logs to the logging folder, relative to the path you've run Judy from.
  • The main log you should monitor (for example, using tail -f) is logging/judy.log, because when Judy is done, the results will be there, printed out: 
    2014-01-25 23:02:37,004  INFO   Computing mutation result... Done 
2014-01-25 23:02:37,023  INFO    
2014-01-25 23:02:37,023  INFO   Classes                      : 222 
2014-01-25 23:02:37,023  INFO   Score                    [%] : 15.51 
2014-01-25 23:02:37,023  INFO     killed mutants             : 2661 
2014-01-25 23:02:37,023  INFO     all mutants                : 17161 
2014-01-25 23:02:37,023  INFO   Duration                 [s] : 675 
2014-01-25 23:02:37,023  INFO     initial tests duration [s] : 69 
2014-01-25 23:02:37,023  INFO     mutation duration      [s] : 605 
2014-01-25 23:02:37,023  INFO    
2014-01-25 23:02:37,023  INFO   Analysis duration (parallel) 
2014-01-25 23:02:37,023  INFO     generation duration       [s] : 51 
2014-01-25 23:02:37,024  INFO     test duration             [s] : 1030 
2014-01-25 23:02:37,024  INFO   Test classes used 
2014-01-25 23:02:37,024  INFO     initial tests runs         : 61 
2014-01-25 23:02:37,024  INFO     tests runs                 : 61 
2014-01-25 23:02:37,024  INFO    
2014-01-25 23:02:37,024  INFO   Saving result...  
2014-01-25 23:02:44,302  INFO   Saving result... Done
    Basically, the higher the score, the better. By default, an xml file with detailed results called judy-result.xml is saved in the folder from which judy was ran.

Documentation

Prerequisites

Judy is a mutation tester written in Java for Java, so you'll need a Java Runtime Environment (JRE) at least 1.6. Tested flavours include Oracle and Icedtea. Supported platforms are Windows XP and Linux (32 and 64 bit), but it should work wherever the JVM works.

Download and install

Head down to the download section and download the latest stable version of Judy. The installation process consists only of unpacking the archive to a suitable location (and maybe adding the bin folder to PATH, for convenience).

Preparing the project

Judy works directly with your project's bytecode, so you need to make sure that your Java source code has been compiled and you have the resulting *.class files somewhere at hand - both the main code's and unit tests' *.class files are needed.

Running Judy

Judy is a command line tool - so fire up cmd if you're on Windows or your favourite Linux terminal.
When invoked with either of the -?, -h or --help parameters, Judy will print out all the possible parameters and a short explanation for each of them (look for the output in the logging/judy.log log).

Mandatory parameters

The following parameters are mandatory - without them Judy will refuse to perform any mutation testing on a project.
  • -w, --workspace PATH - defines the absolute path to the project's workspace. All the other paths are relative to this one.
  • -c, --classes PATH;REGEX;CLASS - defines the relative PATH to the project's classes. This parameter can also take two optional parameters: REGEX and CLASS. REGEX defines the pattern (Java's regular expression) that will be used to find matching classes. CLASS will be used to exclude the specified classes. While there can be one REGEX, there can be many CLASSes. Note that in order to use the CLASS parameter, the REGEX parameter is necessary - it should suffice to set it to .*. For example:
    • -c project1/classes - use all the classes from project1/classes.
    • -c project1/classes;.* - same as above.
    • -c project1/classes;.*Test - use all the classes from project1/classes that end with Test.
    • -c project1/classes;.*Test;SuperLongTest;AnotherLongTest - use all the classes from project1/classes that end with Test, excluding SuperLongTest and AnotherLongTest.
  • -t, --test-classes PATH;REGEX;CLASS - same as --classes but for test classes. The optional parameters have the same meaning.
  • -s, --sources PATH - the relative path to the project's Java sources.
  • -l, --libs PATH - the relative path to the project's libraries: either to the whole folder housing all the dependencies or just a single JAR.

Optional parameters

The following parameters are optional.
  • -r, --result-path PATH - the absolute path to the report file that will be generated by Judy. The default value will be judy.result.xml.
  • -p, --properties PATH - the absolute path to a special file with additional settings for Judy. The file is based on the standard format of a Java properties file. It contains key=value pairs, where value can span many lines and contain a list of values separated by ;. Lines beginning with # are treated as comments and ignored.
    Currently, this file is only used to control the mutation operators used by Judy via the judy.operators key - see selecting mutation operators.
  • -k, --killed - adds the information about killed mutants to the report. By default only the non-killed mutants are reported.
  • -d, --debug - enables additional prints that might help pin-point issues during debugging.
  • --cluster - this parameter forces Judy to run in client mode. This option is only useful when you want to run Judy in a cluster.

Output

Dates and log level were removed from log output for brevity.
Judy, Sat Jan 25 22:51:21 CET 2014 Parsing input... Class deemed as unmutable, removing: org.apache.commons.collections.keyvalue.AbstractMapEntryDecorator Class deemed as unmutable, removing: org.apache.commons.collections.keyvalue.AbstractMapEntry ....... No judy.operators property found, will add all... Parsing input... Done Initial analysis... Initial analysis... Done Computing mutation result... Progress ClassKind Class Killed All Test classes Test methods Tests duration [ms] [ 0.4%] N com.example.Test1 6 316 3 533 2617 [ 1.2%] N com.example.Test2 1 20 5 264 764 [ 1.6%] N com.example.Test3 388 539 3 1341 46560 ......... Computing mutation result... Done Classes : 222 Score [%] : 15.51 killed mutants : 2661 all mutants : 17161 Duration [s] : 675 initial tests duration [s] : 69 mutation duration [s] : 605 Analysis duration (parallel) generation duration [s] : 51 test duration [s] : 1030 Test classes used initial tests runs : 61 tests runs : 61 Saving result... Saving result... Done

Advanced options

Running Judy in cluster

The speed of our mutation tester has always one of our key concerns, and while we've introduced a lot of optimizations, some projects just might be too big or have to many tests to be handled by a single machine in a reasonable time. If your project fits that description and you've done your homework right, you probably have a cluster of Jenkins/Hudson slaves at hand or other machines that can be used to distribute the workload of mutation testing.
To get Judy up and working in a clustered environment, you need three ingredients:
  • Judy Client - of course, you need Judy running somewhere, let's call it the master host. You run it as usual, with the bin/judy.(sh|bat) script. The only difference is the --cluster option.
  • Judy Server - this is the Judy server running using Cajo, a "a small, simple, powerful framework; to dynamically link Virtual Machines into a seamless continuum". It's a gateway between the client and workers. It is stared using the bin/judy-server.(sh|bat) and doesn't take any parameters.
  • Judy Worker - this is the guy doing all the hard work. The Judy client pushes jobs to do to the server and the server sends it out to the workers. Of course, you can have as many workers as needed. A worker is started using the bin/judy-worker.(sh|bat) script and takes the following parameters:
    • -w, --workspace PATH - required option that has the same meaning as for the client. It is very important that this location is a mirror image of the workspace on the client - all the other paths are relative to this one, so it's obvious that the folder's contents must match the ones on the client and other workers.
    • -x, --threads x - optional, specifies the number of threads to use.
You need to open UDP port 1198 and allow multicast of packets in order to get Cajo working - more details are available on their webpage.
There's no restriction as to the location of the client, server or worker instances of Judy - they can all even be on the same host (but you can hardly call that a cluster...).
A simple example, using three computers could look like this:
  • Host 1 - this one has the client and the server:
    bin/judy.sh -r "/var/lib/hudson/workspace/project1/report.xml" -w "/var/lib/hudson/workspace/project1" -l "lib" -s "src" -c "bin/classes;com\.example\.data\..*" -t "bin/tests;;UnitTest1" --cluster
    bin/judy-server.sh
  • Host 2 and 3 - both are running workers, notice that the workspace path doesn't have to be the same (but the contents does).
    bin/judy-worker.sh -w /var/lib/worker -x 4
The resulting report will be saved to /var/lib/hudson/workspace/project1/report.xml on Host 1. This file is a good candidate for a Jenkins' job artifact (just remember that the folder report-xslt is needed to properly display this file).

Selecting mutation operators

Due to the nature of mutation testing, some mutation operators might work better on your particular project (they are better at finding real bugs/not covered parts of code), while others might just introduce noise and waste your time.
If you feel adventurous, you can try disabling some mutation operators - to get "better", more tuned to your project or just faster results. The judy.operators key is used to specify the mutation operators to use, for example:
judy.operators=IOD;IOP;IOR;IPC;ISD;ISI; # inheritance
OAC;OMD;OMR;PLD;PNC;PPD;PRV; # polymorphism
EAM;EMM;EOA;EOC;JDC;JID;JTD;JTI; # Java-specific


©2013 mutationtesting.org Website's design based on gitscm.
e-Informatyka.pl - conferences, workshops
Wrocław University of Technology