Using EMMA with ANT for JUnit test coverage reporting

Author: Neale Rudd, Metawerx

Date: 13-Feb-2007

Emma Test Coverage Report
http://www.metawerx.net/images/screenshots/emma.png Screenshot of EMMA, showing JUnit test coverage. Green lines are tested lines, red lines are code missed by the unit tests.

This guide shows how to add EMMA tasks to your ANT build script to produce coverage reports for your JUnit test classes.

Most of the text and elements below are taken directly from the EMMA example build scripts and example application. This guide explains some of the steps in more detail, and includes some troubleshooting tips and common problems you may experience while trying to add EMMA to your own build.xml file.

The following is a quick summary of how EMMA works. We will add sections to build.xml to accomplish each of the following tasks.

  • Your ant script will compile your classes normally (preferably with debug="on" in the <javac> element, so that EMMA can show the source files in the reports)
  • The emma <instr> tag will add information to your classes, and place the converted classes in a folder called /target/emmainstr
  • The instrumented classes are tested by JUnit, instead of your normal classes
  • JUnit creates a "coverage file" during execution
  • The EMMA reporting system reads the coverage file and produced HTML based reports

Setup

The first thing you will need to do is download EMMA. You can run the example build.xml to see some example reports.

Emma Sourceforge Website

Project Setup

In this example, the EMMA jar files will be kept inside the project, rather than in a common area of your build environment. This allows easy upgrade of EMMA or other components on a per-project basis. It is also possible to use a shared EMMA installation, in which case you will need to modify the path of the EMMA jar files accordingly (see below).

We will assume you have a src folder in your project folder, as with most projects that use ANT.

The EMMA jars will be placed into a new top-level folder, at the same level as /src, called /lib/emma.

After installing the EMMA jar files, your project folder layout should look something like this.

projectname/
    src/
        com/
            example/
                class1.java
                class2.java
                class3.java
    target/
        classes/
            com/
                example/
                    class1.class
                    class2.class
                    class3.class
    lib/
        emma/
            emma.jar
            emma_ant.jar
    build.xml

After successfully integrating EMMA using this tutorial, your folder structure will look like this:

projectname/
    src/
        com/
            example/
                class1.java
                class2.java
                class3.java
    target/
        classes/
            com/
                example/
                    class1.class
                    class2.class
                    class3.class
        emmainstr/                  // Classes created by EMMA's instrumentation process
            com/                    // Classes created by EMMA's instrumentation process
                example/            // Classes created by EMMA's instrumentation process
                    class1.class    // Classes created by EMMA's instrumentation process
                    class2.class    // Classes created by EMMA's instrumentation process
                    class3.class    // Classes created by EMMA's instrumentation process
    reports/
        emma/
            coverage.html           // The coverage report
            _files/                 // The coverage report
                etc...              // The coverage report
    lib/
        emma/
            emma.jar
            emma_ant.jar
    build.xml

Add an EMMA target to your build.xml file

Add the following at the top of your build.xml file. Customise the paths if necessary.

Note that the coverage dir is passed to a forked JVM, so it is important that it is an absolute path (use ${basedir} to achieve this, as in the example below).

<!-- ====================================================================== -->
<!-- EMMA SETUP -->
<!-- ====================================================================== -->

<property name="coverage.dir" value="${basedir}/reports/emma" />     <!-- directory which emma coverage reports will be written to -->
<property name="instr.dir" value="${basedir}/target/emmainstr" />    <!-- directory which emma instrumentation classes will be written to -->
<property name="emma.dir" value="${basedir}/lib/emma" />             <!-- directory that contains emma.jar and emma_ant.jar -->

<!-- Set emma.lib to refer to the list of EMMA jar files -->
<path id="emma.lib" >
    <fileset dir="${emma.dir}">
        <include name="*.jar"/>
    </fileset>
</path>

<!-- Define which classes will be reported in the coverage reports (by default, we will include all classes and assume -->
<!-- that your project's output folder is target/classes -->
<path id="emma.coverage.classes" >
    <pathelement location="target/classes" />
</path>

<!-- Load <emma> and <emmajava> custom tasks so that they can be used in ANT -->
<taskdef resource="emma_ant.properties" classpathref="emma.lib" />

<!-- Enable Emma -->
<target name="emma" description="turns on EMMA's instrumentation/reporting" >

    <property name="emma.enabled" value="true" />

    <!-- EMMA instr class output directory (it is important to create this property only when EMMA is enabled) -->
    <mkdir dir="${instr.dir}" />
    
    <!-- this property, if overriden via -Demma.filter=<list of filter specs>
     on ANT's command line, will set the coverage filter; by default,
     all classes found in 'emma.coverage.classes' pathref will be instrumented:
    -->
    <property name="emma.filter" value="" />

</target>

Add EMMA to your <target> element, as the first element.

We have added a new target called "emma", so add this to your target element at the top of your build.xml

For example, if this is your <target> element ...

<target name="all" depends="clean, update, compile1, compile2, compile3, test, jar"/>

... change it as follows ...

<target name="all" depends="emma, clean, update, compile1, compile2, compile3, test, jar"/>

Tip: run your build now, to make sure nothing is broken.

Add EMMA instrumentation to your test target

EMMA works by modifying certain classes, then monitoring them during an execution of the java binary. The modified classes are called "instrumented" classes.

In this step, we will add an element which creates the instrumented classes.

Inside your test target, before performing the <junit> task, add the following:

<!-- ======================================================================= -->
<!-- EMMA INSTRUMENTATION -->
<!-- ======================================================================= -->
<!-- Instrument the classes, this takes the normal classes, and makes instrumented classes instead -->
<!-- Note that EMMA takes regular ANT path elements as instrumentation input, which is exceedingly convenient -->
<emma enabled="${emma.enabled}" >
    <instr instrpathref="emma.coverage.classes"
         destdir="${instr.dir}"
         metadatafile="${coverage.dir}/metadata.emma"
         merge="true"
    >
        <filter value="${emma.filter}" />
    </instr>
</emma>

Tip: run your build now, to make sure nothing is broken.

If everything went well to this point, your project should now be creating:

  • a target/emmainstr folder, containing converted class files.
  • a folder called /reports/emma with a file called metadata.emma inside it

Change JUnit so that it forks and runs the tests against the instrumented classes

To use EMMA, java needs to run as a separate process (ie: a new JVM) which is passed system properties.

In addition, we need to change your existing JUnit classpath so that it tests the new instrumented classes which were created in the step above.

Do the following:

  • In your <junit> element, add a fork="true" attribute. You may also want to add forkmode="once", or forkmode="perBatch" and wrap the tests in a <batchtest> element so that only one VM is spawned (this will speed up the testing considerably).
  • In the <classpath> element of <junit>, change the locations of your real classes, so that you are pointing at the ${instr.dir} attribute instead of your normal target/classes folder. For example, change:
<pathelement location="target/classes" />
... to ...
<pathelement location="${instr.dir}"/>

Next, add in the emma libraries we defined above. This goes into the <junit> <classpath> element as well.

<path refid="emma.lib" />

The last thing we have to do here, is to pass two special system.properties to the new forked junit task.

These elements go just before your closing </junit> tag.

<jvmarg value="-Demma.coverage.out.file=${coverage.dir}/coverage.emma" />
<jvmarg value="-Demma.coverage.out.merge=false" />

Note that the coverage dir is passed to the forked JVM as a property, so it is important that it is an absolute path. If you followed the example, coverage.dir will already have been set using ${basedir}, so will already be absolute.

If the path is relative, the coverage data will be created in the folder the VM was run from, and not relative to the build.xml file like other relative paths are. In this case, the reporting will not be able to find the coverage.emma file and will give the following error message:

[report] nothing to do: no runtime coverage data found in any of the data files.
This can be very difficult to debug in some cases, such as when the build works from the command line, but won't build the reports when run from CruiseControl (which runs the VM from the CruiseControl base folder instead of the project folder).

Tip: run your build now, to make sure nothing is broken.

If everything went well to this point, your project should now be creating:

  • a target/emmainstr folder, containing converted class files.
  • a folder called /reports/emma with 2 files called metadata.emma and coverage.emma inside it

Add the Coverage Reports

The next step is to add the coverage report creation, which is the whole point of this exercise.

EMMA has now created our coverage.emma file, which contains all the coverage data. The following element converts this into HTML reports.

Add this element after your <junit> element, before the closing </target> tag.

<!-- if enabled, generate coverage report(s): -->
<emma enabled="${emma.enabled}" >
    <report sourcepath="src"
          sort="+block,+name,+method,+class"
          metrics="method:70,block:80,line:80,class:100"
    >
    <!-- collect all EMMA data dumps (metadata and runtime)
         [this can be done via nested <fileset> fileset elements
         or <file> elements pointing to a single file]:
    -->
    <fileset dir="${coverage.dir}" >
        <include name="*.emma" />
    </fileset>
    
    <!-- for every type of report desired, configure a nested
         element; various report parameters
         can be inherited from the parent <report>
         and individually overridden for each report type:
    -->
    <txt outfile="${coverage.dir}/coverage.txt"
        depth="package"
        columns="class,method,block,line,name"
    />
    <xml outfile="${coverage.dir}/coverage.xml"
        depth="package"
    />
    <html outfile="${coverage.dir}/coverage.html"
        depth="method"
        columns="name,class,method,block,line"
    />
    </report>
</emma>

If everything went well to this point, your project should now be creating:

  • a target/emmainstr folder, containing converted class files.
  • a folder called /reports/emma with 2 files called metadata.emma and coverage.emma inside it, as well as coverage.txt, coverage.xml and coverage.html.
  • The coverage.html file is the actual coverage report. View this using your browser to measure the coverage of your
JUnit tests. Click on the file names to see each file in more detail, or to see the source code and lines that have not been covered by your tests.
  • If the report is not produced, or you are getting errors in your build, check the troubleshooting section below. The Success section below shows typical messages that should appear in your logs if each stage progressed successfully.

Troubleshooting

Emma task could be created. Message: Could not create task or type of type: emma.

WARN  ScriptRunner     - /YOUR_PROJECT_PATH/build.xml:268: Could not create task or type of type: emma.
WARN  ScriptRunner     -
WARN  ScriptRunner     - Ant could not find the task or a class this task relies upon.
WARN  ScriptRunner     -
WARN  ScriptRunner     - This is common and has a number of causes; the usual
WARN  ScriptRunner     - solutions are to read the manual pages then download and
WARN  ScriptRunner     - install needed JAR files, or fix the build file:
WARN  ScriptRunner     -  - You have misspelt 'emma'.
WARN  ScriptRunner     -    Fix: check your spelling.
WARN  ScriptRunner     -  - The task needs an external JAR file to execute
WARN  ScriptRunner     -      and this is not found at the right place in the classpath.
WARN  ScriptRunner     -    Fix: check the documentation for dependencies.
WARN  ScriptRunner     -    Fix: declare the task.
WARN  ScriptRunner     -  - The task is an Ant optional task and the JAR file and/or libraries
WARN  ScriptRunner     -      implementing the functionality were not found at the time you
WARN  ScriptRunner     -      yourself built your installation of Ant from the Ant sources.
WARN  ScriptRunner     -    Fix: Look in the ANT_HOME/lib for the 'ant-' JAR corresponding to the
WARN  ScriptRunner     -      task and make sure it contains more than merely a META-INF/MANIFEST.MF.
WARN  ScriptRunner     -      If all it contains is the manifest, then rebuild Ant with the needed
WARN  ScriptRunner     -      libraries present in ${ant.home}/lib/optional/ , or alternatively,
WARN  ScriptRunner     -      download a pre-built release version from apache.org
WARN  ScriptRunner     -  - The build file was written for a later version of Ant
WARN  ScriptRunner     -    Fix: upgrade to at least the latest release version of Ant
WARN  ScriptRunner     -  - The task is not an Ant core or optional task
WARN  ScriptRunner     -      and needs to be declared using <taskdef>.
WARN  ScriptRunner     -  - You are attempting to use a task defined using
WARN  ScriptRunner     -     <presetdef> or <macrodef> but have spelt wrong or not
WARN  ScriptRunner     -    defined it at the point of use
WARN  ScriptRunner     -
WARN  ScriptRunner     - Remember that for JAR files to be visible to Ant tasks implemented
WARN  ScriptRunner     - in ANT_HOME/lib, the files must be in the same directory or on the
WARN  ScriptRunner     - classpath
WARN  ScriptRunner     -
WARN  ScriptRunner     - Please neither file bug reports on this problem, nor email the
WARN  ScriptRunner     - Ant mailing lists, until all of these causes have been explored,
WARN  ScriptRunner     - as this is not an Ant bug.
This error means the emma <taskdef> tag has not been included, or that the emma libraries cannot be located.

Check your <taskdef> tag, and make sure the emma jar files can be located, as in the following example:

<!-- Create a property which specifies the location of the emma jar folder -->
<property name="emma.dir" value="${basedir}/lib/emma" />            <!-- directory that contains emma.jar and emma_ant.jar -->
<!-- Specify the path element used by EMMA taskdef -->
<path id="emma.lib" >
    <pathelement location="${emma.dir}/emma.jar" />
    <pathelement location="${emma.dir}/emma_ant.jar" />
</path>
<!-- Loads the <emma> and <emmajava> custom tasks, making them available for use by ANT -->
<taskdef resource="emma_ant.properties" classpathref="emma.lib" />

Reports are not produced. Message: nothing to do: no runtime coverage data found in any of the data files

INFO  ScriptRunner     -    [report] processing input files ...
INFO  ScriptRunner     -    [report] 1 file(s) read and merged in 81 ms
INFO  ScriptRunner     -    [report] nothing to do: no runtime coverage data found in any of the data files

This message means that your data files do not contain coverage information, so the reporting system can not create the coverage reports. The normal cause for this is that only the metadata.emma file is being scanned, because your junit task has used normally compiled classes instead of the emma-instrumented classes.

  • Check your <junit> <classpath> element has included the instrumented-classes folder. Do not include the normal classes. JUnit needs to run the tests using the classes that were created by emma's <instr> element (the destdir attribute in the <instr> element).
  • Check that your <junit> <classpath> element has included the emma libraries
  • Check that your <junit> element is forking (fork="true") and that the VM is being passed the emma.coverage.out system properties, as follows:
<jvmarg value="-Demma.coverage.out.file=${coverage.dir}/coverage.emma" />
<jvmarg value="-Demma.coverage.out.merge=false" />

As a reminder, the way emma works is as follows:

  • You compile your classes normally (preferably with debug="on" in the <javac> element)
  • The emma <instr> tag adds information from your classes to new classes, and places them in the <instr> destdir location. It also makes a metadata file specified by the metadatafile attribute (usually metadata.emma)
  • The instrumented classes from destdir are tested by JUnit, instead of your normal classes
  • JUnit must fork, and must have access to the emma jar files and must be passed the emma.coverage.out system properties. It then creates a coverage file (usually coverage.emma)
  • The reporting system's <report> tag reads the coverage file(s) specified by its <fileset> tag, to produce the reports

Success

If everything is working properly for each of the 3 stages, you should see the following messages in your CruiseControl logs. When everything has worked correctly, your coverage reports folder will contain an HTML, XML and TXT file, as well as a folder called _files containing additional HTML files used by coverage.html.

Instrumentation

INFO  ScriptRunner     -     [instr] processing instrumentation path ...
INFO  ScriptRunner     -     [instr] instrumentation path processed in 923 ms
INFO  ScriptRunner     -     [instr] [112 class(es) instrumented, 0 resource(s) copied]
INFO  ScriptRunner     -     [instr] metadata merged into [/YOUR_PROJECT_PATH/reports/emma/metadata.emma] {in 464 ms}

Runtime

INFO  ScriptRunner     -     [junit] ------------- Standard Output ---------------
INFO  ScriptRunner     -     [junit] EMMA: collecting runtime coverage data ...
INFO  ScriptRunner     -     [junit] ------------- ---------------- ---------------

Coverage Report Creation

INFO  ScriptRunner     -    [report] processing input files ...
INFO  ScriptRunner     -    [report] 2 file(s) read and merged in 94 ms
INFO  ScriptRunner     -    [report] writing [txt] report to [/YOUR_PROJECT_PATH/reports/emma/coverage.txt] ...
INFO  ScriptRunner     -    [report] writing [xml] report to [/YOUR_PROJECT_PATH/reports/emma/coverage.xml] ...
INFO  ScriptRunner     -    [report] writing [html] report to [/YOUR_PROJECT_PATH/reports/emma/coverage.html] ...


Great Article!!! helped me a lot!!! tks

--Passaro, 23-May-2007


This article was superb. It gave me a tremendous headstart. Thanks!

--AnonymousCoward, 27-Jun-2007


perfect article! very well explained!

--Anonymous, 12-Jul-2007


Excellent article on ANT and Emma!!!!

I searched for such an article and finally found it.

Thanks very much.

--AnonymousCoward, 29-Nov-2007


hey i'v done everything as told but the coverage.emma report is nt being generated :( Can anyone help me plzzzzzzzzz!!!!

--AnonymousCoward, 31-Jan-2008


i tried everything being told here but the coverage.emma file is not being generated. kindly help in this regard. thanks in adv.

--NeedOfHelp, 22-Feb-2008


A very good article.Thanks.

--sudhu, 30-May-2009

navigation
metawerx specific
search
Share
tools
help

referring pages

Share