Introduction to Jacoco
Jacoco provides a wide range of coverage(analysis of instructions, lines, methods, types ) along with the code complexity(cyclomatic complexity). Jacoco is a free, open-source and possibly the best-maintained package till date.
Code coverage and Jacoco
Code coverage metric indicates the percentage of lines of code executed during automated test execution. As a result this metric actually improves the quality of the code. This is alternatively known as Line coverage.
Say we have 20 lines of code in a method and if 10 lines get executed then line coverage is 50%.
Branch coverage actually covers how many decision points and conditional logic has been tested.
Code coverage provides a measurement technique by which we can check and determine how much our code has been executed via automation tests.
The objective of using jacoco or any other code coverage tool is to find and keep track parts of our code lines that got executed or missed.
Primarily in the Java environment, there are four tools available.
Features of Jacoco
- Simple to use – easy to integrate with new and existing build scripts.
- Provides coverage analysis based on JVM instructions.
- It is based on java byte code hence can work without the source file.
- Simple, easy, clean integration via jacoco agent.
- It can be plugged into any framework.
- It supports all releases of java.
- It can be integrated with leading IDEs(Eclipse, IntelliJ, etc)
- It can provide reports in multiple formats like HTML, CSV, XML, etc.
- It supports any JVM languages.( like java, groovy, AspectJ.)
- It supports remote protocols and JMX controls.
- Supports several build tools like Jenkins, Teamcity, etc.
- Lightweight and having good performance.
- It is built on JUnit hence It supports a variety of code lines- developer’s branches, smoke test, regression tests.
- It provides huge online documentations.
- It can work offline and ad-hoc instruction.
Jacoco Coverage Counters
Jacoco uses the following different counters to calculate coverage metrics:
- Instructions– It is often referred to as C0 coverage. It provides details about how much code is executed or missed. It counts the number of instructions in the scope.
- Branches– It is often referred to as C1 coverage. It provides details about the total number of branches of a method to determine how much code is executed or missed. It counts the number of branches in the scope.
- Lines- If a source code line is having one instruction and it is executed, Jacoco provides line details. It counts the number of lines in the scope.
- Methods-Jacoco provides details of all solid (implemented) method. It counts the number of methods in the scope.
- Classes-Jacoco provides details of each class(if one of its methods is executed). Jacoco also considers constructors as static initializers as methods. It counts the number of classes in the scope.
Other metrics that can be obtained from Jacoco are as follows:
- COVEREDRATIO: The ratio of covered items to uncovered items (i.e. the percentage of total items that are covered)
- COVEREDCOUNT: The absolute number of covered items
- MISSEDCOUNT: The absolute number of items not covered
- MISSEDRATIO: The ratio of items not covered
- TOTALCOUNT: total number of items
- Cyclomatic complexity- Jacoco provides cyclomatic complexity for every solid/implemented method and also provides a summary for classes, packages, groups’ complexity.
Jacoco also provides the below customizations
- 100% line coverage for all the classes except few excluded ones.
- It allows us to combine multiple rules(goals) to a specific rule.
- It allows us to exclude classes from being reported into the HTML report(final).
- It allows us to exclude code generated by Lombok framework.
How actually Jacoco works?
When Jacoco is configured as run from the plugin(maven or any other build tools), the jacoco agent configuration is done by invoking the prepare agent or prepare agent integration tasks(goals). It is done even before running the tests.
This action sets the property called argLine that points to the jacoco agent. This property is passed as a JVM argument to the test runner.
The test runner initiates the JVM with the argLine argument. By this time the Jacoco agent configuration is already completed. So the code and instructions can be performed very clearly.
Jacoco modifies the class file by creating a hook that actually counts the coverage.
When the JVM process is terminated, the jacoco agent captures the code coverage report to a file(jacoco.exec) for unit tests and to another file(jacoco_it.exec) for integration tests.
both these files are placed inside the target directory. The jacoco code coverage data file now uses these two reports to create a new and final report with the following details:
- Instruction coverage
- Branch coverage
- Line coverage
Jacoco reports are available in the following formats-
- CSV etc
Jacoco can very well work standalone mode but its actual power can be seen when used with several build tools. (Like-ANT, Gradel, Maven, IntelliJ IDEA project etc Refer here for 3rd party integration)
How to Integrate with Eclipse?
To integrate Jacoco, we have to follow the below steps:
- Open Eclipse
- Go to help
- Select Install New Software
- Use the link to download Jacoco plugin
- This will list out EclEmma plugins in the below space
- Select it and click on Finish.
- Now you must see the code coverage button
- Now either we need to right-click on the class click Code Coverage > Run As or alternatively select the class and click on this code coverage button.
How to integrate Jacoco with Maven?
In the pom.xml, we need to put the below entry:
<dependency> <groupId>JUnit</groupId> <artifactId>junit</artifactId> <version>4.9</version> </dependency>
As Jacoco runs on JUnit, we need to incorporate Junit into the POM.xml file.
In the next step, we need to incorporate the Maven Jacoco plugin:
<plugin> <groupId>org.jacoco</groupId> <artifactId>jacoco-maven-plugin</artifactId> <version>0.x.0</version> // x is the current version <executions> <execution> <id>default-prepare-agent</id> <goals> <goal>prepare-agent</goal> </goals> </execution> <execution> <id>default-report</id> <phase>prepare-package</phase> <goals> <goal>report</goal> </goals> </execution> </executions> </plugin>
How to execute Jacoco from maven?
Execution of jacoco can be done via the command line. like
- mvn: test
- mvn jacoco: report
Once we provide the command, maven runs JUnit test cases in each module. Finally, jacoco will provide a report.
mvn: test generates the jacoco.exec file. jacoco: report generates a report in HTML file under target/site/jacoco folder.
How to Connect Jacoco with Teamcity?
Jacoco is also supported by TeamCity. The supported versions are
How to Connect Jacoco with SonarQube?
Sonarcube by default provides jacoco plugin to attach jacoco with SonarQube.
How to Connect Jacoco with Jenkins?
Jenkins can connect jacoco via jacoco Jenkins plugins.
Reporting in Jacoco
The colour code Jacoco uses are as follows:
- Green – It indicates that fully covered lines.
- Yellow– It indicates that these are partially covered lines and some of the instructions are missed.
- Red It indicates that fully ignored lines that are fully ignored lines.
To obtain the coverage data jacoco uses ASM for code execution on the fly, by receiving the JVM tool interface in the same process.
Jacoco reports can be found at \workspace\Parent\target\site\jacoco folder. So inside the workspace, there is a Parent folder inside the Parent folder there is a target folder. Once we navigate to the target folder, we need to find the folder called site followed by jacoco. Inside the jacoco folder, the index file shows us the report.
Incase if a test case is failed,jacoco captures that as well.Jacoco checks if the code has been covered or not. All codes pass/fail/ignored will be reported.
Note- Jacoco test report needs to be triggered as soon as the tests are executed and finished. If we want to perform the reporting portion later point of time, a failing build (due to some failed step) may stop the build process itself. In that case, we will not be able to generate the report.
How jacoco is different from SureFire or FailSafe?
Maven plugins work differently than the jacoco. SureFire or FailSafe executes the test cases in a given JVM. But the jacoco actually latches itself to the JVM, creates hook then uses its agent to monitor test execution.
So unlike plugins, jacoco can not work on its own, it needs a trigger application.
On the other hand, the advantage of the hook is that it can detect byte code instructions instead of line check as a result the final report contains actual coverage.
Where do I get other resources on Jacoco?
Jacoco has a huge online presence. below are a few important links:
- Jacoco Maven repository- https://mvnrepository.com/artifact/org.jacoco/jacoco-maven-plugin
- Jacoco group- https://groups.google.com/forum/#!forum/jacoco
- Jacoco faqs: https://www.jacoco.org/jacoco/trunk/doc/faq.html
Few more important discussion:
- How to instrument .jar programmatically?- https://groups.google.com/forum/#!topic/jacoco/3XG-oGU8zwg
- jacoco output file has nothing when run inside docker container- https://groups.google.com/forum/#!topic/jacoco/RAldwHsmeVY
- Jacoco Java agent destfile output in xml format-https://groups.google.com/forum/#!topic/jacoco/B-cehXVa_SI
- Partially covered InputStream.bufferedReader()-https://groups.google.com/forum/#!topic/jacoco/bm21z2snpDs
- Jacoco Coverage not reporting- https://groups.google.com/forum/#!topic/jacoco/8tEfdx6XECo
- Using JAVA API to make a simple report- https://groups.google.com/forum/#!topic/jacoco/hDO5G0ua2k4
Drawbacks of Jacoco
Just like any other code coverage tools jacoco also have some drawbacks. They are as follows:
- Jacoco takes a class file as a source but can not take a file as a source.
- Statement coverage is not that effective.
- Modified Condition/Decision Coverage (MC/DC) is not supported.
- Mutation coverage is not supported.
- Reporting in PDF, JSON and text not/partially supported.
- SPOC integration is not out of the box.
- Grails command-line interface is tough.
- Does not support Bamboo and Hudson.
- Does not support Jira.
Code coverage via any tool or with Jacoco can provide greater insight into the developed code with the following benefits
- It provides a way to improve the code or tests.
- It helps to set up development goals and achieve the same.
- It uncovers potential error paths.
However note that higher coverage means only that higher amount code gets executed but it does not talk about the quality of the code.