Quality analysis on Eclipse plugins with Tycho, Sonar, Jacoco and SWTBot

Building tools brings a lot more that compilation. You can go much further.

For instance, have a look at maven : there’s a plugin for almost everything (Here, here, and here). And with the rise of Tycho, you can bring all the power of maven to build OSGI/Eclipse based projects.

Let’s see how to leverage the use of Sonar (a great open source quality analysis tool) to deal with the QA of your project.

Install the Jacoco plugin for Sonar

I assume you have a running instance of Sonar. (If not, have a look here : Install Sonar).

Since Eclipse convention is to separate plugin and test in two separate projects (in order to not pollute plugin dependencies with test dependencies), “traditional” code coverage tools (such as Cobertura) won’t be able instrument Eclipse plugin classes because test classes are not in their classpath.

Jacoco adopts “on the fly instrumentation” by the means of a JavaAgent (http://www.eclemma.org/jacoco/trunk/doc/implementation.html), this enables the ability to maintain plugin code and test code in separated project : that just what we need in our case! Great!

So to install Jacoco plugin just follow that guide : http://docs.codehaus.org/display/SONAR/JaCoCo+Plugin.

Now we are ready to configure our Tycho build.

Tycho build

I assume you have a running tycho build. So you just need to add the following section in the pluginManagement section of the parent pom :

<!-- Testing -->
<plugin>
   <groupId>org.eclipse.tycho</groupId>
   <artifactId>tycho-surefire-plugin</artifactId>
   <version>${tycho.version}</version>
   <configuration>
      <useUIHarness>false</useUIHarness>
      <includes>
         <include>**/*Test.java</include>
      </includes>
      <!-- Kill test JVM if tests take more than 10 minutes (600 seconds) to finish -->
      <forkedProcessTimeoutInSeconds>600</forkedProcessTimeoutInSeconds>
   </configuration>
</plugin>

And then provide the following profile :

<!-- This profile is used to gather code coverage for Sonar -->
<profile>
  <id>codeCoverage</id>
  <properties>
    <!-- Properties to enable jacoco code coverage analysis -->
    <sonar.core.codeCoveragePlugin>jacoco</sonar.core.codeCoveragePlugin>
    <sonar.dynamicAnalysis>reuseReports</sonar.dynamicAnalysis>
    <sonar.jacoco.reportPath>../org.demo.camp.nantes.parent/target/jacoco.exec</sonar.jacoco.reportPath>
  </properties>

  <build>
    <plugins>
      <!-- Enabling use of jacoco -->
      <plugin>
        <groupId>org.jacoco</groupId>
        <artifactId>jacoco-maven-plugin</artifactId>
        <version>0.5.3.201107060350</version>
        <executions>
          <execution>
            <goals>
              <goal>prepare-agent</goal>
            </goals>
            <configuration>
              <!-- Where to put jacoco coverage report -->
              <destFile>${sonar.jacoco.reportPath}</destFile>
            </configuration>
          </execution>
        </executions>
      </plugin>
    </plugins>
  </build>
</profile>

And that’s all for the parent pom configuration.

For unit testing you do not need to do more that providing the right packaging :

<packaging>eclipse-test-plugin</packaging>

But for UI testing with SWTBot, you need some more configuration.

Tycho configuration for UI testing project with SWTBot

In order to execute your UI tests, you need to specify which product/application you need to start. Basically, for an Eclipse plugin, you’ll need to run the following couple :

<product>org.eclipse.sdk.ide</product>
<application>org.eclipse.ui.ide.workbench</application>

So you just need to add the following section to your UI test plugin :

<build>
	<plugins>
		<plugin>
	        <groupId>org.eclipse.tycho</groupId>
	        <artifactId>tycho-surefire-plugin</artifactId>
	        <version>${tycho.version}</version>
	        <configuration>
	          <useUIHarness>true</useUIHarness>
	          <useUIThread>false</useUIThread>
	          <product>org.eclipse.sdk.ide</product>
	          <application>org.eclipse.ui.ide.workbench</application>
	        </configuration>
      	</plugin>
	</plugins>
</build>

Now your UI test can run through Tycho and being analyzed by Sonar. Great!

Limitations

Wokring on Mac has some advantages, and some drawbacks, here we are facing a drawback. Running SWT applications on Mac require the following JVM argument : -XstartOnFirstThread (http://www.eclipse.org/swt/macosx/).

So you need to configure the the tycho-surefire-plugin as below in the parent pom :

<plugin>
	<groupId>org.eclipse.tycho</groupId>
	<artifactId>tycho-surefire-plugin</artifactId>
	<version>${tycho.version}</version>
	<configuration>
		<useUIHarness>false</useUIHarness>
		<includes>
			<include>**/*Test.java</include>
		</includes>
		<!-- Kill test JVM if tests take more than 10 minutes (600 seconds) to finish -->
		<forkedProcessTimeoutInSeconds>600</forkedProcessTimeoutInSeconds>
		<argLine>-XstartOnFirstThread</argLine>
	</configuration>
</plugin>

But if you do that, the tycho-surefire-plugin argLine will overwrite the Jacoco instrumentation argLine.

But thanks to this following tycho-surefire-plugin configuration parameter : appArgLine you can add other arguments without overwriting the ones from the argLine configuration parameter. So you can define your MacOs profile as below :

<!-- MacOS specific vm arguments for UI testing -->
<profile>
	<id>osx</id>
	<activation>
		<property>
			<name>java.vendor.url</name>
			<value>http://www.apple.com/</value>
		</property>
	</activation>
	<build>
		<pluginManagement>
			<plugins>
				<plugin>
					<groupId>org.eclipse.tycho</groupId>
					<artifactId>tycho-surefire-plugin</artifactId>
					<version>${tycho.version}</version>
					<configuration>
						<appArgLine>-XstartOnFirstThread</appArgLine>
					</configuration>
				</plugin>
			</plugins>
		</pluginManagement>
	</build>
</profile>

You now can handle the execution of your UI tests seamlessly on any platform.

To be done

If you have a look at the Sonar analysis, you’ll see that code coverage is calculated from source code and test code (Sonar analysis).

So we need to use the sonar.exclusions pom property : http://docs.codehaus.org/display/SONAR/Advanced+parameters

Another thing to investigate is to differentiate Code coverage and IT coverage : http://www.sonarsource.org/measure-code-coverage-by-integration-tests-with-sonar/

Add continuous integration (Jenkins or Hudson) : here is a good explanation on how to run UI test on a headless server : http://blog.dahanne.net/2011/07/18/run-ui-tests-on-a-headless-jenkins-hudson-continuous-integration-server-running-ubuntu/ by @anthonydahanne

Resources

You can have a look to the source code of a small sample : https://github.com/xseignard/tycho-demo-camp

You can run mvn clean install -PcodeCoverage and then mvn sonar:sonar -PcodeCoverage from the parent project.

And check the sonar analysis here: http://ec2-79-125-43-108.eu-west-1.compute.amazonaws.com:8080/sonar/ It should work out of the box.

You can edit/remove the sonar connection settings in order to use your own Sonar instance.

Feel free to provide pull request in order to ameliorate the project and/or drop a comment.