abstract:

[...] collecting metadata set that represent links between classes under test and test classes (and eventually more fine grained links between methods) and using them for a lot of purposes. Metadata can be collected from different sources: annotations on classes under test, instrumentation of test classes during tests’ execution (in future perhaps even from a dedicated user interface). These metadata can be used for various goals: first of all it becomes possible to run only the tests that stress just a particular set of classes (for instance the classes changed since last compilation) [...] We have a Maven plugin, while Eclipse and Hudson ones are in our roadmap.

Full story:

A while ago I’ve written a post about a new approach to unit tests; it has been a quite read and discussed post and it gives the momentum for starting a new community discussing about the idea and working on a software project implementing this idea. The project was named TestedBy and hosted in google code.

That idea has evolved and changed and we have decided to change also the name of the project as a remark of these changes, calling it SibillaTest. Why Sibilla? Because, as you can read in this post and you can get trying our samples, Sibilla (italian name for Cumae Sybil) is always able to prophesy and predict which tests you need to run.

This post has been written to announce the release of a first beta version of SibillaTest and describe how our thoughts about the original idea have changed in these months and how they have been  implemented in SibillaTest.
I’m going to recall the main concepts of the original idea, If you haven’t read the original blog would be nice to take a look to it, to its numerous comments and also to the update I’ve posted few days after. If you are interested in all the genesis of the project you could also have a look to the discussions continued in our forum

Generally speaking what I’ve depicted in my old blog post was a system to keep annotations in classes under test pointing test classes (or in some case to specific test methods). Advantage of this approach were mainly 2:

• Design By Contract (viewing test as contract definition). The sugar here was the opportunity to put TestedBy annotation also on super classes (even if they are abstract or even interfaces) and inherit test annotations.
• Run only tests stressing a specific class. IOW run test stressing a class of your interest (i.e just compiled) giving you the confidence you aren’t breaking test suite without running the entire suite itself.

The main concerns about this approach was about code cluttering putting a lot of annotations in production code (of course you could have a lot of test stressing a class or even a method).

So we have changed a bit the focus from an annotation centric approach to a more general metadata
approach. The idea is to collect a set of metadata that represent link between classes under test and test classes (and eventually more fine grained links between methods) and use them for a lot of purposes.
The pluses of this approach are mainly two:

• Metadata can be collected from different sources: annotations , instrumentation of test during test execution, maybe in the future from a dedicated  user interface.
• Metadata can be serialized and/or used in a second step for various goals: run “right” tests of course will remain a central feature, but also
“graphical” representation, code navigation in IDE, more dynamic use of it

I’m trying to keep this post not too long trying to keep your attention focused on the usefulness of SibillaTest,  I’m going to describe in this the feature already implemented in SibillaTest. But, of course, we have a lot of ideas that will be developed in next future that we would like to discuss with the community and maybe have some contribution from the community. Here you have a mind map depicting a lot of these working areas to get you an idea. If you are interested in a more detailed description of them you  can find a page in our wiki describing it.

So, what is SibillaTest today?

aims at changing the point of view regarding test classes and classes under test. What we get is the focus being moved to the classes under test; from those classes, which of course are the most important ones of your projects, we obtain links to your test classes and test methods. In other words we define the classes and methods’ contract using tests, while also keeping track of tests that need to be run when a class/method under test has been modified. This is possible by collecting metadata sets that represent links between classes under test and test classes (and eventually more fine grained links between methods) and using them for a lot of purposes. Metadata can be collected from different sources: annotations on classes under test, instrumentation of test classes during tests’ execution (in future perhaps even from a dedicated user interface). These metadata can be used for various goals: first of all it becomes possible to run only the tests that stress just a particular set of classes (for instance the classes changed since last compilation), moreover a graphical representation of classes’ links can be derived. We have a Maven plugin, while Eclipse and Hudson ones are in our roadmap. Further cool idea around this is supporting generic tests to inject on existing production code (to verify commonly situation like don’t accept null parameters) and mock verification (are your mocks respecting the contract you have defined on mocked classes with your tests?)

Practically speaking in this beta release we have:

• a core supporting both annotation based metadata definition and instrumentation of classes during test execution to collect metadata. With annotation we support metadata on test stressing a whole hierarchy of classes/interfaces: you can define tests to be run against all implementation of your interface and SibillaTest take care of it Instrumentation do the magic of knowing which test is stressing particular classes and run only tests needed to validate last changed classes (i.e just compiled)
• a maven plugin (docs here) you can use to run test stressing only last changed classes.
• support of Junit tests (not yet testng or other frameworks)

In our wiki you can find some documentation of how to include Sibilla into your maven project, how to run it and documentation about our annotations if you decide to give TestedBy design by contract approach a try. Here you can also find a small project with trivial classes under test and test classes using SibillaTest to run them. Check it out and give it a try…it’s the key to understand the idea and to say “wow it’s cool!”

Have a look and send us comment through this blog or, much better, through our issue tracker and/or our forum. We have also an IRC channel #sibilla active on FreeNode for discussions and suggestions.

Artifact are available on maven central repository, see the docs too.

Some other post with detailed descriptions of instrumentation and annotation based metadata and their use and plans for next versions will follow during next days. Moreover we will discuss on forum and post here also plans of integration with the very cool Arquillian (I had a very interesting discussion about that with Aslak at last JUDCon)….

….stay tuned.

Of course if anyone is interested to join us and help (especially for eclipse plugin) please write us.

Stay tuned, join us and show some love around on the net!

http://sibilla.javalinux.it

(Forum – IRC – GitHub – Sample(github) – Maven Plugin(github) – @SibillaTest on twitter – issues)