Sibilla first release: experience a new approach to unit testing


abstract:

[...] collecting metadata set that represent links between classes under test and test classes (and eventually more fine grained links between methods) and using them for a lot of purposes. Metadata can be collected from different sources: annotations on classes under test, instrumentation of test classes during tests’ execution (in future perhaps even from a dedicated user interface). These metadata can be used for various goals: first of all it becomes possible to run only the tests that stress just a particular set of classes (for instance the classes changed since last compilation) [...] We have a Maven plugin, while Eclipse and Hudson ones are in our roadmap.

Full story:

A while ago I’ve written a post about a new approach to unit tests; it has been a quite read and discussed post and it gives the momentum for starting a new community discussing about the idea and working on a software project implementing this idea. The project was named TestedBy and hosted in google code.

That idea has evolved and changed and we have decided to change also the name of the project as a remark of these changes, calling it SibillaTest. Why Sibilla? Because, as you can read in this post and you can get trying our samples, Sibilla (italian name for Cumae Sybil) is always able to prophesy and predict which tests you need to run.

This post has been written to announce the release of a first beta version of SibillaTest and describe how our thoughts about the original idea have changed in these months and how they have been  implemented in SibillaTest.
I’m going to recall the main concepts of the original idea, If you haven’t read the original blog would be nice to take a look to it, to its numerous comments and also to the update I’ve posted few days after. If you are interested in all the genesis of the project you could also have a look to the discussions continued in our forum

Generally speaking what I’ve depicted in my old blog post was a system to keep annotations in classes under test pointing test classes (or in some case to specific test methods). Advantage of this approach were mainly 2:

• Design By Contract (viewing test as contract definition). The sugar here was the opportunity to put TestedBy annotation also on super classes (even if they are abstract or even interfaces) and inherit test annotations.
• Run only tests stressing a specific class. IOW run test stressing a class of your interest (i.e just compiled) giving you the confidence you aren’t breaking test suite without running the entire suite itself.

The main concerns about this approach was about code cluttering putting a lot of annotations in production code (of course you could have a lot of test stressing a class or even a method).

So we have changed a bit the focus from an annotation centric approach to a more general metadata
approach. The idea is to collect a set of metadata that represent link between classes under test and test classes (and eventually more fine grained links between methods) and use them for a lot of purposes.
The pluses of this approach are mainly two:

• Metadata can be collected from different sources: annotations , instrumentation of test during test execution, maybe in the future from a dedicated  user interface.
• Metadata can be serialized and/or used in a second step for various goals: run “right” tests of course will remain a central feature, but also
“graphical” representation, code navigation in IDE, more dynamic use of it

I’m trying to keep this post not too long trying to keep your attention focused on the usefulness of SibillaTest,  I’m going to describe in this the feature already implemented in SibillaTest. But, of course, we have a lot of ideas that will be developed in next future that we would like to discuss with the community and maybe have some contribution from the community. Here you have a mind map depicting a lot of these working areas to get you an idea. If you are interested in a more detailed description of them you  can find a page in our wiki describing it.

So, what is SibillaTest today?

aims at changing the point of view regarding test classes and classes under test. What we get is the focus being moved to the classes under test; from those classes, which of course are the most important ones of your projects, we obtain links to your test classes and test methods. In other words we define the classes and methods’ contract using tests, while also keeping track of tests that need to be run when a class/method under test has been modified. This is possible by collecting metadata sets that represent links between classes under test and test classes (and eventually more fine grained links between methods) and using them for a lot of purposes. Metadata can be collected from different sources: annotations on classes under test, instrumentation of test classes during tests’ execution (in future perhaps even from a dedicated user interface). These metadata can be used for various goals: first of all it becomes possible to run only the tests that stress just a particular set of classes (for instance the classes changed since last compilation), moreover a graphical representation of classes’ links can be derived. We have a Maven plugin, while Eclipse and Hudson ones are in our roadmap. Further cool idea around this is supporting generic tests to inject on existing production code (to verify commonly situation like don’t accept null parameters) and mock verification (are your mocks respecting the contract you have defined on mocked classes with your tests?)

Practically speaking in this beta release we have:

  • a core supporting both annotation based metadata definition and instrumentation of classes during test execution to collect metadata. With annotation we support metadata on test stressing a whole hierarchy of classes/interfaces: you can define tests to be run against all implementation of your interface and SibillaTest take care of it Instrumentation do the magic of knowing which test is stressing particular classes and run only tests needed to validate last changed classes (i.e just compiled)
  • a maven plugin (docs here) you can use to run test stressing only last changed classes.
  • support of Junit tests (not yet testng or other frameworks)

In our wiki you can find some documentation of how to include Sibilla into your maven project, how to run it and documentation about our annotations if you decide to give TestedBy design by contract approach a try. Here you can also find a small project with trivial classes under test and test classes using SibillaTest to run them. Check it out and give it a try…it’s the key to understand the idea and to say “wow it’s cool!” :)

Have a look and send us comment through this blog or, much better, through our issue tracker and/or our forum. We have also an IRC channel #sibilla active on FreeNode for discussions and suggestions.

Artifact are available on maven central repository, see the docs too.

Some other post with detailed descriptions of instrumentation and annotation based metadata and their use and plans for next versions will follow during next days. Moreover we will discuss on forum and post here also plans of integration with the very cool Arquillian (I had a very interesting discussion about that with Aslak at last JUDCon)….

….stay tuned.

Of course if anyone is interested to join us and help (especially for eclipse plugin) please write us.

Stay tuned, join us and show some love around on the net!

http://sibilla.javalinux.it

(Forum – IRC – GitHub – Sample(github) – Maven Plugin(github) – @SibillaTest on twitter – issues)

Google code issue tracker integration with version control using git commit template

I’m using google code for one of my open source project. And I think it is definitively a good code site.

All open source community code site have its pros and cons (I’ve tried a lot of them sourceforge, github, jboss.org, google code etc), and every one have some beautiful unique feature. But one of my most loved feature is definitively google code’s integration of issue tracker and version control. In a nutshell it permit to create, update and close issues just adding some keyword on your commit.

It’s great in general, but it’s really amazing if you are using a distributed Version Control. And I’m using git for local repository and git-svn to synchronize it with central subversion repository.
Imagine this situation: you are working with your laptop off-line (i.e into an air plane where also GSM connection isn’t available) and you find a bug or you have an idea for an enhancement. What do you do in this case? Of course take care of the bugged code, fix it, write a test for that and then commit it into your git local repository. What is the missing step in this work flow? Of course create an issue into your issue tracker system to keep community up to date, and don’t forget it if you can’t completely finish it now. At this point come very useful the google code feature permitting you to create, update, or fix an issue when you are committing a change.

There is only one problem with this integration: remember the structure the commit message have to fit and all (or at least the most useful) values you can assign to various fields. I’ve solved this problem creating 3 template for the commit messages and I use them with git. Using template during a git commit is very easy: just make a commit like this

git commit -a -t tempalteFile

Git will Use the contents of the given file as the initial version of the commit message. The editor is invoked and you can make subsequent changes. If a message is specified using the -m or -F options, this option has no effect.

Here you have my 3 template file:

NewIssue.tmpl

#commit comment here

#next line doesn't need any number. ssue number will be automaticaly assigned after commit
New issue
#A summary is always needed. Please complete next line
Summary:
#Uncomment the rigt Status
#Status: Accepted
#Status: Fixed
#Status: New
#Uncommet right Labels and/or add labels to the list (comma or space separate)
#Labels: Type-Defect Priority-Medium
#Labels: Type-Defect Priority-High
#Labels: Type-Defect Priority-Critical
#Labels: Type-Enhancement Priority-Medium
#Labels: Type-Enhancement Priority-Low
#Labels: Type-Enhancement Priority-High

#Comment text for the issue tracker goes here

UpdateIssue.tmpl

#commit comment here

#Complete next line with issue number
Update issue
#summary is rarely updated. If you need that uncomment and complete next line
#Summary:
#Uncomment the new Status you would get your issue. No uncomment status will keep the previous status
#Status: Accepted
#Status: Fixed
#Status: New
#Uncommet Labels and/or add labels to the list (comma or space separate) if you want update/add Labels
#Labels: Type-Defect Priority-Medium
#Labels: Type-Defect Priority-High
#Labels: Type-Defect Priority-Critical
#Labels: Type-Enhancement Priority-Medium
#Labels: Type-Enhancement Priority-Low
#Labels: Type-Enhancement Priority-High

#Comment text for the issue tracker goes here

FixIssue.tmpl

#commit comment here

#Complete next line with issue number
Fixes issue

#Comment text for the issue tracker goes here

The same thing should be possible also with svn and other SCM.

Hoping it could help some other people enjoying this google code feature developing open source code during their trips :)

Have fun!

Writing an irc bot for svn commit notification

Recently I’ve started working with a new team. Since we are based in 2 different site a bit far each other we are using extensively an IRC channel to communicate.

We are using subversion as SCM and we need to keep all members of the team up to date about svn commits. The solution I’ve put in place during an insomniac night in an hotel is a post-commit hook invoking an irc bot script written in perl connecting to the server and shotting a message there. Quite simple, and it is taking its goal.

In $SVN_REPOSITORY/hooks edit and make executable the file post-commit

#!/bin/sh

REPOS="$1"
TXN="$2"
SVNLOOK=/usr/bin/svnlook

# get last commit message
COMMIT=`$SVNLOOK log "$REPOS"`
USER=`whoami`

# call bot with arguments reposname, revison and commit message in one string
/usr/bin/perl /usr/local/bin/svn_irc_bot.pl "$USER $REPOS r$TXN: $COMMIT"

# all checks passed, so allow the commit
exit 0

then edit and make executable the file /usr/local/bin/svn_irc_bot.pl

#!/usr/bin/perl -w
#svn_irc_bot.pl

my $server = ""; #put here your address
my $port = 6667;
my $nick = "svn_bot";
my $ident = "svn_bot";
my $realname = "svn_bot";
my $chan = "#YourChannel";#put here your channel name
my $pass = "svn_bot";
my $svn_commit = $ARGV[0];
use IO::Socket;

my $irc=IO::Socket::INET->new(
PeerAddr=>$server,
PeerPort=>$port,
Proto=>'tcp') or die "DEAD!";

#print $irc "USER $ident $ident $ident $ident :$realname\n";
print $irc "NICK $nick\r\n";
#print $irc "PRIVMSG nickserv/@/services.dal.net :identify $pass\n";
print $irc "USER $ident 8 * :P erl IRC Hacks Robot\r\n";

print $irc "join $chan\n";

while(my $in = <$irc>)
{
if($in=~/004/)
{
print $irc "PRIVMSG $chan :$svn_commit \n";
last;
}

if($in=~/^PING(.*)$/i)
{
print $irc "PONG :$1\n";
}
}
close($irc);
#EOF

If user named “fooUser” make a commit on “fooRepository” for release 409 with a comment like “this is a fooComment” on irc channel you will get something like:

<svn_bot> fooUser fooRepository r409: this is a fooComment

I took the base of code here. Then I’ve modified it a little to get result I like.

[book-review] Pragmatic Thinking and learning

Some weeks ago I’ve been at my usual book store looking for some new book to read. I couldn’t resist when I see this catchy title: “Pragmatic Thinking and Learning. Refactor your wetware”…I had to buy it!

In fact the book have been even better than my expectations, giving me a lot of hint and idea around the day by day improvement and refactor of my brain.

As usual I wouldn’t bother you making just another complete book review (BTW you can find a great one here), but I would prefer to spot just 5 point that totally catch my attention and I’m applying (every day a bit more) in my work and life, hoping to add something to the discussion and stimulate your curiosity for the book:

  1. A greater awareness of R-mode to L-mode flow. Ok it’s not the first time I’ve heard about right vs left brain. In a nutshell (please keep in mind it is a terrible reduction to keep the post brief!!) R-mode is responsible of intuition and creativity, L-mode of verbal and logic. But Hunt’s explanation and hint to favour R-mode to L-mode flow are really great. There are some of his hint I’ve been already doing because they are natural for me (i.e. large use of metaphor and humor during explanatio ; step away from my keyboard when thinking to hard problem or pattern matching; describe problems to someone to get the solution). But now I have a more precise awareness that I’m working with these habit on R-mode to L-mode flow, and it make easier to understand what is happening into my brain and eventually try to drive this process.
  2. Mind map. I already know them. Perhaps because I’m italian: it’s quite impressive how much infos about you can find in italian language about on internet, much more than in english  (if you undertsand italian  just try this wikipedia page and follow external link). What Hunt’s book have added is the fantastic idea that simply make decoration on them expose in some way your R-mode thought. I’ve tried that and it’s quite impressive which results I’m getting.
  3. personal wiki to keep track and organize idea. Of course I already known wiki, but I’m trying to use it extensively to write down idea and knowledge of various kind. One more time I’m getting quite impressive results, getting much more ideas than ever and having a single place to correlate and link them. BTW I’m using zim desktop wiki…really a cool software for this purpose
  4. meditation to increase focus and attention. Well I’ve been already curious about. It just give the hint to try vipassana meditation. I like it, try to leave your prejudice (it’s nothing about religion!) and give it a try the results is tangible.
  5. keep the focus, avoid distraction in your environment. Hunt give some hints on how to avoid distraction starting from desktop workspace organization, email frequency and so on. The reason to avoid distraction is that our brain would spend a lot of energy and time (about 20 mins!) to change context and get back i.e to move away from IDE to answer an email and then switch back to programming activities. At a first look it seems impossible to limit the frequency of email check or just use a script to put a todo in your todo list without review the whole list before, but if you try that you will notice productivity will definitively increase.

They aren’t the only good points in the book, and I’m applying more hints, but for sure these 5 are the most effective for me. Moreover the second chapter of the book (you can read it here) is influencing a lot how I’m managing my team and considering expertise of people working with me and expertise of myself too.

The book of course contain much more than these, perhaps something that work better for you (no one of us have the same brain…luckily). Take a look to the linked book review, or much better leave your chair right now for few minutes, walk to your book store and pick up the book. Walking will re-activate  your Rigth Brain, and the book will do much more for it ;)

Thanks to Mr. Hunt for all the hints.

Open source is matter of ideas not of software

Hi all,
A long silence from me here.

I’m sorry for that but time is very very few. It is one of the reason I’m twitting more than blogging. Twitter require a lot lesser time than blogging. I’m loving it men! It make possible to write some notes without wasting a lot of time in a blog post and it’s very very effective in terms of feedbacks and information exchange.
But I’m not here to describe all the resons I’m loving twitter.
I’m here to say I have no time :) . A little counter sense wasting time in a blog post to say I haven’t time to waste :) . But counter senses are fun because drive us to think different, open the mind and ask our selves where is the real problem.
I have no time for what?
To realize my ideas. I have a lot of ideas (as all people have, I’m sure of that), but it seems I haven’t time to put them in practice. And it’s frustrating. But I decided t return to regular post some ideas here to don’t loose them.

Perhaps I’m not having a revulotionary ideas, I’m not saying E=mc2! But ideas are important per se and share ideas could be the key to give them a spin, an make them fly. Isn’t it the real spirit of open source movement? Open source isn’t IMHO a matter of software, but more generally is matter of ideas.

Innovation have a simple recipe: ideas +time to realize it. But innovation in a open source world could become ideas + some opportunity to realize and share them. One of my favourite blog is exactly this: enunciating ideas and realize them (Bob is sufficient lucky to have time itself to realize hi idea).
In the past I have some ideas became an open source project, and some stimulated some discussions and I’d like to implement it in near future as open source project (I haven’t yet got time, but I love too much TestedBy underlying idea, it is one of my first priority!).

Anyway I can’t switch off my brain just because I haven’t time to realize my ideas.
Could be more interesting and helpful for my curiosity to write down these ideas and share them for an open discussion. At least I’ll have take note of them to work on in some future, or maybe I’ll stimulate others’ ideas in virtuos circle. Let me remark one more time, that is the real underlying idea of open source.

Moreover I’m reading an interesting book “Pragmatic Thinking and Learning“. It say a lot of interesting things, but the one I’m totally sold out is that ideas will be lost if you leave it in a corner of your brain, it’s much better to write it down and eventually discuss it with someone. I would raise the bidding discussing my ideas not with someone, but virtually all ;)

Stay tuned, I’m going to post much more regulary, with really brief post, perhaps someone could be interesting for you and you would like to discuss about it in a virtuous circle ;)