A new approach to unit tests

What does “a new approach to unit tests” mean? Isn’t JUnit or TestNG enough and fine? JUnit (from here on I’ll nominate it only for briefness, but TestNG is the same for my discussion) puts test classes on focus and starts from them all tests. This means in fact that classes under test are considered only in test classes code, the only way a programmer can keep an eye on classes under test is using some kind of naming convention.

With older versions of JUnit you were forced to design your test classes extending a framework’s class and calling methods starting with “test”. So the convention has been to name test classes and test methods with words which could “connect” them with class and method under test. I think you would agree with me TestNG and Junit 4 give us a lot of freedom removing these requirements. Anyway the problem of logical connecting class and methods under test to our tests still remain and most test classes still respect that old convention.

But there are a lot of better ways to name classes and methods! Let me introduce you to Behaviour Driven Development (BDD). Please note BDD is not going to be the main focus of this article, anyway it makes a perfect wedding with my idea. So, let’s go into BDD in as fewer words as possible.

BDD is not only a new way to write tests but also a new form of design by contract

Let me start my introduction quoting the behaviour-driven.org:

BehaviourDrivenDevelopment grew out of a thought experiment based on NeuroLinguisticProgramming techniques. The idea is that the words you use influence the way you think about something

The whole idea is to ask programmers to concentrate on words used to describe a test class or method, because selected words will influence their point of view on the problem. In practice test we will write with a BDD approach will be much more concentrate on the behaviour of class/method under test then on the method itself. Of course this will change the way we test our code a lot, i.e. we will test a method multiple time to verify each behaviour is valid for it.

OK, what if I don’t believe on Neuro Linguistic Programming? Well, from a pure developer point of view, we are defining with our behaviour tests contracts of the class and methods. And moreover the tests results will be absolutely clear (i.e. “shouldAcceptNullValue fails” is a very clear statement also without complex reporting). Let me just provide a simple example to get you an idea:

@Test( expected = IllegalArgumentException.class )
    public void shouldNotPermitMethodNull() throws Exception {
    [..]
    }
    @Test( expected = IllegalArgumentException.class )
    public void shouldNotPermitEndPointNull() throws Exception {
    }
    @Test
    public void shouldInitWebParams() throws Exception {
    }
    @Test
    public void getHoldersResultShouldReturnHolderForRightParameters() throws Exception {
    }
    @Test
    public void getHoldersResultShouldIgnoreUnknowntParameters() throws Exception {
    }
    @Test
    public void getHoldersResultShouldIgnoreINParameters() throws Exception {
    }
    @Test
    public void shouldRuninvokeForOneWayMethod() throws Exception {
    }
    @Test
    public void shouldRuninvokeForMethods() throws Exception {
    }
    @Test
    public void shouldRuninvokeForMethodsApplyingMapping() throws Exception {
    }

Do you need a brief introduction about how BDD can be successfully applied to Java (original idea comes from Ruby’s RSpec)? Have a look at this excellent post.

So is BDD enough?

IMHO the answer is no. BDD is great and you should try it, but if you try to put it really in practice you will soon completely loose relations between classes under test and test classes. In BDD not only method names loose testXXX convention, but also test class names may loose their conventional name. Moreover you can have more than one test method insisting on the same method and/or class. For example the previous example isn’t perfect, maybe a specific test class to test getHolder method behaviour would be finer.

Do you need help? Here is my new project TestedBy

What about an annotation to mark classes and methods under test with a reference to test classes and methods? Not bad, isn’t it?

My thoughts started more or less from here, but there is much more than an annotation. I’ve loved so much this idea that I decided to start a new open source project providing testing tools that put class under test at the centre.
Continue this read and I’ll demonstrate you how this annotation and related tools may totally change your approach to tests.

In a nutshell TestedBy aims at changing the point of view regarding test classes and classes under test. What we would obtain is to put classes under test (the most important classes of projects) at the centre and link test classes and methods from them. A code snippet may help much more than any explanations:

public class TestedBySample {
 
    /**
     * @param args
     */
    public static void main( String[] args ) {
        TestedBySample sample = new TestedBySample();
        System.out.print(sample.add(1, 2));
 
    }
 
    @TestedBy( testClass = "it.javalinux.testedby.TestedBySampleTest", testMethod = "addShouldWork" )
    public int add( int i,
                    int j ) {
        return i + j;
    }
    @TestedByList( {@TestedBy( testClass = "it.javalinux.testedby.TestedBySampleTest", testMethod = "addShouldWork" ),
        @TestedBy( testClass = "it.javalinux.testedby.TestedBySampleTest", testMethod = "addShouldWork2" )} )
    public int add2( int i,
                     int j ) {
        return i + j;
    }
 
}

Oki, it’s nice, but can it really change your approach to unit tests? I think so.
How? At least in two different manner.

1. Design by interface and contracts

A famous adagio in software design says “Design by interface”. And it’s a sweet song for my ears. But of course you should “Test interface” too.

More formally “Design by interface” means defining interfaces of your API and then asking all implementors (maybe and a lot of times are different people or even companies) to implement these interfaces. Anyway all you can enforce is what a strongly typed language as Java can ensure: an interface implementation must respect its interface signature both in types and parameters. What you cannot enforce are behaviours. And of course it’s a big limitation in API design as this could drive to implementation with totally unpredictable behaviours.

Here comes to my mind Eiffell language and its design by contract (DbC) approach where contracts guide redefinitions of features in inheritance. There are a lot of tools providing DbC in Java, but my thought is that we already have a consolidated way to test contracts on methods and also finer behavior even of smaller piece of code: Unit tests.

With TestedBy, tests declared on an interface (or a superclass) can be run upon all implementers to verify they’re respecting the behaviour and contract defined in the super type. IOW the API designer not only provides the interfaces, but also a set of test classes verifying expected behaviour. Then every implementer would supply its own implementation and run the tests provided by the API designer against its concrete classes to verify they are respecting not only type safety but also beahviour/contract safety for which the API was designed. TesteBy here invokes a test defined for an interface passing to test class a concrete instance of class implementing the interface under test.

Here is an example on how this is achieved with TestedBy: we have added @TestedBy annotation on APIInterface and then provided a @BeforeTestedBy annotation to set the interface instance on which tests run. TestedBy will run shouldAddTwoAndThree both for APIImplOne and APIImplTwo, succeding on the first one and failing on the second

public interface APIInterface {
   @TestedBy( testClass = "it.javalinux.testedby.APITest", testMethod = "shouldAddTwoAndThree" )
   public int add(int a, int b);
}
 
public class APIImplOne {
   public int add(int a, int b) {
	return a + b;
}
 
public class APIImplTwo {
   public int add(int a, int b) {
	return a - b;
}
 
public class APITest {
 
  private APIInterface instance;
 
  @BeforeTestedBy
  public beforeTestedBy(APIInterface instance) {
	this.instance = instance;
  } 	  
 
  public void shouldAddTwoAndThree() {
	assertThat(instance.add(3,2), is(5));
  }
 
}

Of course I have kept the example simple, but TestedBy has some more annotations, for instance factory of classes under test can be specified when simple reflective invocation of no argument constructor isn’t enough.

2. Run test on current working class

You write a test to verify your code correctness. Moreover you are using unit test to ensure your changes isn’t breaking working code and mainly works of other people. How are you doing this? You make your changes and then you run all your tests against code you are modifying. Then at the end you run all tests to be sure you haven’t broke anything else.

What about having your IDE do the first step for you then? As it compile your modified classes it could (and IMHO it should) run tests against these classes. TestedBy makes this possible since you or your IDE could ever run tests against changed (aka compiled) classes. Here an eclipse plugin can do the magic of running tests insisting on you modified classes and verifing you aren’t breaking your test suite, not only your compilation, during code development. And this should not be too heavy, since the plugin could use TestedBy’s annotations to run only few tests insisting on your modified classes (or even methods).

Moreover running tests on a particular class under test you’ll get a clear report saying something like

"ClassUnderTest.methodUnderTest shouldThrowExceptionWithNullParameter doen't pass"

or even better

"Failure: methodUnderTest in ClassUnderTest doesn't throw exception with null parameter, but it should!"

Moreover you can of course totally break (if you like or need) conventions on class/method names and find, run and navigate your tests staring from your project classes.

Of course massive launches running all tests insisting on all classes will still be possible and your tests will be also runnable by JUnit since a TestedBy test is a JunitTest.

Some features

Well, let me detail some features I have in mind for TestedBy

1. The first one maybe the most trivial, but one of the most useful: within your ide you can navigate sources starting from the class/method under test. This feature comes out of the box with Eclipse 4.4, since it makes full qualified class names navigable also if they are inside a string. Of course it’s just a starting point, and a specific Eclipse plugin may be developed to navigate the code, construct the tree of test classes from a class under test, execute tests insisting on the open/changed class (see point 3 too) and so on. I’m not an Eclipse guru, so any contributions on this area are more than welcome.
2. You will find this annotation in you javadoc. Think how much advantage you can get if you are using a BDD approach, defining test methods like shouldNotAcceptNull() or shouldThrowsExceptionIfEmpty and so on. With a BDD approach you can in fact define and verify contracts and with TestedBy annotation in JavaDoc document it to your API users
3. You can run test starting from class under test and not test class. I have already said a lot about this in the previous chapter.
4. A design by interface and contracts. I have already introduced this idea in a dedicated paragraph.
5. Test class generation. A tool (ant, maven, or eclipse too) can use our TestedBy annotation to generate test classes. Read also point 6.
6. Ant and/or maven task/plugin to run tests starting from class under test. This tools will manage also round tripping (it’s important to ensure test classes and annotations don’t go quickly out of sync):
   • Reverse engeneering: starting from existing tests will add TestedBy annotations to class under test
   • Running tests from TestedBy annotations will verify they run all tests and that no annotation point to not existing test.
   • Generate (empty) test classes and methods starting from annotations.

Ok, that is not all, but I think it can give you the whole idea. I’m working on some other ideas behind this one, but I need to define them a little better before presenting them to the community.

Conclusions

So can this approach interest you? Let me known what you think about and share with your friends this post…more feedback I get, better refinement I can do.

Now the question is…which is the status of the project?
Well have a look to project homepage on javalinuxlabs. On googlecode you’ll find also some classes under svn. It’s not a really first implementation of the project…it’s more or less a proof of concept. But we are working hard to kick it out of the door soon.

Subscribe this feeds. I’ll make announcement here very very soon, and moreover other posts regarding evolutions of this idea.

Would you like to contribute? Have a look to this page to better understand in which area we need help.