Software Engineering
Unit Testing and Continuous Integration

Course Map

Agenda

What is black box/white box testing?

The Testing Process

Testing Process Goals

System Testing

Integration Testing

Components: ((A, B) -> C) -> D

Rerun the test for previous increments as well as the new tests - regression testing.

Component Testing

Object Class Testing

Testing Approaches

Test Case Design

Partition Testing

Equivalence Partitioning

Equivalence Partitions

Example:
A program specification states that the program accepts 4 to 8 inputs that are five-digit integers grate than 10,000.

Why Junit?

Install JUnit

Below are the installation steps for installing JUnit:

  1. download from JUnit.org or directly
  2. unzip the junit4.0.zip file
  3. add junit-4.0.jar to the CLASSPATH. For example: set classpath=%classpath%;INSTALL_DIR\junit-4.0.jar;INSTALL_DIR
  4. test the installation by running java org.junit.runner.JUnitCore org.junit.tests.AllTests.

Notice: that the tests are not contained in the junit-4.0.jar but in the installation directory directly. Therefore make sure that the installation directory is on the class path

Important: don't install the junit-4.0.jar into the extension directory of your JDK installation. If you do so the test class on the files system will not be found.

JUnit Annotation Types

@Retention(value=RUNTIME)
@Target(value=METHOD)
public @interface Test

A simple test looks like this:
public class Example {
  @Test public void method() {
    System.out.println("Hello");
  }
}

Simple Test Case

How do you write testing code?

The simplest way is as an expression in a debugger. You can change debug expressions without recompiling, and you can wait to decide what to write until you have seen the running objects. You can also write test expressions as statements which print to the standard output stream. Both styles of tests are limited because they require human judgment to analyze their results. Also, they don't compose nicely—you can only execute one debug expression at a time and a program with too many print statements causes the dreaded "Scroll Blindness".

JUnit tests do not require human judgment to interpret, and it is easy to run many of them at the same time. When you need to test something, here is what you do:

  1. Annotate a method with @org.junit.Test
  2. Methods annotated with @Test that are also annotated with @Ignore will not be executed as tests.
  3. When you want to check a value, import org.junit.Assert.* statically, call assertTrue() and pass a boolean that is true if the test succeeds
For example, to test that the sum of two Moneys with the same currency contains a value which is the sum of the values of the two Moneys, write: 
@Test public void simpleAdd() {
    Money m12CHF = new Money(12, "CHF"); 
    Money m14CHF = new Money(14, "CHF"); 
    Money expected = new Money(26, "CHF"); 
    Money result = m12CHF.add(m14CHF); 
    assertTrue(expected.equals(result));
}
If you want to write a test similar to one you have already written, write a Fixture instead.

Fixture

What if you have two or more tests that operate on the same or similar sets of objects?

Tests need to run against the background of a known set of objects. This set of objects is called a test fixture. When you are writing tests you will often find that you spend more time writing the code to set up the fixture than you do in actually testing values.

To some extent, you can make writing the fixture code easier by paying careful attention to the constructors you write. However, a much bigger savings comes from sharing fixture code. Often, you will be able to use the same fixture for several different tests. Each case will send slightly different messages or parameters to the fixture and will check for different results.

When you have a common fixture, here is what you do:

  1. Add a field for each part of the fixture
  2. Annotate a method. eg., setUp with @org.junit.Before and initialize the variables in that method
  3. Annotate a method with @org.junit.After to release any permanent resources you allocated in setUp
    For example, to write several test cases that want to work with different combinations of 12 Swiss Francs, 14 Swiss Francs, and 28 US Dollars, first create a fixture: 
    public class MoneyTest { 
    private Money f12CHF; 
    private Money f14CHF; 
    private Money f28USD; 
    
    @Before public void setUp() { 
        f12CHF = new Money(12, "CHF"); 
        f14CHF = new Money(14, "CHF"); 
        f28USD = new Money(28, "USD"); 
    }
}
  4. Once you have the Fixture in place, you can write as many Test Cases as you'd like. Add as many test methods (annotated with @Test) as you'd like.
  5. Sometimes several tests need to share computationally expensive setup (like logging into a database). While this can compromise the independence of tests, sometimes it is a necessary optimization. Annotating a public static void no-arg method with @BeforeClass causes it to be run once before any of the test methods in the class. The @BeforeClass methods of superclasses will be run before those the current class.
  6. If you allocate expensive external resources in a @BeforeClass method you need to release them after all the tests in the class have run. Annotating a public static void method with @AfterClass causes that method to be run after all the tests in the class have been run. All @AfterClass methods are guaranteed to run even if a @BeforeClass method throws an exception. The @AfterClass methods declared in superclasses will be run after those of the current class.

TestRunner

How do you run your tests and collect their results?

Once you have tests, you'll want to run them. JUnit provides tools to define the suite to be run and to display its results. To run tests and see the results on the console, run: 

   public static void main(String args[]) {
        org.junit.runner.JUnitCore.main("Example");
   }

Use this invocation for programmatic testing:

   public static boolean wasSuccessful() {
        Result result = org.junit.runner.JUnitCore.runClasses(Example.class);
        return result.wasSuccessful();
   }
You make your JUnit 4 test classes accessible to a TestRunner designed to work with earlier versions of JUnit, declare a static method suite that returns a test. 
   public static junit.framework.Test suite() {
       return new JUnit4TestAdapter(Example.class); 
   }
	
   public static void testIt () {
	junit.textui.TestRunner.run (suite());
   }

Expected Exceptions

How do you verify that code throws exceptions as expected?

Verifying that code completes normally is only part of programming. Making sure the code behaves as expected in exceptional situations is part of the craft of programming too. For example: 

new ArrayList<Object>().get(0); 
This code should throw an IndexOutOfBoundsException. The @Test annotation has an optional parameter "expected" that takes as values subclasses of Throwable. If we wanted to verify that ArrayList throws the correct exception, we would write: 
@Test(expected = IndexOutOfBoundsException.class) public void empty() { 
    new ArrayList<Object>().get(0); 
}

Examples

SimpleTest.java

ListTest.java

CollectionAllTest.java

Using JUnit 3.8 - without Annotations

Assuming that you want to test a class called Parser. The following are the general steps to use the JUnit framework to test this class:

  1. Write a class (let's call it TestParser) to test the Parser class. This class must extend the class TestCase which is defined by the JUnit framework.
  2. Create a constructor for this class, passing a name that is representative of the set of tests for this class as the parameter.
  3. Create a fixture. A test fixture is a set of sample objects that you want to (re)use during testing. For example, you might create a few sample source files for the Parser to parse. JUnit provides a setUp and a tearDown method to manage the fixture. Therefore, you can eg. create file objects in setUp to open the source files and release these resources in the tearDown method. The important thing to note is that setUp and tearDown will be called for every test that you run.
  4. Each test you perform is represented by the implementation of a method in the test class. For example, if you want to test whether the parser extracts the tokens correctly, you can implement a method called testGetToken. An important point is that each method name begins with the word test. This is necessary because JUnit uses reflection to know which tests to run. The collection of test methods you implement forms a test suite.
  5. In each test method you create, use the assertion mechanism provided by JUnit to compare the results of running the tests and the results you expected. This will enable you to create repeatable tests as well as saving you lots of time from visually inspecting the results.
  6. Finally, run the tests. There are two ways of running the tests. JUnit provides a TestRunner tool that can be invoked from the command line to run the tests and display the results. (there are both text and graphical versions). Alternatively, you can create a main method which invokes the TestRunner for execution. Internally, JUnit creates a test suite object that contains all the test methods of the testing class and execute each method. As each test is run, JUnit will provide feedback on whether the test run successfully, or the test failed, or an exception has occurred.

Example - JUnit

In this section, I will describe how you can use JUnit using an example. Take a few minutes to examine the following two Java files to see what the two classes are doing: Basically, there are two files: Course.java contains the class definition of a course. Each Course object contains a name eg. CS3214s and an integer grade which ranges from 0 to 100. Student.java contains the class definition of a Student. Each student has a name, a number as well as a list of course grades. You can add the grade that a student scores at a particular course using the assignGrade method and retrieve the grade of a particular course using the getGrade method. The following is the test file I wrote for the Student class (called StudentTest.java): 
import junit.framework.*;    // Note 1

public class StudentTest extends TestCase {    // Note 2
  // constructor - Note 3
  public StudentTest(String name) {
    super(name);
  }
  
  // method to test the constructor of the Student class
  public void testConstructor() {    // Note 4
    String student_name = "Jimmy";
    String student_no = "946302B";
    
    // create a new student
    Student stu = new Student(student_name, student_no);
    
    // verify that the object is constructed properly - Note 5
    assertEquals("student name wrong", student_name, stu.getStuName());
    assertTrue("student no. wrong", stu.getStuNumber().equals(student_no));
    
    // create some illegal inputs - Note 6
    try { 
      Student s = new Student("Jimmy", null); 
      fail("Constructor allows null student number"); 
    } catch (RuntimeException e) {}

    try { 
      Student s = new Student(null, "980921C"); 
      fail("Constructor allows null student name"); 
    } catch (RuntimeException e) {}
  }
    
  // method to test the assigning and retrieval of grades
  public void testAssignAndRetrieveGrades() {
    // create a student
    Student stu = new Student("Jimmy", "946302B");
    
    // assign a few grades to this student
    stu.assignGrade("cs2102", 60);
    stu.assignGrade("cs2103", 70);
    stu.assignGrade("cs3214s", 80);
    
    // verify that the assignment is correct
    assertTrue("fail to assign cs2102 grade", stu.getGrade("cs2102") == 60);
    assertTrue("fail to assign cs2103 grade", stu.getGrade("cs2103") == 70);
    
    // attempt to retrieve a course that does not exist
    try { 
      stu.getGrade("cs21002");
      fail("fail to catch non-existent course name");
    } catch (RuntimeException e) { }
  }
  
  // method create a test suite - Note 7
  public static Test suite() {
    return new TestSuite(StudentTest.class);
  }
  
  // the main method - Note 8
  public static void main(String args[]) {
    junit.textui.TestRunner.run(suite());
  }
}
Notes for the preceding code: To run the test suite, simply type: 
java StudentTest
and you're off doing your unit testing :-)

Exercise

The best way to learn JUnit is to use it. So, here's a small exercise you can do to get some hands-on practice. Let's say we now extend the Student class by adding a method to find the average grade of all the courses taken by the student. You can add the following piece of code to Student.java: 
// method to find the average grade of all the courses taken by the student
public float findAveGrade() {
    // if the student has not taken any courses, return 0 marks
    if (course_grades.isEmpty()) return 0.0f;

    // otherwise, find the average grade
    int total = 0;
    for (Enumeration e = course_grades.elements(); e.hasMoreElements();) {
        Course c = (Course)e.nextElement();
        total += c.getCourseGrade();
    }

    return (float)total / course_grades.size();
}
Your job is to write a method in StudentTest.java to test this newly created method. Give it a try and see whether you really know how to use JUnit ;-)

Packaging your source code

Organizing the project code into packages

All your sources go under src. Create meaningful package structure under the src directory. All your class files go under classes directory. To compile, just do the following.
java -classpath c:\classes -d c:\classes c:\src\........

Organizing the test code

Following the guideline of "Code a little, test a little, code a little, test a little,...", you'll soon find the amount of test code increasing rapidly as the project progresses. A recommended way to organize your test code is to put all the test code for classes of a particular package as a sub-package. Let's say I now want to organize the tests for my tutorial package:
  1. Create a subdirectory call test as follows: 
    mkdir C:\src\junit\tutorial\test
  2. Place the StudentTest.java file in the subdirectory just created
  3. Ensure that StudentTest.java has the following (you will need to recompile after editing the file): 
    package junit.tutorial.test;
import junit.tutorial.*;
  4. To run the test, type the following: 
    java junit.tutorial.test.StudentTest
In this way, you will find it easier to locate your test code and do regression testing.

 

References

  1. I need to remind you that what I have just covered is the minimum that you need to know to get started using JUnit.
    2. JUnit Cookbook
        A cookbook for implementing tests with JUnit.
    Javadoc
        API documentation generated with javadoc.
    Frequently asked questions
        Some frequently asked questions about using JUnit.
  2. The following documents still describe JUnit 3.8.

Test Infected - Programmers Love Writing Tests
    An article demonstrating the development process with JUnit.
JUnit - A cooks tour