Nevertheless, I attempted to implement the same few tests I wrote for Jubula, and you can view the results in the piece of code below:

package de.johoop;

import static org.junit.Assert.*;

import org.eclipse.jface.bindings.keys.KeyStroke;
import org.eclipse.swtbot.eclipse.finder.SWTWorkbenchBot;
import org.eclipse.swtbot.swt.finder.keyboard.Keystrokes;
import org.eclipse.swtbot.swt.finder.utils.SWTBotPreferences;
import org.eclipse.swtbot.swt.finder.widgets.SWTBotShell;
import org.junit.Before;
import org.junit.Test;

public class FirstSwtBotTest {

  private SWTWorkbenchBot bot;

  @Before
  public void setUp() throws Exception {
    bot = new SWTWorkbenchBot();
  }

  @Test
  public void mailToolbarButtonShouldOpenMessageDialog()
      throws Exception {

    bot.toolbarButton(1).click();
    assertThatCorrectDialogIsDisplayed();
  }

  @Test
  public void openMessageMenuEntryShouldOpenMessageDialog()
      throws Exception {

    bot.menu("Open Message").click();
    assertThatCorrectDialogIsDisplayed();
  }

  @Test
  public void keyboardShortcutShouldOpenMessageDialog()
      throws Exception {

    SWTBotPreferences.KEYBOARD_LAYOUT = "EN_US";
    bot.activeShell().pressShortcut(
        Keystrokes.CTRL, KeyStroke.getInstance("3"));
    assertThatCorrectDialogIsDisplayed();
  }

  private void assertThatCorrectDialogIsDisplayed() {
    final SWTBotShell openDialog = bot.activeShell();
    assertEquals("Open", openDialog.getText());
    assertTrue(bot.label("Open Message Dialog!").isVisible());

    bot.button("OK").click();
    assertFalse(openDialog.isOpen());
  }
}

It’s a set of three tests opening the “Open Message” dialog via toolbar button, keyboard shortcut or menu entry. Simple.

At first glance, this looks neat, especially if you’re a developer: few lines of code, all directly in Java, using the already familiar JUnit framework, well supported from within the Eclipse IDE.

Nevertheless, here’s my list of things I don’t like about this:

1. Let’s begin with a minor nitpick: I’m using a German keyboard layout, and SWTBot complains that it can’t find my layout. This means that I either have to somehow create one (which might be easy, I haven’t checked), or I have to set my keyboard layout to the US one (it doesn’t matter for my simple Ctrl-3 test).
2. Next, I can’t write these tests in Scala for some reason (I’d really like to use Scala for my tests, so very much more concise): In Scala, the following line:

   new SWTWorkbenchBot().toolbarButton(1).click()

   fails with a compile error in Eclipse: “Access to protected method click not permitted because enclosing class object ScalaUser is not a subclass of class AbstractSWTBot in package widgets where target is defined”. – What’s that?

   If you ask me, it’s somehow caused by the SWTBotToolbarButton having an abstract public method click() returning an SWTBotToolbarButton, and inheriting from AbstractSWTBot<T>, which has an abstract protected method click() returning an AbstractSWTBot<T>.

3. Also, I don’t want to use JUnit for system tests. JUnit is a great framework if all you do is unit tests. If you’re testing on the level of integration or system tests, however, TestNG is a way better fit if you ask me. The execution model of TestNG is a lot more flexible than that of JUnit (even if you use decorators and stuff there). For example, have a look at the data providers of TestNG. Unfortunately, there’s no TestNG integration for SWTBot.
4. Finally, here’s what I’d call a showstopper. Have you noticed how I say toolbarButton(1) in my code instead of the toolbarButtonWithTooltip("Open Message") you’d probably expect? That’s because using the latter method will require another dependency in your plug-in project. That dependency will have to get injected into your test object, too (the RCP mail application in my case). And unfortunately, it will add toolbar buttons and keyboard shortcuts to your test object!

   In my case, this leads to Ctrl-3 suddenly not working as expected anymore: The tests have changed the behaviour of the test object! I absolutely can’t have that happening.

For me, these four reasons unfortunately mean that SWTBot is not a viable tool for automating UI tests. I would really like to use it, especially combined with Scala, for quick and easy developer UI testing; I’d be quite happy if you could refute any of my four reasons above.

I’ve still got hope that maybe I’m just too stupid to use SWTBot correctly.

Before starting up Jubula today, look for the “AUT Agent” service and start it. The AUT Agent is our connection to the “real world”. It takes commands from Jubula and executes them on the application under test (you can even run Jubula and the agent on different boxes for distributed testing). On Windows, you might have to start the agent via “Run as Administrator”.

Next, start Jubula and open your project. Now, look for the “Connect to AUT Agent” toolbar button and press it in order to establish the connection to the agent we started before:

Next, start the test object via the “Start AUT” toolbar button. You should then see its ID listed in the “Running AUTs” view:

The application under test is running now, and you can view it on your screen. However, we still can’t run any tests, as the logical components we specified last time are not yet mapped to their real component counterparts within the RPC application. Let’s fix this by starting the “Object Mapping Mode”, again from the toolbar:

Now activate the test object and move the mouse around within it. Notice the green borders around the various components? That’s the Jubula Object Mapper showing you the components it has detected. In order to add any of these to your object mapping, you have to press Ctrl-Shift-Q.

You can collect all components at once like that if you want to, but I’d advice against it because it can get crowded and confusing quickly in Jubula when you do that. Instead, start with the few components we currently require for our tests to run. In my case, this would mean capturing the mail icon in the toolbar, and – after clicking it – the dialog message text and the dialog “Ok” button.

When done, switch back to Jubula. You should see a new editor called “OM / <something>”, already containing your logical components on the left side, and the newly recorded test object components on the right side:

All that’s left to do now is to map the logical to the technical component names. You can do this by dragging each logical component name over the respective technical one and drop it there. The editor will change into this:

Aaand… – we’re done with the object mapping. Your “Problems” view should be empty. This means that we can run our tests, at last! – Ok, nearly. First, we have to stop the object mapping mode again. I’m sure you’ll find the respective toolbar button by yourself.

Select your “System Test” test suite in the “Test Suite Browser” view and run it:

Hopefully, your automated tests will run now. – You can switch into the Test Execution afterwards and check the results of the test run:

Oops, my tests failed. What’s the matter?

I’ll go check the screenshot Jubula captured when the “wait for window” action failed. It shows a second message tab being open, and zero “Open Message” dialogs. Hmmm…

Oh right, I accidentally mapped the wrong toolbar button to my logical “Open Message” component… – I should have mapped the mail button to it, instead of the green plus sign.

I’ll correct that (for some reason, I had to restart Jubula before it accepted my fixed object mapping), and maybe you’ve got a few things to correct, too, and then we’re done!

You’ll notice that there will always be some debugging of your test specifications going on here, unfortunately, but that’s to be expected: Specifying tests is like developing in a lot of ways. Also, finding out what exactly made your tests fail can sometimes be harder than it looks, screenshots or not.

Alright, let’s summarize what we’ve achieved up to here:

• In the first part of my blog series, there was some configuration to do in order to get the application under test to cooperate with Jubula. A one time effort.
• In the second part, we specified a few tests. This is the part you’ll have to put the most effort into. You absolutely want to get your test specifications right: Clear, concise, complete, robust, maintainable. This is always difficult to get right. However, in my humble opinion, Jubula gives you a lot of good tools for structuring and refactoring your test specifications to simplify this.
• In the third part, we mapped the components and executed the tests. Yay!

This should be enough about Jubula to get you started with your own experiments. I hope I tempted you into trying it out! – It’s a tool with a lot of potential, at the least.

Before using Jubula in a real-world project, there’s still a lot of things to clarify, of course, like:

• What kind of database should you use with Jubula (it defaults to an embedded H2)?
• How well does it cope with multiple team members specifying tests at the same time?
• How to integrate it into an Ant/Maven/SBT/Gradle/Buildr/whatever build?
• How to run it on an Continuous Integration server?
• How well do the exported test specifications play together with version control systems?
• … and probably more stuff like that…

But it’s a start.

What’s that about specifying tests? – Can’t you just capture and replay them, perhaps? – Well, no, not with Jubula, and not with me, either! The reason is that test scripts created via Capture/Replay have lots of problems with maintainability and robustness. And when you try to refactor Capture/Replay scripts into a good automation, you’ll notice that it’s not easier than writing those scripts from scratch with modern test automation tools like Jubula.

Also, Capture/Replay is only possible after the test object already exists (which is normally too late in the lifecycle of a project), and it misleads you into adapting the tests to fit the current behaviour of the test object.

So, let’s start up Jubula and begin specifying a few tests. After the welcome screen, Jubula greets you with the following prim perspective:

If you look at it very closely, you’ll find a small hint about what to do next within the Problems view:

However, it doesn’t tell you how to create a new project. Surprisingly, you won’t find it in the File menu, but in the Test menu (Test/New…):

Choose “rcp” as toolkit, give your project a good name and click the “Next” button. Now you’ve got to tell Jubula about your test object (Jubula calls it “AUT” for “application under test”):

Choose a name for your test object and ensure, that the “rcp” toolkit is selected. Don’t worry about AUT IDs for now. Press the “Next” button for the last step: configuring your test object:

Give your configuration a good name, and set an “AUT ID” for it, too. But mainly, select the executable of your RCP application, then click “Finish”. – At last, we’re done with creating the test project.

Note (in the top right) that we’ve already got the correct perspective open: the “Functional Test Specification” perspective. When specifying tests in a top-down way, you’ll at first mostly work with the “Test Case Browser” view (bottom left), with the editor pane, of course, and with the “Properties” view (top right).

To begin with, let’s create two levels of abstraction for structuring our tests (for complex applications, you probably want more than two): “Use Cases” and “Basic Interactions”. Create these as new categories in the Test Case Browser. Notice how the “Use Cases” gets sorted (alphabetically) after the system library test cases of your toolkit (“unbound modules…”). If you don’t like that, you can always prefix your categories.

Let’s test the opening of mail messages first. I created another category for that under the “Use Cases”. What are our use cases for opening mail messages in that simple application? Well, we can open a message by menu, by keyboard shortcut or by toolbar button. And afterwards, we can check whether the “Open Message Dialog” appears correctly. That’s it. Here are my test cases:

Alright, now we can open our first test case “Open Message via Menu” in an editor:

A lot of empty space to fill… – what are the basic interactions this test case consists of? I’d say we have to find and select the correct menu entry first, and then check for the dialog next. So let’s create these two basic interactions (I already created the basic interactions for the other use case level test cases, too):

Now that we have declared the required basic interactions, we can drag and drop them into the editor, and even rename the references afterwards in the Properties view (not very necessary in this simple case, where there are no parameters, but in general, I’d always do that):

Up to now, we’re still on an abstract level without any toolkit interactions. But that’s going to change when we specify how our basic interactions work.

Let’s start with opening the “File” / “Open Message” menu entry. We’re looking for a way to select a menu bar entry. So, let’s open up the “unbound_modules_concrete” tree in the Test Case Browser, and then look under “Actions (basic) / Select / Menu Bar / ub_mbr_selectEntry_byTextpath”. That should be the right one. Drag it into the Test Case Editor of your basic interaction.

Nearly done. Now give it a good name in the “Properties” view, and set the two parameter values. The value for TEXTPATH should be /File/Open Message*, and the value for OPERATOR should be simple match (or you could do the same with matches and a proper regular expression in TEXTPATH).

Now your test case should look similar to this:

Next let’s specify how to check the dialog: First, we want to wait until an “Open” dialog appears, then we want to check that it displays the correct text, and finally we want to close it again by pressing the “Ok” button. This means, that for this test case, we need a sequence.

So, find the basic concrete action “Wait / Application / Wait for Window / ub_app_waitForWindow” and add it to your test case. You now have to configure a few properties again, namely TITLE, OPERATOR, TIMEOUT_IN_MILLISECS and DELAY_AFTER_VISIBILITY:

Then, find the action “Check / Component with Text / ub_ctx_checkText” and append it to your test case. Enter the parameter values as always. But now, there’s something new to do. We have to tell Jubula the text of which component we want to check, and we do that in the “Component Names” view (in the lower right). In there, under “New Name”, just enter an identifier that makes it easy for you to remember which component it is (I chose “open_message_dialog_text”):

Later, when you want to execute your tests, you’ll need to map these component identifiers to the physical components of your test object. This is called object mapping, but we’ll handle that later on.

Alright. Now, only the button click is missing. It’s in the basic actions category, too, under “Click” / “ub_grc_clickLeft_single”. There’s no properties to set this time (except for the reference name), but again, we have to tell Jubula that we want it applied to the “Ok” button of the dialog by telling it how we want to call our logical component (I chose “open_message_dialog_ok_button”):

And finally, we’re done with our first check.

I’ll leave specifying the two other basic test cases “Open message via toolbar” and “Open message via shortcut”, and filling in the two other use case level test cases to you, as an exercise.

When you’re done, we can collect the fruits of our work in a test suite. Within the “Test Suite Browser”, create a new test suite called “System Test” (for example), and open it in an editor. Now drag all your use cases into the test suite:

Don’t worry if you see problems in the “Problems” view. They probably tell you that you haven’t done the object mapping yet. – We’re done specifying our tests, but we can’t execute them yet!

And that means that there’s more left to do in my next blog entry…

I won’t go into comparisons with other tools in this post, but in short, imho, though there may be better tools for web testing, though there may be alternatives for Swing testing, though there may be better commercial tools for testing RCP and SWT, when you’re looking for free tools for testing RCP/SWT, I think you should start with Jubula.

Unfortunately, working with Jubula turns out to be a bit unwieldy at first. – So I’ll try to give a detailed step-by-step guide to your first GUI test automation with Jubula here, using a simple Eclipse RCP sample project as the test object.

So, let’s start with setting up the test object first: Open up Eclipse and create a new plug-in project (File/New/Project…/Plug-in Development/Plug-in Project):

Next, tell Eclipse that it’s a rich client application:

And finally, choose the “RCP Mail” template:

Now, let’s see what we’ve got in our workspace:

Hey, wait, what’s that, warnings about raw types? – We can’t tolerate that. I’ll quickly fix these by turning the List into a List<TreeObject>:

That was easy. Actually, why do they have these warnings in their sample code at all? – Anyway, what’s next? Right, let’s try if it works:

Looking good. This should be simple enough for our first steps, yet complex enough to play around a bit. – However, for exporting the application, we’re missing a product configuration (File/New/Product Configuration):

Alright, now we’re set for exporting the RCP application as Eclipse product (File/Export…/Eclipse product):

Finally, we’re done with preparing our test object. Or are we?

Actually, we’re not, unfortunately. The reason for this is that sadly, the Eclipse JFace/SWT libraries haven’t got any easy way to access/instrument them for automation on the fly (unlike Swing).

So, Jubula has to somehow hook itself into our RCP application, and we have to help with that: Look for the archive rcp-support.zip in your Jubula installation directory and extract it into your product directory. This will add a Jubula instrumentation plugin to your product.

However, that’s not all. You’ve still got to activate that plugin, and you can do that by editing the configuration/config.ini file in your product directory. There’s a long line in that file listing the osgi.bundles, and you have to add the Jubula plugin at the end:

Phew. Now we’re really really done with preparing the test object. Of course, for real projects, you’ll probably want to automate all these preparations in your build.

Let’s start up Jubula and do some testing! – Or wait, actually, let’s do that in my next blog post.