Automated Testing

Hey there

Lately, I’ve been looking at automated testing. I went through a whole gaggle of tools, mostly FOS (Free/Open Source). Automated testing has been something that interested me for a while.

The project I’m currently working on involves an AJAX web app written in .NET 3.5. As such, I’ve had to contend with the difficulties of background activity after the page has loaded. Today, I want to share with you some of the lessons I’ve learned.

Firstly, you should know I’ve been working a lot on a tool called Selenium IDE. It’s a plugin for FireFox and is free and open source. You can take a look at it here if you’d like.

Hunting Wascally Controls

AJAX introduces a whole new fun. When your page loads, depending on the control, it will be displayed and available immediately or later as things are downloaded and executed in the background. Other times, it may be that a control is unhidden because of a user-selection, or because of some other event. In general, the page loading has no immediate bairing on the availability and visibility of controls! Yay!

Don’t get depressed. Selenium has many commands. Where a command acts on a control, there is likely a “waitFor” command available as well. So, for instance, if I trigger an event, I can “waitForElementPresent” before I inspect it, or “waitForTextPresent”. These commands are specifically intended for an AJAX web app.

Held Up In Traffic

Often, AJAX web apps download so many things in the background, executing scripts and so forth, that the page has not finished loading in the background. Sometimes, this just means a faster user experience not having to wait for things to download but other times it can mean being held up by background traffic.

Selenium often uses the “clickAndWait” command when navigating. This command is generally intended to click, i.e. instigate the navigation, and then wait for the page to load. Selenium, however, is not intending to wait forever and by default will only wait 30 seconds. With the project I’m looking at especially, 30 seconds is no where near enough time. As such, I use the “setTimeout” command to increase the duration that Selenium will wait for; I set it at 5 minutes or 300000 milliseconds when passed to the command. This tells Selenium that things will likely take longer and you get less timeouts.

Starting From The Beginning

Another thing I’ve learned is to always assume the script is not starting in the right place within the application or website. As such, I always have the first few steps navigate back to the beginning or starting point for all scripts. This means I don’t have to worry about where one script ends ready for the next script.

Absolutely!

The last point I’d like to share is to always use an absolute URL for the “open” command. The “open” command simply sends the browser to the correct location to begin testing. By using an absolute URL, it means I can just open a browser window and hit Start. Even if the browser is displaying the Quality in Software blog, the “open” command will take the browser to the right server ready to begin testing.

A Few More Comments

Just to finish, I’d like to make a few comments as to why I like Selenium IDE. Firstly, it is free and open source. There are a lot of arguments both for and against FOS software but the best argument for is that it’s accessible to more people; I like that. The second thing is that it uses a real browser to perform the testing; none of this pretend stuff where it simulates accessing the website or uses its own rendering engine. Although its recording is confined to FireFox, it is possible to execute using all the popular browsers. That makes sense since it means testing the scripting and functionality on the platforms they will be deployed to!

Anyway, I’ll leave it at that. Let me know of your experiences with Selenium or with automated testing in general.

Kind Regards
Glenn

Scope

Hey

One thing important in software is scope. Scope can be the difference between success and failure in some circumstances. Often though, scope can be a little confusing when testing.

In software, we often have two sources of expectations. The first is the explicit requirements as documented. The second is the implicit understanding of usability that we have learned throughout life. Sometimes, these two sources can compete and that’s when decisions need to be made. Today, I want to discuss how I approach navigating this stretch of water.

Scope is defined by the requirements of the software product; that’s a given. Anything outside of those requirements is outside of scope: however, much of the time, requirements are deliberately or otherwise written in such a way as to provide room for our experiences to shape the usability of the product. Herein lies the line between scope creep and usability. Often, these are the little things that would make requirements documents too long if all were documented.

We need to return to the first note we discussed: scope is defined by the explicit requirements of the product. This is the first test as to whether it is in scope or not. Really, there is anything to discuss on this point. If a customer pays 1 million googaflops for an application with red polka dots, then the client gets red polka dots; end of story! So if the requirements say the product behaves in a certain way, that’s the way it must work.

The next thing to consider is whether it is whether there is industry convention or standard that everyone else conforms to. This means you’re applying familiarity to the product as a method of usability. This is the second test.

Now, if you don’t find yourself being motivated by either of those two above tests, you probably are compelled only by your personal preference and it’s not likely to be what the client wants. This is important. We can sometimes feel like we have a vested interest because of how many hours we spend with these products but really, it’s up to the client.

So, next time you find yourself at a loss for what you should do, just think about these two rules: is it in the spec and is it convention. If your answer is No to both of these, then it’s out of scope.

Kind Regards
Glenn

Update

Hey all

It’s been a while since I last posted; sorry about that. Recent times have found me quite busy as test lead. Still, I’ve managed to come across a few things that are of interest.

Visual Studio 2010

Microsoft are already pushing the bandwagon out. VS2010 will introduce some exciting new features including requirements management. In their mission to democratize the software engineering landscape, they introduce a host of features around the area of managing requirements, test cases, test runs, and defects. Additional to the VS we all know, there ability to record requirements, perform traceability analysis and coverage analysis. Microsoft say the pricing model will stay largely unchanged. I haven’t played with the preview yet but you can find out more on Channel 9.

comp.RISKS

You know, I like to keep in touch with what’s happening in the world with regards to quality. This means both the successes and the failures. One source I found to be interesting and with a good deal of content is the comp.RISKS on Usenet. Check it out some time. It provides you with the latest on quality done well and quality done badly.

ISTQB Certification

International Software Testing Quality Board (ISTQB) Certification is an internationally recognized certification in software engineering. It’s not just testing but most of it is; the foundations can be studied by anyone who wants to know why we test. I’ve started pursuing this certification in a self-study format as there are no courses offered in Adelaide.

Anyway, I’m on the bench again for a short time, so I should be able to blog a lot more than I have. Again, sorry for the lag, life happens; I’m sure you know how it is.

Kind Regards
Glenn

Presentation

Well, I did say I was going to be busy. Yes, I’m still here and Quality in Software is still active. Hooray!

These last weeks have seen me feeling like my time has been pretty thinly spread. Nevermind, still, what’s happening?

Well, tomorrow I’m giving a presentation on Testing Components with Complex Associations. It’s a dive in to the world of Mock Objects, patterns, and other things like that. Of course, I’ve held back because my space is only 20 minutes. Quality is such a big topic.

Just lately, I’ve been in a test lead role. It’s been good. I’m surprised just how much I think about my test lead back at Motorola; Chris Lawn. Very dedicated person who I believe has had an impact on how I approach the role of testing.

In the not too distant future? Work, I’m real busy right now, more blogging, and of course one of my favorite professional development activities, reading books. They’re so invaluable!

I’ve been meaning to have a page to share book titles. That will still go ahead. I look forward to sharing book titles with you all as I’m always on the lookout for a great book on quality.

Anyway, for now, it’s back to work. I just got an updated spec I have been waiting for so I’m on the quality trail yet again.

Kind Regards
Glenn

More Than Testing

One thing I’ve been wanting to write about lately is what types of activities a modern software QA person does these days. As you can probably work out from the name of this post, it’s more than testing.

Now, before I go any further, if you haven’t read my About page yet, I work as a software consultant for a company called DWS. As such, I do development as well but most of what I do is quality related tasks.

Anyway, let’s look at what happens in modern software QA. Now, I’m not going to say this is everything that everyone does, but it is some of what some people do, especially this person. So, let’s dive in!

Testing

Ok, so you already knew that. But what does that involve?

• Unit Testing
• System Testing
• User Acceptance Testing
• Performance Testing

You probably know at least some of these but I will detail them further in later posts. Essentially, testing is where you get your blue ribbon for everything working or direction on what to fix.

Persona Development

Persona what? Persona development is where we establish different personas of people using software. This generally means we understand who will be using software in a holistic way.

Developing a persona involves identifying the needs, purpose, habits, etc of people to identify behaviours. This of course allows for better performance testing, acceptance testing, and even helps with defining requirements.

Requirement Traceability Analysis

Who ever said that a text editor should make coffee? Traceability analysis looks at requirements and traces them from concept through to the developed property.

Traceability analysis really helps with identifying priorities and stops scope creep. It’s also a great way of giving confidence to managers that a project has addressed all the requirements when used in conjunction with test reports.

Requirements Coverage Analysis

Although all your test cases may pass, it doesn’t mean all your requirements have been tested. Requirements coverage analysis examines requirements for test coverage at each applicable phase of testing. Focusing on requirements that haven’t already been tested can save big on time, money, and help your team look good.

Testware

Testware is all the tools used to test software. These may include load generators for performance testing or an automation framework. The QA function is also about developing testware as well!

Reviews

Code and document reviews are another function of the QA role. Reviews may include using tools such as the Orthogonal Defect Classification, checklists, code analyzers, etc.

Define Requirements

Yes, QA defines requirements as well, not just test them! Where there are issues of technical or organizational acceptance, the QA role can come in and help define requirements to encourage success as well as promote testability.

Advising

The QA role is often called upon for advice as their job often means they are familiar with a lot of the issues that can go wrong. Furthermore, testing efforts later on can be greatly reduced with QA input early on.

Metrics

Quality often involves metrics. Metrics are great at helping us to find where we need to focus our testing attention by keeping track of what has needed our attention in the past. Metrics can also help us with identifying where risk is increased just by metrics, such as where there is more code in a method or class. Testing is often performed in a risk-based approach, focusing on what poses the greatest risk.

Risk Analysis

Risk analysis is a task for many people working as a group. Without a QA presence, many risks may go unnoticed.

---

So, if you’re thinking of a role in QA or maybe you’re just interested in what the QA role involves, hopefully, I’ve helped you have a clearer understanding. It really isn’t just about testing, it’s about quality.

What sort of activities do you perform in your QA life? I’d be interested to know what other activities people perform to improve quality in software.

Kind Regards
Glenn

MVP Pattern

Hey!

I haven’t been blogging as much as I’d like to because I’ve been busy writing my Wedding Application. You can find more about it in my previous post but if you’d just like a quick synopsis, in my spare time, I’ve started writing an application around my coming wedding. Being busy with this is not all bad because it’s helped me to revisit a technology I learned about on-site. Let me tell you more…

If you don’t already know about design patterns, they’re essentially descriptions of what works most frequently in programming. They’re based on a lot of the work by Christopher Alexander with his architecture patterns. Now, before you go googling for “software architecture patterns by Christopher Alexander”, you might like to know his patterns are for buildings, suburbs, and other general living spaces: i.e., not software. Although Christopher Alexander’s work wasn’t designed for software, it’s inspired a generation.

Anyway, this post isn’t about all patterns, I don’t have that much spare time; it’s a very large topic. Instead, I want to talk about the pattern I’ve been using for the GUI of my wedding application. This particular pattern is called the MVP Pattern or Model View Presenter.

So, the whole idea behind this model is to decouple the UI controls from the UI logic and from the business model. It doesn’t do this by burying the UI logic with the business model, that would be a step in the wrong direction, but instead, it introduces a new layer. Let me explain more…

The MVP pattern is really about three parts. The first is the Model, the business model. This is where your business logic exists. It’s not unreasonable for the model to be a complete module containing all the non-UI stuff: business logic, persistence logic, etc, etc. It knows how to do stuff, like create records and apply business rules.

The next part of the MVP pattern is the View. This is what you get to see, literally. This is your forms and windows, your dialog boxes and your console screens. Now, what I need to mention here is that it is developed as an implementation of an interface. This is because it decouples so greatly, it allows you to swap and change your views: think of one application with one UI logic yet interchangeably displayed on Windows Forms, WPF, Console, ASP.NET, or Silverlight. The application isn’t changed, the IView (typically) interface is simply implemented differently.

Lastly, the MVP pattern introduces the Presenter. This is where your UI logic exists. It’s actually the driver for the UI and the main conduit between the view and the model. It’s responsible for updating data displays, formatting displays, and notifying the business model of user commands.

What I like most about the MVP pattern is that it makes your UI logic testable. Unlike UI drivers that send window messages to your application, MVP allows you to unit test your UI logic and that’s what I like about it. It’s really early in the development cycle, meaning less cost to fix any defects that are injected in the product at that stage.

You might like to look at more here. You will definitely want to know that Martin Fowler has decided to split the pattern in to two separate patterns called Supervising Presenter and Passive View. Definitely worth the read.

If you haven’t already, I really urge you to get in to design patterns. They really do help with software architecture and enough people know them that you can use them by name in conversation with other software engineers. If you’d like another great resource, try Do Factory, it doesn’t include MVP but it is good none the less and reflects the GoF (Gang of Four) design patterns.

Kind Regards
Glenn

Glenn’s Wedding App – Using Test Driven Design

Here at DWS, I’m “On the Bench”. This essentially means I don’t have any client work to do so I’ve been doing things for DWS internally as well as professional development. With this spare time, I’ve been practicing my skills in XML, .NET 3.5, and TDD (Test Driven Design).

I’m not going to pretend that I know everything about TDD, I don’t. What I will do, however, is tell you about my understanding of this fast way of designing good quality software that has fantastic code coverage as well!

I used to code in .NET 2.0 a lot at home and in TAFE (a technical college where I learned software development): as well as .NET 3.0 as part of my job here at DWS. Having said that, much of my work for the last 9 months or so has been testing, specifically leading UAT at a client’s site, so my coding skills are a little rusty.

I wanted to sharpen these coding skills but first I needed a project. Too easy! My upcoming wedding to my darling fiancee Jenna is a great inspiration for a project! So, I started a project that tracks wedding-related events: you know those events, engagement party, pre-wedding parties, organising the wedding, etc. My project will tell me which events are coming up next as well as how much time there is until the event.

Now, where does Test Driven Design come in to this? Well, it’s a great way of getting software written quickly with high quality, so that’s the method I have chosen to write my program. So, what is Test Driven Design?

Test Driven Design as I understand it is a process where you begin with a single concept. This concept says that any piece of software may either be complete or incomplete. Any defects are just where the software is incomplete. To demonstrate this, a test is written that confirms the software is incomplete and provides a proof for when it is complete. It starts with writing a test case that describes a little of the functionality of the final product.

Once the test case is written, the test case is compiled. At this stage, there is no production code and so the compilation of the test case fails. Let’s take a look at an example:

[TestMethod]
public void DoStuffTest()
{
  Something target = new Something();
  bool expected = true;

  target.DoStuff();
  bool actual = target.StuffDone;

  Assert.AreEqual(expected, actual);
}
Example 1: Initial Test Case

So, there’s a test. The idea here is that Something doesn’t exist, so the compiler fails to compile the test. Let’s add Someting.

public class Something { }
Example 2: Inital Production Code

Ok, so I know what you’re thinking, “He’s left out all the boring stuff.” Not at all, that’s it! There isn’t anything more. Let’s compile… Compilation Failed! Error: DoStuff() is not defined.

That’s right, we have stubbed the class but haven’t filled in the methods yet. Simply put, we didn’t do that because it wasn’t necessary to fix the problem at hand. That’s what TDD is about, fixing what is at hand to the very minimum required to satisfy the tests. As a result, you get very high code coverage, as in greater than 90%, often 100%, and the code is simple and testable.

We continue down this path until we finally get a compiled program: Compilation Successful! Now we run the test…

Test Failed: Expected: true, Actual: false

What happened? Let’s look at the code.

public class Something
{
  public void DoSomething()
  { }

  public bool StuffDone
  {
    get { return false; }
  }
}
Example 3: Production Code - Meeting the minimum requirements of the test case

Ah, this is easy to fix:

...
get {return true;}
...

Now it passes! Fantastic! But now we have only proven that we can know when something has happened, what about before it’s happened? Let’s change our test:

[TestMethod]
public void DoStuffTest()
{
  Something target = new Something();
  bool expected1 = false;
  bool expected2 = true;

  bool actual = target.StuffDone;
  Assert.AreEqual(expected1, actual);

  target.DoStuff();
  actual = target.StuffDone;
  Assert.AreEqual(expected2, actual);
}

Example 4: Updating the Test Case

Now, see the change? We check target.StuffDone before we call DoStuff() and expect it to return false. Of course it should be false, we haven’t “done stuff” yet! Let’s compile: Compilation Successful. Not a problem, it only exercises existing production code. Let’s run the test:

Test Failed: Expected: false, Actual: true.

Ok, so what happened? Remember how we made our code work by statically returning true from the StuffDone property? We’re going to have to be a little smarter now.

How can we get the test to pass with the simplest change to the production code? Really, the best way is to change the result of StuffDone when we call DoStuff(). Because the first Assert statement expects StuffDone to be false, that’s what we’ll initialize it to satisfy that test.

// Something that does stuff.
public class Something
{
  private bool mStuffDone = false;

  // Do something.
  public void DoSomething()
  { mStuffDone = true; }

  // Has something been done? Return true if yes.
  public bool StuffDone
  {
    get { return mStuffDone; }
  }
}
Example 5: The Completed Production Code

Easy! Now, I know what you’re thinking, it’s easy because it’s just some simple example. Well, yes, that’s right but it’s also how TDD works. Everything is broken down in to simple bite-sized pieces. Now, think about what we have here. We have a fully tested piece of production code, a simple design, and a test that proves that we have delivered on the requirement described by the test case: let’s just say, green ticks are hard to argue with.

You’ll likely want to refactor your code at some stage but at least you won’t be refactoring too early and you have test cases to check that your refactoring doesn’t break anything. Try writing test cases around the requirements in your projects. I find it easy and rewarding.

Kind Regards
Glenn

SharePoint Webpart Test Cases

When I don’t have a lot of work to do here at DWS, I often find the time to look around for testing topics. Here’s a resource I found today: http://msdn.microsoft.com/en-us/library/ms916830.aspx .

Dated 2004, it’s old but it should be a good starting point for building your own SharePoint test suite.

Kind Regards
Glenn

Test Case Template

Here at DWS in Adelaide, I’m known as the test guy. Most of my industry experience has been in software testing from when I first started at Motorola to where I am now; I have done .NET development too if you’re wondering but mostly, it’s been testing.

Anyway, yesterday, I was asked for a test case template. Interestingly, I didn’t have one. Now, you might be curious as to why I might not have such a simple template: well, because my work has largely been with Quality Center and I didn’t think configuration settings were in the same league. Anyway, I ended up whipping one up shortly after being asked; they’re easy to write.

Now, I hear you say you want to know what information is important to include in a test case! Here’s a nice list of information you should have as a minimum:

• Header
  • Test ID
  • Test Name
  • Creation Date
  • Description
  • Prerequisites
  • Requirements Covered
• Test Steps
  • Step Name
  • Task
  • Expected Result

So, there you have it. Remember, you almost always want to get someone’s signature as well as who wrote each test case: it leads to someone being responsible.

Kind Regards
Glenn