Test automation is one of my favourite subjects as a software tester. I have been planning for long to write a series – “Notes on Test Automation” – through which I can document and share my learning from designing and implementing test automation frameworks as well based on the literature I read for the purpose and test automation tools/frameworks that I analyse & use.
Initially it was supposed to be a series of blog posts, but later I thought to get it published in a testing magazine.
I approached Jose Diaz, the editor of Testing Experience magazine with the idea. He showed a lot of interest in publishing this as a series.
The first “episode” of the series has been published in the Q3’10 edition of Testing Experience magazine as – Test Encapsulation – Enabling Powerful Automated Test Cases.
I have copy pasted some contents from the published article in this blog post to give you an overview of what the series is about and what Test Encapsulation is about.
Introduction about the Series – Notes on Test Automation
Test Automation Frameworks (TAFs) do not get the treatment in terms of design that they deserve. Spending time on understanding the precise requirements and then designing/choosing the framework becomes much more important when a general purpose framework is required. Such a situation is unavoidable when your organization is looking at the possibility of a single underlying TAF that could be employed by multiple teams involved in strikingly different products; their testers are conversant with different programming languages, and they choose or needto employ different tools to support testing.
This series is focused on the technical aspects of designing a scalable, general-purpose, object-oriented TAF. This in turn means that the contents would mainly comprise concepts that directly translate into an implementation in an object-oriented language. One could choose to write the same thing in the traditional programming way, but there would be further work required on part of the reader to translate the design.
The series will not attempt to answer the questions of why to automate, when to automate, which tests to automate, automated tests versus manual tests, ROI of automation, etc. There are quite a few books available on the subject of test automation, and these topics are discussed at length in them. So, if the reader chooses to follow this series, it is assumed that he/she already understands the importance of test automation. This series focuses on how to automate, whereby the “how part” is not about using an existing tool or framework, but rather about designing one.
The name of the series “Notes on Test Automation” purposefully does not use any adjectives to indicate how universally efficient or effective its ideas are. This series will evolve over time, and I am going to share practical nuggets of test automation knowledge with you that I have learnt from my practical experience. We will occasionally strike out a sentence or two which revokes our initial understanding, and you might find me challenging my own ideas from the previous or even from the same article.
The Problem of Dumb Tests (or Why We Need Test Encapsulation)
In the design of TAFs, most of the important information that a test requires is kept outside the TAF, as many times such frameworks are designed using traditional programming rather than employing object-oriented programming. A simple example is grouping tests into platform-based folders, after which the scheduling logic copies in only the tests meant for a test execution machine based on a specific platform. There are multiple problems with this approach. One is redundancy, multiple copies of tests across folders. If you think about it, we could have a common test platform and then specific platform folders, but my question is how many such “commons” would you have? Even if you havethe patience, what happens when a test gets modified and is not meant to be run on Platform A? How do you know to which folder you should go to change it? So, there should be an external means to map tests to folders! By not making the information available within the test about which platforms it should (or should not) run on, you have the following situation:
- You need a complex folder structure, which is virtually impossible to maintain as the number of supported platforms grows
- You need an external means to map tests to the above folder structure.
- If the test writer is not the test framework administrator, it means the person who requires this is different from the one who implements it.
- You have a “dumb” test, which does not know on which platform it can run. So, if anything goes wrong, you have an incorrect test case that could crash the system or, at worst, lead to an incorrect result.
So far, we have only talked about the platforms. We have not talked about categorization of tests, priority values associated with tests, known bugs, API version checks. Can you see the complex and impossible folder structures that these options would lead to if kept outside the test?
This article focuses on the concept that a test is meant to be the most complex part of the framework in terms of its power and flexibility. Test encapsulation is at the heart of building a general purpose framework, which can be used by testing and development teams as a common test automation platform for whitebox and black-box tests. To know about what exactly is meant by test encapsulation, its approach and benefits, keep reading!
Not clear about the exact meaning of “encapsulation”. Is it same as simply hinding?
if put all that a test case needs (metadata, configuration, logging settings, execution properties, post processing instructions) in a separate file and have execution logic process this file and run the test — Have I achieved encapsulation?
Shrini
@Shrini,
Information hiding is the principle behind encapsulation. Encapsulation is a term widely used in Object oriented programing world, which you can consider as a technique for information hiding. Encapsulation is the way information hiding is achieved for modules/objects. It’s about a combination of external interface in the form of “public” methods and object properties & internal methods which are not exposed publicaly.
If the data read from the input files mentioned by you is populated outside of the Test object, you haven’t achieved test encapsulation.
If you place the data in a text/XML/CSV file and (for example) in the constructor of the Test class you populate object variables by reading from this input file, it is in line with test encapsulation.
Test class would expose a method e.g. run() which would be called from outside the test object. This method would have all the information it needs to execute based on the Test properties already populated.
The above means the interface for all tests would remain the same and every time the outside logic would call run() method irrespective of which “Test” it is dealing with.
Rahul,
Thanks… that helps … another naive question related to OO – How doe you distinguish between encapsulation and abstraction.
If Encapsulation is information hiding (for the purpose of making the life simple for the users of the thing being encapsulated) – what is abstraction?
Abstraction to me is process of representing idea and relate it to a group of real/concrete objects.
File system, String Class/Library, TCP/IP framework, certain types of code generators, JVM etc are abstractions to hide the inherrent complexities of the real object that they “attempt” to abstract so that users can work with them easily.
How do you now compare and contrast the process of abstration and encapsulation in the light of test automation .. some or most of the discussion is in the OO design realm — but we are talking about test automation … hence aski’n you …
shrini
@Shrini,
Following is my understanding of these terms (Please validate the same against official OOP resources):
Representing a real world test case with a TestCase object is an abstraction.
This object would needs to solve the purpose of the real test case. For the same we would need to think of how the “state of test” can be represented and different actions that it can perform. Some of these actions should be available externally so that other objects can make a TestCase object perform some of these actions.
Encapsulation is about implementation of this abstraction by providing its structure (object/class variables) and its behavior (external and internal methods) in a language of choice. As per Grady Booch – “encapsulation serves to separate the contractual interface of an abstraction and its implementation.”
For real world implementation, you would think of an “abstract TestCase class” that has abstract methods like setup(), run(), cleanup(), report(). Any new test case would inherit from this base class and override all these abstract methods to provide concrete implementation of a unique test.
Nice Post. I have one question for you though. Do you think, it is possible to use more than one OO language (to tie components together & to describe tests) in TAF?
Rahul,
The article is a very good read. Good to know test automation development approaches is actively looking into object oriented approach. One of the problems we’ve encountered in test automation development is the request from testers to incorporate newer even more complex test scenarios even after the development of automation is complete. The newer test cases even are in most cases incremental changes over an existing test case that’s already automated. One of the ways we’re able to solve this situation is thorough ‘extending’ a test case using object oriented concept. By approaching the test automation problem as object oriented design problem, we were able to model application under test as business classes and associated methods. The first and most noticeable benefit we’re able to achieve is to hide the details of the GUI within business level classes and this has lessened our test case maintenance. I’ve written a blog post about this approach. http://elusivebug.blogspot.com/2009/05/test-automation-approach-object.html
Regards
Rajesh
@Verand,
This is a very important question (and probably a little unusual for many
)
When we want to support multiple OO languages in a single TAF, there has to be a “middle tier” or “translation tier”. Following are some examples:
- Jython – Can integrate Java/Python code
- IronPython – Python/DotNet
- ctypes – Python and C (procedural)
- Boost Python – Python on top of C/C++
- SWIG – Python (and several other langs) on top of C/C++
….
The above depend on third party tools/ languages. There can be simpler means of integration via STDOUT/File output, though in such cases you can’t directly talk to the objects/functions from one language to another – it’ output based integration between components. I have used this approach successfully in writing a framework which can possibly run tests in any language.
Another way (and a little complex one) is employed usually by developers to provide a scripting interface on top of e.g. compiled Code. You can find several such examples of Lua and Python. In such cases the language interpreter is “embedded” in the software written in another language.
I’ve experimented with some (and not all) of the above ideas with a fair amount of success.
Hope it helps! I’m planning to write on this subject in the near future as a part of this article series.
Regards,
Rahul
@Rajesh,
Good to hear that you liked the article.
I agree to your comments. I especially liked the point on business classes. It is very important to keep business classes separate from the generic class hierarchy of the framework so that the core framework can be used by multiple projects.
Object oriented thinking (though I consider myself at a very preliminary stage as of now) has helped a lot in my test automation framework design efforts.
Thanks for sharing the link to your post. I’ll read it and share my thoughts.
Regards,
Rahul
Rahul, excellent high level overview – thanks for sharing.
>>”I have used this approach successfully in writing a framework which can possibly run tests in any language.”
I am glad that you already experienced with multi OO language support TAFs.
I have been developing a few “Hybrid” test automation frameworks using Java & C#. But TAF of multiple OO language support is something that I’m really looking forward to apply in a manner consistent with its purpose.
I’m interested in your thoughts on some more details on this topic in your next series. All the best!:)
@Verand,
It’d be interesting for us to know what’s the context in which you have a requirement for using multiple OO languages? Is it an GUI application or API that you are developing the framework for?
@Rahul,
We’d be very interested in understanding the architecture and design of your framework. Looking forward to the subsequent posts in this series.
I’d also come across an interesting article by Michael Hunter http://www.thebraidytester.com/downloads/CaptureTheEssenceOfYourTestCases.pdf on Object Oriented approach of test automation.
Regards
Rajesh
@ Verand, Rajesh,
Thanks for visiting again. Please subscribe via the RSS feed so that you are informed regarding the news on further articles of the series.
FYI, my next article in the series is on high level models of distributed testing.
I’ve got a related talk scheduled for PyCon India’2010 as well which is scheduled this month at Bangalore, India.
@Rajesh.
Thanks for sharing the link.
Regards,
Rahul
@Rajesh,
My requirement is “to learn”
. Don’t you think there is power in information.