Authors: Rahul Verma, Chetan Giridhar
Performance tests are instrumental in keeping a check on product quality in terms of reliability and scalability. One essential area of performance measurements is application launch time which is the time elapsed from the operation which triggers the launch (double-click/command-line etc.) and the moment at which the application becomes responsive. This measurement would be useful in atleast two contexts:
Measurement of launch time of the application under test
Measurement of launch time of applications on a system with and without your software installed (e.g. for AV products)
The scenarios could fall under the category of performance testing or benchmarking based on whether you are conducting the test to:
measure the time to launch (and compare against the performance criterion in requirements, if you are lucky!)
measure and compare against the previous version of the product
measure and compare against a competitor product
The above could be achieved with a stopwatch, which wouldn’t be a very reliable method most of the times as:
start-up time would be in a couple of seconds
knowing exact point of “responsiveness” is a little tricky in manual testing
what falls after the decimal point is interesting in such tests
such tests require multiple readings which would be good enough to force someone to resign in a few days
AppTimer is a free tool for the purpose and this article discusses how you can use the same for the performance testing and benchmarking in terms of application launch time.
PassMark Apptimer is an easy-to-use tool to measure application launch times on Windows platform. It gives the provision of configuring the tests with GUI or configuration files, wherby latter can enable easy automation around this tool as discussed in the later part of this article. It saves a log of the measurements which could be easily tied with a simple text parser and plotter. Alternatively, the log can be chosen to be an Excel file for which analysis and plotting could be automated via Excel Macro and scripting modules that interface to Excel. It is a standalone executable and one could carry it in a portable device like a hard drive.
|Binary Size||130 KB (59 KB compressed)|
|Supported OSs||2000/WinXP/ 2003/ Vista|
AppTimer will run an executable ‘N’ (user configured) times and measures the launch times of the application under test. For example, if user sets the application winword.exe and configures it to run for 10 times, itr maintains a record of launch times for 10 executions. After it is able to take this measurement, it attempts to close the launched application; the way it closes the application is configurable.
Number of measurements taken in a performance testing exercise is a very essential aspect but often neglected. There’s no rule for the same. It would vary from one application to other, one platform to other and so on. One should understand that one has to establish this number via experimentation.
You should run AppTimer for a given application till you see some level of consistency in results for majority of the measurements. This is a rough picture of statistically equivalent measurements. You would have to do some ground work here establishing the acceptable standard deviation and things which are beyond the scope of this text. One rough rule is that do not rely on plain averaging of whatever measurements you collect. You must remove the outliers before you do that.
Download the file (AppTimer.zip) from http://www.passmark.com/products/apptimer.htm. Unzip and launch the executable. Let’s check out the welcome screen:
Here’s something in short about the parameters to be given to the AppTimer application.
|Application||Path of the executable|
|Cmd Line||Represents any command line arguments to be given to the application under test.|
|Log File||Path of the file where the user wants to store the log of measurements.|
|Window Name||Apptimer uses this string to detect which window to be closed (out of all the windows active on the computer). For example, in case your application is Notepad.exe, you could use ‘Notepad’ string to tell AppTimer to close the notepad application it has launched. Advantage here is that AppTimer, tries to match the part of the string and doesn’t look for an exact match.|
|Executions||Number of times a user wants to launch the application. (expects a natural number)|
|Delay||It is the time delay between when the application is launched and ready for use to the time when AppTimer tries to close it. (It expects time in milliseconds. 1 seconds = 1000 milliseconds)|
|Window Detection Method||AppTimer uses three methods of detecting the window it has to close after the application has launched:
- Input Idle: In this method, AppTimer identifies the window to be closed when the user starts using the launched application. Like in case of winword, AppTimer understands it has to close the winword window only when user starts writing on it.
- Window Name: In this, AppTimer uses the WindowName as entered in ‘WindowName’ section and searches the windows open on the computer. If the Window Name gets matched, it closes the matched window.
- Visible: In this technique, AppTimer checks whether the window name that is provided by the user, is ‘visible’ on the user system. If visible, AppTimer closes the active window.
|Window Close Method||There are three ways of doing it
* WM_CLOSE: WM_CLOSE message is sent to the application from AppTimer.
* WM_SYSYCOMMAND: WM_SYSYCOMMAND is to the application from AppTimer.
* ALT-F4: As name suggests, it uses ALT+F4 to close the application.
Test parameters can be configured with a help of configuration file placed in AppTimer.exe folder path. File name should be named as “config.txt”.
C:\Application.exe /MIN C:\log.log WindowName 5 1000 1 1 0 1 0 1
This configuration file suggests the following:
Cmd Line: /MIN
Log File: C:\log.log
Window Name: WindowName
Delay Time: 1000 milliseconds
Window Detection Methods: Input Idle and Window Name
Window Close Methods: WM_CLOSE and ALT-F4.
For this example we take Winword.exe (located at: C:\Program Files\Microsoft Office\Office12).
Above is a screen shot of AppTimer that is configured to run Winword.exe and store the logs in C:\log.log file. It uses Window Name detection method and uses ALT-F4 method to close the application. Number of execution is set to 2. Click Run App to start the test.
During execution, AppTimer launches winword.exe waits for the dealy time and then closes the window with ALT-F4 method. It then logs the launch time in C:\log.log file. Various other combinations of application, executions, delay time and window detection and close methods can be used depending on user requirements.
Log.log file looks like this:
Using the log file: Take the log file and take the average of all the measurements. For example, in this case we can safely say that the average time it takes to launch winword.exe application is: (0.0230 + 0.0214)/2 = 0.022 seconds.
From the automation point of view, what we did was to keep a confog.txt file in the AppTimer.exe path location, launch AppTimer and then click on “Run App” button. Clicking can be achieved using an AutoIt or any scripting language. This will start the application with configuration tester needs.
Here’s an example code in AutoIt:
Run("D:\Tools\AppTimer\AppTimer.exe") WinWaitActive("PassMark AppTimer") ControlClick("PassMark AppTimer", "Run App",1010)
Keep the script where the AppTimer executable is present.
Script launches AppTimer with the configurations in config.txt file and waits till the “PassMark AppTimer” window pops up.
It then clicks on “Run App” to start the test.
To extend the utility of this tool further, we thought of enhancing it with AutoIt to enable measuring launch times of different applications in a single execution.
The Automate AppTimer binary will help you achieve the above.
Download the binary and keep it at a location where AppTimer.exe exists.
Create a “Config” folder in the above location.
Add ‘.txt’ files with user defined configurations (format same as “config.txt”.)
Double-click and run the binary.
Tool will first browse to Config folder.
Picks one of the .txt file and copies the file in to AppTimer location.
Renames the .txt file to config.txt, launches AppTimer and clicks on ‘Run App’
Once the test gets executed, it will start working on the other .txt file till all config files are used up.
Meant only for GUI applications
Meant for Windows platform
As per the PassMark website:
Some applications may have strange behaviour at startup making it difficult to tell when they have fully loaded and are ready for use. It may not always be possible for AppTimer to be used to time these applications however trying different combinations of the window detection methods may provide better results