Performance tests are instrumental in keeping a check on product quality in terms of reliability and scalability. One essential area of performance measurements is application launch time which is the time elapsed from the operation which triggers the launch (double-click/command-line etc.) and the moment at which the application becomes responsive. This measurement would be useful in atleast two contexts:

1. Measurement of launch time of the application under test
2. Measurement of launch time of applications on a system with and without your software installed (e.g. for AV products)

The scenarios could fall under the category of performance testing or benchmarking based on whether you are conducting the test to:

• measure the time to launch (and compare against the performance criterion in requirements, if you are lucky!)
• measure and compare against the previous version of the product
• measure and compare against a competitor product

The above could be achieved with a stopwatch, which wouldn’t be a very reliable method most of the times as:

• start-up time would be in a couple of seconds
• knowing exact point of “responsiveness” is a little tricky in manual testing
• what falls after the decimal point is interesting in such tests
• such tests require multiple readings which would be good enough to force someone to resign in a few days

AppTimer is a free tool for the purpose and this article discusses how you can use the same for the performance testing and benchmarking in terms of application launch time.

PassMark Apptimer is an easy-to-use tool to measure application launch times on Windows platform. It gives the provision of configuring the tests with GUI or configuration files, wherby latter can enable easy automation around this tool as discussed in the later part of this article. It saves a log of the measurements which could be easily tied with a simple text parser and plotter. Alternatively, the log can be chosen to be an Excel file for which analysis and plotting could be automated via Excel Macro and scripting modules that interface to Excel. It is a standalone executable and one could carry it in a portable device like a hard drive.

AppTimer will run an executable ‘N’ (user configured) times and measures the launch times of the application under test. For example, if user sets the application winword.exe and configures it to run for 10 times, itr maintains a record of launch times for 10 executions. After it is able to take this measurement, it attempts to close the launched application; the way it closes the application is configurable.

Number of measurements taken in a performance testing exercise is a very essential aspect but often neglected. There’s no rule for the same. It would vary from one application to other, one platform to other and so on. One should understand that one has to establish this number via experimentation.

You should run AppTimer for a given application till you see some level of consistency in results for majority of the measurements. This is a rough picture of statistically equivalent measurements. You would have to do some ground work here establishing the acceptable standard deviation and things which are beyond the scope of this text. One rough rule is that do not rely on plain averaging of whatever measurements you collect. You must remove the outliers before you do that.

Download the file (AppTimer.zip) from http://www.passmark.com/products/apptimer.htm. Unzip and launch the executable. Let’s check out the welcome screen:

Here’s something in short about the parameters to be given to the AppTimer application.

Test parameters can be configured with a help of configuration file placed in AppTimer.exe folder path. File name should be named as “config.txt”.

C:\Application.exe
/MIN
C:\log.log
WindowName
5
1000
1 1 0
1 0 1

This configuration file suggests the following:

• Application: C:\Application.exe
• Cmd Line: /MIN
• Log File: C:\log.log
• Window Name: WindowName
• Executions: 5
• Delay Time: 1000 milliseconds
• Window Detection Methods: Input Idle and Window Name
• Window Close Methods: WM_CLOSE and ALT-F4.

For this example we take Winword.exe (located at: C:\Program Files\Microsoft Office\Office12).

Above is a screen shot of AppTimer that is configured to run Winword.exe and store the logs in C:\log.log file. It uses Window Name detection method and uses ALT-F4 method to close the application. Number of execution is set to 2. Click Run App to start the test.

During execution, AppTimer launches winword.exe waits for the dealy time and then closes the window with ALT-F4 method. It then logs the launch time in C:\log.log file. Various other combinations of application, executions, delay time and window detection and close methods can be used depending on user requirements.

Log.log file looks like this:

Using the log file: Take the log file and take the average of all the measurements. For example, in this case we can safely say that the average time it takes to launch winword.exe application is: (0.0230 + 0.0214)/2 = 0.022 seconds.

From the automation point of view, what we did was to keep a confog.txt file in the AppTimer.exe path location, launch AppTimer and then click on “Run App” button. Clicking can be achieved using an AutoIt or any scripting language. This will start the application with configuration tester needs.

Here’s an example code in AutoIt:

Run("D:\Tools\AppTimer\AppTimer.exe")
WinWaitActive("PassMark AppTimer")
ControlClick("PassMark AppTimer", "Run App",1010)

• Keep the script where the AppTimer executable is present.
• Script launches AppTimer with the configurations in config.txt file and waits till the “PassMark AppTimer” window pops up.
• It then clicks on “Run App” to start the test.

To extend the utility of this tool further, we thought of enhancing it with AutoIt to enable measuring launch times of different applications in a single execution.

The Automate AppTimer binary will help you achieve the above.

1. Download the binary and keep it at a location where AppTimer.exe exists.
2. Create a “Config” folder in the above location.
3. Add ‘.txt’ files with user defined configurations (format same as “config.txt”.)

1. Double-click and run the binary.

1. Tool will first browse to Config folder.
2. Picks one of the .txt file and copies the file in to AppTimer location.
3. Renames the .txt file to config.txt, launches AppTimer and clicks on ‘Run App’
4. Once the test gets executed, it will start working on the other .txt file till all config files are used up.