TestComplete and BugBuster test automation tools

This blog describes two automated testing tools – TestComplete and BugBuster.

Testing the TestComplete tool

The primary objective was to analyze the ability of this tool to test applications that employ the ExtJs framework in their presentation layer and compare it to the IBM Rational Functional Tester, which itself has many rather significant issues with dynamic object identification that render the writing (and the usefulness) of test scripts tedious and produce questionable value.

Final critique: For the purpose of testing ExtJs applications – not worth it.

TestComplete, in comparison to the IBM Rational Functional Tester (RFT), is perhaps slightly more intuitive (particularly for inexperienced users) and more easily solves problems with dynamic object identification, and yet those advantages are not substantial enough to merit migrating to TestComplete. The same/similar problems that arise with RFT are observable in TestComplete and thus it does not impart any significant advantages. A detailed overview of the advantages and disadvantages follow:

Pros (+) :

+ Great tutorials and general support, quite precise error detection, even offers solutions to particular problems.

+ The possibility of writing scripts in a number of languages: VB, JavaScript, C++, C#, Delphi

+ Supports Flex, Flash, Sencha ExtJS, Silverlight, jQuery UI controls.

+ Contains simple script recording suitable for testers with less programming knowledge, where only objects are displayed (they deem these “keywords” and I’ll employ that expression later in the article), nicely displayed and easy to read, does not contain any code, and it is possible to add some basic actions to those objects, such as calling other scripts, etc. Though, there do exists some limitations here regarding actual program scripts. It’s a great asset that keyword scripts can be converted into straight code with a single click, but the reverse is not possible. There also exists a problem with synchronization which will be described in the cons section.

+ For every action there is a corresponding screenshot (this relates only to keywords scripts) that displays the expected result of the action. A great asset for someone new to a project.

+ Reproduction of recorded scripts is quick, significantly faster than RFT, goes through objects quicker, is stable and unlike RFT, doesn’t lose focus easily (if, for example, someone clicks accidentally or it loses focus when an IM window pops up:-) )

+ Test logs are recorded, making it easier to find errors with that method than with RFT, logs are saved with each test execution and are marked with a checkmark or an X if the test was successfully/unsuccessfully executed, respectively.

+ If an object has already been opened/made visible, or for example, the requested tab is already open, it will not click repeatedly as RFT does but will simply display in the logs that it had already been opened and had skipped it.

+ Verification points exist as they do in RFT. It is possible to define conditions that can be adjusted to execute actions based on user input.

+ Interesting graphs displaying test results, I hadn’t thoroughly tested this out, but it seems to be a nice feature.

+ Integration with JIRA, HP Quality Center, Visual Studio Team System

Cons (-) :

– All web browsers are supported, yet each one needs to be adjusted slightly (disabling protected mode in IE and similar adjustments, though it is mostly like that with RFT as well), IE is best supported while others need plugins to be installed, which I did not try personally.

– Even though its fantastic to be able to simply translate keywords scripts into programming scripts, there is a massive problem in that there is no synchronization, such that they’re two completely separate scripts. Once a keywords script is translated into straight code, if new controls (or others) are added to the code, it will not be synchronized with the original keywords script. Likewise, if a new part of the test is added to the existing script, there will be no synchronization between the keywords script and the programming one. Due to this, in practice it is best to use a straight programming script, at least for ExtJs applications for which additional code is necessary for everything to work as intended. I contacted SmartBear’s development team in regards to this problem they reported to me that they were working on it, but that it would not be finished any time soon.

– When recording new scripts or even extending existing ones, all objects are saved into a new DOM tree. This is a huge problem for dynamic identification and for this reason alone it is not worth using TestComplete. Important changes are made to objects in the DOM tree (call it browser1) in order to ease dynamic identification, then as the script is extended, new objects are saved into a new tree (call it browser2) that loads objects from the original and now all changes made in browser1 are irrelevant and it is necessary to copy much of it over, creating an insurmountable barrier to productivity. SmartBear’s developers gave me very vague and unclear answers to my questions in regard to this issue, so I see this as a significant shortcoming. Every loading goes into a new DOM tree and until this changed or a different loading method is enabled I would personally avoid this tool.

– An option similar to RFT’s “Update recognition properties”, where you can easily search the tree for the location of a specified object is and where recognition fails, is omitted.

– Object maps are saved as Object tree and Alias tree. The idea is that aliases are shorter titles for objects and are as such easier to handle, and when new objects are read they are saved in both trees. That’s alright, but massive problems arise when changes are made to either map. They don’t synchronize themselves and then there is chaos. SmartBear’s development team explained to me how it functions, but it is certainly a bad thing for dynamic identification where there is much “tinkering” with objects. Overall, it may be that in my limited time I wasn’t able to discover all the possibilities, yet this seems to me to be another problem with this tool.

Dynamic Identification:

Even if it was already mentioned within the pros and cons, I will summarize some methods for (partially) solving problems with dynamic object identification.

1. Just like RFT, it is possible to use regular expressions for the problematic and variable parts of DOM trees. It’s a bit simpler than in RFT
2. It’s possible to disable comparisons by ID (the most crucial property in ExtJs applications) and putting, for example, by classIndex or something else, or of course, more than one property.
3. We can physically move (literally drag ‘n’ drop from the name mapping display) an element next to the first stable element in the DOM hierarchy that the tool always finds. We then mark it as “Extended find”. With this method the tool directly finds the object by its property and it is irrelevant to the tool where it is located within the the tree (this is how we eliminate unnecessary accessing of branches where there aren’t any objects and when it frequently does not find anything because it’s running time runs out or something other issue). The problem with this is when additional changes are made, frequently there is some issue and the object can no longer be found. Another object expects its existence in a specific place etc, there are many problems and in the limited time I had I was not able to figure out how to cover all scenarios. The point is that this solution is very sensitive.
4. It is also possible to create a template, for example, for every textbox allow the last element to look for its static ID, while for the parent object does not search by ID but by some other property. I was not able to study this extensively, in theory it sounds great, though I’m sure that here there would also be many problems in practice.

Naturally, in using these options there also exists the same problem as in RFT where the same script will 3 times out of ten find a certain element, and the other 7 times fail at dynamic identification, pass over an object and of course, not necessarily at the same location.

My final judgement concerning this tool is that it will not solve the problems related to testing ExtJs applications, but perhaps with time they will further repair and develop this tool to the point where we may give it a second chance.

Testing the BugBuster tool

Firstly, it is important to state that this is not about the classic desktop tool, but the Google Chrome plug-in based on Webkit. Aside from that, this tool was mainly designed for for testers that will solely click on the application, without writing any additional code. BugBuster claims to be “the most user-friendly testing platform”, with which I don’t entirely agree, but more on that later.

In order to start testing with BugBuster, it is first necessary to register, and thereafter to login every time onto the BugBuster page. There you can open a list of applications, test plans, test environments, and test scenarios from which you can start executing tests or edit existing tests. Aside from the BugBuster page, tests can be recorded and started from the BugBuster icon in Google Chrome. Each application is associated with corresponding scenarios, execution results, test environments, yet in the trial version you are limited to just one application and only three test scenarios.

As previously stated, BugBuster is designed for test recording by clicking, in order to simplify the process for the user. Every step is recorded and it’s easy to choose whether to to use either an inputted value or a random one. There is no data pool from which to fetch data, instead the random.word() function is employed, which inputs a randomly chosen word. After recording a script and exporting it to BugBuster Test Plan (its own equivalent of storing a recorded scenario), it is translated into a nicely commented code (JavaScript). Here we arrive at the alternative method of writing scripts, by manually typing code. There are more possibilities with this method than with mere clicking, though it’s pretty awkward since there are no maps containing test objects and instead all values must be manually entered, leading me to believe that it isn’t realistically feasible in practice. It might be useful for adding code to existing scripts, but I don’t see the point in writing scripts from scratch.

The programming language employed in BugBuster is JavaScript, as well as the rather rich BugBuster API that contains a good number of useful functions. That JavaScript is separate from the JavaScript utilized to test pages such that there are no “code collisions”. Aside fom that, there is an API for DOM queries that is almost identical to jQuery and is named DomQuery.

Pros (+):

+ The API is extensively described, and in BugBuster in general is well documented

+ Simple recording of scenarios by clicking, not demanding for users without ‘technical backgrounds’

+ Integration with JIRA, Github, Jenkins

+ Displays a screenshot at each step of the test script

+ Test results and similar notifications are sent by email, easily filterable and has many possibilities

+ The test schedule is straightforward

+ Detects broken links, non-existant recursions, even small JavaScript errors

Cons (-):

– Can only be used in Google Chrome and Safari

– Limited testing of applications that are on a private network, VPN or localhost, there is a way to circumvent this using the secure tunnel option, but regardless this is a potential problem

– No object map

– All values are generally hardcoded in the code itself, which makes maintaining scripts difficult and resistant to changes

– No data pool

– Once recorded, a script cannot be modified with additional recording, only by modifying code

– Cannot use scripts repeatedly, redundant code piles up

– Weak forum representation (rather hard to find comments and experiences, incomparable to RFT, even to TestComplete)

My final (personal) judgement is that BugBusters capabilities are too limited. It mainly has to do with the fact that it is only supported on Google Chrome and with the problems concerning testing applications that are not publically available on the internet (localhost, VPN…), keeping in mind that that is where the majority of applications are tested, particularly in the development process. In order to test those applications, it is necessary to install node.js, connect it with the command line to BugBuster’s built-in solution for setting up a secure tunnel, and allow outgoing conncetions in your firewall on port 443 to sesame.bugbuster.com so that BugBuster can access your private network. There is no other way, since we are using an ‘outside’ tool. Considering that it claims to be oriented for simplicity and for users without ‘technical backgrounds’, this isn’t exactly in that spirit, especially since this procedure is necessary for most applications.

I would say that BugBuster is suitable for testing static web pages, but with dynamic pages it will encounter quite a few problems. In regards to dynamic identification, to touch upon ExtJs, BugBuster will naturally have problems similar to those of RFT and TestComplete (which I have described earlier). Particularly problematic is that the DOM hierarchy is hardcoded into the code itself such that even the regular expression solution for object maps doesn’t apply to this tool.

Therefore, this is a good tool for testing static webpages or simple web applications, but is still nevertheless limited for testing complex applications.