What tools do you use (if any) to do GUI testing; not browser testing, but windows type applications
Hi,
A friend asked me this question. Unfortunately I don't do much of GUI testing. Whatever
little I do is browser testing and I use curl or wget for that. I suppose QTP (by Mercury tools now acquired by HP) could be used.
Here is more about the question from my friend:
If you use anything, what are the down falls (price, limitations, etc). Know of any free
tools that are effective?
The application he wants to test is similar to Network management applications in Windows.
thanks,
Swam
Good Answers (15)
I have tried most of the tools here and found them extremely lacking.
They packaged a lot of poorly and incomplete features but didn't focus on doing a smaller set of features correctly.
Then while evaluating I came across Ranorex.
We are extremely pleased and I am pleased that the light of reality has begun to shine in my company. Previously they blindly followed the bandwagons of popularity contests and went for the QTPs, Compuwares, and their ilk.
Remember to evaluate, evaluate, evaluate. Write down criteria for what you need to do, perhaps on a spreadsheet and keep track of each product's delivery in that area. You may indeed find that QTP works great for you if its gaping (IMHO) flaws and bloat do not affect you.
QTP and its ilk are massively priced.
Some of our criteria (outside of specific control types, which you should watch out for with QTP especially like date dialogues, scrollbars, etc)
* price
* support (ask around, google, get ratings, you will find that Mercury's legendary bad support didn't change with their acquisition by HP, at least not yet)
* plugability (didn't find much of this)
* cross platform scripting languages that are elegant and powerful like Python
* speed (includes speed of recording, bloatware locked the machine on many)
* support for externally running (i.e. automate the automation)
* run results outside of just a log or static report. Sure you can scrape these but why should you have to?
* customizable reports
* advanced logging features (like to files, standard out, web calls, email, etc.) Like using XMLRPC or REST calls to update my controller with results, Python supports this fantastically with its logging module.
* support to extend and integrate with others (or said another way, Play Nicely With Others... this doesn't include only that vendor's tools)
* variables and modularity (maintenance and reuse ability are key)
* scalability
With Python, I can easily integrate these test scripts within a larger framework (testing Unit, Functional, Integration, etc) that is all automated and follows the continuous integration principle. My results are processed not as reports but as real results per test and I report in the end as desired. (e.g. customizable HTML)
In closing, never take advise from someone who asserts notions like 'You get what you pay for' as that goes against reality and age old wisdom. Any fool can charge a lot for an inferior product. The bigger fools are those that buy it. Caveat emptor (Let the buyer beware).
If HP is smart, they will look to IBM's support of the Eclipse project and its suite of testing tools. (Plus, Rational is really good but expensive)
Also note this please. I noticed with many that you could not automate the running, much less obtain the return/result of each run with QTP and friends. This was totally unacceptable to integrate into a larger suite of testing. Which leads to my last point.
Please keep in mind that GUI (especially non web GUI) testing is a strange beast and difficult to maintain. Especially if your front end developers are undisciplined and like to change form or control and element names without first working with you. Perform White Box testing and leave GUI testing to 20% at most of your testing due to its limitations. GUI scripts are easily broken to show false negatives due to the flakiness of GUIs (all of them, not just gdi and its derivatives).
Make sure you adopt a more agile like approach in your relationship with development. You should be involved in discussions on changes up front so that development knows the ramifications of their choices and you can prepare for them. Having worked 'both sides of the fence' I can say this is the optimal relationship that all dev houses have. This is what open source most frequently uses and why it is such a superior development model (usually, there are always exceptions). Communication and collaboration are key.
Links:
QTP is far an away the industry leader, but that comes at a very hefty price. From my experience, you know if QTP is the one for you and finding the justification for the price is the hardest part.
I was recently turned on to Eggplant from Redstone Software, but have not gotten a chance to spend much time with it.
All in all, I have not found a suitable solution in either of my last two jobs and have instead taken bits and pieces from various solutions (some don't supprt VB, some don't support Java, etc) to make an in-house suite that covers all bases.
Jessica M.
Product Director at NIIT China Ltd.
Best Answers in: Business Development (1), Biotech (1), E-Commerce (1), Enterprise Software (1), Computers and Software (1), Computer Networking (1), Databases (1), Information Security (1), Software Development (1), Web Development (1)
XBO Soft is specialized in that, you can read this article or contact them
Links:
Michael Y.
CEO and Founder, Conflair Inc
Best Answers in: Commercial Real Estate (1), Offshoring and Outsourcing (1), Antitrust Law (1), Computers and Software (1)
There are few:
(1) QuickTest Professional, or QTP, by Mercury (now HP). Works well for Windows GUI testing. Another big plus is a large number of people who know how to use it. This factor is rarely taken into consideration during purchasing, yet it’s a very important one for any dynamic organization. The price tag is high, though.
(2) TestPartner by Compuware. Similar to QTP in the declared abilities, yet somewhat less mature and with fewer people who can work with it.
(3) TestComplete by AutomatedQA. My recent find. Seems to work well with both regular Windows GUI and browsers. Has a different approach to organizing tests. Can use a number of scripting languages: VBScript, JavaScript, and some others. Cheap: around 15% of the price tag of the other tools.
(4) Visual Studio Team System Code Name Rosario will have Windows GUI testing capabilities once released. Right now available for community preview. Something to watch for, given excellent features of the larger Visual Studio Team System product.
My test teams have been using QTP. This allows both record & replay as well as scripting for GUI testing. The cost is high and the RoI comes in primarily when you have a large number of users.
One issue that we have faced with QTP is that they don't recognize the latest the latest .NET 3.x controls and we've been working with Mercury/HP for fixes. Response time was faster before the HP acquisition but now it has been slow.
I use Rational Functional Tester. I have had a lot of success with Perl's Win32::GuiTest module too.
Links:
You can give a shot with UIA API available in .NET 3.0 framework. However you would need to build up your own solution on top of that and might need to do some workarounds but I see two advantages:
1. More control over what is going on and in the worst case you can fall back to native WINAPI in case the UIA is not at rescue.
2. Its free :)
The cons to this would be UIA bugs and limitations as I don't see it is adopted heavily. There are issues and limitations with this so it would be better if you can define the scope of testing and do a POC before adopting this as a solution.
why dont u try winnrunner 7.x, it was really good one, i used it a couple of year back. i found it very cool. i think u must try to Test GUI with latest Winrunner.
hope u ll get ur solution.
bye with regards
surya
Clarification added May 6, 2008:
why dont u try winnrunner 7.x, it was really good one, i used it a couple of years back. i found it very cool. i think u must try to Test GUI with latest Winrunner.
hope this will help u ?
bye
with regards
surya!
Rémon S.
Consultant Software Engineer at EMC, member of OASIS XACML Technical Committee
TestComplete is a very nice tool that has served me well for many years. Be sure not to depend on the macro recording facility too much. Instead, build up your own library of scripts, and compose tests from that.
Links:
Bogdan C.
.NET Project Manager
Best Answers in: Databases (2), Blogging (1), Computers and Software (1), Software Development (1), Web Development (1)
Our choise is TestComplete.
Anthony W.
Experienced Software Developer, Maintainer of the Boost Thread Library for C++, Author of C++ Concurrency in Action
Best Answers in: Software Development (5)
If you're after free tools, AutoIt works quite well. Designed as a GUI automation tool, it provides full control of the GUI of just about any Windows app with a basic-like programming language.
Links:
Since this is not something the tool vendors tend to stress, allow me to point out that it is not just the tool that determines if automated testing will be effective.
Sure, the tool needs to have various functions to test your product. But the maintainability of your testware is just as important: All your automated tests are going to end up gathering dust on a shelf when keeping the tests working starts taking too much time and is dropped. Then all the time and money invested in the automated testing, often a lot, is lost. And this is something many, many testers have experienced, unfortunately.
It is easy to avoid, though: Do NOT use record & playback functionality. Except maybe to learn the scripting language or if you are quite sure your test cases will require no maintenance over the life time of the product you are testing (but that is rather rare). Otherwise, chances are that you will end up with a maintenance nightmare. Many changes to the product will require changes to test cases. If you rely completely on record and playback, you will have to re-record the test cases, which is time consuming and becomes annoying after a while. If you prefer to edit the test cases, you still have to do it separately for every affected test case (assuming that you can identify them, this can also be tricky).
A much more flexible way is writing test cases according to an action based testing approach. I have always preferred TestFrame, but your taste may differ. This requires that all (automated) test cases are written rather than recorded. But the bulk of the writing should NOT be done by the tester. The tester should only write the test cases, not the underlying actions (that are shared between many test cases). The implementation of the actions is for someone with software engineering skills, so the tester can stick to what he does best - identifying WHAT to test rather than all the details of HOW to test it. And maintenance is no problem if the actions are implemented using standard software engineering practices. Then changes often result in maintenance in a single place, rather than in many test cases and automated testing of a product can remain effective for years.
QTP or winrunner can be used for windows based applications. Mercury tools are expensive but documentation is extensive.
thanks,
Rajesh
Tejesh N.
Associate Manager
Best Answers in: Software Development (2), Personnel Policies (1), Staffing and Recruiting (1), Currency Markets (1), Equity Markets (1), Computer Networking (1)
You can use QTP for this.
Otherwise Squish is good for this kind of Testing
Using the comfortable Squish IDE, tests are created using Squish's event recorder. Verification and synchronization points can be easily inserted. Squish allows the user to choose between popular and open scripting languages such as Python, JavaScript, Perl and Tcl for test scripts. Therefore the complete set of language features, in addition to Squish's testspecific APIs can be used to create powerful and robust tests. Squish for Java recognizes all standard Java GUI controls and offers special support for complex widgets such as tree, table, list and menu controls. In addition Squish for Java recognizes custom Java controls. Squish for Java's mechanism to identify Java GUI widgets is very robust to make sure Squish tests will keep working while the application evolves. Squish for Java provides access to the complete Java API via its test scripting languages and offers access to all objects and properties via the Spy and verification point editor. Additionally, it is possible to access the application's API from test scripts for even more advanced tests and verifications. Squish for Java supports is completely crossplatform and natively runs on Windows, Linux®, Mac OS X, Embedded Linux and other Unixbased systems. All tests created with Squish are crossplatform and run without modifications on each supported platform.
Squish is an open system that can also be driven remotely. A set of commandline tools allow an easy integration in any test management system. Squish also offers special APIs and tools to create and run dataand keyworddriven tests.
