Application Testing using virtualised clients
As the planet is going VM mad at the moment I was pondering usage at my site and also what level of usage had filtered into the testing world in general so far.
Some pronent development environment providers are working up virtualisation clients into their IDEs for more effective unit/link testing purposes.
At most sites I have worked at there are problems configuring testing clients (everything else too but that's what I'm interested in here). This significantly impacts planning and execution timings on occassion costing a lot more in terms of catch up time for the test team.
Is anyone seeing benefits by using virtualisation technology to deploy testing clients to PCs in their labs?
I'd also be interested in any experiences with test automation on their virtual client/s.
Good Answers (6)
VM's are one of the best things that have happened to Testing. It allows you to have multiple OS's and configurations setup to test against. It cuts down the time for state restoration of the test environments an both the server and workstation levels. This allows for tight control of the environment and reduces the issue of version conflicts, thus eliminating another factor in the problem resolution equation.
For Functional and Configuration (as long as the software doesn't have a hardware dependency) testing it is the way to go. And that is where the majority of testing occurs anyway.
For Performance/Load testing I do agree that VM is not ideal and that you need to have the full environment to test against. BUT... you can have your functional and configuration testing done in a VM environment and then move to the Performance environment for that work. So in essence you could do parallel work in the late phases of the project.
This arraignment can act as another gate for the testing process. One of the biggest issues and gripes in the testing world is availability of systems (hardware & configurations of software) to use. So instead of having 20 workstations and 4 servers in your test lab you could have about 8 beefy server machines (with lots of memory and disk space) running VM's to support your 'servers' and 'workstations' to test against.
For UAT testing you could then go to a more conventional test environment with physical machines.
It does prove itself to be cost effective from the standpoint of money and time during a project. And that is a key consideration.
Virtualization can be pretty handy for both development and testing. With regards to testing it allows you to have (for the most part) hardware agnostic environments that can be put on ice when not in use. VMware has some neat tools ($$$) for their virtualization platform that makes the storage of the environment quite light on resources. So when your QA team finds an egregious issue that the dev team says they can't recreate, the QA team can pull up their stored state of the virtual machine.
The other general benefits I like about virtualization (it took me the longest time to convert, I was very skeptical at first,) is that unless you've got either an application that has high performance considerations or dependencies on the actual hardware that it's running on, the notion that the virtual machine abstracts the hardware means that the VM image is portable. This can be a huge win for example when setting up new employee machines as your standard dev environment can simple be transferred onto their machine without all the time wasted of downloading, making sure they've got the right versions, etc. And for those times when your machine takes a dive and leaves you stranded, providing it's not your harddrive that's dead all you have to do is get your new machine, copy over the VM image and it's as if you've not dropped a beat.
VMware has some great enterprise solutions if you've got the $ for it. If you're looking for good, reliable free solutions I highly recommend VirtualBox (which was recently acquired by Sun).
Eric J
Chief Architect and COO at Canonic Corp
Best Answers in: Software Development (2), Computers and Software (1)
We provide an integration between Sparx Enterprise Architect and HP Quality Center. Since we support three versions of Enterprise Architect and two versions of Quality Center, with three different database backends for Enterprise Architect, we have quite a large combination of environments to test in.
Using VM's makes a daunting task much more manageable.
Virtualization software has helped me by allowing me to test all sorts of different OS's without affecting my own. Plus, if another developer can't recreate the problem on his/her own system, the virtualized image is portable allowing him/her to experience the exact error. Most virtualization clients also support the idea of rolling back to a certain point. If you're testing an expiring key or something along those lines it can come in really handy.
I have also found it handy when wanting to test an application that uses multiple machines. You can run a few different virtual servers at the same time allowing you to test communication between the two. Taking one out of the loop can help you determine when, where, or how errors are going to be handled.
The main benefit is just the portability of the images. I can create a base image, that I can then copy and customize for each app I'm testing. This saves me a ton of time watching progress bars during the setup phase. And once you have the base image set up, it is easier to tweak the config rather than having to start from scratch.
I had VMWare at my old job, I don't at my new job where I/we spend alot of time preparing/rejigging our machines to get them to a state where we can test. I only have the one machine and so this can take up a huge amount of time and effort.
I miss VMWare, it saved me alot of time and hassle.
I also miss having two monitors but that's another story.
VMs are not going to be suitable for all situations and as someone has already mentioned, if you don't think there will be benefits/value you probably won't get any.
On the other hand, VirtualBox is opensource, won't cost you anything to have a play with it.
Scott A
Software Quality Assurance Leadership, currently a VP & Manager of Enterprise Delivery Quality at Wellington Management
There are some great answers here that I agree with wholeheartedly. One topic not covered though has been the use of automation as a VM enabler. If you configure your environment correctly, particularly the automated technologies used within your testing framework, you can realize significant gains in testing throughput with virtualization, these gains certainly include a reduction in the cost of testing labs/environments. There has been a lot written on this subject and some of the QA-centric boards on the web cover these ideas well...hope this helps you get started in the right direction.
More Answers (3)
Michael K
Very Principal Scientist at Medtronic MiniMed
Best Answers in: Computers and Software (2), Software Development (2), Information Storage (1), Telecommunications (1)
Virtualization is often used to work around poorly written and malfunctioning software. In general, if the world goes mad around something, but you do not see why even after trying, - let the world to do its thing. Stay cool. Virtualization has its applications, and testing is one possible application, but if you do not see why, then it means that you do not need it. As simple as it gets.
I can understand if the server software is in a VM, in fact I am currently working on a project like that. But if your client app is in a VM, unless the user is going to use the VM, why test it in a VM? One possible reason may be early proof of working software. If the client software will be used as a native install, however, it should be tested as such.
My perference in testing is to replicate as close as possible (or practical) the environment in which the application is going to be used.
So, if its going to be used in a virtual setup, then yes defintely test on a virtual machine.
If the application was to be used in a more traditional environment I would suggest that you test on that. Or at least test a subset of the critical functionality in a traditional environment.
