A big part of the testing done on Warewolf so far has been done using manual testing for the Studio User Interface and functional tests using SpecFlow. We use unit testing to handle the business layer of the system as both cannot test the UI (see Unit Testing vs UI Testing).
Advantages of Manuel Testing
Manual testing, although tedious and time consuming, adds a necessary layer to the overall quality of the system. Manual testing isn’t as strict as automated testing because it allows you to do a lot more “out of the box” testing. This adds to the variety of test scenarios that need to be done for excellent stability.
Out of the box testing really makes a greater impact on the software as it raises more of the right questions and discussions around the product and its features. It highlights inconsistencies that would be easily missed by a developer as an oversight. It also allows the tester to relay feedback to the developers on how the user interface (UI) would be experienced by an end user, and increases the odds of finding real user bugs. This kind of testing caters to the system by ensuring that the presentation of the UI stays exactly as intended. This is good for smaller systems but not what Warewolf needs right now.
The above diagram simply represents how a user interacts with a system while manually testing, versus how Coded UI tests replace the user in automated testing.
Why Coded UI Testing became necessary
Warewolf is a growing system and recent developments within the Studio means a lot more needs to be done to ensure quality continues to improve to meet the expectations of Version 1 and beyond.
A new layer of testing is needed to make sure the UI of the system can be tested in the same way the business functionality layer is tested. The growth of the system means more time would need to be spent manually testing. Of course this can be automated to save time, as tests run without the need of a person to watch over them. This is why we opted to revisit Coded UI tests.
Advantages of Automated Testing
One of the best advantages of having automated tests in our system is that we can cover areas that were previously under-developed, and enhance them. It also allows us to get more coverage of the system and automate it to avoid human error and oversight, especial considering the volume of work being done to the Studio on a daily basis. Another advantage is that repetitive tests can now be programmed, enabling a greater variety of tests that can run using different configurations in a shorter space of time.
Automated testing instead of manual testing means more time to work on enhancing system stability. Naturally, this benefits both the developers and testers. For example, changes that could cause event firing failures or other side effects of code changes can now be traced and handled better without needing to revisit every component after a code change.
If you would like to see our implementation of Coded UI tests come check out Warewolf in GitHub.