July 31st, 2018 by Adam Sandman
Someone on Quora asked the expansive question - "What are examples of automated testing?". When I read this question, it made me think - that is such as vast question, since it can encompass so many different types of testing (hardware, software, network, security, performance, compatibility), all of which can be done automatically or manually depending on the situation. However I thought it might be useful to describe some different examples...
Hardware Testing
Automated testing of hardware systems has a long pedigree, with industrial systems having developed "test harnesses" pretty much right from the inception of the systems engineering industry. After all, if you are creating a complex electronic system with many different circuits, boards, and components, you needed a way to test different parts of it isolation. Similar to unit testing software, you create an electronic rig that simulates the inputs and measures the various outputs from the device being tested. It can send a large number of different signals, measure the results and compare the values. This is much easier than manually trying them and recording the output voltages on paper.
Similarly, for testing the entire device you may have a large-scale test rig that lets you put a vehicle or machine into a test environment that can simulate the real-world usage without having to actually use the device in the wild. This is particularly useful for vehicles or safety systems that you cannot easily test without a lot of precautions (test driver, closed track, actual power station, etc.). You will eventually test the devices in the real world, but automated solutions to test them in a lab environment reduces cost and improves quality.
Software Testing
When you are testing software systems, it is a very similar approach, you need to isolate specific functions (unit testing), then test entire modules (functional testing), then test entire systems (end to end system testing), and finally test all the external interfaces (API testing and/or UI testing). This is described more detail in our Testing Methodologies Whitepaper.
For specific examples of API testing and UI testing, we have some specific examples:
Finally, for general information on automated software testing, here are some more examples:
Performance Testing
Load testing, stress testing and performance testing are all different names for a set of types of automated testing where you use a load testing tool to simulate load by lots of users (called virtual users or VUs in the parlance) on an application being tested. The main difference in the terms is what the objectives are:
- Load testing - you put the expected number of users against the application and see if it works OK.
- Stress testing - you put an increasing number of users as load and see where it fails.
- Performance testing / engineering - you put a varying number of users as load, measure performance, and re-engineer the application to improve response times.
Security Testing
Security testing, vulnerability testing and cyber testing are basically automated tests that you run against a piece of software, a network device, or an entire IT infrastructure and look for vulnerabilities that could be exploited by a hacker. Sometimes the testing will be looking for known versions of systems that are vulnerable (e.g. old versions of a web server), looking for specific attack vectors (stored XSS, CSRF), or even attempting to overload a system to see if it will reveal information (DDoS, Brute Force).
In addition, systems may try more active methods that actually try to hack into a system, rather than just looking for vulnerabilities passively. For example, testing password forms to see if they can be broken by brute force attacks, dictionary attacks, etc.
Compatibility Testing
Finally, another type of automated testing is compatibility testing. In the software world, you may need to do cross-browser testing, to test that the same web page or application works on different web browsers.
You may also have to test the same application on different mobile devices (iOS, Android) or a hardware system may need to work on different voltages (230V for Europe, 115V for North America), different USB versions, etc. This kind of testing is called compatibility testing and can be complex and expensive to perform because you have to maintain so many different types of device. So for automating this kind of testing you may want to use simulators that can simulate different devices, browsers or operating systems. In the hardware world it gets trickier, but you can use develop emulators and test labs that can test different possible environments.