Black Box Testing – Challenges and Opportunities
So, what do we mean ‘Black Box Testing’?
The term refers to a common situation in which a person interacts with a system solely through its external interfaces, without looking inside. That is why it’s referred to as ‘black box testing’, we have no idea what’s inside, it is as if we had a magical black box to test.
A good example of Black Box Testing may be a physician who tries to diagnose and treat an illness using the external markers of a disease. This approach is called a black box and it stands in contrast to an approach that a surgeon takes, when the latter “opens” the box and takes a peek inside.
But how accurate is the above comparison when we talk about software?
In fact, there are many situations when this analogy may be quite accurate:
- Most of the time, in the software world, you don’t have the source code. The product is shipped as a binary. Or it is assembled from 3rd party components that are themselves binaries. So, all you have is a User Interface (UI) or command line interface.
- Sometimes the sources are available, but they are either too complex, minimized, and/or obfuscated. So, it is easier to assume that there are no sources rather than trying to read them.
- The System Under Test (SUT) may be very complex and include multiple components. Its internal state is also complex and includes memory triggers, files, database entries, etc.It is like an automobile. Its interface (dashboard) is much simpler than its internals (engine). Furthermore, if it is unable to go we can easily detect it. However, if the hood is open and we look at it we are unlikely to be able to say if it will move or not. It is much quicker to put in the key and just try to drive it first.Is it ready to drive?
- Sometimes you are buying the tool as a whole, with a goal of doing acceptance tests. You need to make sure it is functioning for you. It may have a broken functionality that you don’t care about, so that may not be an issue for you.
When to Choose Black Box Testing
You may be an internal member of the development team with access to all the application’s source code. You may have the skills to understand the code and to do whatever is needed to improve the quality of the application. Does it mean you should always do “white box” testing in this case?
Well, even in this case “black box” testing can still be a fruitful approach with its own unique benefits:
- It gives you different view of the subject. When your focus is on the internal logic of a software, you may not see the whole picture and lose the ability to view your product from the outside.
- It forces you to see the SUT as if you are its user - not a developer. To use a car analogy, you have to drive with a hood closed, just like your users do.
- Even if you know the system well and know that all parts are functioning properly, there could be a problem in how pieces are connected. Or there could be incompatibility among the parts. The only way to find out is to test the system as a whole.
What About the Internals of the Application?
So, having decided to use Black Box Testing; from all of its features described earlier, it seems that we don’t care about the application internals. But… But the quality of our tests will vary greatly depending on how the application is designed. So, if you have influence on the application design, then there are things that may help you make testing more effective (and your life much easier).
- The application should provide output, health check and signals. That is extremely useful for both testing and support. For example, if a car has broken brakes then having ‘BRAKE signal blinking on your car’s dashboard may mean a great deal. With Black Box Testing your attempt to drive may have catastrophic results: you simply can’t stop! But with a signal you quickly know it without any risk.
- In many cases such signals are implicit and shown somewhere on the side or in the corners. A good example is the status bar available in many applications:
- There are lots of similar features, and you may see different types of status bar in other applications:
- Another signal is the application Title Bar. In his example with Notepad++, when a document is not saved, you will see an asterisk:
- Another useful tool is verbose output and file logs. Usually developers enable logging for their own purposes. The information in the Log files is typically for internal consumption. However, it may be used in testing; e.g. one of the standard tests is to look for error messages and exceptions in the logs.
- About 20 years ago assistive technologies completely changed automated UI testing and now Microsoft Active Accessibility, UI Automation, ARIA, Automated UI are designed to allow developers to provide more information about application UI structure and purpose. The goal is to make an application more screen reader-friendly.A screen reader is a tool enabling interaction for people who are visually impaired and it will read contents of the screen, narrate to the user and allow them to interact with the application using simple commands. But this, in turn, makes it much friendlier for automated UI testing, because all widgets become clearer, better attributed, easier to recognize and then interact with. So if you can insist on enabling assistive technologies in your app you will achieve two goals: make an application useful for more people and make it more testing-friendly.
- It is a common practice for developers to use the default internal IDs and object names suggested by their development tools. Some believe that no one sees them anyway and users will never know, so no need to waste time by giving them meaningful names. These IDs are visible for UI automation tools and make script development much easier because the script that types text into field ‘username’ is simpler that one doing the same with ‘text27’.
Approaches to Black Box Testing
Now that we have decided to go ahead with Black Box Testing, how do we actually do it? The set of approaches that may be considered as ‘Black Box’ is actually quite wide. Here is a list of suggestions, but these approaches are not exclusive: usually they are combined in the same test scenario. Sometimes the same approach may be used as both Black Box and White Box depending on the parameters.
- Manual UI testing. You just work as a user. Click, type, press, drag. You check data and error messages. e.g. SpiraTeam is an awesome solution that makes this approach easy and smooth.
- Automated UI testing. Do the same but using some kind of record/learn approach. Usually you need to record and make test once and then play it daily, weekly or on demand. Rapise is a great example of a tool for automated UI testing.
- Documentation testing (or specification testing, if there is a spec). Read and try to re-produce. It is purely-manual approach.
There are more advanced and formal methods for Black Box Testing:
- Model based testing. In this approach, we implement a model that performs the same functionality and exhibits the same behavior as the application being tested. It is useful when the application has a calculation algorithm that may have several different implementations. So, we implement our own version of the algorithm, prepare the array of inputs and compare the results and outputs of our model version against the real application.For example, the application may have an optimized high-speed version of some algorithm and you may instead testing with a trivial, simple and slower one. That is not good for production but makes testing much easier.
- Load testing may be considered a Black Box approach. This type of testing is mostly intended for multi-user systems. The goal is to find crashes, slow downs, bottlenecks, memory problems – everything that kills the performance of the system. A fully-featured web application is in fact a great example of a “Black Box”. It has authentication, authorization, system caching, browser caching, protocols/network, a web server tier, an application server tier, and a database tier all working together. Potential problems are usually intermittent, hard to re-produce, locate and understand. The key here is to generate proper data and well-thought-out load scenarios that address the most probable patterns of the real load.
What About Other Types of Testing?
Indeed, Black Box Testing has many useful features. So why should we bother with other types of testing? It’s because here we have a classic example of devil being in the details:
- The system may be fast enough for users, but a user is usually much slower than an automation tool. And we want to try different versions of input data and different combinations to make testing more thorough. That’s where the system response time becomes more critical for automated testing than for an end-user. It forces us to limit the number of tested scenarios. So, we have to split efforts in our test cases according to well-known testing pyramid:
- Unit tests and are much cheaper and faster than Black Box Tests. One of the goals should be to have a proper test set design to optimize efforts, execution time and costs by combining different variations.
- UI tests are “flaky”. This means they can have rare intermittent failures. There are too many factors making it possible to have sporadic failures in the test case. Unit tests are also flaky, but at a much lower rate. So, the more complex the system the more instability is going to be caused by the environment.
- Flakiness by itself is not a critical problem and we may live with it. At this point it is important to have good test management system (SpiraTest is a terrific option for this). Dealing with flaky tests is no rocket science. It is manageable if your test management system provides visibility, history of runs, attributes and structure, which allows you to easily make decisions about each particular test run.
In an ideal world, all tests would be Black Box, but obviously the approach has its own intrinsic limitations. There are alternatives, however. Sometimes unit testing and API level testing is considered as an alternative to UI and Black Box testing. It is mostly free from the drawbacks described earlier. But it is also missing some seriously awesome benefits:
- Black Box Testing improves all aspects of the product. It gives an overview of the functionality. It hits complex, intermittent, design level issues that are not visible from the API or unit test level.
- Only full, system Black Box testing gives a confidence that the product is ready for real life usage.
- With API level tests, you may find issues that are, in fact, never or rarely hit by actual users and system could ship with them unfixed. So, your product releases may be blocked due to issues that are in fact not really showstoppers. A system level and black box test is the true judge for when you should consider releasing the product.
With a good set of tools, black box testing is a must-have link in the software life-cycle chain. It gives you real confidence in the software as a whole. You see the software exactly as it is seen by the users. It also helps you understand if your application is ready with all its known issues and what problems will affect the end-user experience. And you can understand and prioritize better what needs to be resolved first.
The key success factor here is the right methodology and tools to organize your development process. Inflectra’s set of ALM tools include all the pieces to get maximum benefits from the different tools and types of testing.