IoT has dramatically changed the skill set and focus in which traditional testing teams operate. The interconnection of embedded software and microchips that exchange data over various networks across appliances and hardware which a user controls remotely, IoT complicates testing. Should you use the traditional desktop applications and systems integration strategy? A firmware test strategy, or a combination of embedded software, security and communications test strategies? What components do you need? Are their unique characteristics?
A test strategy for IoT is not static but a living document that evolves and changes as the needs and technologies of the product change. Let’s look at some approaches to creating an IoT test strategy.
There is no one template of structure that fits all IoT testing. A strategy can vary from one product to the other and from one version of the product to another. First, establish the scope of what to test. In IoT there are four basic components: the devices and hardware, the communication between them, the data transferred across the communication between the machines, and the applications that process the data. Second is knowledge about dependencies with third-party products that operate with the product under test’s IoT ecosystem. A deep understanding of how the product under test operates in these components is essential to building a robust test strategy.
The strategy designer must understand the domain in which the product will function. Keep in mind that the purpose of IoT is to allow people and machines to make more intelligent decisions that improve the user’s daily life. In IoT testing, think about the user. They don’t care why an appliance isn’t working, and they especially don’t care about dependencies on third party networks, applications, and devices. The company’s name is on the thing that’s not working correctly, so that company is to blame.
Users expect flawless execution, without exception. This dynamic makes user experience a priority in IoT testing. No longer is it satisfactory to just test the requirements. Testers must know how the user uses the product, in what kind of environment the product is being used, and why they user is using it in the way they do.
Components to Test
As with all comprehensive test strategies, the standard elements such as roles and responsibilities, governance, quality metrics and reporting, priorities, reference documentation, risks, assumptions and expectations, and schedules still apply. Collaboration with key decision makers and product leadership is still essential for populating these areas of the strategy. The engine of the test strategy is the test methods and how they will be applied in the product’s evaluation. The methods are not new to product testing, but the tools, environments, and processes usually are. Test methods used for IoT often include:
- Data Protection
- Data Encryption
- Storage Data Security in local and remote clouds
Using these methods can vary greatly depending upon the needs and culture of the organization building the IoT product. Tests can be individually executed on targeted aspects of the product such as the hardware or the data. Or as a group of tests for just the machine interactions and communications, while others are for the user interfaces and data. Understand why a test is being used and how the test will help prevent and identify product defects.
Tools to Use
There are a variety of tools available for the various components of IoT testing. I won’t list them here, but unless the tools have been validated through a good proof of concept they should not be part of the test strategy. Use automation tools that provide thorough code coverage, easy execution with each build, rapid response results, and metrics. A well-structured test framework that has automation driving many of the testing processes should be part of every IoT strategy.
The next step in building a strategy is addressing the challenge of constantly changing test scenarios which cover all the various permutations, data types and various usage environments. Also, is the product dealing with an app to machine, a machine to machine, AI, or cloud aspect with communication? Replicating and executing every possible scenario can be daunting. Concentrate on the most common and critical aspects of the product. Detail out the scenarios in each test sprint and provide addition tests to evaluate the product in the sprint. Don’t weigh the strategy down with all the possibilities.
The Test Environment
An IoT environment should mimic the hardware and devices and have a good test data management solution. Networking of device protocols, hardware, operating systems, and firmware need to be set up to represent a full ecosystem. Use virtualization to create the test environment. Generate realistic test data. This can be a separate document or included in the strategy, but it is essential to test viability.
I hope this guides you to build and effective IoT test strategy. Establish scope, understand the domain, and the third party dependencies. Then apply that knowledge to the types of tests to evaluate the product in an environment that mimics the real ecosystem in which the product will operate with real data.