D4.1 – Framework for studying existing IoT testing solutions - probe-it

Transcription

Project Deliverable
Project Number:
Project Acronym:
288315
PROBE-IT
Instrument:
Project Title:
Pursuing ROadmaps and BEnchmarks for the Internet
of Things
Thematic Priority
Support Action
Internet of things
Title
D4.1 – Framework for studying existing IoT testing
solutions
Contractual Delivery Date:
Actual Delivery Date:
September 2012
Start date of project:
October 2012
Duration:
October, 1st 2011
Organization name of lead contractor for this deliverable:
UR1
24 months
Document version:
V1.5
Dissemination level ( Project co-funded by the European Commission within the Seventh Framework Programme)
PU
PP
RE
CO
Public
Restricted to other programme participants (including the Commission
Restricted to a group defined by the consortium (including the Commission)
Confidential, only for members of the consortium (including the Commission)
X
288315
PROBE-IT
D4.1 – Framework for studying existing IoT testing solutions
Authors (organizations) : UR1
Arulnambi Nandagoban (UR1)
César Viho (UR1)
Abstract :
The document defines a framework for a worldwide study of existing testing activities including evaluation
criteria for comparison of existing IoT testing solutions.
Keywords :
Framework, IoT testing, IoT technologies, Interoperability,
Disclaimer
THIS DOCUMENT IS PROVIDED "AS IS" WITH NO WARRANTIES WHATSOEVER, INCLUDING ANY WARRANTY
OF MERCHANTABILITY, NONINFRINGEMENT, FITNESS FOR ANY PARTICULAR PURPOSE, OR ANY WARRANTY
OTHERWISE ARISING OUT OF ANY PROPOSAL, SPECIFICATION OR SAMPLE. Any liability, including liability
for infringement of any proprietary rights, relating to use of information in this document is disclaimed. No
license, express or implied, by estoppels or otherwise, to any intellectual property rights are granted
herein. The members of the project PROBE-IT do not accept any liability for actions or omissions of PROBEIT members or third parties and disclaims any obligation to enforce the use of this document. This
document is subject to change without notice.
 PROBE-IT – Page 2/42
288315
PROBE-IT
Revision History
The following table describes the main changes done in the document since it was created.
Revision
V1.0
Date
October 2011
V1.2
V1.3
V1.4
Sept 2012
V1.5
Sept 2012
Description
Author (Organisation)
Creation
C. Viho (UR1)
Contribution:
Introduction,
stakeholder
A.Nandagoban (UR1)
analysis, need for testing, Different means of
testing
Modification & Contribution: approach for
A.Nandagoban (UR1), C.
choice technology, test types & methods, Viho (UR1)
Framework definitions for studying IoT testing
solutions, Evaluation criteria
Update testing methodology and framework
C. Viho (UR1)
definition, document reorganization
Update with comments obtained from
A.Nandagoban, C. Viho
partners and finalization of the document
(UR1)
288315
PROBE-IT
Table of Content
1. A CRONYMS AND DEFINITIONS
6
2. INTRODUCTION
7
3. WHAT IOT TECHNOLOGIES TO CONSIDER
9
3.1. Stakeholder analysis ............................................................................................................... 9
3.2. Considering top IoT domains, applications and services ......................................................... 10
3.3. Market feedback on top IoT technologies .............................................................................. 11
3.3.1. Bluetooth Low Energy..........................................................................................................................11
3.3.2. Zigbee ..................................................................................................................................................12
3.3.3. IETF communication standards (6LoWPAN & CoAP) ...........................................................................13
3.3.4. Very short range wireless communication (NFC & RFID): ...................................................................14
3.3.5. Wireless standards for process control industry (WirelessHART and ISA 100.11A): ..........................16
3.3.6. Wireless standard for home and building control (KNX & Z-Wave) ...................................................17
3.3.7. Lower layer technologies (IEEE 802.15.4 & WiFi IEEE 802.11) ...........................................................18
3.3.8. Machine to Machine (M2M) ...............................................................................................................19
3.4. Considering books and surveys on IoT technologies ............................................................... 20
3.4.1. Interconnecting Smart Objects with IP ................................................................................................21
3.4.2. Architecting the Internet of Things......................................................................................................21
3.4.3. The Internet of Things –Key applications and Protocols .....................................................................21
3.4.4. M2M communications –A systems approach .....................................................................................22
3.4.5. What recent surveys says ....................................................................................................................22
3.4.6. News, articles & other information .....................................................................................................24
3.4.7. 2011 best IoT consumer product awards ............................................................................................24
3.4.8. Studying existing reference models for IoT .........................................................................................25
3.5. Synthesis and selected IOT technologies ............................................................................... 27
4. TYPES OF TESTING TO CONSIDER
28
4.1. The need for methodology in testing ..................................................................................... 28
4.1.1. Cost reduction issue ............................................................................................................................28
4.1.2. Time reduction issue............................................................................................................................29
4.2. Different types of testing ...................................................................................................... 29
4.2.1. Conformance testing ...........................................................................................................................29
4.2.2. Interoperability testing ........................................................................................................................30
4.2.3. Robustness testing...............................................................................................................................30
4.2.4. Safety/Regulatory testing ....................................................................................................................30
4.2.5. Synthesis and conclusion .....................................................................................................................31
4.3. Different approaches for interoperability testing ................................................................... 31
4.3.1. Empiric approach for testing ...............................................................................................................31
4.3.2. Methodological approach for testing ..................................................................................................31
4.4. Different means for testing ................................................................................................... 33
4.4.1. Test beds ..............................................................................................................................................33
4.4.2. Test houses ..........................................................................................................................................34
288315
PROBE-IT
4.4.3. Certification programs .........................................................................................................................34
4.4.4. Interoperability events: Plugtest/Connecthaton .................................................................................34
5. F RAMEWORK DEFINITION FOR STUDYING IOT TESTING SOLUTIONS
36
5.1. Technical overview of a technology ....................................................................................... 37
5.2. Testing overview of technology ............................................................................................. 37
5.3. Evaluation criteria ................................................................................................................ 38
5.3.1. Test coverage .......................................................................................................................................38
5.3.2. Test method.........................................................................................................................................38
5.3.3. Test tool ...............................................................................................................................................38
5.3.4. Means for testing.................................................................................................................................39
5.3.5. Cost of testing ......................................................................................................................................39
5.3.6. Time for testing....................................................................................................................................39
5.3.7. Return on investment ..........................................................................................................................39
5.3.8. Discussion on evaluation criteria .........................................................................................................39
5.4. Metrics for evaluation criteria ............................................................................................... 40
5.4.1. Metrics for evaluating test coverage ...................................................................................................40
5.4.2. Metrics for evaluating test method .....................................................................................................40
5.4.3. Metrics for evaluating test tool ...........................................................................................................40
5.4.4. Metrics for evaluating means for testing ............................................................................................40
5.4.5. Metrics for time for testing .................................................................................................................40
5.4.6. Metrics for evaluating cost of testing ..................................................................................................40
5.4.7. Metrics for evaluating return on investment ......................................................................................40
5.4.8. Synthesis on metrics for evaluation criteria ........................................................................................40
5.5. Conclusion and synthesis ...................................................................................................... 41
6. BIBLIOGRAPHY
42
288315
PROBE-IT
1. Acronyms and Definitions
Acronyms Affaires
BTLE Bluetooth low energy
CoAP Constrained Application Protocol
CoRE Constrained RESTful Working group
IEEE - Institute of Electrical and Electronics Engineers
IETF Internet Engineering Task Force
IoT Internet of Things
PHY Physical layer
PROBE-IT Pursuing Roadmaps and Benchmarks for the Internet of Things
RF Radio Frequency
RFC Request for Comment
WSN Wiresless Sensor Nodes
1.1.
11
13
14
30
12
7
13
7
11
13
14
Definitions
Testing: Take measures to check the quality, performance, or reliability of (something), esp. before putting
it into widespread use or practice.
Conformance: The ability of a component to behave as foreseen in specifications or standards it is based
on.
Interoperability: The ability of two or more systems or components to exchange data and use information
successfully
288315
PROBE-IT
2. Introduction
The main purpose of this document is to provide a framework for studying worldwide existing
testing solutions for the Internet of Things (IoT). The interest of PROBE-IT (Pursuing Roadmaps and
Benchmarks for the Internet of Things) in engaging this work is to provide elements for future
research agenda that supports IoT testing and validation. This is a huge challenge as there are
different approaches to be used for this work. Indeed, IoT covers numerous domains with multitude
of applications and services, based on different kinds of technology and protocols. Depending on the
approach used, one may cover different aspects of the IoT. In this study, we decided to use IoT
technology-based approach. We will explain in details reasons that bring us to choose this approach.
On the other side, there are several types of testing. Since IoT is a very big complex paradigm with
several domain involvements, we argue that a basic key requirement of IoT deployment success is
interoperability. We stress the importance of interoperability from the IERC definition of IoT which
states that “The Internet of Things is an integrated part of Future Internet and could be defined as a
dynamic global network infrastructure with self-configuring capabilities based on standard and
interoperable communication protocols where physical and virtual “things” have identities, physical
attributes, virtual personalities and use intelligent interfaces, and are seamlessly integrated into the
information network.” Though there are different types of tests involved in a product design and
development, we focus on conformance and interoperability testing issues and its future prospects.
Both these testing avoids human made errors in development, interoperability issues and
conformance issues from an implementation. Since from our expertise, we know the important
impacts of conformance and interoperability issues in market deployment of a product. Today, we
have different communication technologies and we like to use it for various IoT applications. It
creates new requirements and complexities in adopting existing technologies to be used in IoT
paradigm. It makes testing as more important thing for IoT to ensure the quality of product or
service to be deployed. As the number of communication technology standards g row day by day, it is
more and more difficult for a company to select a standard that satisfy their stakeholder
requirements and it needs to be interoperable with the existing solutions.
Another argument that motivates this work is that we also came to see the testing (more
particularly, conformance & interoperability) as an important perspective that enables the standards
to mature and push its deployment. It is well known fact that for any product to be interoperable,
the product should have developed from a matured standard and tested with standardised testing
method with accurate tests as depicted in Figure 1
INTEROPERABLE PRODUCTS
CONSISTENT
STANDARDS
STANDARDIZED
TESTING
METHODOLOGIES
WELL-SUITED
TESTING TOOLS &
ACURATE TEST SUITES
Figure 1: Pillars of interoperability
The work done in this PROBE-IT work-package (WP4) aims at providing an extensive overview of
existing test solutions with possible issues to be met to the stakeholder. PROBE-IT Work-package 4
organized the work with an objective of providing support to IoT testing and validation by studying
the test requirements needed for a smooth worldwide IoT deployment. In the end, with knowledge
acquired regarding main issues and requirements, it is possible to draw a research agenda for the
future of testing.
288315
PROBE-IT
In order to realize this work, PROBE-IT collects technical information on conformance and
interoperability testing of different IoT communication standards. This deliverable is a study plan for
carrying out the work for deliverable D4.2- Report on state of the art on testing of IoT technologies.
This work was a result of interaction with PROBE-IT international partners, test engineers and
knowledge obtained from workshops & test events arranged by PROBE-IT consortium. This
document was developed with an intention of providing readers with precise information on





Why we need testing for IoT communication standards?
What is the importance of conformance & interoperability testing?
What are best practices tests in testing?
What are the different approaches today in testing?
How to evaluate a test solution and what are criteria to be considered for this evaluation?
Before developing the framework, the first two questions to answer are: (i) who can be interested by
this work? (ii) Who are the main actors of this domain of IoT testing? The answer to these two
questions is: stakeholders. A study was done to identify the latters. The approach used and the
results are given in Section 3.1. . Other following parts of Section 3 explain in more details the
approach used to select IoT technologies to be studied. For each selected technology, we then give a
general description together with application domains, information on current deployment,
stakeholders, etc.
In the Section 4, different existing types of testing are explained. It then gives necessary information
on conformance and interoperability testing allowing the reader to understand the importance of
these two types of testing in the deployment of IoT technologies.
The Section 5 contains the criteria to be used to compare the IoT existing testing solutions for the
selected IoT technologies, leading by this way to the purpose of this document the definition of a
“framework for studying existing IoT testing solutions”.
288315
PROBE-IT
3. What IoT technologies to consider
The purpose of this work-package 4 is to provide a state of the art on existing IoT testing solutions.
There are so many approaches possible to carry out the study on IoT testing solutions. When we
narrow out these approaches with keeping in mind the interoperability issues of IoT, we landed in
selecting technology centric approach. To do so, we have looked at various report and studies in
particular:



Top technologies as promoted by the market, indicated in market study reports, considered by IoT
research projects or by standardization bodies, etc
State of the art technologies as analyzed by other projects like the lighthouse IOT-A IP project
Comprehensive books and surveys on IoT technologies as promoted by ETSI M2M
3.1.
Stakeholder analysis
Before developing the framework for studying IoT testing solutions, the first two questions to
answer are: (i) who can be interested by this work? (ii) Who are the main actors of this domain of IoT
testing? So the first step of our work was to identify stakeholders. The objective of this stakeholder
identification is to get a clear understanding of who are the main stakeholders and their interest for
this work. The main stakeholders for Work package 4 are listed in the figure 2. Two steps help us to
identify stakeholders and their interest.
1. First, we analysed partners of different existing IoT projects. This allows us identifying main
actors on IoT from academia/research entities, industry and standardization bodies. For all
these actors, we tried to identify what are their main interests in participating in those
projects and their main activities during those projects.
2. Second, we participate to several IoT related workshops, interoperability test events and IoT
forum events where we have direct discussions/interviews with actors identified in the first
step.
With these work, we came into identifying the stakeholders. The stakeholders indicated in the
document are the direct benefiters of this work. Table 1 below describes the stakeholders’ interests:
Stakeholder
Stakeholder interest
Policy makers
Current testing activities; Focus on issues of quality;
Investment in future research on IoT
Standardisation organisation
Testing solutions that help the standard to mature
Companies (Application developers, device IoT standards with good testing solutions.
manufacturers, Network designers)
Research organisation
The current research activities on IoT testing
Table 1: Stakeholders’ interest
288315
PROBE-IT
Stakeholders
Policy makers
Standardisation
organisation
Research
organisation
Companies
Standard
developers
Academia
Application
developers
Industrial fora
Governement
research
agencies
Device
manufacturers
Vendors
Figure 2: Stakeholder analysis
3.2.
Considering top IoT domains, applications and services
To consolidate the choice of technologies to be studied, we considered different criteria such as IoT
business models (most popular applications), networking capabilities & power consumption of the
devices used to implement the IoT applications, cost for developing the application or cost of
chip/hardware used to develop those applications and size of the device to select different
technologies for this study. We emphasized on few criteria for selecting technologies such as: IP
connectivity (by means of gateway or adaptation layer support), Power consumption factors, Cost
and size of the device (these standards can be implemented different types of hardware still we
considered the most smallest). There are two more major factors that drive the choice of
technologies: market occupation and future market growth.
Figure 3 summarizes the approach for selection of the communication technologies. Figure depicts
different criteria for selecting a technology from the diversified pool of communication standards.
288315
PROBE-IT
Smart metering
Networking
capabilities
Environment
Addressing
capabilities
Smart cities
IP conectivity
Retail
Logistics
IoT applications
Battery powered
Power consumption
Cost
Huge number of
devices deployment
Industrial automation
Deployed in complex
environment
Consumer electronics
Automobile
Branding
Well-known
applications
Push from big
companies
Existing sucessful
market occupation
Huge deployment
Choice of IoT techologies
Size
Home automation
World wide
acceptance
Market research
Potential future
market growth
Expert talk
Figure 3: Approach for selecting IoT technologies
3.3.
Market feedback on top IoT technologies
Selecting an IoT technology from the pool of existing communication technologies is challenging. We
consider that, whatever approach is used to make this kind of study, it is important to take into
account current deployment and market acceptance and forecast. Moreover, objective is to analyse
testing activities of technologies which are already present in IoT eco system or which may find a
place in IoT eco system in future. We investigated the criteria such as market feedback, cost,
connectivity and size to verify their potential presence in IoT arena. First criteria we investigated
were market feedbacks and forecasts. Here we list the technologies with above said criteria to
understand their promising presence in IoT area. The factors, which allow a communication
technology to be IoT enabled, are: Cost, Convergence towards IP, Connectivity (interoperability),
energy consumption, computing power and smart applications. To analyze a list of top ten IoT
technologies, we have looked at various reports and studies. The findings are described in the
following.
3.3.1. Bluetooth Low Energy
The introduction of Bluetooth low energy (BTLE) technology has seen considerable interest from IoT
enablers due to its technical features compared to Bluetooth classic. It is quite different from
Bluetooth classic in power consumption, number of nodes as well as cost wise. The technical
features of BTLE could satisfy the requirements of IoT application domains like Industrial
automation, Medical application, Home automation etc. It supports large networks with more
number of nodes, low latency, robustness towards RF (Radio Frequency) interferences, short wake –
up time etc.
288315
PROBE-IT
Cost: As we all know, when first Bluetooth chip introduced in the market, the price point was
around $20. Ten years later, it has fallen to $1 with export volume of a billion chips per year. The
price matters when a range of similar products (like BTLE) is available in the market. It was designed
to be low cost even in low volume. BTLE in 2013 will be less than $1, after ten years it may reduce
less than $1. It is worth using a BTLE instead of cable technology(Creative connectivity, 2010).
Connectivity & IP Convergence:
Figure 4: BTLE wireless connectivity & IP convergence (Source: Nokia)
The network topology used in BTLE is star-bus. It supports one-to-many connections. Billions of
devices can be connected using star-bus network topology. This network scale is one of the factors
that enable IoT in BTLE. The basic idea is: things have the data and web applications need those data
to process. How is it possible with BTLE? BTLE enabled sensors communicate locally with a central
node such as Smart phone, personal computers, tablets etc.(Nieminen, 30 May 2012). Today, IP
convergence is enabled by using those central nodes such as smart phones to act as a gateway
between Internet and BTLE as shown in Figure 4. Internet Engineering Task Force (IETF) develops a
specification for transmitting BTLE over IPv6. It adopts 6LoWPAN standard to support BTLE since
BTLE and IEEE 802.15.4 has similar technical characteristics.
Next, we talk about energy consumption in BTLE. We all know how important is energy consumption
in IoT enabled devices. Most of the IoT enabled devices are sensors. They are deployed in a location
where human intervention is complicated as well as size of device is so small that it cannot
accommodate bigger batteries. These issues were clearly understood in designing BTLE which
operate with coin sized small batteries for many years.
Future market for BTLE: There are so many smart applications and devices available in the market,
which already proved BTLE as an IoT enabled communication standard. Analyst research firm IMS
estimated by 2013, 1 billion BTLE devices will be sold every year. Increased health conscious among
the people will create a huge market for devices that monitors their health(htt). IMS research recent
study forecasts that 35 % of wireless-enabled consumer medical devices shipped in 2016 will contain
Bluetooth Smart(Senior Analyst Lisa Arrowsmith , 2012).
3.3.2. Zigbee
IEEE 802.15.4 is a lower layer technology that supports the applications that requires low bit rate
and lower energy consumption. It is mostly used for sensor and other control applications. In reality,
those applications require a mesh networking capabilities and standardised message exchange. In
288315
PROBE-IT
order to fulfil these requirements, Zigbee alliance was formed to develop a standardized network
and application layers for sensor applications. Will Zigbee account for enabling IoT? The answer is
yes. How does it do? We see it in the following lines.
Cost: The cost is the major driver in successful market deployment of a product. What means a low
cost for user? It is not only lower power consumption but also low retail, maintenance and
installation costs. The reason for low cost Zigbee is its low cost IEEE 802.15.4 PHY/MAC adaptation,
self-healing network capability that requires less maintenance, extensive use of reduced function
devices and its simple design(Setting Standards for Energy-Efficient control networks). The
procurement of considerable volume of Zigbee chips costs around $1 in the current market.
Connectivity & IP convergence: Zigbee is success because of its networking capabilities. The
important challenge for IoT devices is to overcome the complexity of communication in very large
networks. Mesh networking capability of Zigbee allows overcoming the problem of communication
in large scattered network. It also supports star and bus network topology. Current requirement of
exchanging data between Zigbee enabled sensor nodes with web application is made possible by
Zigbee smart energy profile and Zigbee IP gateway.
Future market for Zigbee: According to ABI research, nearly 55% of IEEE 802.15.4 chip shipments will
be comprised of Zigbee. Another research agency called IMS research forecasts 1.1 billion of IEEE
802.15.4 chip shipments between 2009-2015 in that 85% of the chips contains Zigbee software stack.
ON World says that Zigbee will capture half of the global home management market by 2014(www3).
The great contribution of Zigbee in IoT applications (such as Home automation, smart energy, Light
control applications, smart metering etc.) ensures its place in future IoT market.
3.3.3. IETF communication standards (6LoWPAN & CoAP)
The concept emphasized by IETF is IP over everything and everything over IP. IETF works to enable
the constrained nodes to communicate with IP network. It designs protocol for constrained nodes to
adapt IP network layer, transport layer and protocols for routing. 6LoWPAN is popular protocol
designed to adapt IP functionalities for lower layers of sensor nodes. Same way, Constrained
Application Protocol (CoAP) is designed to provide HTTP like functionalities to sensor nodes. IoT view
of IETF is connecting anything and everything to IP.
6LoWPAN: IETF developed the 6LoWPAN standard under Request for Comment (RFC) 4944. The
complexity of carrying IPv6 packets over IEEE 802.15.4 is PHY (Physical layer) packet of IEEE 802.15.4
is 127 bytes whereas IPv6 header information itself account for 40 bytes. This problem handled
successfully by compressing IPv6 Header compression under IETF Request for comment 6282. These
specifications are open and developers can use open source codes, or develop their own thus saving
money. The key feature of 6LoWPAN is it enables communication with IP device directly without
need any gateway or translation device. This way it reduces the cost spend on gateways.
“One of the most disruptive developments over the past couple of years is the emergence of
IPv6/6LoWPAN,” says Mareca Hatler, ON World’s research director
288315
PROBE-IT
igure 5: Comparison between 6LoWPAN, Wi-Fi tag and Zigbee (source: MIMOS Bhd, Malaysia)
The requirements that enable IoT in communication standards are Low price, Low power, network
scalability, mobility and mesh networking capacities. From the figure, we can see a comparison
between Wi-Fi tag, Zigbee and 6LoWPAN and we can note that 6LoWPAN achieves above said IoT
requirements today. Today we can see many propriety communication standards adopt 6LoWPAN
for connecting their devices to IP networks. The list includes Zigbee, BTLE, and ISA100.11a etc.
CoAP: IETF Constrained RESTful Working group (CORE) involved in the standardisation of CoAP
protocol. It is an application layer protocol developed for sensor networks where minimum usage of
memory and power is required. CoAP protocol works with 6LoWPAN and IEEE 802.15.4 in its lower
layers. 6LoWPAN enables Internet and CoAP enables web service functionalities to the resourceconstrained nodes. 6LoWPAN with CoAP reduces the complexity of developing application for
Wiresless Sensor Nodes (WSN).
Market research forecasts of IEEE 802.15.4 devices again favours 6LoWPAN and CoAP devices success in
the future market.
3.3.4. Very short range wireless communication (NFC & RFID):
One of the important applications of IoT is security & tracking application. RFID find its mass
deployment in retail, consumer goods, aviation, healthcare and tracking applications. NFC finds its
market in mobile payment. Both these short-range wireless technologies enter into IoT paradigm
with its application domain. Will these technologies satisfy IoT requirement? The answer is definitely
yes.
Cost & future market of RFID: According to Bridge project, a technology breakthrough will reduce
cost of an RFID tag to less than 1 cent. The number of RFID tags on food items could grow to
hundreds of billions. ID tech EX forecasts that RFID can find a profitable market in animal, food and
farming domains by 2021.
Figure 6: Animals, Food and Farming Systems value vs Tag value globally in 2021 (source: ID tech EX)
288315
PROBE-IT
38.8% of animal, food and farming domain in 2021 will account for RFID tag value. It also forecasts 1.35
billion RFID tags in 2031 for retail domain. The requirement of RFID Transit ticket will keep growing.
The growth of sales for passive RFID tags will reach $10 billion by 2019.(www4)
Table 1: RFID market & deployment (source: BRIDGE project)
Bridge project analysis predicts in five years the deployment of passive RFID readers will be 170,000
at 30000 locations in Europe. These readers require 3 Billion tags to process. This number will grow
significantly in the future. This project expects a deployment and purchase of 6 million readers with
86 Billion tags per each year after 2022.
Connectivity:
Figure 7: Communication scenario for RFID (source:(Sandra Dominikus and Jorn-Marc Schmidt) )
IoT put emphasis on connectivity but the real value of RFID business is data. RFID has the
information about the type of an object and its environment. Thus, it allows us to manage the
objects automatically and cost-effectively. RFID majorly account for collecting information than
networking it. But when it enters into IoT paradigm, it needs to communicate the data it collected.
How is this possible? It is possible by connecting RFID to any Internet device, cellular device or other
WSN. This way RFID accomplishes the connectivity to enter into IoT arena. Figure 7 shows an
example communication scenario where RFID connects to IPv6 network. In this concept, RFID reader
acts as a translator between RFID tags and IPv6 network. The usage of IPv6 solves th e problem of
addressing complexity for RFID(Sandra Dominikus and Jorn-Marc Schmidt).
Cost & Future market of NFC: The interest shown by big business giants such as Google, Apple,
Amazon and Facebook towards NFC will be a positive value for NFC in future. NFC found its place in
consumer devices in 2012 through its implementation in newly arrived Samsung Galaxy S3 smart
phone. According to strategyFacts report, NFC will be standard component on all smart phones in
the future (90%). The NFC will find a major market in contactless payment in the future. NFC will coexist with other technologies but better business model will make it a big success in the future.
288315
PROBE-IT
Connectivity: Similar to RFID, NFC uses other communicating devices as a gateway to share the
information it collected. Libelium has introduced a new RFID/NFC module for its Waspmote sensor
platform. In this system, NFC interface is used to collect the information from the passive tags and
communicated to web via Zigbee, which acts as a gateway between Internet and NFC. It is an
example and this way NFC enables the end-to-end communication to web applications.
Energy consumption of RFID & NFC: The key feature of both these technologies is its energy
consumption property. The size and its passive device, which doesn’t use any external energy
source, will make these technologies to boost up in the future market.
3.3.5. Wireless standards for process control industry (WirelessHART and ISA
100.11A):
While we discuss about WirelessHART and ISA 100.11A, we may have many questions in our mind
that why this two similar technologies exist for process control. We know that there is no
considerable difference between both these technologies from technical po int of view. Both use
IEEE 802.15.4 in their lower layers. Both offer mesh-networking capabilities. Both use frequency
hopping on top of the direct sequence spread spectrum. Both operate on 2.4 GHz ISM band. The
question is how they co-exist: big process control instruments and control valves giants support
WirelessHARt, which makes WirelessHARt to co-exist with ISA 100.11A.
Future market of WirelessHART & ISA 100.11A: WirelessHART dates back from 80’s and has a large
deployment in the process control in the industry. It still shows growth even its competitor Profibus
and Foundation fieldbus, have started using wireless technology. HART constantly improving its
standard and taking care of interoperability issues which makes it a prominent leader in the Process
control sector. According to ONWORLD, despite the market battle between WirelessHART and ISA
100.11A, WirelessHART users got tripled with comparison to 2010. The interest of ISA 100.11A in
the market is its flexibility to adopt any wireless hardware technologies and IP addressable devices.
When an application needs latency of less than 100ms, then in that case IEEE 802.15.4 is unusable. In
that case, ISA 100.11 A ‘s higher layer standards designed to adopt other wireless technologies that
can provide latency less than 100ms. The design of ISA 100.11 A supports the adoptability of
different physical layers, has mesh networking capabilities with IPv6 addressable devices which
enables ISA 100.11a to be success in the future in the M2M market.
Table 2: Expected trend in wireless networks. (source: VDC research )
From the table, we can see the future trends for WirelessHARt and ISA 100.11a that grows
consistently in the market. According to the recent research indicated in VDC research predicts the
288315
PROBE-IT
growth of 40% adoption of WirelessHART and 31% (double in 2014) adoption of ISA 100.11a
technologies.
Connectivity:
WirelessHART
ISA100.11a
IPv6
6LowPAN
IEEE802.15.4
MAC
2.4GHz DSSS
IEEE802.15.4
MAC
2.4GHz DSSS
Figure 8: Protocol stack of WirelessHART & ISA 100.11a
As per as connectivity, ISA 100.11a seems to be a better solution than WirelessHART. ISA 100.11a
adopts 6LoWPAN in its lower layer for providing IPv6 connectivity. Main disadvantage of
WirelessHART is not compatible with other networks. It requires gateways and adapters for
communication. WirelessHART is seems to be a wireless enhancement to the HART devices, which
are already deployed in the industry environment.
3.3.6. Wireless standard for home and building control (KNX & Z-Wave)
KNX is worldwide standard for home and building control. KNX uses its own propriety solution for all
layers from physical to application. The main advantage of this technology is it can support a variety
of physical layer technologies including wireless communication. Like KNX, Z-Wave is also a key
player in home control market. It has its own layer design from physical to application. Are KNX and
Z-Wave part of IoT? : The answer is definitely yes because of their application domain and other IoT
properties such as connectivity, power consumption and cost. KNX was designed to support smart
home and building applications. It also provides a fair interoperability through its certification
program.
Market for KNX & Z-Wave: The recent study made by BSRIA says that KNX account for 70% of smart
home market in Europe as shown in the Figure 9. This study analyses different European market and
summarizes it. Smart home market still has room for growth and is expanding day by day. According
to BSRIA, Germany home automation manufacturers are market leaders and the n come France and
UK. Luxury proprietaries count for two thirds of market. The second important focus group is small
and medium sized businesses, which use these smart home solutions in their commercial buildings,
hotels and restaurants.
288315
PROBE-IT
Figure 9: Current market situation of Smart home/Home automation (source: BSRIA)
Joost Demarest is one of two Directors of KNX International says: “If the Internet of Things is not yet
a fact in five years' time, KNX will at least continue to show that its proven technology is the best of
its kind in the area of home and building control when it comes to interoperability, openness of
technology, tools and connectivity to the Internet. If the Internet of Things is already a fact, KNX will
still have been able to contribute its interoperability and its more than 20 years of ex perience in the
area of home and building control fully up to level of IP.”(Guest Interview: Joost Demarest, the KNX
Association, 2012)
Z-wave certified 600 products on January 2012 and it announced its 650th product certification on
June 2012 [10]. This certification demonstrates the rapid growth of Z-Wave. It is key enabler in IoT
arena. Similar to WirelessKNX, it plays its important role in home and building co ntrol sector.
Verizon, which is multi sectorial industry giant, joined Z-Wave alliance recently after the launch of its
connected home solution, which uses Z-Wave. Two domestic markets, japan and Brazil started to
implement the Z-wave technology from August 2012. Vesterbet indicated: Many home automation
experts and business analyst expects that home automation market will expand significantly in the
years to go in Japanese and Brazilian market.
Connectivity & Power consumption: Whenever we talk about IoT, we indicate IP convergence or
communication with IP network as an important criterion. In case of KNX, IP support is provided in
the routing device. It is positive sign for KNX to enter into IoT world. But still it has potential issues
in IP communications. For example, communication from KNX Subnetwork to router always uses
multicast addressing. If an IP datagram got lost, the sender cannot detect it since multicast address
is not acknowledged. IP tunnelling is handled in KNX which enables the access from any point in the
network is possible. From connectivity point of view, KNX devices are IoT enabled. Since KNX
standard can support wide variety of RF hardware, it offers variety of choice for users to select their
desired hardware, which provides lower power consumption feature. Z-Wave supports IP
networking by using a gateway. Thus enabling the control from some web applications.
3.3.7. Lower layer technologies (IEEE 802.15.4 & WiFi IEEE 802.11)
Market for IEEE 802.15.4 & IEEE 802.11: Most of the technologies discussed earlier uses IEEE
802.15.4 in their lower layers. IEEE 802.15.4 is the most important technology in WSN, M2M and
directly in IoT. It enabled numerous upper layer standards to enter into IoT paradigm. According ABI
research, IEEE 802.15.4 IC market foresees a growth rate of 60% every year from 2010 to 2016. It will
288315
PROBE-IT
expand its export by 850 million units by 2016. The growth will be seen in all popular domains of
IoT.
Figure 10: Wireless LAN & WiFi phones market forecasts (source: Infonetics Research )
According to an article from Herald online(www5), “Wireless LAN has had a very good run over the
last couple of years, even outperforming wired LAN, and the WLAN market is now approaching the
$1-billion-per-quarter mark,” says Matthias Machowinski, directing analyst for enterprise networks
and video at Infonetics Research. WiFi market grew 16 % from second quarter of 2012. In Europe,
Middle East, Africa, it grows 1/3 every year. This research tracked potential companies involved in
WiFi market.
Connectivity & Power consumption of IEEE 802.15.4 & IEEE 802.11: Since IEEE 802.15.4 allows its
upper layers to implement all types connectivity option; it plays a vital role in IoT market. We see so
many protocols such as IPv6, WirelessHART, and Zigbee etc. are using IEEE 802.15.4 as their lower
layer protocol. Since its development IEEE 802.15.4 was playing a potential role in WSN. Most of
IEEE 802.15.4 devices are battery powered and deployed in most complex locations. There were
numerous study were done to analyse and improve its power consumption techniques.
How IEEE 802.11 contributes to IoT regarding connectivity? – Most of the cases, IEEE 802.11 device
serves a gateway for sensor nodes and other devices of standards such as Bluetooth, NFC, RFID etc.
This way it acts as a translator for the constrained nodes to communicate to Internet. For power
consumption, various techniques were used still it is a challenge for this type of protocols that carry
enormous amount of data.
3.3.8. Machine to Machine (M2M)
The definition of IoT says that the objective is connecting things everywhere and every time. Here
‘thing’ refers to the device with less computational functions or even without any computational
functions. M2M is defined as connecting computing devices each other everywhere and every time
from point of view cellular technologies. We can clearly say that M2M is a subset of IoT.
Market:
288315
PROBE-IT
Figure 11: M2M cellular market by ABI research
Like the diversified technologies involved in M2M, the M2M market is also diversified from retail to
healthcare. Connecting anything and everything is possible with great amount of technical challenges.
The number of M2M connections will rise from 110 millio n connections in 2011 to approximate 365
million by 2016 as per ABI research as shown in the Figure 11. The number of connected cellular
units amounts 35 million in 2011 and it is expected to triple in 2015 with a number of 95 million
units(www6). Another forecasts predict, 12 to 50 billion M2M devices by 2020. Machina research
forecasts an eight-fold increase in revenue from 91 billion dollar in 2010. How a M2M market look
like and where to find opportunities for M2M? It finds its place in the market where M2M
functionalities to be implemented to the existing non-communicating products and new applications
which needs M2M functionalities. Whatever the business model is for M2M, smaller and cheaper
device will be driver for future market growth. The increase in number of smart phone and tablet
devices with thousands of applications will secure a separate market place in the future.
Figure 12: M2M architecture (Source: ETSI)
Connectivity: Connectivity is a potential driver for M2M growth. Connectivity with interoperable
products enhances its position in the market. The communication in a M2M network comes from
different types of media such as wireless or wired lines, electrical lines, satellites etc. A simple M2M
architecture is depicted in the Figure 12. M2M network consist of numerous gateways to ease the
communication between different communication standards.
3.4.
Considering books and surveys on IoT technologies
In order to have consolidate the identified top technologies as from market size, we also consider
the point of views indicated in important IoT related books such as “Interconnecting Smart Objects
with IP by Jean-Philippe Vasseur & Adam Dunkels”, “Architecting the Internet of Things ”,”Internet of
288315
PROBE-IT
Things- Key applications and Protocols” & “M2M communications-A systems approach”. In addition
to that, we considered IoT-A deliverables and different IoT related articles & surveys. A short
description of information gathered is given in the following.
3.4.1. Interconnecting Smart Objects with IP
This book describes the IP based smart objects architectures with discussing
IoT application domains. It was written by JP Vasseur that is an active
member of IETF and co-author of more than 3 RFCs/Drafts. He is also chair
of the Technology Advisory Board of IPSO alliance & Adam Dunkels who
authored more than 40 papers on embedded IP and author for open source
Contiki operating systems. This book gave us in understanding about IoT and
technologies involved in building with different application domains. It
discusses about IP in smart objects and other technologies involved in
enabling IP in smart objects.
3.4.2. Architecting the Internet of Things
This book discusses about different IoT architecture and business
models. It is edited by Dieter Uckelmann (MD of the LogDynamics Lab at
the University of Bremen), Mark Harrison (director of Cambridge Auto-ID
lab) and Florian Michahelles. (Project Manager & Associate director of
the Auto-ID lab). This helps us to precise the definition of IoT for our
work. It discusses different business opportunities for IoT. This book
helped us to take technology centric approach for our task in PROBE-IT.
3.4.3. The Internet of Things –Key applications and Protocols
This book point out different technologies and their ability to part of IoT for
industrial automation and smart grid. Standardization Organizations such as
ETSI are also in direct contact with market demand and need as most of
their members are from industry. IoT is an important topic addressed by
one of the ETSI committees called M2M (Machine to Machine). Omar
Elloumi, Alcatel Lucent vice-chairman of ETSI M2M, David Botswawick , the
ETSI technical officer of M2M together with Olivier Hersent one of the active
member have published a book revised in January 2012 called “ The Internet
of things: key applications and protocols”
The book is very useful to give point of views from this standardization body
and important IoT applications and protocols such as: IEEE 802.15.4, BACnet
protocols, LonWorks, ModBus, KNX, Zigbee, Z-Wave, 6LowPAN, Zigbee
Smart energy etc.
288315
PROBE-IT
3.4.4. M2M communications –A systems approach
This book discusses about the M2M business developments and
technological developments regarding M2M. It also points out the latest
technologies currently in development from ETSI and 3GPP. Regarding the
information for Probe-IT WP4, we were interested in knowing about the
market interest for the different M2M technologies. This book is released
very recently but still considered it useful to verify our selection of
technologies.
3.4.5. What recent surveys says
Thanks to participation of Brazilian partner Perception into Probe-IT, an IoT competitive association
was created (http://www.iotbrasil.com.br/) to help gathering IoT interest in Brazil as well as helping
to contribute to the Probe-It project. A survey made and presented during the IoT seminar held in
April, which used as a input for discussion forum about SWOT analysis of wireless and wired
standards for IoT reveals the potential technologies for two important application domains of IoT
such as Consumer electronics and Smart grid. We already saw billions of smart phones acts as a
platform for enabling IoT technologies such as Bluetooth, Wifi, NFC etc. found in the market today.
Figure 13 & Figure 14 shows an extract from the IoT event presentation held on 2012.
Figure 13: Survey on IoT technologies for CE
288315
PROBE-IT
Figure 14: Survey on IoT technologies for Smart metering
From the survey, we can see clearly that the occupation of WiFi in Consumer electronics and Zigbee in
smart metering.
On august 2012, PROBE-IT made a survey for its interoperability workshop planned for 2013. More
than 40 companies participated in the survey.
Figure 15: Survey by Probe-IT
From the survey we can conclude that the most popular technologies in the area of IoT. Again WiFi
wins with more number of implementations when compared to other technologies.
288315
PROBE-IT
3.4.6. News, articles & other information
Recently, Libelium released an article, which lists 54 most popular
IoT applications, which uses Libelium hardware platform classified
by their application domains. Interesting factor in this article is that
it gave awareness to the IoT society to create new ideas for IoT
applications. The most captivating thing for us was the different
communication technologies listed for each application. It gave us
an insight of popular technologies from viewpoint of applications
and applications domains. We can see the technologies such as
NFC, RFID, Bluetooth, WiFi etc. found in many applications listed in
the article.
Postcapes is a popular website which was created from
inspiration from different IoT thinkers. They are passionate
about providing information around IoT. They provide with
timely news, experts views, news regarding IoT events etc. We
read an article on postcapes about IoT communication
technologies and found different technologies such as NFC,
6LoWPAN, WiFi, etc.
3.4.7. 2011 best IoT consumer product awards
Postcapes announced best consumer product award in 2011. The list of product s can be seen from
postcapes website. Here we list the winner and runner up product to give an idea of IoT products,
which we used for selecting our approach for this work.
Rymble is the winner of 2011 best IoT consumer product award.
Rymble can connect to almost any Internet service using its
embedded Wi-Fi module and the Symplio service platform. The
services provided by this product are: It connects to your social
network and reflects the new trends by changing its shape and
colour. We can see the WiFi acts as a backbone for this device
to connect with Internet.
288315
PROBE-IT
Olly is the runner up in the contest. It is a nice device, which
connects to the facebook and twitter account. Whenever, a
new post, notification or comment given to the account, it will
release a good scent to inform you. It has arduino hardware in
it.
Big giants like IBM, HP and Vodafone are also involved in developing IoT solutions for the current
market with use of different communication technologies.
3.4.8. Studying existing reference models for IoT
Many research projects (including FP7 European projects), industry alliances and standardisation
organisations proposed various IoT reference models as those proposed by the CASAGRA2 project,
IoT-I, IoT-A, etc. At this point of time, there is no common agreement about an IoT reference model.
The CASAGRAS project has proposed an inclusive model and CASAGRAS2 has extended the model to
consider Internet independent and latent IoT applications. In the final report of CASAGRAS2, they
proposed a nine layers protocols stack model. The ITU-T proposed also a reference model for IoT
with four layers and two transversal capabilities called management and security capabilities.
Another possible model is the one provided by the ETSI M2M specifications work that is more
application domains oriented. After studying all these models, the one that appears to us to be more
accurate for studying existing IoT testing solutions is the one proposed by the IoT -A project.
3.4.8.1. The IoT-A reference model
From (IoT-A : Deliverable D1.3 –Updated reference model for IoT v1.5, 2012), IoT-A tree represents
IoT applications as leaves and flowers while interoperable communications standards as roots.
Taking this tree representation, we see that business as light observed by the leaves (IoT
applications) and testing as water nutrient to the roots (interoperable communication standard).
Both these energy sources serve for the life of the tree (IoT system) that is depicted in the Figure 16
with IoT-A reference tree.
288315
PROBE-IT
Business innovations
(Light)
Testing (water)
IoT applications
(Leaves)
Communication
technologies (roots)
Drivers (energy source)
IoT (Tree)
Figure 16: IoT-A tree and PROBE-IT WP4 approach
According to (Dieter Uckelmann, Mark Harrison, & Florian Michahelles), research on Internet
protocols and communications technologies overlaps partially with IoT research. Anyways, even we
follow the application centric approach; we end up with analysing the technologies that support
those applications. So, we selected the communication technology centric approach for this task.
Moreover, technology centric approach provides the deep insights of testing activities than
application centric approach. However, we bring out some of the popular IoT applications in the
market to showcase the importance of technologies underneath and to manifest our choice of
technologies.
3.4.8.2. State of the art of technologies analyzed by IoT-A
The aim of IoT-A project is to provide reference architecture for
Internet of Things. In order to provide different approaches to
define architecture for IoT, deliverable D1.1 was developed. The
aim of D1.1 is to provide a state of the art report on existing
integration frameworks/architectures for WSN, RFID and other
emerging IoT related Technologies. Here, they investigated
different IoT technologies that belong to the internet of things
area. Even though the concept of IoT is well established now, the
problem of determining the set of technologies that could be
included in the IoT arena remains argumentative. In our case,
since our approach is based on evaluating testing activities from
technology viewpoint, it is necessary to determine a set of
communication technologies. For this, we considered IoT-A is a
potential source for making this decision on selecting different
IoT technologies. The technologies discussed in IoT-A context are
WirelessHART, Bluetooth, BACnet, KNX, Lonworks , Zigbee and
Sensinode implementations which supports 6LoWPAN.
288315
PROBE-IT
3.5.
Synthesis and selected IOT technologies
Our approach for selecting technologies to be considered in our study is from point of view of
market deployment and business solutions. Our approach is very close to the tree model proposed
by IoT-A. We took the tree model and our point of view to justify our approach for this work.
What is ‘technology centric approach? - It is study method to follow testing activities from
communication standards point of view. Communication standards are backbone for IoT eco system
and they create IoT’s greatest challenge ‘interoperability’. So, we see that it is holistic to carry our
study with this approach. By this way, we also map our approach with different IoT reference models
proposed today. The preference is given to the IoT tree reference model proposed by IoT-A which is
well suited for studying and analyzing existing testing solutions.
These are the part of research we did before making a conclusion regarding the selection of IoT
communication technologies. During recent years, several studies have been done on IoT, leading to
several classifications in top-ten applications or services, top-ten domains of IoT, top-ten protocols
for IoT, etc. For this work, we took 12 most important communication protocols and M2M for
analysis after a good time spent on research from books, EU projects, and markets’ studies. In the
above lines, we argued and justified our choice of communication technologies that might have a
huge market deployment in the future for IoT.
Listed below are the technologies that are expected to have a great acceptance in the future IoT
market arena.













Bluetooth low energy
Zigbee
6LoWPAN
IEEE 802.15.4
WirelessHART
CoAP
Z-Wave
ISA 100.11A
NFC
KNX
Wi-Fi
RFID
M2M
For those technologies, we will provide in the deliverable 4.2, the state of the art on interoperability
testing. For each technology, we analyse the testing aspects concerning its test methods and test
tools. To do so, we will give in the following sections, more details on conformance and
interoperability testing.
288315
PROBE-IT
4. Types of testing to consider
4.1.
The need for methodology in testing
Testing holds its important position in any product development cycle. A communication device is
generally developed based on a communication standard. These standards state the functionalities
to be implemented in a particular device. So, what do we need to test? We need to verify and
validate the functionality stated in the standards, interoperability with other device and more ov er
human error in the development. Overall testing improves the product quality and ensures the
quality of the final product to the consumer. The goal of testing is to find the error prone areas in
the development cycle. Early testing is proportional to saving money. For example, when an error is
found in the early stage of a development, it saves money and quickens the time to market for the
product. Cost and time are two important evaluation criteria for testing. Above statement can be
rewritten as testing is an investment for any company.
Interoperable products
Quality improving
Standardised method &
testing
Time reducing
Cost reducing
Figure 17: Golden triangle for testing
As discussed in more details later, testing can be realised manually without any test specification
(free style by developer themselves) or formalised with some test specification. The interest of using
a standardised or formal testing is to cover up another set of criteria such as functionality coverage,
test time, test cost etc. How these criteria become important in testing? - Since testing is an
investment for a company, it should not introduce additive neither cost nor time to the company. In
order to achieve this goal, testing should cover up above criteria. First we see about the important
criteria such as cost and time.
4.1.1. Cost reduction issue
Cost in testing refers to two things: one thing is money spent on testing and another is money
wasted because of mistakes and errors made in developing an implementation. Both these costs play
an important role in marketing as they impact the price of the final product. For a product to be
successful in the market with tremendous competition, it needs to have a best quality. In any
product or service, nowadays quality is an important scale for success. To improve quality, a rigorous
testing should be practised for a product to pass different quality milestone s to reach end user
288315
PROBE-IT
market. So, an important amount of money is spent in testing to enhance the quality of the product
delivered to the end user. Money spent on testing should be reasonable and effective. That is why
we need research on testing to innovate new methods of testing which can co nsiderably cut down
the cost of testing and find errors in the early stage to cut down the development costs to improve
the return of Investment for testing.
4.1.2. Time reduction issue
Time spend on development of a product is divided into time spen t on design, time spent on
development, time spent on testing. These different times decide the time to market for a product.
In these, testing time has a great impact on design and development time of products. How to
minimise the testing time? - First and important option is automating the tests. Early testing and
coverage saves a lot of time. For instant, an error found in the conformance testing saves a lot of
time and money than one found during the interoperability testing.
For a company, when it positions its product in the market in correct time and cost, it finds its return
of investment. So, correct time and cost is decided partly by testing phase of the product. A
standardised testing provides the reduction of cost/time to market of a product .
4.2.
Different types of testing
We discussed earlier the need for standardised testing. Now, we discuss about different types of
testing involved for development of a product. Choosing a right product for their business solution
need is a challenging job for a company and for that they need to see the types of testing underwent
by the product.
4.2.1. Conformance testing
ISO/IEC DIS 10641 defined conformance testing as “test to evaluate the adherence or non-adherence
of a candidate implementation to a standard” in 1991. Some standards come with additional
documents, which have explained test methodology and features to be tested.(Gray, Goldfine,
Rosenthal, & Carnahan)
Standard
Conformance issues
Tester
IUT
Figure 18: Conformance testing
Conformance testing is done to verify conformance of the implementation against the standard. It is
done to prove that the implementation is correct and complete with respect to its specification.
Figure 18 explains the general methodology used to perform conformance testing.
288315
PROBE-IT
4.2.2. Interoperability testing
Vendor A
Vendor B
Standard
Errors & ambiguities
Design & development
Design & development
Errors (Programming,
interpretation of
standards)
IUT - A
IUT - B
Interoperability
issues
Figure 19: Interoperability testing
“The ability of two or more networks, systems, devices, applications or components to exchange
information between them and to use the information so exchanged” – EICTA Interoperability task
force (Georg Lütteke, 20 Feb 2010)
The current Information and communications world provides plenty of different software and
hardware that implements the same specifications. It is known fact that even following same
standard/specification, two different implementations failed to interoperate. In order to solve this
issue, interoperability testing is performed in addition to conformance testing. The major
interoperability issues faced because of unstable standards, human errors during development cycle
as shown in Figure 19. Having a single method to address the interoperability problem is difficult at
this point of time, which requires much more research activities should be carried in the
future.(Probe-IT, March 2012 )
4.2.3. Robustness testing
Robustness is defined in IEEE glossary as “The degree to which a system or component can function
correctly in the presence of invalid inputs or stressful environmental conditions.”(IEEE Standard
Glossary of Software Engineering Terminology, September 28,1990)
Due to the inherent nature of development in the wireless personal area network arena, a wide
variety of interacting products and performance situations are possible. In order to ensure
interoperability and crash-proof performance, implementations/devices must not only meet the
measure of specifications, but also stand up to increasingly complex user scenarios. In order to verify
the product or measure up in the field (in real scenario) when it has to interact with all the other
product and to see how the system behaves in midst of RF interference and attacks from some
malicious hacker who tries to purposefully cause disruption. This type of verification is called
Robustness testing.
4.2.4. Safety/Regulatory testing
For instance, every region has certain amount of regulations regarding RF spectrum usage, Powe r
level emission, Environmental safety requirement etc. The objective of this type of test to convey
the government or particular body that the product fulfils all necessary requirement to be la unched
in the particular market(htt1).
288315
PROBE-IT
4.2.5. Synthesis and conclusion
In the context of the Probe-IT project we will focus mainly on interoperability, this framework
document will concentrate mainly on interoperability testing. As said before, interoperability testing
cannot be separated from conformance testing. So, we will also discuss existing conformance testing
solutions. By this way, we will not avoid giving information on other types of testing (robustness,
performance, etc.) when they are available.
4.3.
Different approaches for interoperability testing
Testing can be done in many ways that can be regrouped in two main approaches for testing:
1. Empiric approach: This approach regroups all kinds of testing where non -real method is
defined. They are also called ad hoc testing.
2. Methodological approach: A specific method is used to develop the test tools and test suites,
and to execute them against components to be tested.
4.3.1. Empiric approach for testing
This approach of testing regroups several ways to do testing. Since this kind of testing is informal,
they are generally carried out while coding. There is no set procedure for informal testing, and this
is entirely up to the coder to implement without the need to submit the test reports. The coder feels
confident that his code works as required and contains no obvious bugs.
Empiric approach for testing encompasses tests that are done while developing the product to
identify bugs, as well as those that are done on the fly, for example during IETF meetings. Two or
more companies decide to bring their products (that they are developing) for a meeting or a
conference they are participating in. They agree quickly on an interconnection/configuration and on
some scenarios. They execute generally manually those scenarios and draw conclusions on the
degree of maturity of their components regarding their conformance to standards and their ability to
interoperate with other products.
The main advantages of this empiric way to do test are:

Tests can be done very earlier while developing the products, allowing detecting errors/bugs
in the earlier stage of the development.

Tests can be setup very quickly, without huge constraints such as having reports to prepare
etc.
These advantages are totally canceled out by numerous drawbacks:

No ideas on test coverage. As there is no real methodology, part of the properties to be
tested cannot be measured. Thus, errors/bugs may not be detected leading to potential non interoperability of the product at the end.

No real value to the market. As these tests have been done informally, end users will have
difficult to trust the final product.
4.3.2. Methodological approach for testing
This approach of testing generally encompasses different steps leading to the execution step where
executable test suites are executed against products. These products can be at different degree of
their development.
Three main steps can be seen in this approach:
1. Abstract Test Suite (ATS) specification. Here, several methods exist leading to a set of
properties to be verified on the implementations to be tested, together with corresponding
test scenarios called also test cases. These sets of test cases are called abstract test suites.
They are abstract meaning that they are independent from any kind of environments that
they will be executed. The ATS can be written in shape of paper version or in a test language
288315
PROBE-IT
like the standardized language TTCN-3 proposed by ETSI. Sometimes there exist some tools
that help in writing easily ATS, like TPlan tool provided by ETSI-CTI.
2. Derivation of executable test suites (ETS). The test cases of ATS obtained in the previous
step are then translated into executable test suites (ETS). To do so, necessary libraries,
coding and decoding functions are provided. Some environments that help in easing this step
exit, like the T3Devkit provided by the IRISA/Université de Rennes 1 research laboratory; see
in Figure 20 an example of application to the 6LOWPAN IoT protocol.
Steps towards executable test
suites
TM-UI
PIXITS
TM
TL
PIXITS
TTCN-3
ATS
CODETS
TM: Test Manager
TM-UI = Test Manager – User I nterface
TL= Test logging
CH= Component Handler
SA = System Adapter
PA =Platform Adapter
T3DevKit
Codec
Generator
Compila on
tools
Compiled
test suite
T3DevKit (CD)
Compiler
Generated Codecs
Editors
CH
Test suite
System
Under Test
CoAP, …
ICMP, UDP, ....
TE
IPv6
T3DevKit (SA&PA)
SA
PA
6LoWPAN
IEEE 802.15.4
Tes ng
environment
Figure 20 Deriving executable test suite: The example of IRISA T3DevKit environment
3. Test execution and results’ analysis. This last step is dedicated to the execution of the
obtained ETS (see step 2 above) on components to be tested. The test architecture and
configuration together with technical parameters (from tester side and from components’
side) that help in this step are provided. Depending on the results, the corresponding verdict
(Fail, Pass, or Inconclusive) is emitted (see Figure 21). Sometimes, trace analysis is done (on
logs obtained during the test execution) to confirm verdicts and/or to avoid false positives
and false negatives.
Advantages of methodological approach for do test are numerous:

Test coverage. As there is real methodology, properties to be tested can be measured. Thus,
it may help in determining more precisely how to cover important parts of the components
under test. By this way, it may reduce non-interoperability of the product at the end.

Real added value to the market. As these tests have been done formally, end users will trust
more easily the final product.

Tests can be done very earlier in parallel with products’ development, allowing detecting
errors/bugs in the earlier stage of the development.
288315

PROBE-IT
Results of tests can be provided through reports, etc.
The main drawback is that it may take time to obtain final executable tests. But, there are more and
more tools that help in facilitating this step.
TTCN-3 ba sed a pproa ch: test execution
the exa mple of 6LoWPAN
Component
Under Test
Test Equipment
Test Verdict
PASS, FAIL or
INCONCLUSIVE
TTCN-3
Test Suite
CoAP, …
Send
s muli &
observe
response
ICMP, UDP, ....
IPv6
6LoWPAN
IEEE 802.15.4
IEEE 802.15.4
8
Figure 21 Test execution step: the example of 6LoWPAN testing using TTCN-3 based approach
4.4.
Different means for testing
4.4.1. Test beds
A testbed is an experimental platform where different types of testing are carried on to avoid testing
in a live or production environment. By this method of testing, one can test a unit/module or many
modules or full implementation before it gets deployed for business purpose. WSN testbed is a
perfect example for IoT. It has all necessary elements such as software, hardware and other
networking components for realizing the test. Anything can be tested in a testbed, for example, a
communication device can be tested for its functionality, interoperability, robustness etc. a software
implementation can be tested for development errors against the standard, a networking device can
be tested for its networking capabilities. Testing in a testbed environment doesn’t risk more money
and time for a company. Many applications can be tested for user acceptance that leads its success
in the market. Testbed method is widely used during the research phase of the product. It allows the
researchers to develop and test their own innovated applicat ions. Nowadays, many test beds are
federated for providing wide range of test scenarios for researchers to carry on their testing process.
The tests carried out with a test bed are mostly free style testing i.e. without any standardized test
specification. Test case or test scenarios for testing in a testbed is solely depend on the developer or
researcher.
288315
PROBE-IT
4.4.2. Test houses
Test house is a business model evolved to use the demand for testing a product. A test house
provides testing solutions for their stakeholders. Engineers and experts who work in a test house
know all possible techniques of testing and different certification programs. They involve in test
planning, test reporting, test tool development etc. Test house is a place where a developed product
is given for testing. It provides testing as a service to their clients. Different types of testing service
are carried out in a test house staring from conformance to safety testing. For IoT, the challenges are
consolidating different services into a single paradigm, integrating new features to the existing
device and arrival of new technologies like IPv6, 6LoWPAN, NFC etc. New application enablers from
diversified IoT domains require end-to-end function of their device. A test house can be means for
providing support by verifying the functionalities, performance, and user acceptance of the
particular device and enable its success in the market. Test planning is the main role for a test
house. It optimizes the test process by a clear and strategic planning. Test house is a positive
element for a company that wants its product to be successful in the market.
4.4.3. Certification programs
A certification program is a process to verify a product for compliance with a standard. Most of
certification programs are developed by the standardization organizations like Zigbee alliance,
Bluetooth SIG, Hart foundation etc. It follows a methodological approach to carry out the test. The
process starts with receiving application for certification from a device manufacturing comp any or
implementation developing company. Once the application is received, the certifier instructs the
company to conduct with a test house to carry out the tests. Or, a company directly approaches a
test house for carry out the test and return to the product certifier with test results and reports.
Depending on the test results, the decision is made by the product certifier.
4.4.4. Interoperability events: Plugtest/Connecthaton
Test events are one of the means of testing a product interoperability and conforman ce. Many
companies come together to arrange such events to test their prototypes or full implementations
with their partners and competitors. It has benefit of enhancing the specification and speed up the
standardisation process. This results in reducing the time to market for the product and enable the
deployment of the technology in the market. For example, ETSI conducts plugtests event with aim of
satisfying the interoperability between the implementations. A test event look like in the Figure 22.
288315
PROBE-IT
Figure 22: CoAP plugtests, Paris-March 2012
A test event provides an opportunity to vendors, researchers, developers and test developers to
identify potential issues related to their product/implementation. It also provides a means to
improve the features of a technology standard for technology consortium and standardisation
bodies. Moreover, this is a fruitful opportunity for business network ing, technical knowledge sharing
etc.
288315
PROBE-IT
5. Framework definition for studying IoT testing solutions
Section 3 provides the arguments and justification for our choice of ‘per technology’ approach from
different other approaches such as ‘per application domain’, per ‘test enablers ’, etc. We decide also
to adopt a model that is closer to the models proposed in other research and industrial projects. We
have strong consensus about our approach. The open question is: What is the information to be
collected such that in the end we snapshot the current status of existing testing solutions in IoT?
This information collected should be coherent with testing evaluation criteria .
Testing related public
projects
Analysis
Technical overview
IoT communication
standards
State of the art
Future research
Testing overview
Industrial fora
Standardisation
bodies
Other standardisation
bodies
Figure 23: Basic information flow for state of the art
Figure 23 shows the basic information flow for developing state of the art of IoT testing. The idea
was to capture the current testing activities for each communication standard chosen for this work.
In addition to that, we would like to look into the current research activities in IoT testing. Finally
testing information will be collected from the standardisation bodies such as Bluetooth SIG, Zigbee
alliance etc. The methodology followed to design the deliverable D4.2 – “First report on state of the
art on testing of IoT technologies” is reviewing/gathering the information on testing for each
identified key IoT technology. And then we analyse, synthesize and interpret it to provide the
current status of protocol testing for each technology. The analysis done could help to develop a
research roadmap for IoT testing. The method used for this document is explained through the
following sub sections. The information gathered for each technology is split into two categories:
1) Technical overview and
2) Testing overview.
288315
5.1.
PROBE-IT
Technical overview of a technology
Protocol stack
Stakeholders
Technical overview of IoT
communication standards
Application domain
Network architecture
Figure 24: Technical overview
As said in the earlier section, the first part provides extensive information about the technology. The
idea is to make the reader more confortable while reading this document. If a technical detail on a
technology remains not provided in the document, readers are oriented to some other document for
that information. What we discuss in this section? We discuss about the stakeholders promoting the
technology, protocol stack, and network architecture as well as about IoT application domains
connected with the considered technology.
5.2.
Testing overview of technology
Certification process
Cost of testing
Certification
programs
Test tools
Time of testing
Testing overview of IoT
communication standards
Test houses
Test events
Figure 25: Testing overview
In the testing section, we try to gather most possible information such that in the end we could do
an analysis to draw a state of the art and also research plans for the future for testing. There are so
many sources for collecting information on testing. Each source will give a different point of view
with others. We collect the information about the certification programs, test events, test tools used
and information about the test houses. When we collect these information, we come across
information on different evaluation criteria such as coverage, methodology, cost, time etc.
288315
PROBE-IT
5.3.
Evaluation criteria
Product from R&D
Quality
Coverage
Conformance testing
Method
Time to market
Test automation
(Test tool)
Return of investment
Time of testing
ad
Ro
fo r
Cost of testing
ess
su c
Interoperability
testing
Market price
Other testing
Market sucess
Figure 26: Evaluation criteria
5.3.1. Test coverage
Test coverage is a very important criterion in evaluating a testing solution. This parameter says
whether tests cover all functionalities of an implementation. If test coverage is weak, then there is a
great possibility of implementation under test is verified incorrectly. Although this criterion is very
difficult to evaluate, it is important to take it into account for the evaluation of a test solution. It also
allows comparing different test solutions.
5.3.2. Test method
Test method is a criterion, which determines the quality of the product. A good test method will
produce a quality product. Test method consists of test planning, test specification development,
test reporting etc. What is the metric for measuring this criterion? We consider that for empiric
approaches for testing, it is difficult to obtain methods that are used for testing. Thus, this criterion
will indicate whether a formal approach (see section 4.3.2) exist for testing the considered
technology.
5.3.3. Test tool
Time is precious for anybody. It is the same for any company small or big. What make a test tool as
evaluation criterion? – A test tool is used in testing for saving the time spent on testing. For
example, suppose that we have 1000 test cases to test on an IUT. If you do it manually, it takes more
time than automating it. In production lines, when there is need to test 1000 product s with 1000
test cases, if manual testing is practised, then of course it is time and money loss for the company. A
test tool is introduced in testing to automate the tests to save time.
288315
PROBE-IT
5.3.4. Means for testing
As stated before (see Section 4.4. ), test beds, test houses, test certification programs and
interoperability events (such as ETSI Plugtest events) are important means for testing that help in
ensuring more widely interoperability of final products. By this way, they contribute in pushing a
technology towards its fast deployment. Those means of testing have to be considered when
studying IoT testing solutions.
5.3.5. Cost of testing
Money spent on testing a product is an investment for a company. When they invest money on
something, they normally expect a good return on investment. Investment on testing will impact on
the market price of product. This cost includes money spent on certification programs, test house
fees, purchase of test tools, etc. It also includes the money spent on correcting errors found during
testing. This information is very difficult to obtain from a company. Mostly these types of
information are confidential and difficult to gather.
5.3.6. Time for testing
Time of testing has impact on the time to market and market price o f a product. Time of testing is
the time spent on testing a product. Using test tool to automate the test will considerably reduce
the time spent on testing.
5.3.7. Return on investment
The test team manager in a company mostly assesses this criterion . He is the one who allocates the
resource for the test team. He will calculate the return on investment regularly to avoid the risk of
more money testing without getting any return from it.
5.3.8. Discussion on evaluation criteria
The deliverable 4.2 “State of the art on IOT testing solutions” will provide information on each
technology selected in Section 3.5. It will use the selected criteria indicated above in Section 5.3. . To
do so, we need metrics to evaluate those criteria. There will be trade-offs to make in selecting
criteria. For example, criteria that appear “direct” (like “test coverage”) tend to be less "operational'
as they may be difficult to estimate or to model. The criterion "test method" may not be
understandable to non-technical decision makers. We found this interesting characterization of good
evaluation criteria. “They have to be:

Accurate and Unambiguous, meaning that a clear and accurate relationship exists between
the criteria and the real consequences.

Comprehensive but concise, meaning that they cover the range of relevant consequences
but the evaluation framework remains systematic and manageable and there are no
redundancies.

Direct and ends-oriented, meaning they report directly on the consequences of interest and
provide enough information that informed value judgments can reasonably be made on the
basis of them.

Measurable and Consistently Applied to allow consistent comparisons across alternatives.
This means the criteria should be able to distinguish the relative degree of impact across
alternatives. It does not exclude qualitative characterizations of impact, or impacts that can't
be physically measured in the field.

Understandable, so that consequences and trade-offs can be understood and communicated
by everyone involved.

Practical, meaning that information can practically be obtained to assess them (i.e., data,
models or expert judgment exist or can be readily developed).
288315
PROBE-IT

Sensitive to the Alternatives under consideration, so that they provide information that is
useful in comparing alternatives.

Explicit about Uncertainty so that they expose differences in the range of possible outcomes
(differences in risk) associated with different policy or management alternatives.”
Among all these characteristics, the one that will be used in the deliverable 4.2 is that a criterion has
to measurable, meaning that we have to provide for each selected criterion metrics that will be used
to measure or evaluate it. This is done in the following section.
5.4.
Metrics for evaluation criteria
We give here metrics that will be used to evaluate each criterion described in Section 5.3.
5.4.1. Metrics for evaluating test coverage
A metrics for test coverage is very hard to define, as there are several ways to evaluate test
coverage. Depending on the technology and the testing approach used (empiric or methodological),
it may be even sometimes not measurable. In this first version of the framework, we will indicated if
we consider that the existing test solutions do not cover or cover partly, totally functionalities of the
considered IoT technology.
5.4.2. Metrics for evaluating test method
We will indicate if the existing testing solutions are based on methodological approach (see Section
4.3.2) or not. Information on the level of completeness of those solutions will be given.
5.4.3. Metrics for evaluating test tool
We will indicate here if there are testing tools provided for the considered technology or not. For the
existing tools we may also provide how mature they are.
5.4.4. Metrics for evaluating means for testing
We will mention here if there are test houses, test events, certification programs and test beds for
the considered technology.
5.4.5. Metrics for time for testing
This metrics is straightforward. It will be given in shape of hours, days, months or years spent in
testing.
5.4.6. Metrics for evaluating cost of testing
It includes cost of errors’ correction and cost of test labs and certification programs. It is generally difficult
to evaluate or to obtain. But if they are available they are given in terms of euros or dollars per year or for
the entire testing duration.
5.4.7. Metrics for evaluating return on investment
This criterion will correspond to the gain obtained when using test solutions comparing to the
situation where these testing solutions are not used. Sometimes stakeholders explicitly g ive them.
Sometimes it can be inferred from different information.
5.4.8. Synthesis on metrics for evaluation criteria
As described above, all these metrics are sometimes difficult to obtain. But based on the information
collected on the selected technologies, we were able to draw a good evaluation of a specific
288315
PROBE-IT
criterion of a technology in regards with the same criterion of other technologies. For each criterion,
we have all the information to indicate the synthesis described in the following table.
Criterion
metrics
Corresponding
symbol
Very good
++
The criterion is very well covered (close to the best that can
be obtained) by the technology compared to other
technologies
Good
+
The criterion is well covered by the technology compared to
other technologies but they are some aspects that make it
not close to the optimum
Average
Explanation
The criterion is satisfactory but can be improved compared to
the same criterion observe for other technologies
Below average
-
The criterion is under what can be expected and compared to
what is observed for other technologies
Worst
--
The criterion is totally not covered and close to the worst
situation that have been observed for other technologies
Not Available
NA
This corresponds to cases where we did not find information
on the considered criterion because it is confidential or not
yet available
5.5.
Conclusion and synthesis
The purpose of this document is to define a framework for studying existing solutions for IoT testing.
In this document, first a study was done to identify the stakeholders for IoT testing solutions. As IoT
domains are numerous, it is important to determine which IoT domains and/or technologies will be
considered. In Section 3, we explain in more details the approach used to select IoT technologies to
be studied. For each selected technology, we then give a general description toget her with
application domains, information on current deployment, stakeholders, etc. In the Section 4,
different existing types of testing are explained. It then gives necessary information on conformance
and interoperability testing allowing the reader to understand the importance of these two types of
testing in the deployment of IoT technologies. The Section 5 contains the criteria to be used and
associated metrics to compare the IoT existing testing solutions for the selected IoT technologies,
leading by this way to the purpose of this document: the definition of a “framework for studying
existing IoT testing solutions”. It contains elements that can be found in the deliverable 4.2, which
will contain a state of the art on IoT testing solutions. It will be used to collect more information
possible regarding testing to carry out the analysis, which in turn will produce at the end a roadmap
for IoT testing. We are mostly interested in collecting information regarding formal test method and
try to identify bottlenecks to provide ideas for future development in testing activities.
288315
PROBE-IT
6. Bibliography
(n.d.). Retrieved from http://extremesoftwaretesting.com/Info/TestingDictionaryQRS.html
(n.d.). Retrieved from http://www.bluetooth.com/Pages/Medical.aspx
(n.d.). Retrieved from www.zigbee.org/About/AboutTechnology/MarketLeadership.aspx
(n.d.).
Retrieved
from
www.printedelectronicsworld.com/articles/printed-rfid-in-201000001961.asp?sessionid=1
(n.d.). Retrieved from www.heraldonline.com/2012/09/04/4234966/infonetics-research-wireless-lan.html
(n.d.). Retrieved from www.techvibes.com/blog/machine-to-machine-the-future-of-business-2012-02-08
(2010, July). Retrieved from Creative connectivity: http://www.nickhunn.com
Dieter Uckelmann, Mark Harrison, & Florian Michahelles (Eds.). Architecting the Internet of Things (2011
ed.). Springer.
Georg Lütteke. (20 Feb 2010). Standardization to ensure interoperability. Philips CE.
Gray, M., Goldfine, A., Rosenthal, L., & Carnahan, L. (n.d.). (National Institute of Standards and Technology)
Retrieved from http://www.nist.gov/itl/ssd/is/conformancetesting.cfm
Guest Interview: Joost Demarest, the KNX Association. (2012, June). Hiddenwires .
(September 28,1990). IEEE Standard Glossary of Software Engineering Terminology. IEEE Standards Board.
(2012). IoT-A : Deliverable D1.3 –Updated reference model for IoT v1.5.
Nieminen, J. (30 May 2012). Connecting IPv6 capable Bluetooth Low Energy sensors with the Internet of
Things. Nokia.
Phillip
Maddocks.
(n.d.).
Retrieved
from
http://imsresearch.com/blog/Bluetooth_low_energy_Set_to_Lead_Standardization_in_Health_and_Fitness
_Devices_by_2015/165
Probe-IT. (March 2012 ). IoT CoAP plugtests WHITE PAPER. Paris.
Sandra Dominikus and Jorn-Marc Schmidt. Connecting Passive RFID Tags to the Internet of Things. IAIK,
Graz University of Technology.
Senior Analyst Lisa Arrowsmith . (2012, June ). Bluetooth Smart To Be the Wireless Technology Most Used
in Consumer Medical Devices by 2016.
Setting Standards for Energy-Efficient control networks.
Sigma Designs, Inc. (2012, June). Z-Wave Alliance Announces 650th Certified Product; Extends Leadership
of Wireless Control Market. Yahoo finance .

D4.1 – Framework for studying existing IoT testing solutions - probe-it

Transcription

Similar documents

Internet of Things Security Implications v1.4

Why connected?

Igniting Growth in Consumer Technology

Internet Of Things Lightweight Embedded Security

The PDF

Internet of Things in Logistics

Delivering on the IoT customer experience Business white paper

Internet of Things:

how the internet of things will impact you and the law

Software Defined Network for Internet of Things

La cuarta revolución industrial: Internet de las cosas y sistemas