Validation plan and report on first interoperability tests

Transcription

Validation plan and report on
first interoperability tests
Deliverable D5.3
GLOBAL ITV ID: GLOBALITV-D5.3-ValidationPlanandInteroperabilityTests
Version: 13
Deliverable number: D5.3
Authors: Marianne Kogut Eliasquevici, Marcos César da Rocha Seruffo, Carlos Natalino, Fernanda Chocron Miranda, Maria Ataide Malcher,
Jane Aparecida Marques (UFPA).
Contributors: Christian Keimel (IRT), Jean-Paul Bernoux (TDF).
Internal reviewers: Christian Fuhrhop (FOKUS).
Work Package: WP5
Task: T5.2
Nature: R – Report
Dissemination: PU – Public
Status: Living
Delivery date: 04.09.2015
Version of
2015-09-04
D5.3 Validation plan and report on first interoperability tests
Version and controls:
Version
Date
Reason for change
Editor
1
2015-06-15
Table of content
Marianne K. Eliasquevici, Fernanda Chocron Miranda, Marcos Seruffo and Carlos Natalino
(UFPA).
2
2015-07-10
Report of the first block of tests (IRT
and TDF)
Christian Keimel (IRT) and JP.
Bernoux (TDF).
3
2015-07-21
Systematization of results achieved
by partners during the tests
Marcos Seruffo, Carlos Natalino, Marianne K. Eliasquevici
and Fernanda Chocron Miranda
(UFPA).
4
2015-07-29
Contributions from TDF
JP. Bernoux (TDF).
5
2015-07-29
Systematization of TDF contributions
and UFPA adjustments
Marcos Seruffo, Marianne K.
Eliasquevici and Carlos Natalino
(UFPA).
6
2015-08-06
Contributions from IRT
Christian Keimel (IRT).
7
2015-08-17
Final review from UFPA (Subjective
tests and Final Remarks) and format
improvements
Marianne K. Eliasquevici, Jane
Marques, Fernanda Chocron
Miranda and Maria Ataide Malcher (UFPA).
8
2015-08-18
Internal review
Christian Fuhrhop (FOKUS).
9
2015-08-19
Version taking into account internal
review remarks
Marianne K. Eliasquevici
(UFPA).
10
2015-08-20
Final contributions from TDF
JP. Bernoux (TDF).
11
2015-08-21
Final review and format improvements from UFPA
12
13
2015-08-24
2015-08-31
Additional comments on document
Version taking into account remarks
from IRT
Marianne K. Eliasquevici, Fernanda Chocron Miranda and
Carlos Natalino (UFPA).
Christian Keimel (IRT).
Fernanda Chocron Miranda,
Maria Ataide Malcher and Carlos Natalino (UFPA).
Acknowledgement: The research leading to these results has received funding from the European Union's Seventh Framework Programme (FP7/2007-2013, call FP7-ICT-2013-10.2) under grant agreement
n° 614087 and from the Brazilian Conselho Nacional de Desenvolvimento Científico e Tecnológico
(CNPq) under grant CHAMADA MCTI/CNPq Nº 13/2012 and project number 490088/2013-9.
Disclaimer: This document does not represent the opinion of the European Community, and the European Community is not responsible for any use that might be made of its content.
This document contains material, which is the copyright of certain GLOBAL ITV consortium parties, and
may not be reproduced or copied without permission. All GLOBAL ITV consortium parties have agreed to
full publication of this document. The commercial use of any information contained in this document may
require a license from the proprietor of that information.
Neither the GLOBAL ITV consortium as a whole, nor a certain party of the GLOBAL ITV consortium
warrant that the information contained in this document is capable of use, nor that use of the information
is free from risk, and does not accept any liability for loss or damage suffered by any person using this
information.
page i
Version of
2015-09-04
Executive Summar y
This document summarises the outcome of the first block of interoperability tests, and, based on these
experiences, modifications to the initial evaluation plan of Deliverable 5.1 - Testing and evaluation plan.
This activity was part of Task 5.2 - Conducting user test evaluation, and conclusions, inside Work Package
5 (WP5).
This document is organised in five sections: (i) Introduction; (ii) Section about the approaches how the
objective and subjective tests were designed in the Evaluation Plan, emphasizing the multimethodological
matrix developed to guide the tests; (iii) Section for the descriptions about the results of the first block of
objectives tests performed by TDF, IRT and UFPA and with a brief discussion about the results to serve
as a basis to revalidate the platform specification; (iv) Section that describes the update or confirmation
of the actions provided in the Evaluation Plan; (v) Section with final remarks.
page ii
Version of
2015-09-04
Table of Contents
1 Introduction....................................................................................................................................... 1
2 Design of the tests ........................................................................................................................... 2
3 Description of the first and pilot block of tests ............................................................................. 4
3.1 Platform tests................................................................................................................................ 5
3.2 Discussion and results ............................................................................................................... 15
4 Validation Plan and final tests ...................................................................................................... 17
4.1 Objective tests ............................................................................................................................ 17
4.2 Subjective tests .......................................................................................................................... 18
5 Final remarks .................................................................................................................................. 20
Annex A – Survey about the market for the GLOBAL ITV architecture ......................................... 21
Glossary ............................................................................................................................................... 26
References ........................................................................................................................................... 27
List of Figures ...................................................................................................................................... 28
List of Tables ....................................................................................................................................... 29
page iii
Version of
2015-09-04
1 Introduction
The GLOBAL ITV project has, as its main objective, the definition of an interoperable interactive TV system between Brazil and Europe. This implies the definition of new mechanisms for signalling, transportation and receiving systems to allow Set-Top Boxes (STBs) to handle signal broadcast in DVB and ISDBTb standards. The STB prototype also needs to handle both application platforms (i.e., HbbTV and Ginga)
in order to provide interactivity in both TV standards.
In order to increase the effectiveness and benefits of the GLOBAL ITV platform, it is essential for the
project to adopt a consistent assessment. It is expected that the results from the evaluation process will
be an important input to the exploitation plan. The system performance, user acceptance, impacts, costs,
and benefits measurements should be designed to help any market assessment or product marketing.
Considering this, Deliverable 5.1 - Testing and evaluation plan [1] presented the Evaluation Plan, which
focusses on all efforts necessary to achieve the objectives of Work Package 5 (WP5). In view of the need
to validate the system/services developed in other work packages, it is a document or map which defines
the scope and objectives to be evaluated, the requirements to be validated, the strategies and resources
to be employed in the activities: “A well-structured evaluation plan allows to ensure that the needs of the
project are met and platform performance assessments are properly executed from different points-ofview, aiming at providing feedback to the other work packages (especially the WP3 and WP4) that can
enhance the technology developed. As a management tool for the project, an evaluation plan should
describe the core activities in a simple and coherent way and, above all, also integrate with the final results
planned for the whole project”. [1]
At this stage of the project when the first block of objective tests are performed, it is necessary to revaluate
the actions foreseen in D5.1, as well as report the results achieved with the tests to serve as feedback for
other work packages of the project.
The Validation Plan is the document that will update or confirm the planned actions of the D5.1 and the
validation process will be carried out with respect to two main areas:
(i) verification and technical assessment regarding to the final objective tests (conformance,
integration and interoperability tests); and
(ii) validate the platform via end-user acceptance tests, and identify user behaviour regarding the
applications and the prototypes.
page 1
Version of
2015-09-04
2 Design of the tests
Research to develop new technologies needs to be concerned about aspects beyond technical performance, given the increasingly competitive markets and the growing realization of the importance of acceptance and ownership by end users.
For GLOBAL ITV, entrepreneurs and scholars with different skills and backgrounds are involved in all
stages of platform development. Regarding staff responsible for testing and evaluation, there is participation of researchers from the Communication, Engineering and Computing departments, in a dialogue that
has led to discussions on testing technology for a new methodological approach. In this process of exchange of knowledge, more than a sum of procedures, the integrated work of a multidisciplinary team has
shown the possibility and the need for innovation in driving tests covering both technical and functional
aspects, regarding the use of technology experience for users, given that public acceptance is crucial to
the development and competitiveness of the product on the market.
Considering these perspectives and scope of the project, which is to present an innovative technological
solution for different television markets, a multimethodological matrix was designed, as shown in D5.1 [1],
to guide the testing and evaluations of the GLOBAL ITV platform, which brings together and coordinates
qualitative and quantitative procedures, in order to undertake a more comprehensive assessment. In this
matrix, the evaluation process adopted is based on two interconnected sub-processes: (i) one called
validation phase is the moment the operation of the system is tested to verify its technical performance
and additional tests are performed to validate its use by end users; (ii) and the other, exploitation, refers
to the stage at which all the above results are analyzed together with the cost of the technology to create
strategies, and incorporating commercial exploitation of such products on the market.
These two sub-processes aim to understand and evaluate three key elements of any innovation process:
technology (restrictions, performance, operation), the user (sociodemographic profile, habits and consumption patterns, desires, interests, satisfaction, usability levels) and market (strategies, interest, costs,
segmentation, customers and prospect for new markets).
For each of these elements, some research questions were considered during the designing process of
the tests: (i) what are the test´s objectives? (ii) What types / testing phases will be developed? (iii) What
public and / or products will be involved in testing? (iv) What are the procedures for data collection and
analysis? (v) What theoretical and methodological frameworks are the base for procedures?
In this sense, the developed matrix intends to evaluate the adoption of new technologies in a broad way
(from product design to the reception).
Regarding the underlying technology, the development of an interoperable architecture requires achieving
several stages of integration of the existing technologies. In the case of GLOBAL ITV architecture, we
consider four steps for migration (i) coexistence (consists of two separate iTV system stacks, one for
HbbTV 1.x and one for Ginga in one device); (Ii) interoperability (the coexistence is extended with interoperability between applications in the same service, i.e, an application in the active iTV system is able to
start an application in another, currently non-active iTV system, providing interoperability on application
level between the two iTV systems); (Iii) dual stack and unified architecture (the dual stack approach from
the previous two versions is combined with the unified architecture, presenting the solution converging
towards the GLOBAL ITV system); and (iv) unified architecture utilizing a single stack (only the unified
architecture of the GLOBAL ITV system remains, representing the ultimate step towards a convergent
system) [2].
“For all of them, the global architecture of the services (and by consequence of the platforms that handle
them) will be the same. Linear TV programs will be broadcast with an interactive services linked to that
program. The data services can be composed of one or several applications. They will be program related
or of public interest” [1]. The tests outlined in the methodological matrix are in line with the expected
implementation of coexistence for step (i). For the other steps they are dependent on the development of
prototypes.
page 2
Version of
2015-09-04
In multimethodological matrix proposed, it was understood that the end user is the central element of new
technologies tests. Therefore, in GLOBAL ITV project, the main objective of testing with users is to observe and analyze the experience of end users interacting with the platform prototypes and applications
in development.
From the point of view of the market, the proposed multimethodological matrix also provides for the collection of information that may help in developing a long-term strategy for incorporating GLOBAL ITV
platform in the market. A fundamental issue is understanding what types of business models will succeed
for multimedia sites. Understanding consumers’ willingness to pay in these new media environments
would be an important start” [3]. The objective of this testing phase of is to check possible difficulties in
incorporating the platform in a broadcaster’s routine and in the workflow of the content producer. It is
known that any change in established patterns and embracement of the possibilities of access to new
technologies must be accompanied by equally interesting content that enable and leverage the use of
these technologies.
As will be detailed in the next section, the GLOBAL ITV project is in the prototype development stage and
the first interoperability tests have been carried out, following the design proposed in the matrix and described in detail in Deliverable 5.2 - Detailed Test sites Specification and First Set-Up [4]. Based on these
experiences, a new set of objective tests and verification of acceptance of the developing platform are
been planned in parallel with the outcomes from WP3 and WP4.
page 3
Version of
2015-09-04
3 Description of the first and pilot block of tests
The objective tests have three separate goals:
 Validation of the feasibility GLOBAL ITV concepts. The end-to-end platform will be deployed including play-out and GLOBAL ITV receiver. The validation will be carried out in both interactive
systems, HbbTV and Ginga, corresponding to the coexistence and interoperability phases. An
important point will be the verification that the requirements defined by WP2 are met.
 Validation of the interoperability with current equipment deployed in the network: this condition is
mandatory if we want GLOBAL ITV studies adopted by the broadcaster and network operators;
 Non-disturbance of the different receivers: This condition is mandatory to adopt GLOBAL ITV by
different standardisation bodies and local authorities.
The first block of tests is part of the objective tests and verifies if the specified platform (according to WP2
and WP3) fulfils the interoperability requirements. It means that a signal generated using the set of components specified by GLOBAL ITV must be correctly played over a variety of TV sets and the STB developed on the project works with DVB or ISDB-Tb standards; i.e. the interoperability tests are required to
verify that the proposed signalling scheme does not impact on existing STBs and that the GLOBAL ITV
STB works properly for the current addressed signalling schemes (DVB and ISDB-Tb).
This section describes how TDF, IRT and UFPA have carried out the first and pilot block of tests using
playout and STB prototype of the project with applications and equipment that are currently in operation.
And considering the purpose of the objective test, the functional requirements and the two phases in
which the tests are divided.
For this goal, the tests were conducted in two ways: signalling interoperability tests and STB interoperability tests. The signalling interoperability tests were done with by broadcasting the GLOBAL ITV proposed
signalling to different TV sets and STBs from different manufactures to verify the GLOBAL ITV signalling
reception capability on such devices.
Figure 1 shows the overall strategy used to perform the GLOBAL ITV signalling interoperability tests
developed by TDF and IRT. In this case, the different TV sets must receive, handle and correctly present
audio, video and applications broadcasts using DVB and signalled according to the GLOBAL ITV proposed signalling. The tests ensure the standard proposed by GLOBAL ITV works correctly in existing (i.e.,
single standard) DVB TV sets.
STB 1
STB 2
GLOBAL ITV TS
Playout
STB 3
.
.
.
STB n
Figure 1: GLOBAL ITV signalling interoperability tests
page 4
Version of
2015-09-04
The STB interoperability tests were performed using the GLOBAL ITV STB prototype to receive a signal
from current broadcasters and this was the base of the tests held in Brazil by UFPA team. It is possible
to ensure that the STB works not only under GLOBAL ITV signalling but also under DVB and ISDB-Tb
signalling, which increases the flexibility of the GLOBAL ITV compliant equipment.
Figure 2 shows the overall strategy to verify the STB interoperability.
Broadcaster 1
Broadcaster 2
Broadcaster 3
GLOBAL ITV STB
.
.
.
Broadcaster n
Figure 2: GLOBAL ITV STB interoperability tests
The purpose of the testing is to ensure the usability of GLOBAL ITV concepts and devices with an existing
environment and not to single out manufacturers, so the name of the manufacturer and the references of
the receivers are anonymized in this deliverable.
3.1 Platform tests
This section describes the first results of the interoperability testing.
3.1.1 TDF platform
TDF focused on tests applicable to the French context:
 The configuration of the interacting devices is consistent with French regulations (except
GLOBALITV specific features).
 The interoperability is tested with DVB-T TV sets.
 Additionally, as a preliminary test, TDF verified that the GLOBAL ITV set-top box decodes the
test signals correctly.
3.1.1.1 TDF test signal
TDF has generated a multiplex of three TV services:
 GLOBAL ITV program: Audio and video sources are Brazilian programs that have been recorded
in Belem. Ginga and HbbTV services are signalled. HbbTV application was a broadband service
and no Ginga application was broadcast.
page 5
Version of
2015-09-04
 Channel 1: a commercial TV program with HbbTV applications with a broadcast and a broadband
component.
 Channel 2: a commercial TV program with HbbTV with a broadband application.
The test signals were generated as follows:
GLOBAL ITV program generation:
Transport streams are streamed and recorded. It consists of:
 TV program with audio/video components as previously described.
 Signalling: equipment taking charge of multiplexing the components of the service and the signalling. All the DVB tables are generated in accordance with the French regulation, with an
amendment to enable simultaneous broadcasting of HbbTV and Ginga application. This is done
according to the specification of WP3. The HbbTV URL refers to a commercial portal. The broadcast video is resized and displayed in a window.
The signalling used uses two AITs: one for HbbTV one for Ginga.
Multi-program multiplex generation:
The GLOBAL ITV program is played in loop using a DVB playout tool. It is multiplexed with two live
programs currently broadcast in France. Both feature include HbbTV services.
Multiplexing is carried out with a standard network multiplexer used on the French DTT networks. It generates a transport stream that is then recorded, like shown in Figure 3. For further testing only the recorded
multiplexed stream is used, as it is necessary to make replicable tests in standardised conditions to guarantee that the content or the generated signals are always identical.
Recorded
GLOBALITV
multiplex
multiplex.ts
Live channels
Multiplexor
Stream
Recording
PAT
à PMT service S
à Video component
à Audio component
à AIT Ginga
à AIT HbbTV
à PMT service X
à Video component
à Audio component
à AIT HbbTV
à à etc.
Figure 3: Multi-program Multiplex generation
Multiplex play-out and modulation:
For the compatibility testing itself, the recorded multiplex is played-out to feed modulators at different
standards:
 DVB-C to test the GLOBAL ITV receivers and check that the generated stream is interoperable
with the set-top box. Modulation is a 64QAM modulation, RF frequency is 120MHz.
page 6
Version of
2015-09-04
 DVB-T to test interoperability with existing TV receivers. The modulation was in Channel 58,
770MHz. The modulation features are those defined in France:
o
OFDM modulation with 64QAM
o
FFT: 8K
o
Guard Interval:1/8
o
Code rate: 3/4
The Figure 4 shows the architecture.
Recorded
Multi-program
multiplex
multiplex.ts
Dektec DVB-C & ISDB-T GLOBALITV
receiver
modulator
DVB-T
Modulator
Figure 4: Multiplex play-out and modulation
3.1.1.2 Test procedures
3.1.1.2.1 GLOBAL ITV STB
The STB has been tested with different HbbTV broadband applications developed by different service
providers. Test were carried out using the URL of the main French broadcasters (France Television, Arte,
M6, I>tele, Gully. Pratal) in the AIT. Access and navigation and video streaming were tested.
The same signal used in these tests is used to test the interoperability with standard HbbTV sets.
3.1.1.2.2 Interoperability with standard TV sets
The DTT signal is distributed to all the TV sets at the same time. The procedure is the following:
 TV set is switched on. A set-up “search for new TV programs” is launched. This to confirm that
the result is not affected by previous settings of the device.
 We check that the three different programs (GLOBALITV, Channel1, Channel2) are found.
 We switch between channels:
o
We verify the video and audio quality of the three programs.
o
If the TV set is unconnected, we verify that:
 No application is presented for the program that requires broadband HbbTV
service.
 The broadcast-only HbbTV application is available.
o
If the TV set is connected to broadband:
page 7
Version of
2015-09-04
 We verify that the services are available on the different services and running
properly (navigation in the menu, access to streamed the video).
Fifteen HbbTV TV sets from the five manufacturers with the highest market share in France have been
tested, covering a large portion of the devices used in France.
3.1.1.3 General information on the tests
The main effort went into the preparation of the tests, requiring 4-5 person days for:
 Development of the play-out with the GLOBAL ITV signalling (in the framework of WP3).
 TV set test-bed set-up.
Once the test bed has been set-up, the execution of the tests could be performed comparatively quickly:
File generations and verification:
 20 to 30 minutes depending on the length of the sequence, for testing existing broadcasts with
the GLOBAL ITV STB.
 Tests with multiplex required 15 to 20 minutes per TV set
3.1.1.4 Test Results
In order to keep test result anonymous in a public document, the name of the TV set manufacturer will be
replaced by a letter A, B C….
Tests were carried out with TV set from 2011 that correspond to HbbTV launch in France to 2014.
3.1.1.4.1 GLOBAL ITV set-top box
The application was launched rapidly with no problems. The displayed service was in conformance with
what was expected. When zapping between two programs with HbbTV applications, the individual application life cycles were respected. The interoperability between the play-out and the STB has been validated in that context.
3.1.1.4.2 Interoperability with standard TV sets
TDF has carried out tests with the stream file provided by IRT (see next section) with the six TV sets
presenting problems. HbbTV service, with the GLOBAL ITV signalling was launched for all of them.
Manufacturers A, C, G were completely interoperable with GLOBAL ITV signalling. Existing programs
were installed without problems and HbbTV applications were running correctly
Manufacturer B was suppressed from the tests. The TV set was of the first generation but do not provide
reliable behaviour even with conventional HbbTV. The market share of this manufacturer is very weak.
Manufacturer D did not run the broadcast application of the conventional HbbTV system. It has been
checked that the services were correctly received with the conventional “on-air” signal.
Manufacturer F was completely interoperable for one receiver, but did not launch the HbbTV application
with the GLOBALITV signalling. We had no disturbance for the conventional HbbTV services.
We can remark that the problem occurs on the latest version of the receivers.
Further investigation has to be done to verify the impact of the double AIT broadcasting.
It is important to register that the device names are in the format <device manufacturer><tester><device
number>, so GTDF03 denotes the second device by manufacturer ‘G’ in the tests performed by TDF, as
is reported in tables 1, 2 and 3.
page 8
Version of
2015-09-04
Table 1: First interoperability test results from TDF (GLOBAL ITV services)
Results with GLOBAL ITV TS
Equipment ID
Broadband
Audio
Video
HbbTV
ATDF01
OK
OK
OK
ATDF02
OK
OK
OK
ATDF03
OK
OK
OK
ATDF04
OK
OK
OK
CTDF01
OK
OK
OK
CTDF02
OK
OK
OK
CTDF03
OK
OK
OK
DTDF01
OK
OK
OK
FTDF01
OK
OK
OK
FTDF02
OK
OK
NOT OK
FTDF03
OK
OK
NOT OK
FTDF04
OK
OK
NOT OK
GTDF01
OK
OK
OK
GTDF02
OK
OK
OK
GTDF03
OK
OK
OK
Table 2: First interoperability test results from TDF (Channel 1 services)
Portal A
Equipment ID
Broadcast Broadband
Audio
Video
HbbTV
HbbTV
ATDF01
OK
OK
OK
OK
ATDF02
OK
OK
OK
OK
ATDF03
OK
OK
OK
OK
ATDF04
OK
OK
OK
OK
CTDF01
OK
OK
OK
OK
CTDF02
OK
OK
OK
OK
CTDF03
OK
OK
OK
OK
DTDF01
OK
OK
OK
NOT OK
FTDF01
OK
OK
OK
OK
FTDF02
OK
OK
OK
OK
FTDF03
OK
OK
OK
OK
FTDF04
OK
OK
OK
OK
GTDF01
OK
OK
OK
OK
GTDF02
OK
OK
OK
OK
GTDF03
OK
OK
OK
OK
Table 3: First interoperability test results from TDF (Channel 2 services)
Portal B
Equipment ID
Broadband
Audio
Video
HbbTV
ATDF01
OK
OK
OK
ATDF02
OK
OK
OK
ATDF03
OK
OK
OK
ATDF04
OK
OK
OK
CTDF01
OK
OK
OK
CTDF02
OK
OK
OK
DTDF01
OK
OK
OK
page 9
Version of
2015-09-04
FTDF01
FTDF02
FTDF03
FTDF04
GTDF01
GTDF02
GTDF03
OK
OK
OK
OK
OK
OK
OK
OK
OK
OK
OK
OK
OK
OK
OK
OK
OK
OK
OK
OK
OK
The Table 4 lists the functional requirements that were checked during the first block of testing conducted
at TDF premises Rennes (France), on July 9th, 9:00-19:00. At the table the functional requirement tested,
the test signal and the equipment used during the tests are reported. The fulfilment and occurrence of
errors is also reported. All the tests were performed in a controlled environment (i.e., broadcast and reception at the same building). No recording equipment was used.
Functional Requirements
#2
HbbTV 1.x application support
#4
HbbTV 1.x application signalling and handling commands support
#6
HbbTV 1.x application receiving
#8
Set-top-box
DVB support
#10
HbbTV 1.x
Broadband
Support
#11
Table 4: First block of tests - Functional requirements tested - TDF
(X) indoors
( ) outdoors
Kind of tests
Equipment Used
(X) Playout with GLOBAL ITV signalling
(X) GLOBAL ITV Set-top box
(X) TV
(X) Multiplexer
(X) indoors
( ) outdoors
(X) TV
(X) Multiplexer
(X) indoors
( ) outdoors
(X) TV
(X) Multiplexer
(X) indoors
( ) outdoors
( ) TV
(X) Multiplexer
(X) indoors
( ) outdoors
(X) TV
(X) Multiplexer
(X) indoors
Test signals
Requirement fulfilled
( ) Live TV streams
(X) Recorded TV streams
( ) Broadcast interactive application
(X) Broadband interactive application
( ) Yes
( ) No. Why not?
(X) Partially. Some TV sets do not launch
HbbTV: with GITV signalling and standard signalling. Probably link to signalling (TBC)
( ) Live TV streams
( ) Yes
( ) No. Why not?
HbbTV: with GITV signalling and standard signalling
( ) Live TV streams
( ) Yes
( ) No. Why not?
(X) Partially. Broadcast application not tested
yet.
( ) Live TV streams
( ) Yes
(X) No. This is a DVB-C set-up box. The STB is
compatible comply with the main DVB feature.
( ) Partially.
( ) Live TV streams
( ) Broadband interactive application
(X) Yes
( ) No.
( ) Partially.
( ) Live TV streams
page 10
Version of
2015-09-04
Playout
#12
Application
Availability
( ) outdoors
Kind of tests
Equipment Used
( ) Playout with GLOBAL ITV signalling
( ) GLOBAL ITV Set-top box
( ) TV
( ) Multiplexer
(X) indoors
( ) outdoors
(X) TV
(X) Multiplexer
Test signals
( ) Recorded TV streams
( ) Yes
( ) No.
(X) Partially. Investigation has to be done to explain some TV set behaviour.
( ) Live TV streams
(X) Yes
( ) No.
( ) Partially.
3.1.2 IRT platform
IRT focused on testing the following aspects:
 Acceptance of the dual AIT signalling proposed in the GLOBAL ITV specification for interoperability and coexistence with existing terminals, in particular if this concept interferes with the terminals’ normal operations.
 Assessing a range of DVB-C capable TV sets and STBs from different manufacturers.
3.1.2.1 IRT test signal generation
IRT generated a multiplex of one TV service that included both a Ginga and HbbTV application on the
same service using the dual AIT approach described in Clause 4.6.2.1 of Version 1.1 of the GLOBAL ITV
specification as documented in D3.2.
Both applications were signalled as auto-start in their respective AIT, but since the terminal devices tested
were only capable of executing HbbTV applications, only the HbbTV application should always be started.
The test signals were generated using the first version of the GLOBAL ITV play-out prototype. The tables
used for the DVB signalling were simplified and parameters not necessary for this first test were omitted,
e.g. network parameters in the NIT, since the focus was on the interoperability of the dual AIT signalling
approach with existing terminals
3.1.2.2 Play-out and modulation
The generated multiplex is played-out using a DekTec DTU215 USB VHF/UHF modulator supporting
DVB-C modulation as depicted in Figure 5 with the following parameters:
 530 MHz
 64QAM
page 11
Version of
2015-09-04
Figure 5: Block diagram of IRT platform
3.1.2.3 Test procedure
The terminals have been tested with a basic Ginga application (provided by UNICAMP) and an on-air
HbbTV application from a public broadcaster in Germany (ZDF).
Auto-start and subsequent access to the HbbTV portals and navigation within these portals were evaluated in order to assess the non-interference with existing terminals
In Table 6, the overall behaviour of all examined terminals is summarised. Details regarding encountered
errors will be explained in detail for the affected terminals in Section 3.1.2.5.
Note, that some of the requirements in Table 5 focused originally in D2.3 only on the GLOBAL ITV STB.
For the purpose of this interoperability assessment, we applied the same requirements more generally to
all available terminals.
The tests detailed on Table 5 was performed at the IRT premises Munich (Germany), on July 8th, 9:0019:00. At the table the functional requirement tested, the test signal and the equipment used during the
tests are reported. The fulfilment and occurrence of errors is also reported. All the tests were performed
in an indoor environment (i.e., broadcast and reception at the same building). No recording equipment
was used.
Functional Requirement
#2
HbbTV 1.x application support
#4
HbbTV 1.x application signalling and handling commands support
#6
HbbTV 1.x application receiving
Table 5: First block of tests - Functional requirements tested - IRT
(X) indoors
( ) outdoors
Kind of tests
Equipment Used
(X) TV
(X) STB
(X) indoors
( ) outdoors
(X) TV
(X) STB
(X) indoors
( ) outdoors
(X) TV
(X) STB
Test signals
( ) Live TV streams
( ) Yes
( ) No. Why not?
HbbTV: with GLOBAL ITV signalling and standard signalling Probably link to signalling (TBC)
( ) Live TV streams
(X) Yes
( ) No. Why not?
( ) Partially. Some TV sets do not launch
HbbTV: with GLOBAL ITV signalling and standard signalling Probably link to signalling (TBC)
( ) Live TV streams
(X) Yes
( ) No. Why not?
( .) Partially
page 12
Version of
2015-09-04
Functional Requirement
#8
Set-top-box
DVB support
#10
HbbTV 1.x
Broadband
Support
(X) indoors
( ) outdoors
Kind of tests
Equipment Used
(X) TV
(X) STB
(X) indoors
( ) outdoors
(X) TV
(X) STB
(X) indoors
( ) outdoors
#11
(X) TV
(X) STB
(X) indoors
( ) outdoors
Playout
#12
Application
Availability
(X) TV
(X) STB
Test signals
( ) Live TV streams
(X) Yes
( ) No. Why not?
(. ) Partially.
( ) Live TV streams
(X) Yes
( ) No.
( ) Partially.
( ) Live TV streams
( ) Yes
( ) No.
(X) Partially. Investigation has to be done to explain some terminal behaviour.
( ) Live TV streams
(X) Yes
( ) No.
( ) Partially.
The signal was distributed sequentially to all terminals. The procedure was as following:
 Play-out started from PC using StreamXpress
 Terminal is switched on and a “search for new programs” is initiated.
 Confirming that the GLOBAL ITV service has been found.
 Tuning to the GLOBAL ITV channel:
o
Confirm that HbbTV application is auto-started
o
If the HbbTV application was automatically started, verify that the application accepts
user input
o
If the HbbTV application was not automatically started, check whether the terminal recognizes that a HbbTV application is signalled in the service
3.1.2.4 General information on the tests
The preparation of the test-bed consisted of:
 Preparing the play-out with the GLOBAL ITV signalling (in the framework of WP3).
 Terminal test-bed set-up, in particular marshalling of the devices.
Once the bed test has been set-up, the test can be executed efficiently and the test per terminal took
about 15 to 20 minutes.
page 13
Version of
2015-09-04
Eight HbbTV terminals, two STBs and six TV sets, from overall six different manufacturers, representing
the major brands in the German market have been tested.
Note, that the names of the manufacturers and terminal type numbers have been anonymized in this
deliverable.
3.1.2.5 Test results
Manufacturers A, F, J, K: terminals from these manufacturers were completely interoperable with the
GLOBAL ITV signalling: the HbbTV application was automatically started and executed correctly.
Manufacturer D, G: terminals from these manufacturers did not automatically start the signalled HbbTV
application, even though no transmission errors were encountered. In particular, the terminals from these
two manufacturers did not recognize any applications in the service at all.
Closer inspection revealed that terminals from these two manufacturers are sensitive to the order of the
AITs in the service’s component loop if the PMT:
 If the AIT with the Ginga application type (application_type = 0x09) is signalled before the
AIT with the HbbTV application type (application_type = 0x10), no application is recognized and consequently no application is automatically started. Note, that the first AIT containing
the Ginga application is completely ignored, implying that the terminals seem to check for the
application type, but also assume that a service may contain only one AIT and thus as soon as
an non-usable AIT has been detected, the terminals assume that no application is signalled in the
service at all.
 If the AIT with the HbbTV application type (application_type = 0x10) is signalled before
the AIT with the Ginga application type (application_type = 0x09), the HbbTV application
is automatically started and executed correctly.
Using a PMT in which the HbbTV AIT is signalled before the Ginga AIT, all terminals passed the test as
listed in Table 6. For all the terminals tested, audio, video and HbbTV application could be used without
problems.
Table 6: First interoperability test results from IRT (Results with GLOBAL ITV TS)
Results with GLOBAL ITV TS
Comments
Equipment ID
Audio
Video
HbbTV
JIRT01
OK
OK
OK
STB
KIRT01
OK
OK
OK
STB
AIRT01
OK
OK
OK
TV set
GIRT01
OK
OK
OK
TV set
AIRT02
OK
OK
OK
TV set
DIRT01
OK
OK
OK
TV set
FIRT01
OK
OK
OK
TV set
DIRT02
OK
OK
OK
TV set
3.1.3 UFPA platform
Tests at the broadcast level could not be performed during the first block of tests, as expected, since the
playout was not available to UFPA on time for testing. This fact also made it difficult to gather information
that could support the business plan and the validation of the technology. Due to this, we used only the
signalling of a local broadcaster to perform initial testing with the STB of GLOBAL ITV that was available
(reception level). However, it must be highlighted that at the time of testing the STB did not have the
functionality of running Ginga applications, making it impossible to test and analyse the flow generated
over the network. It was only possible to do the interoperability tests, analysing some of the STB hardware
requirements and the reception of the ISDB-Tb signal in STB. Considering those facts, the UFPA team
performed GLOBAL ITV interoperability tests and the results as reported in Table 7.
page 14
Version of
2015-09-04
The initial schedule of tests of UFPA team comprised functional requirements #1, #2, #3, #4, #5, #6, #7,
#8 and #11 at the first phase of tests (as reported at section 3.5.3 of deliverable 5.2 [4]). However, the
Ginga-NCL application support on the GLOBAL ITV Set-Top Box and the GLOBAL ITV Playout are not
available at the time of these tests. For this reason, the only functional requirement that could be tested
was the #7.
Table 7: Functional and interoperability requirement tests (UFPA)
ApPhase
Requirements
Test
proach
Broadcast (RBA) and reception (labora- First
First
#1, #3, #5, #7, #11
tory in UFPA)
Transmission and reception at RBA TV First
First
#1, #2, #3, #4, #5, #6, #8, #11
The first procedure was to search for the available broadcaster channels. Then, two available channels
were chosen for the test. Each of the channels were tuned for one hour in order to ensure the #7 functional
requirement is fulfilled.
The tests detailed on Table 8 was performed at UFPA premises (Brazil), on June 27th, 13:00-17:30. At
the table the functional requirement tested, the test signal and the equipment used during the tests are
reported. The fulfilment and occurrence of errors is also reported. All the tests were performed in an indoor
environment (i.e., broadcast and reception at the same building). No recording equipment were used.
#7
STB system
support
Table 8: Fist block of tests - Functional requirements tested - UFPA
(X) indoors
( ) outdoors
Kind of tests
Equipment Used
( ) Playout with GLOBAL ITV signalling
(X) TV
( ) Multiplexer
Test signals
(X) Live TV streams
(X) Yes
( ) No. Why not?
( ) Partially.
The remaining untested functional requirements will be tested in the second phase of tests, once the
Ginga-NCL support will be available at the GLOBAL ITV STB and the GLOBAL ITV playout will also be
available.
3.2 Discussion and results
The partners of the GLOBAL ITV project have performed the initial phase of testing, both transmission of
GLOBAL ITV signalling to existing (DVB) STBs and TV sets, and reception from broadcasters to the
GLOBAL ITV STB were tested to ensure the functionality of the prototype with the current European and
Brazilian standards. At the moment only the reception of broadcasters to the GLOBAL ITV STB could be
tested in Brazil, since the GLOBAL ITV playout was not available in time for the scheduled test in Brazil.
This way, the interoperability tests using the GLOBAL ITV signalling broadcast to market STBs will be
done in the next phase of tests. Moreover, also existing ISDB-Tb STBs and TV sets will be included in the
non-interference testing in the next phase.
Based on the presented test results, it can be determined that the GLOBAL ITV architecture fulfils most
of the interoperability requirements on the tested equipment. It means that a deployment of services based
on the GLOBAL ITV architecture is very likely to not have any negative effects on existing equipment.
However, some of the components from GLOBAL ITV prototypes were not available at the time of this
first phase of tests, so some parts of testing will be postponed to the second phase of tests.
page 15
Version of
2015-09-04
For the second phase of tests, not only objective but also subjective tests are planned. The results will
include feedback regarding possible improvements on the GLOBAL ITV platform and the application prototypes, indicating also strategies designed for the incorporation of this platform by the market.
page 16
Version of
2015-09-04
4 Validation Plan and final tests
As mentioned in the introduction, the objective of the Validation Plan is to update or confirm the planned
actions of the D5.1 [1], regarding to the objective and subjective tests. As this is a live document, it is
reviewed against each stage of the Project.
The results of the objective test were to ensure that the platform is operating in accordance with the
specifications and functional requirements. While user participation is not involved, it ensures that the
system is acceptable technically and will be available for testing with users.
4.1 Objective tests
The final objective tests will be carried out using playout, STB and apps developed inside WP3 and WP4.
The results of these tests will be reported in the D5.4 – Running test site. Based on the results of the first
block of objective tests, each partner will review what to test.
4.1.1 TDF test plan
In the second block of tests, further investigation will be carried out to adapt the signalling so that 100%
of the receiver will be compatible.
TDF will include other network equipment in the signal generation platform to guaranty that they do not
impact the GLOBAL ITV signalling. With these tests the functional requirements shown in Table 9 will be
tested.
Table 9: Functional and interoperability requirements of the final objective tests (TDF)
Test
Approach
Phase
Requirements
Integration test
First
Second
#2, #4, #6, #8, #11
4.1.2 IRT test plan
The test in the first phase have confirmed that the dual AIT approach with separate PIDs for each AIT is
feasible and is non-interfering with existing terminals. The number of tested devices and manufacturers,
however, is limited. In particular, the tested devices do not necessarily represent the state-of-the-art, as
most of the devices tested at IRT were older than two years.
Therefore, a more in-depth testing of the signalling approach is planned during the next interoperability
workshop at IRT, where manufacturers attend with current and future HbbTV terminals. Moreover, using
the final version of the GLOBAL ITV STB prototype, GLOBAL ITV STB specific requirements shall be
tested as well. These tests, will allow checking the functional requirements shown in Table 10.
Table 10: Functional and interoperability requirements of the final objective tests (IRT)
Test
Approach
Phase
Requirements
Test signals
First
Second
#7, #8, #9, #10
Interoperability workshop
First
Second
#4, #6, #8, #10, #11
page 17
Version of
2015-09-04
4.1.3 UFPA test plan
In the second block of tests, the UFPA intends to conduct integration testing in terms of broadcast and
reception in environments 1 (Broadcast at RBA TV and reception at laboratory in UFPA) and 2 (Broadcast
and reception at RBA TV), as described in D5.2 [4]. For the environment 1, the GLOBAL ITV signal will
be broadcast using the RBA TV infrastructure, and such signal will be received and analysed at the UFPA
laboratory. For the environment 2 the signal broadcast and reception will be performed at the RBA TV
facilities. It is intended that the results obtained can be used to validate the architecture proposed in
GLOBAL ITV.
Considering the functional requirements, UFPA tested only functional requirement #7. The remaining untested functional requirements will be tested at the second phase of tests as detailed in Table 11).
Table 11: Functional and interoperability requirements of the final objective tests (UFPA)
Test
Approach
Phase
Functional requirements
Broadcast (RBA) and reception (laborFirst
Second
#1, #3, #5, #7, #11
atory in UFPA)
Transmission and reception at RBA TV
First
Second
#1, #2, #3, #4, #5, #6, #8, #11
4.2 Subjective tests
The subjective tests include all procedures to validate the use and approval by the end-users of the platform, and also tests involving people who will have access to the platform in the future and will therefore
have their workflow impacted. It is planned to integrate the efforts of UFPA and A-CING during the tests,
to make possible compare the results, as presented in D5.2 [4].
Regarding to the acceptance tests with end-users, the sample selection, both in Brazil and in Europe, will
be made using invitations focused in social media and in universities (located in Belém and in Madrid)
and possibly using recruitment agencies in Brazil.
As mentioned during Section 2 of this deliverable, this kind of test will be important to understand the user
and the possible of a creation a new market; thus, tests will be done to ensure the most important part in
this process, the end user, will be able to provide input and insights. As Cauberghe and De Pelsmacker
[5] note, interactive digital TV offers new opportunities, but also, “implicate new threats for broadcasters,
marketers and advertisers because the existing TV business model becomes unstable”. There will be
changes for all agents in this market, including manufacturers who need to adjust their production line to
meet the new demands imposed by the interactive digital television.
Considering that the GLOBAL ITV project is in the prototype development stage and based on the effort
of development of the multimethodological matrix, presented in D5.1 [1], regular reviews are made to
realign the methodological procedures to be applied until the end of lifetime of the Project.
The proposal is to realize a final block of tests involving end users from a targeted testing scenario that
will be assembled in Brazil (Federal University of Pará) and Europe (Spain). In this occasion tests of the
functioning of the developed prototype and procedures for prospecting of information on the possible
adoption of technology by participating users will be performed. This will be based on the controlled environments to test the transmission, reception and interaction of participating users with the GLOBAL ITV
architecture. As mentioned in D5.2 [4], we believe that with this type of test environment it will be possible
to create standards for contextual and system factors, in order to establish benchmarks between the tests
performed in the Brazilian and European scenarios. By restricting the tests to a controlled environment,
the only factor that will make a difference between the tests conducted is the human factor, related to the
sociocultural profile end-user.
page 18
Version of
2015-09-04
Similarly, regardless of the finalization of project prototype the tests of reception and interaction on endusers at home will not be performed.
Another adjustment is that in September, interviews will be performed with and one focus group section
of entrepreneurs and professionals of the area of broadband and broadcast in Brazil, specifically the RBA
TV in Belém.
During the review of the Evaluation Plan, members of the GLOBAL ITV Council were incorporated as
domain specialists to opine about GLOBAL ITV. They are a group of independent experts, mainly from
Europe and Brazil but also from other parts of the world, with the objective of promoting interoperable
solutions for advanced interactive digital television services, as well as fostering future common research
priorities between European and Brazilian researchers.
A specific survey (Annex A) was designed by UFPA team in cooperation with A-CING team to be applied
the members of the Council in September, during a regular meeting in order to gather information on
perceptions about the advantages, disadvantages, and challenges of the implementation process of the
GLOBAL ITV architecture and its potential incorporation in the commercial market.
The proposal is that the results of this survey could be evaluated in combination with the qualitative data
collected in the interviews and focus group that will be held with Council members and by others public
who will have access to the platform in the future and will have their work flow impacted, such as entrepreneurs and professionals from the television broadcasters and content producers, as described in D5.2
[4].
The description of the objectives and questions of the survey are presented in Table 12.
Table 12: Question and objectives of survey developed to GLOBAL ITV Council members
Objectives
Questions
Profile identification data of respondents:
1, 2, 3
area and/or professional segment, the country of residence of them and how
they learned of the GLOBAL ITV platform.
Perception of respondents about the GLOBAL ITV platform:
4, 5, 6, 13
knowing the opinion of the respondents about the advantages, acceptability and
resistance about the platform and if they intend to adopt it.
Perception of platform adoption:
evaluation of the probability of the market and users come to adopt the platform, how long it might take to happen and how much the consumers would
pay for it.
Impacts on working routine:
investigating the level of impact and/or changes in the platform use in working
routine.
Indication of other potential respondents:
requesting information of other professionals to answer the survey
7, 8, 9, 10, 11
12
14
page 19
Version of
2015-09-04
5 Final remar ks
This document describes the result of the first block of objective and interoperability tests, performed in
June/July by TDF, IRT and UFPA, located, respectively, in France, in Germany and in Brazil. For this
phase, both transmission of GLOBAL ITV signalling to existing STBs and TV sets and reception from
broadcasters to the GLOBAL ITV STB were tested. In the second phase, these tests are planned to be
extended to additional devices, especially ISDB-Tb terminals.
As mentioned, the GLOBAL ITV project is in the prototype development stage, part of the components
from GLOBAL ITV architecture were not available at the time of this first phase of tests, which postpones
its tests to the second phase of evaluation. In this second and final phase will be possible to complement
early results with results that can give detailed information on how to make improvements on the GLOBAL
ITV platform and the application prototypes.
In this second and final phase, the subjective tests will be performed, which require the completion of the
development of prototypes by WP3 and WP4. Given this, the planning of tests with users presented in
D5.1 [1] and D5.2 [4] was reviewed. In order to prospect important information for the development project, regarding their acceptance by users and subsequent merger of GLOBAL ITV architecture on the
market, we additionally scheduled, interviews with one focus group section of entrepreneurs and professionals of the area of broadband and broadcast in Brazil (RBA TV). We also created a specific survey for
members of the Council, in order to gather information on perceptions about the opportunities of the implementation of the GLOBAL ITV architecture and its potential incorporation in the commercial market.
Considering the effort of development of the multimethodological matrix, presented in D5.1 [1], we invest
effort in the final phase of the project, on the review and realignment of objectives and procedures for the
tests, thus ensuring appropriate testing of the tools, platform and applications developed by the project.
The expectation is that this realignment of the methodological procedures for objective and subjective will
allow testing to be carried out satisfactorily.
page 20
Version of
2015-09-04
Annex A – Sur vey about the mar ket for the GLOBAL
ITV architecture
The survey was developed by UFPA team in collaboration with A-CING team. This for was approved
during the 6th GLOBAL ITV Consortium meeting, held in Munch (Germany), June 1st and 2nd.
In order to facilitate implementation, the survey was made available online to be filled by the members of
the GLOBAL ITV Council.
page 21
Version of
2015-09-04
page 22
Version of
2015-09-04
page 23
Version of
2015-09-04
page 24
Version of
2015-09-04
page 25
Version of
2015-09-04
Glossar y
CA
Consortium Agreement
CoA
Coordination Agreement
DoW
Description of Work, Annex-1 of Project Proposal
EC
European Commission
IPR
Intellectual Property Rights
NDA
Non-disclosure agreement
PO
Project Officer
QA
Quality Assurance
R&D
Research and Development
URL
Uniform Resource Locator
WP
Work Package
Partner Acronyms
Acronym
Name
IRT
Institut fuer Rundfunktechnik GmbH, DE
A-CING
Aqua-Consult Ingenieros, S.L., ES
FHG
Fraunhofer-Gesellschaft zur Förderung der angewandten Forschung e.V. , DE
TDF
TDF, FR
RETEVISION
Retevisión – AbertisTelecom,ES
SYMELAR
Symelar Innovación SLU, ES
TARA
TARA System, DE
BAND TV
Rede Bandeirantes de Televisão, BR
HXD
HXD Interactive Television, BR
LSI-TEC
Associação do Laboratório de Sistemas Integráveis Tecnológico, BR
UCB
Universidade Católica de Brasília, BR
UFABC
Universidade Federal do ABC, BR
UFPA
Universidade Federal do Pará, BR
UNESP
Universidade Estadual Paulista “Júlio de Mesquita Filho”, BR
UNICAMP
Universidade Estadual de Campinas, BR
USP
Universidade de São Paulo, BR
W3C
World Wide Web Consortium at GEIE ERCIM, FR
RBA TV
Rede Brasil Amazônia de Comunicação
page 26
Version of
2015-09-04
References
[1] GLOBAL ITV, "D5.1 – Testing and evaluation plan," 2014.
[2] GLOBAL ITV, "D3.2 - Initial high level architecture and first specifications," 2014.
[3] T. Hennig-Thurau, E. C. Malthouse, C. Friege, S. Gensler, L. Lobschat, A. Rangaswamy and B.
Skiera, "The Impact of New Media on Customer Relationships," Journal of Service Research, 13,
pp. 311-330, 2010.
[4] GLOBAL ITV, "D5.2 – Detailed Test sites specification and First set-up," 2014.
[5] V. Cauberghe and P. De Pelsmacker, "Opportunities and Thresholds for Advertising on Interactive
Digital TV: a view from advertising professionals," Journal of Interactive Advertising, 7(1), p. 12‐23,
Fall 2006.
page 27
Version of
2015-09-04
List of Figures
Figure 1: GLOBAL ITV signaling interoperability tests .............................................................................. 4
Figure 2: GLOBAL ITV STB interoperability tests ...................................................................................... 5
Figure 3: Multi-program Multiplex generation ............................................................................................ 6
Figure 4: Multiplex play-out and modulation .............................................................................................. 7
Figure 5: Block diagram of IRT platform .................................................................................................. 12
page 28
Version of
2015-09-04
List of Tables
Table 1: First interoperability test results from TDF (GLOBAL ITV services) ............................................ 9
Table 2: First interoperability test results from TDF (Channel 1 services) ................................................. 9
Table 3: First interoperability test results from TDF (Channel 2 services) ................................................. 9
Table 4: First block of tests - Functional requirements tested - TDF ....................................................... 10
Table 5: First block of tests - Functional requirements tested - IRT......................................................... 12
Table 6: First interoperability test results from IRT (Results with GLOBAL ITV TS) ................................ 14
Table 7: Functional and interoperability requirement tests (UFPA) ......................................................... 15
Table 8: Fist block of tests - Functional requirements tested - UFPA ...................................................... 15
Table 9: Functional and interoperability requirements of the final objective tests (TDF) ......................... 17
Table 10: Functional and interoperability requirements of the final objective tests (IRT) ........................ 17
Table 11: Functional and interoperability requirements of the final objective tests (UFPA) .................... 18
Table 12: Question and objectives of survey developed to GLOBAL ITV Council members .................. 19
page 29

Validation plan and report on first interoperability tests

Transcription

Similar documents

Representing three decades of some of the best

30 Wilmot young place, brockville $199,900.00 MLS ®

Vista Costa Blanca

HEIDI RANGE

Vista Costa Blanca

Antilope a thematic network - European Health Forum Gastein

Dennis Buckmaster

4DCT - AAPM

ITV plc Corporate responsibility report 2006

Here - TV Cream