- Kuwait Institute for Scientific Research

Integrated Model for Project
Evaluation in R&D Institution:
A Case Study of Kuwait
Dr. Abdulhadi Al-Otaibi
Director General
Dr. Yousuf Al-Sultan
Dr. Wesam Al-Khadra
Assistant Director General
Project Analyst
Division of Policy & Planning
Kuwait Institute for Scientific Research
July 2006
LIST OF CONTENTS
TITLE
PAGE
CHAPTER I: KUWAIT INSTITUTE FOR SCIENTIFIC RESEARCH:
ESTABLISHMENT, OBJECTIVES, AND ACHIEVEMENTS 1
Historical Background
1
Main Mission
1
Principles and Values
1
Vision and Aspirations
2
KISR’s Main Objectives
3
KISR’s Research Programs and Strategic Planning
4
Partners in Re-Building Kuwait
5
KISR's R&D Budget 1977-2005
12
KISR's Patents
13
CHAPTER II: EVALUATION AND SELECTION OF R&D PROJECTS:
THE IMPORTANCE OF PROJECT EVALUATION
14
Introduction
14
Overview Background
14
R&D and Economic Growth
16
R&D Project Evaluation and Selection
17
The Importance of Project Evaluation and
Selection in GCC Countries
17
The Concept of R&D Evaluation
18
Research Progress and Contributions to Literature
19
Research Context: Project Evaluation and
Selection in GCC Countries
20
GCC Background
20
R&D Future Outlook in GCC Economy
22
Research Aims
24
Research Strategies and Aims
24
Research Theme
25
The Significance of the Research
26
CHAPTER III: REVIEW OF R&D PROJECT EVALUATION
AND SELECTION
29
Introduction
29
R&D Project
29
Overview
29
Managing R&D Projects
32
R&D Project Evaluation
37
Background
37
The Importance of R&D Project Evaluation
40
Types of Project Evaluation
42
Planning Evaluation
44
Formative Evaluation
46
Summative Evaluation
47
R&D Project Selection
48
Background
48
Methods of Project Selection
50
Screening Models
50
Evaluation Models
58
Portfolio Models
63
Evaluation and Selection Criteria
Proposal Review Process
Views of Evaluation Methods in S&T Organizations in Kuwait
67
69
71
Background
71
Kuwait Foundation for the Advancement of Sciences
71
Environment Public Authority
73
Kuwait University
74
Review of Evaluation Process in Kuwait Organizations
75
Organization Problems towards R&D
Project Evaluation and Selection
76
Views on Integrated Evaluation Model
76
Literature Limitations
78
Technical Discussion
78
Summary
80
CHAPTER IV: RESEARCH DESIGN AND METHODOLOGY
82
Introduction
82
Research Studies and Propositions
82
Research Design
83
R&D Project Review Cycle
83
S&T Policies
86
R&D Project Processing Management
88
Research Methodology
Analysis of Project Evaluation Systems
89
89
Evaluation of R&D Projects in respect to S&T and
Decision Making
91
Improvement of R&D Project Management Procedures
92
Summary
CHAPTER V: R&D PROJECT REVIEW CYCLE
93
94
Introduction
94
Views on ATP's Efforts in R&D Activities
94
ATP's R&D Evaluation Process
96
Proposals and Projects Preparation
100
Desired Proposal Characteristics
101
Desired Proposal Contents
101
Criteria against which Evaluation Proposals will be Judged
102
Minimum Requirements for Contractors
102
Requirements for Evaluation Reports
103
Proposal Preparation and Evaluation Guidelines
104
Project Review Cycle in KISR
109
Lead System
110
Proposals
111
Management of Projects
113
Project Delivery and Reporting
114
Post-Project Follow-Up
116
New Development of Project Monitoring System at KISR
117
Commercialization of KISR's Projects
118
Summary
122
CHAPTER VI: SCIENCE AND TECHNOLOGY POLICIES
124
Introduction
124
Views on Science and Technology
124
Linking S&T Policy to Project Evaluation
126
Kuwait’s National Development Plan
128
Kuwait- Reconstruction after the Gulf War
130
Economic Reconstruction
131
Political Background
133
KISR's Mission
134
Linking Criteria to S&T Policy
135
An Example of S&T Goals and Objectives in Austria
138
Linking Criteria in USA
139
Economic Prosperity
139
National Security
140
Enhancement of Economic Growth
140
Economic Growth in USA
141
Economic Growth in Japan
143
Relationship between S&T and Society
146
Summary
147
CHAPTER VII: R&D PROJECT PROCESSING MANAGEMENT
148
Introduction
148
Views on R&D Scheme
148
The Concept of R&D Project Management
150
A Case Study of Project Processing Practice
153
Introduction
153
Case Study Approach
154
KISR’s Organizational Structure
154
KISR's Projects
154
Project Approval Process
156
Interview Structure and Findings
157
Linking Project Evaluation to Project Management Practice
The R&D Project Evaluation Peer Committee
Peer Committee Structure
160
161
162
Responsibilities
162
Approval Process
163
Types of Participants
163
On-Going Project Advisory Groups
164
Project Monitoring Committee Structure
164
Responsibilities
164
Types of Participants
165
Project Termination
165
Completed Projects
168
Technical Discussion and Summary
171
CHAPTER VIII: CONCLUSIONS AND CONTRIBUTIONS
OF THE RESEARCH
173
Introduction
173
Evaluation of Findings
173
Introduction
173
A New Approach to R&D Project Evaluation
174
Proposed New Model for R&D Project Evaluation
175
Introduction
175
Model Design and Structure
176
R&D Project Evaluation and Selection Process
177
The Evaluation Model
180
The Analysis Model
184
A Hypothetical Case Study for R&D Project Evaluation
187
Case Study Definition
187
Model Structure and Evaluation Trial and Testing
190
Evaluation Model Structure
190
Analysis Model Structure
191
Assessment and Appraisal of the Proposed
R&D Project Evaluation Model
Evaluation of Research Method and Approach
192
194
Implementation on KISR’s Existing Projects; Evaluation Model 199
Implementation on KISR’s Existing Projects; Analysis Model
202
Total Budget
204
Project Duration
204
Manpower Utilization
205
Client Contribution
205
Project Matrixing
205
REFERENCES
210
APPENDIX.I: PRM EVALUATION FORM
225
APPENDIX.II: APPLICATION OF KISR’S MODEL TO
PROJECT EVALUATION
APPENDIX.III: COMPUTERIZED ANALYTICAL PROGRAM
230
237
ABBREVIATIONS
ARC
-Austrian Research Center
ATP
-Advanced Technology Program
CCF
-Country Cooperation Framework
CV
-Curriculum Vitae
DG
-Director General
DDG
-Deputy Director General
EAO
-Economic Assessment Office
EC
-European Commission
EPA
-Environment Public Authority
EPRI
-Electric Power Research Institute
EPSRC
-Engineering and Physical Sciences Research Council
ESRC
-Economic and Social Research Council
FE
-Further Education
FKA
-Formally Known As
GCC
-Gulf Cooperation Council
GDP
-Gross Domestic Product
GIS
- Geographical Information System
GNP
-General Net Present
GRI
-Gas Research Institute
HE
-Higher Education
HH
-His Highness
IT
-Information Technology
KFAS
-Kuwait Foundation for the Advancement of Sciences
KISR
-Kuwait Institute for Scientific Research
KOC
-Kuwait Oil Company
KU
-Kuwait University
MEW
-Ministry of Electricity and Water
MM
-Man Month
MOE
-Ministry of Energy
MOP
-Ministry of Planning
MRM
-Management Review Meeting
MITI
-Ministry of International Trade and Industry
NBER
-National Bureau of Economic Research
NBK
-National Bank of Kuwait
NIST
-National Institute of Standards and Technology
NISTEP -National Institute of Science and Technology Policy
NPD
-New Product Development
NSF
-National Scientific Foundation
OECD
-Organization for Economic Cooperation and Development
OMPM
-Office of Marketing and Project Management
PACI
-The Public Authority for Civil Information
PERT
-Program Evaluation and Review Technique
PMC
-Project Monitoring Committee
PMIS
-Project Management Information System
PRM
-Proposal Review Meeting
RA
-Research Administration
R&D
-Research and Development
REC
-Research Evaluation Committee
REMI
-Regional Economic Models, Incorporated
RFFG
-Reserve Fund for Future Generations
ROI
-Return on Investment
RSS
-Research Support System
SEB
-Source Evaluation Board
SI
-System Integration
S&T
-Science and Technology
STA
-Science and Technology Agency
TRM
-Technical Review Meeting
V-C
-Value-Contribution
PREFACE
Research and development (R&D) project management is as much about attitudes as it is
about procedures. It is a functional area like production, marketing, or finance; it allows
change and innovation to take place.
R&D project management deals with numerous and complex projects. By improving R&D
project management, an organization will become more efficient. R&D project
management provides a framework for managing work, making it easier to manage
projects, improve communication, broaden experience, drive innovation and so on.
R&D project management has a great role in promoting scientific and technical research by
developing and enabling technologies with strong potential for producing broad economic
benefits. It is a catalyst to enhance and improve R&D research projects and to help in
overcoming challenges and barriers that research institutes may face. It also has a main
role in evaluating research activities to support increasing and measuring the short- and the
long-term impacts of technology development projects in order to optimize the funding and
monitoring the research programs. Project management lies within the R&D organizational
strategy to promote the overall economic well-being. Excellent project management
procedures will, undoubtedly, accelerate the facilitation and commercialization of enabling
technologies that are expected to yield large economic benefits to the organization and to
the community, and to enhance the advancement science in the region.
One of the main purposes of R&D project management, therefore, is to produce highquality research projects. It is important to have a stringent review and evaluation system in
order to ease the selection process and direct limited resources to projects more likely to be
successful. For this reason, most R&D and science and technology (S&T) institutions
consider evaluation and project selection criteria as a first priority and a prime element in
their R&D project management philosophy.
The developing countries have shown little achievement in the implementation of R&D
project evaluation and selection. This has been mainly noticed in the S&T and R&D
organizations within the Gulf Cooperation Council (GCC) and particularly in Kuwait's
R&D organizations.
This book argues that if R&D evaluation practices are not carried out properly, then it is
likely to be inevitable for the allocation of R&D resources to be misdirected, resulting in
wasted scientific effort.
In this regard, the research presented here in will investigate an important R&D project
management issue that could play a more effective role in the selection and implementation
of the right projects at the right time. It involves R&D project evaluation and selection
within R&D institutes and related organizations.
The research presented will provide R&D project management with clear, step-by-step
guidelines on effectively selecting projects so as not to detract from work already in
progress or to overcommit and exceed available resources. At the same time, it will provide
R&D management with respected information on how best to tailor project proposals to
increase the likelihood of contributing to national technological, social and economical
development.
The authors will establish an advanced system, which is easy to use and depends on welldefined evaluation criteria. It will include an up-to-date evaluation and selection process,
which consists of two models, an evaluation model, and an analytical model. It will
demonstrate a tailored evaluation model that matches the unique circumstances, objectives
and goals of developing countries.
In addition, the research presented will investigate ways in which the evaluation and
selection process can be linked in a single integrated model throughout a project's life cycle
in keeping with S&T policies, and by improving project management procedures and
regulations to ease its implementation and facilitate its utilization.
This integrated evaluation and selection model can then be applied efficiently to evaluate
most types of R&D projects and assist in selecting appropriate projects to make optimal use
of resources available.
CHAPTER I
KUWAIT INSTITUTE FOR SCIENTIFIC RESEARCH:
ESTABLISHMENT, OBJECTIVES, AND ACHIEVEMENTS
Historical Background
The Kuwait Institute for Scientific Research (KISR) was established in 1967, and
later reorganized in 1973, at which time KISR became attached to the Cabinet of
Ministers, under the supervision of a Board of Trustees, chaired by the Minister of
State for Cabinet Affairs.
In appreciation of Kuwait, and of the importance of scientific and technological
research in raising the plane of the country’s development, and economic and social
progress, an Amiri decree was issued as Law No. 28/1981, declaring KISR to be a
public institution of an independent and legal nature. It also stated that KISR was to
be supervised by a minister selected by the Cabinet of Ministers. In addition, the
decree specified the objective of KISR to be the development of scientific and applied
research, especially that related to industry, power, natural resources, food resources,
and the other major elements of the national economy. Thus, KISR was to work in the
interest of the State’s objectives for economic, technological, and scientific
development, and render consultations to the Government, including the provision of
a scientific research policy for the country.
Main Mission
KISR aims to conduct scientific research, develop technology, provide technical
consultancies and services, and develop human resources in the interest of national
needs, and to support the economic and social development of the State of Kuwait.
Principles and Values
KISR has always held firmly to the principles and values upon which it was founded:
•
Dedication to the progress and development of the State of Kuwait.
1
•
Application of an intensive research strategy that achieves positive outputs.
•
Adherence to the main principles of scientific research, i.e., accuracy and
acting in accordance with scientific facts at all times and under all
circumstances.
•
Adoption of a team spirit in work with partners and financers, and acting
in accordance with an integrated method free of competition.
•
Provision of a good working environment that allows convenient
opportunities for professional innovation and personal ambition.
Vision and Aspirations
KISR continues to participate effectively in the technological, economic, and social
advancement of Kuwait by conducting research, and rendering scientific consultations
and technical services using its resources. KISR looks forward to achieving
distinction in scientific research as a highly efficient institution and center, so as to
make the Kuwaiti society dependent on modern technology.
In light of the new developments brought about by scientific and technological
advances, KISR endeavors to seize available opportunities and invest in the enormous
capabilities of modern technology with the aim of contributing to the development of
Kuwait’s production and service institutions, promoting its competitive abilities, and
achieving comprehensive development in the country.
KISR’s focus on the future has not been restricted to investing in technology and
promising capabilities for advancement, but it has always involved strategic planning
based on the scientific method that specifies priorities and needs, endeavors to meet
these needs at the appropriate time, and contributes to finding creative solutions with
which to face challenges, seize available opportunities, and lead to the advancement
of the society.
In the areas of scientific consultation, and technical and analytical service, KISR
provides distinctive services to numerous sectors in the country, such as oil, water,
2
construction, energy, agriculture, marine environment and fisheries, information, and
the economy.
In the areas of information and information technology (IT), KISR continues to
develop its applications as part of the e-KISR project, and for knowledge
management. It also continues to develop scientific and administrative information,
and apply Geographical Information Systems (GISs). In addition, it has developed a
Kuwait Distance Learning Center, which is a promising projects expected to have
positive effects on the development of national manpower.
In the area of technical facilities and capabilities for research, KISR continues to
establish and maintain modern research facilities, developing and providing existing
facilities with the latest equipment and instrumentation needed to meet the
requirements of research and development (R&D), and the needs of governmental and
private bodies, institutions, and companies in Kuwait.
KISR’s Main Objectives
KISR seeks to achieve its main objective, i.e.,
the promotion of scientific and applied research, particularly in matters relating to
industry, natural and food resources, and other primary constituents of the national
economy, in an endeavor to serve the goals of economic, technological, and
scientific development.
In addition, KISR offers advice to the government on scientific matters and on science
policy issues.
KISR achieves its objectives by
•
Conducting scientific research and studies relevant to the development
of national industry and the preservation of the natural environment, in
coordination with other concerned parties.
•
Encouraging young Kuwaitis to pursue scientific research.
3
•
Studying natural resources and the best means for their utilization,
including developing water and power resources, and improving
agricultural utilization, in cooperation with other concerned parties.
•
Providing research, and scientific and technological consultations to
governmental and private institutions.
•
Keeping abreast of the latest developments in and adopting modern
techniques for science and technology (S&T).
•
Establishing
and
strengthening
cooperative
relationships
with
institutions of higher education and scientific and technological centers
in Kuwait and around the world, in order to conduct joint research, and
exchange information and expertise.
•
Contributing to the study of the diversification of the nation’s economic
resources, and working towards the development of the country’s
economy.
•
Providing consultative and applied services and expertise, and
conducting scientific and technological research to serve the
development objectives of the Gulf Cooperation Council (GCC)
countries in particular, and of the Arab World in general.
KISR’s Research Programs and Strategic Planning
Since its establishment, KISR has endeavored to achieve its objectives through fiveyear plans, with the first covering the period from 1979 to 1984. KISR has continued
to develop the method for preparing its strategic plans over the past 25 years, in spite
of the damage that affected most of its facilities in 1990-1991. KISR’s five-year
strategic plans are as follows:
•
The First Strategic Program for Research (1979-1984).
•
The Second Strategic Program for Research (1984-1989).
•
The Third Strategic Program for Research (July 1990).
4
Partners in Re-Building Kuwait
The late Amir, HH Sheikh Jaber Al-Ahmed Al-Jaber Al-Sabah, invited a number of
KISR’s officials to participate in the official Kuwaiti meetings that were held in Taif,
Kingdom of Saudi Arabia, to discuss the tasks to be implemented immediately after
the country’s liberation.
On 20 April 1991, KISR’s management started to rebuild KISR’s facilities with the
aim of reinstating its previous status as a reputed scientific center. A committee was
formed, headed by the Director General (DG), and its members included the Deputy
Director General (DDG), Assistant Director General, directors’ of KISR’s scientific,
advisors, and senior researchers. The committee drafted a short-term (one-year) plan
(1991-1992) focused on identifying losses, requesting replacements for equipment
and instrumentation that had been looted by the invaders, rehabilitating the Institute’s
manpower, and operating scientific and support divisions and departments. The
research activities of this period were focused on coping with the effects of the
aggression in the fields of the environment, food resources, oil, and water and energy
resources.
•
The Transitional Strategic Program (1992-1995).
•
The Fourth Strategic Program for Research (1995-2000).
In order for KISR to achieve the above-described goals, its organizational structures
underwent a variety of changes until it was decided, over the course of a decade, to
utilize the following six programs.
1. Petroleum Resources Program
There is no doubt that the petroleum sector forms the cornerstone of Kuwait’s
economy. Therefore, it is important to enhance this sector, activate it, and
allocate resources to support it.
5
Objectives
•
Support petroleum production and refining to achieve higher
economic proceeds, through the provision of required research
development in relevant fields, upgrading of operational processes,
and amendment of the properties of oil-refining products.
•
Utilize foreign expertise in the development and diversification of
oil products.
•
Conduct studies in the field of corrosion.
This program consists of the following three elements
•
Petroleum Production.
•
Petroleum Refining.
•
Petrochemical Processes.
Main Research Activities
•
Characterizing oil reservoirs.
•
Improving oil-production efficiency.
•
Handling environmental problems for oil-production processes.
•
Improving catalyst process techniques.
•
Enhancing oil refining production efficiency.
•
Improving oil products.
•
Developing improved petrochemical products.
•
Handling environmental problems of the petrochemicals industry.
2. Water Resources Program
Water is a scarce and expensive source in Kuwait. The increasing water
consumption urges that the highest priority be given to the need for water. KISR
has taken the initiative in this respect, especially in its desalination projects were
implemented in cooperation with the Ministry of Energy (MOE) (fka the
Ministry of Electricity and Water (MEW)). KISR continues to cooperate closely
with the Government and the other Gulf countries in the field of hydrological
6
studies. As for the demand on water, KISR has focused its study the ideal
balance between water resources and demand in Kuwait.
Objectives
•
Expansion of the supply of water resources.
•
Efficient utilization of water resources.
•
Development of a technological base for reducing water
consumption, and developing related policies and legislations.
•
Building a base of qualified technical national manpower to reduce
the utilization of water resources.
Main Activities
•
Hydrology and Water Resources Management.
•
Water Technologies.
Main Research Activities
•
Water desalination.
•
Primary treatment of charged water.
•
Performance of different types of membranes.
•
Assessment of groundwater stock.
•
Groundwater contamination.
•
Artificial recharge of groundwater
•
Increase of groundwater levels.
•
Assessment of water in residential units.
•
Assessment of scientific and economic feasibility for using
groundwater in mosques.
3. Food Resources Program
With Kuwait’s increased dependence on food imports, the issue of food safety
rose to the top of the nation’s priorities, especially as it relates to the production
of food from marine and terrestrial resources. Thus, it is important to utilize
Kuwait’s limited terrestrial and water resources optimally while preserving their
7
quality. Research conducted in this field was focused on increasing the
production of marine and agricultural resources (plants and animals),
manufacturing foodstuffs, and producing scientific and artificial alternatives to
feed for cattle and fish.
Objectives
•
Production of local food based on strong environmental elements.
•
Enhancement of the rationalized utilization of national resources by
increasing Kuwait’s production of food and animal feed, developing
modern systems to treat food and animal feed through the utilization
of conventional and nonconventional techniques, and optimal
rationalization of the country’s natural resources.
This program consists of the following three elements
•
Aridland Agriculture and Greenery.
•
Aquaculture, Fisheries and the Marine Environment.
•
Biotechnology.
Main Research Activities
• Plantation and poultry production.
• Survey and classification of Kuwait’s soil.
• Drafting of a long-term strategic plan (1995-2015) for the
development of the agricultural sector.
• Animal production.
• Fish and marine resources.
• Initiation of zubaidi culture.
• Oceanography.
• Fish stock development.
• Fish age estimation.
• Preservation of natural resources and biological diversity.
• Analysis of animal feed.
• Genetic engineering of plants, bacteria, and animals.
• Tissue culture.
8
•
Embryo transfer.
•
Production of food products with increased nutritional value.
4. Environmental and Earth Sciences Program
Kuwait’s interest in the environment is increasing day by day. KISR has been
a pioneering national institution in this field conducting a wide range of
studies on dust storms, and biological and chemical pollutants in the air and
water in industrial and marine areas.
Objectives
•
Continuing the preparation and evaluation of a database on the
marine, terrestrial, and aerial environments of the State of Kuwait,
and with definition of natural and anthropogenic problems.
•
Developing mathematical modules to estimate the effect of various
developmental activities on the environment.
•
Providing concerned authorities with proposals for strategic
elements and environmental preservation measures.
The program consists of three elements
•
Environmental Sciences.
•
Coasts and Air Pollution.
•
Desertification and Atmospheric Dynamics.
Main Research Activities
•
Marine pollution.
•
Terrestrial pollution.
•
Aerial pollution and traditional pollutants.
•
Solid and liquid waste.
•
Monitoring the atmosphere’s pollution.
•
Radial pollution monitoring.
•
Biological treatment of oil-polluted soil.
•
Energy modeling.
9
•
Assessment of seismic risk.
•
Coastal and island development.
5. Energy, Building and Engineering Systems Program
KISR has contributed to the infrastructure services’ sector, i.e., electricity,
water, and drainage, with researchers designing and helping to operate the
services. KISR also developed new methods for designing distribution networks,
analyzed operational problems, and supported the efforts of the MOE (fka the
MOE) in energy preservation measures.
Objectives
KISR’s main objective in this program was to provide convenient and costeffective planning and engineering solutions for important national problems,
such as
•
The government housing program.
•
Preservation of energy resources.
•
Electricity distribution networks and treatment of wastewater.
The program consists of the following three elements
•
Energy.
•
Building Technology.
•
Systems and Monitoring.
Main Research Activities
•
Electricity generation.
•
Recycling of industrial and construction waste.
•
Treatment of environmental problems and natural phenomena that
affect building safety.
•
Energy auditing.
•
Solar energy research.
10
6. Techno-Economics Program
KISR owns research tools for the natural and biological sciences, as well as
economics. The two goals KISR has recently achieved, namely, the utilization
of economic analytical methods to benefit the development of Kuwait and the
Gulf region, and support of KISR’s other scientific divisions in the areas of
economic analyses and feasibility studies.
Objectives
•
Establishing a distinctive center for national economic planning,
resources development, computer applications, and industrial
operations research.
•
Developing and enhancing analytical modules related to
economics and applicable systems.
•
Developing applicable systems for imitation modules based on
computer and artificial intelligence, and means for decision
support.
•
Providing support represented by economical assessments, applied
and experimental statistics, and mathematical modeling in
accordance with the needs of KISR’s other research programs.
The program consists of the following two programs
•
Economic Studies.
•
Quantitative Methods and Modeling.
Main Research Activities
• Development of the industrial sector.
• Manpower and employment opportunities.
• Financial and banking sector.
• Foreign trade.
• Economic feasibility studies.
• General and productive services.
• Educational, housing, and health services.
11
KISR’s R&D Budget 1977 – 2005
Research Program
Project Budget (KD) *
Total Budget
No. of
Financers’/Beneficiaries
(KD)
Research
Contribution to the
KISR’s
Financers’/Beneficiaries’
Contribution
Contribution
9,296,702
12,693,423
21,990,124
145
58
Water Resources
8,223,928
8,321,709
16,545,637
84
50
Food Resources
27,093,114
13,927,750
41,020,864
250
34
5,488,208
5,098,226
10,586,434
219
50
3,827,774
7,628,137
11,455,911
176
48
2,209,548
3,677,692
5,887,240
96
62
56,139,274
51,346,937
107,486,210
970
48
Petroleum
Projects
Total Budget (%)
Resources
Environmental
&
Earth Sciences
Engineering
Systems
Techno-Economics
Total
NB: The Environmental, and Earth and Engineering Systems Programs were merged
as the Environment and Urban Development Program in January 2001.
*KD1 = US$3.3
*KD1 = £2
12
KISR’s Patents
KISR has internationalized its research projects through the registration of numerous
patents. KISR has obtained 25 patents in various areas of R&D from Arab, regional,
and international institutions.
Most of the patents registered by KISR are for techniques developed for the
manufacturing of chemicals that have useful application in the petrochemicals
industries in the State of Kuwait, and the Gulf countries in particular, and in the rest
of world in general.
13
CHAPTER II
EVALUATION AND SELECTION OF R&D PROJECTS:
THE IMPORTANCE OF PROJECT EVALUATION
Introduction
R&D is an economic tool for growth and development. In depth, this chapter is detailing
the concept of evaluating and selecting R&D projects within KISR and the GCC
countries. To start the research journey, the chapter also emphasized on the expected
contributions to the literature beside the research strategies, aims, theme, and
significance. Finally, the contents of all other research’s chapters' contents are
summarized to include the overall paths of the research work.
Overview background
R&D project management is different than any other form of management. Project
Management is as much about attitudes as it is about procedures. It is a functional area
like production, marketing, or finance; it allows change and innovation to take place.
A project is a process of coordinated activities with a conception and completion. In
order to achieve certain objectives, a specific set of tasks are implemented which depend
on a limited time, cost, and resources.
R&D project management deals with numerous and complex projects. By improving its
project management, an organization will become more efficient.
R&D project
management provides a framework for managing work, making it easier to manage
projects, broaden experience, improve communication, drive innovation and so on.
Above all these are evaluation and project selection criteria that must be considered as the
prime element in R&D project management.
14
Projects have many different forms (Lockyer and Gordon, 1996), however they all have
common elements, namely: plan, time, budget, cost, objectives and expected output,
identification of any areas of uncertainty, and evaluation for possible risks.
Designing a proposal is similar to designing a laboratory experiment. Just as an objective
is necessary when starting a lab experiment, the first priority in formulating a proposal is
also a well-defined measurable objective.
Typically, each project processes through several phases (Lockyer and Gordon, 1996),
i.e.:
1. Conception: the idea for the new project to the organization. This is the
most important phase.
2. Evaluation: to evaluate the new project by the evaluators/peer reviewers.
3. Development: if the new project is accepted, it will be designed and
specified to execute the project.
4. Realization: to turn the development into reality, such as determining the
project’s progress and achievement.
5. Termination: if analysis of on-going project’s report provides invaluable
progress, then the decision about termination should be considered.
6. Completion: end of project.
One of the main purposes of R&D project management, therefore, is to produce highquality research projects. It is important to have a vital review and evaluation system in
order to ease the selection process. For this reason, most of the R&D institutions
consider this as a first priority of the R&D management system. It is cited in the literature
that National Scientific Foundation (NSF, 1996), Merit Review System, recently had
updated their evaluation system and criterion. In the State of Kuwait, KISR, Kuwait
Foundation for the Advancement of Sciences (KFAS), Environment Public Authority
(EPA), and Kuwait University (KU), are considered with the evaluation system, whereas
15
KISR had previously two steps to evaluate the project, now it is integrated into one step
as an approach to enhance the evaluation system in KISR.
R&D and Economic Growth
Economic growth is an increase in national income over a period of time, normally
national income rises each year. The measure of national income used is gross domestic
product (GDP). In almost every year, GDP has been higher than in the previous year.
About one-thousand years ago, world output was extremely small; it consisted mainly of
food, simple shelter, wood and other simple necessities of life. Today, the output of just
one small economy, that of the UK is well over £600 billion a year. How has this
transformation taken place? What are likely sources of economic growth in the future?
Long-term economic growth is generated mainly by increasing the quantity and quality of
the factors of production in an economy. Some countries have grown because of their
rich resources of land. As we have seen, ‘land’ includes not only land itself, but all
natural resources. Saudi Arabia, for instance, has grown because it has been able to
exploit its oil reserves. The UK, too, is rich in many natural resources, North Sea oil
alone contributed up to 2-3% a year in the 1980s to the national income of the UK. Not
all countries possess rich natural resources, but labour too, can be an important source of
growth (Anderton, 1993).
In this respect, R&D research projects, technology advancement, and innovations are the
major support arms enhancing the world economic growth. Moreover, it is inevitable that
research and technology parks will replace the oil wells in promoting the economic
growth. Therefore, evaluation and selection of the right R&D projects play a vital and a
major role in economic development through implementing the needed advanced R&D
projects that support the national economy and enhance the advancement of science in
the region.
16
R&D Project Evaluation and Selection
Despite the increasing efforts of such evaluation systems, the effectiveness of the various
approaches to the evaluation of research has not been critically assessed (OECD, 1997).
Regardless to the R&D activities, usually the evaluation system and process include one
of the two following complementary approaches: The use of quantitative indicators and
the use of qualitative peer judgment. In addition, there are a number of other factors to
consider in evaluating an R&D project, such as the type of R&D organization, level of
evaluation, S&T concerned in specific areas and so on.
The term ‘project selection’ has many different meanings (Souder, 1975b). In its
narrowest sense, project selection means determining which project will be best
approved. In its broadest sense, project selection may be viewed as a sequential portfolio
determination process. In this broader view, the decision maker wishes to determine the
best allocation of the available resources among the alternative projects (Souder, 1980).
Each project can be funded at several different levels, with each level yielding a different
contribution to the individual or organizational goals.
The Importance of Project Evaluation and Selection in GCC Countries
The emphasis of effectiveness and efficiency has been increased on governmentsupported research through research evaluation, as it is needed for optimizing research
allocation of budget and other resources, re-orienting research support, rationalizing
research programs/elements/projects, augmenting research productivity, etc. This is why
R&D centers have developed and stimulated research evaluation activities in an attempt
to get more value from the resources allocated for research support. Research evaluation
can be performed by several different actions (OECD, 1997), depending on the objectives
and specific context of the evaluation. Basically, there are two major categories of
evaluation performance: self-directed process when implemented by institutions
17
themselves and that which is performed by an external organization. Research
evaluations, regardless of the society culture, depend on two basic approaches: the use of
quantitative indicators such as bibliometrics, and the use of more qualitative peer
judgments.
Therefore, it is important that GCC's R&D institutions highly consider the evaluation of
their research and projects, in order to enhance their project selection practice. Evaluation
should be conducted on the one hand, at an institutional level with full cognizance of the
impacts of research on the other interrelated functions of those institutions. On the other
hand, the evaluation system should be linked to the national system of innovation, where
the public and national entities would be involved in the evaluation and selection and its
related criteria to meet the beneficiary sectors needs as well as to match with the R&D
centers and S&T entities capabilities and research programs.
The Concept of R&D Evaluation
The notion of evaluation has been around for some time. In fact, the Chinese had a large
functional evaluation system in place for their civil servants as long ago as 2000 B.C.
(EHR/NSF, 1993). Not only does the idea of evaluation have a long history, but it also
has varied definitions. Evaluation means different things to different people and takes
place in different contexts. Thus, evaluation can be synonymous with tests, descriptions,
documentation, or management.
Many definitions have been developed; the Joint Committee on Standards for
Educational Evaluation (1981) presents a comprehensive definition:
“Systematic investigation of the worth or merit of an object…”
This definition centers on the goal of using evaluation for a purpose. At the most general
level, the process of ‘evaluation’ is a particular control mechanism and as such it is
inevitable where social formations are concerned (Lapiere, 1954 and Lemert, 1967). Most
evaluations aim to answer the question: Is the program doing well (achieving) what is set
18
out to do? ‘Evaluation’ can be also viewed as an event or as a process. While more
traditional approaches to the evaluation of research initiatives tended to see it as a one-off
(or iterative) resource intensive exercise, more recent experience indicates an increase in
the use of continuous evaluations (EUREKA and COST, 1998; Sand and Nedeva, 1998).
Research Progress and Contributions to Literature
R&D project evaluation is facing an increased interest among R&D and S&T
organizations. This is due to the fact that approving and supporting an R&D project is not
a simple decision, rather a complex process rooted in S&T policy, mostly involving
formulation of economic and research priorities, management of the research process and
monitoring and follow-up on the implementation of research results. Moreover, funding
and allocating limited resources to R&D projects is becoming more important and
sensitive toward R&D project management.
Parker (1999) argued that, on average, only one in seven projects is commercially
successful, where 50-75% of development money was spent on projects that never
actually reached the market.
This research, therefore, will investigate on academic and hypothesis methods that could
enhance R&D project evaluation process, due to the fact that current existing evaluation
methods cited in the literature do not involve an integrated approach to deal with the
R&D projects within the overall review life cycle of the project. It will set out to explore
the proposition that they neither enhance the possibility of successful commercialization
of the project nor reduce the risks of selecting the right projects.
Thus, the expected contribution to the literature of this research will define an important
role of the R&D project evaluation and selection in order to close the gap between the
cited existing literature and what is expected to be contributed from this research work. It
is expected that this research will add a vital contribution to the R&D evaluation theory
emphasizing the procedures and practices needed to introduce a new approach to develop
19
well-designed guidelines, procedures, and a model mainly to be used by all economy
nations in the GCC region, where the GCC region and other developing countries are still
practicing the traditional means, ways, and methods that are strongly needed to be
promoted.
This research will also investigate the development of prevailing R&D project evaluation
practice, linking the evaluation criteria for the project review life cycle within a
systematic framework. In exploring the research propositions and addressing the research
questions, it will help R&D and S&T organizations direct limited resources to projects
more likely to be successful.
The findings of the main tasks of the research will be integrated to construct the R&D
project evaluation model, based on the following:
•
Criteria of evaluation and selection within a project's life cycle;
•
Proposal preparation;
•
Method of evaluation (technical and management);
•
Evaluation procedures; and
•
Project management (monitoring and follow-up).
This task will be followed by examining the efficiency of the model and its utilization, by
implementing its applications on several hypothetical proposals and projects within an
R&D organization. Further modifications will be adjusted on the integrated evaluation
model as needed.
Research Context: Project Evaluation and Selection in GCC Countries
GCC Background
According to its charter, the GCC is a political, social, regional organization. It is a
regional cooperative system between the Arab States of the Gulf created to meet the
challenges imposed by surrounding circumstances. The scope of its work includes
20
economy, politics, security, culture, health, information, education, legal affairs,
administration, energy, industry, mining, agriculture, fishery and livestock. The United
Arab Emirates, the Kingdom of Bahrain, the Kingdom of Saudi Arabia, the Sultanate of
Oman, the State of Qatar, and the State of Kuwait are members of the GCC.
The basic objectives of the Cooperation Council (Man and Development, 1995) are:
•
To undertake the coordination, integration, and interconnection among
member states in all fields in order to achieve unity among them.
•
To deepen and strengthen relations, links and areas of cooperation now
prevailing among their people in various fields.
•
To formulate similar regulations in various fields including the following:
- Economic and Financial Affairs;
- Commerce, Customs and Communication;
- Education and Culture;
- Social and Health Affairs;
- Information and Tourism;
- Legislative and Administrative Affairs.
•
To stimulate scientific and technological progress in the fields of industry,
mining, agriculture, water
and
animal resources; establish scientific
research; establish joint ventures and encourage cooperation by the private
sector for the good of their people.
The GCC countries have wisely recognized that its major natural resources (Petroleum
and Gas) are a finite resource. They have carefully embarked on a structural program to
use the income from developing and marketing this natural resource to improve the living
environment for the local population and provide for future generations. One component
of this strategy is to retain and invest part of the financial flow from petroleum for future
generations. A second component is the improvement of the physical environment,
wherever possible, the effective utilization of the available natural and human resources
21
to produce deliverable goods and services. A third component of this strategy is the full
development of the intellectual potential of their citizens.
GCC countries have recognized that in the modern world, social, physical and life
sciences all play an important role in structuring and sustaining an effective society,
therefore ensuring that their educational system - from the elementary through the
university level - includes all elements of a balanced development program.
They have also recognized that the life and physical sciences are today being used in
many practical ways in order to develop goods and services that society needs. The
mechanisms required to manufacture and supply these products become the engines of a
social and financial growth that make it possible to improve the quality of life in the
society. This evolution of the application of science to produce goods and services has
occurred in the Western developed world over many generations. Similarly, its
educational systems have evolved in such a way that the search for knowledge and the
application of that knowledge have become closely intertwined. In some instances
structures have evolved such that the full range of seeking and applying knowledge
occurs within one institution dedicated to education, research and application.
There is, however, considerable evidence that the flow of information and people within
and between organizations is a critical element in building a viable system for generating,
applying and exploiting knowledge (Robinson, 2002). GCC have established an
educational system (including universities and institutes for scientific research) that
creates and applies science. It has also given a clear mandate to the organizations
involved to develop a viable structure to use science for the benefit of society, and to
make it compatible with their human and physical resources.
R&D Future Outlook in GCC Economy
The research activities are geared to the fact that the new information and advanced
technology-based areas are now booming and will continue to open-up a whole new array
22
of development opportunities well into the 21st century. These new opportunities will
force enormous changes in business/economic structures, bring clarity to new S&T
challenges, and demand more business competitiveness and entrepreneurship than has
been required in the past.
It is also anticipated that societies would be more extensively privatized and that the S&T
infrastructure will have to be improved merely to maintain its position, let alone grow.
a) Development/ Organization Perspective:
•
Privatization;
•
Offset Program;
•
Public Sector Organization Adjustments;
•
Shifts in Types of Industries.
b) Financial or Business Circumstances:
The current funding status, taking Kuwait as an example of the GCC countries, is that
most research funds are allocated directly by the government or through contractual
research with public sectors. Private sector contributions to the R&D activities are
minimal – typically in the range 10 - 15% of the total funds. This may be due to the fact
that the private sectors in Kuwait contribute to one of the funding agencies, known as
KFAS. Founded in 1976, KFAS’s resources are endowed by a 5% deduction of profits of
companies and firms and banking operating in Kuwait. These resources are directed
toward funding (non-reimbursable) several scientific and technological activities, where
funding R&D activities is a major issue.
c) Technological Advancements/Opportunities:
GCC countries research centers focus was initially directed, approximately, on the
creation of a database and the establishment of core competencies - emphasizing ‘core’
focused projects. Today, the emphasis has moved to contractually applied research. It is
anticipated that the focus will shift toward ‘technological adaptation of research results’
23
and the ‘absorption and transfer of technology’ per se for its commercialization, from a
commercial stand point.
Specific areas of emphasis include:
•
Information Technology (IT) and System Integration (SI);
•
Advanced Biological Technologies;
•
Other Cutting-Edge Technologies.
Research Aims
Research Strategies and Aims
The research will study the evaluation and the selection of individual submitted,
portfolio, on-going, and completed projects in order to find ways in which to integrate
them into an evaluation system. The research will review current R&D project
management practice and evaluation systems. In addition, the research will compare and
contrast the process of writing and preparing proposals and their relationship to various
evaluation techniques.
The research design will include several approaches and propositions that would consider
case study, interviews, and analysis of actual data and information that could enhance the
research findings and contributions.
The aims of the research, therefore, are:
1. To develop a conceptual and integrated model for R&D project evaluation
that not only includes selection of proposed projects, but also evaluation of
on-going projects, output evaluation for completed projects as well as
project process management.
24
2. To develop evaluation criteria that includes S&T policies perspectives as
well as social, economic, political, environmental and financial factors.
3. To explore an appropriate evaluation and selection mechanism that could
link R&D capabilities, business sectors and end users needs in an
integrated evaluation process.
4. To make a case study for evaluating the potential of R&D projects with the
proposed new model, methods and selection criteria to demonstrate the
efficiency of the proposed model.
Research Theme
In a study by (Bordley, 1998), entitled R&D Project Selection Versus R&D Project
Generation , the authors state that emphasis centers on the generation of high-quality
R&D projects through effective communication of corporate priorities, implementation
issues, and related technical efforts; Criteria of Project Success: an exploratory reexamination article published by (Lim and Zain Mohamed, 1999), explores the issues
from different perspectives, clarifying the differences between criterion and factors. The
authors propose that a project should be classified using two categories: macro and micro
viewpoints.
The research also will depend on a review of the literature, published by (Schmidt and
Freeland 1992), ‘Recent Progress in Modeling R&D Project-Selection Processes.’ Here,
the authors state that a new stream of research in R&D project selection models emerged
in the 1970’s in response to replace the traditional models that was identified as a
constrained optimization problem, where the organizational context in which decisions
are made was ignored. The authors approach seeks insight rather than outcomes, focusing
on the decision process by reviewing the development of an evaluation model. They
divide the project selection model into two: the traditional ‘decision-event’ approach and
the ‘decision-process’ or ‘system’ approach. In reaching their conclusions, the authors
reviewed the project selection literature in order to provide both researchers and
managers with a summary of current knowledge regarding the success of the systems-
25
oriented approach. As discussed earlier, they conclude that R&D project selection
remains an important managerial issue and the system approach is still in the early stages
of development and, therefore, few concrete results or methods are currently of direct use
to practitioners. The authors highlighted the following areas of research:
•
Addressing problems at high level of aggregation, at the level of corporate
management rather than at the department level.
•
More research to be carried on quantitative modeling of the interactions,
(e.g. R&D/ marketing interface to be studied from quantitative approach).
•
Modeling efforts to include area of strategy development.
•
Improvement of system models to explore the trade-offs between various
aspects of performance as risk, development time, and cost.
Therefore, this research will investigate a methodology with multi criteria perspectives
for the evaluation and selection of R&D projects that depend on the nature of the project
and the decision problem in hand. In addition, the integration will be implemented at
corporate and business level to include all four proposed phases of the project process
cycle as shown in the upcoming figure.
The Significance of the Research
The developing countries are greatly concerned in advancing their status in S&T and
related activities, where selecting research projects gives a high priority in order to utilize
their resources in a good manner and achieve their goals and objectives within their
national plans. Therefore, several organizations have allocated their resources and efforts
to develop new methods of evaluation that goes online with the upcoming technological
and economic changes. On the other hand, the GCC countries are staying away behind
these developments, where their resources are wasted without gaining significant findings
26
Individual
Project
Evaluation
Portfolio Analysis
(Under consideration)
Integrated
Model for
Project
Evaluation
Completed
On-going
Project
Project
Proposed Phases of Project Process Cycle.
27
and outputs from their R&D activities. In this regard, this research will investigate one of
the important R&D management issues that could play a more effective role in selecting
and implementing the right projects at the right time. It will deal with the issue of R&D
project evaluation and selection within the R&D institutes and related organizations.
The significance of the research is to develop a well-suited and tailored evaluation model
to be utilized by the GCC countries, where more effective procedures and systems could
be implemented to enhance the technological advancement in the GCC region.
The GCC countries will benefit from such research works that pave the way towards
improving the R&D project evaluation practice and use their efforts in selecting the most
successful R&D projects that will contribute to the overall region S&T goals and
objectives.
28
CHAPTER III
REVIEW OF R&D PROJECT EVALUATION AND SELECTION
Introduction
This chapter looks into and investigates views of the available literature within the field
of R&D project management and R&D project evaluation and selection. The cited
literatures are analyzed and criticized to broadly understand the R&D project evaluation
and selection importance, types, techniques, methodologies, criteria, evaluation process
and organizational constraints that been explored and utilized in this respect.
Furthermore, the literature limitations will be quoted in order to assess the research work
and to enrich its proceeding and further chapters with ideas and approaches to build up
the proposed evaluation model.
R&D Project
Overview
A ‘project’ is a process of coordinated activities with a conception and completion. It
sets out to achieve certain objectives through implementing specific tasks that typically
depend on limited resources. R&D project management provides a framework in which
to control a large number of complex projects, simultaneously. By improving project
management, the organization will become more efficient.
A critical element of this framework is the way in which projects are evaluated and
selected. R&D performance can be measured at several levels. Typical levels of
assessment are organization division, department, process, project and personal level.
This research therefore focuses on evaluation aspects at the project and management
process levels.
64
R&D projects can be divided into stages in which different evaluation techniques are
applied (Pappas and Remer, 1985).
R&D projects have become increasingly more complex and costly over the past few
years. Today's projects often entail large organizational commitments that can turn into a
significant loss if a wrong choice is made. Thus, it is important to evaluate and select
only the very best projects. Project evaluation and selection involves the determination of
the most appropriate or best ideas to start work on, and to continue working on as time
passes. Many different methods have been developed for R&D project evaluation and
selection during the last decades (Brenner, 1994; Liberatore and Stylianou, 1995; Piippo
and Tuominen, 1990). ‘Acting smart’ and ‘being a smart’ organization may be considered
the best guarantees of business success in this fast changing and competitive world.
‘Being smart’ could be defined as making good strategic decisions. ‘Acting smart’ is the
activity of effectively carrying out those decisions (Matheson and Matheson, 1998).
According to more recent studies (Martino, 1995; Cooper, et al. 1998), the use of R&D
project selection methods is quite low in R&D organizations, where in many cases, the
techniques have been developed in academic settings and they do not adequately reflect
the realities and problems faced by R&D management, especially if they are oriented to
the use of complicated mathematical tools.
Project selection decisions are very closely interrelated with project funding and resource
allocation decisions (Souder, 1984). For instance, the selection of a particular project
usually implies some corresponding commitment of resources to it. Similarly, the
selection of a portfolio of several projects implies something about the allocation of the
available manpower and resources among these projects. Project selection is also closely
interrelated with project control and with project completion and termination decisions.
R&D project evaluation is a critical area of capital budgeting for technologically oriented
companies. As project selection is the means by which technology strategies are actually
64
implemented, the long–term success of a company is often determined by the
effectiveness of its project selection process.
Corporate managements give their time to extramural exercises in the hope that the
information accruing will help their strategy processes. Governments hope to ensure that
the research they fund will be in areas where they have industrial strengths able to exploit
the likely results. Moreover, towards this discussion, the adequate system for evaluation
and selection of R&D research projects is considered to be the stringent mechanism to
support the R&D management in order to achieve their set goals and objectives. This is
due to the fact that usually the R&D managers have long struggled with the evaluation of
R&D projects opportunities. R&D projects are difficult to evaluate for several reasons
(Loch and Bode-Greuel, 2000):
•
Not all possible contingencies or courses of actions are known;
•
Quantitative economic estimates are subject to significant uncertainty;
•
Continued corrective action based on new information is at the heart of
successful R&D management, and central to strategy in general (e.g.,
Markides, 1999) – how should one quantitatively represent such
continued action in financial tools?
R&D managers and public administrators have always defended the merits of long-term
research on qualitative arguments of technology options. Recently, however, economists
have formulated the methodological foundations for appraising such arguments
empirically. It is time for public R&D project managers and policy decision makers to
carefully consider the technology options approach to strategic R&D project selection as
it has the potential of liberating them from wearing ideological battles (Vonortas, 1995
and 1998).
In this respect, as several evaluation approaches and processes have been practical in the
State of Kuwait, and have been implemented in the S&T organizations, it may be
concluded that a better integrated system could be sought and searched to unify the
64
evaluation system and link it to the main issues and factors that effect the overall
evaluation process (Szakonyi, 1988).
Managing R&D Projects
Regardless of implementing a unique R&D project evaluation and selection system,
practicing a parallel stringent management methodology is essential. The question of
why R&D projects succeed or fail remains valid, where managing R&D projects play a
major role in this respect.
Project success is a topic that is frequently discussed and yet rarely agreed upon. The
concept of project success has remained ambiguously defined. It is a concept which can
mean so much to so many different people because of varying perceptions, and leads to
disagreements about whether a project is successful or not. Bacarini, (1999) presented a
logical framework method for defining project success. He emphasizes the following:
Project Management Success: This focuses upon the project process and, in
particular, the successful accomplishment of cost, time, and quality objectives.
It also considers the manner in which the project management process was
conducted.
Product Success: This deals with the effects of the project’s final product.
Project Goal: Is the overall strategic orientation to which the project will
contribute and should be consistent with the strategic plans of the organization.
Project Purpose: Are the intended near-term effects on the users of the project
as a result of utilizing the project’s outputs.
Project Outputs: Are the immediate, specific, and tangible results or
deliverables produced by project activities.
64
Project Inputs: Are the resource inputs and activities required to deliver each
output.
Project Management Success
•
Meeting time, cost, and quality objectives.
•
Quality of the project management process.
•
Stakeholder satisfaction.
Quality of Project Management Process
•
Anticipating all project requirements, having sufficient resources to
meet project needs in a timely manner, and using these resources
efficiently to accomplish the right task at the right time and in the right
manner.
•
Dealing with the issues early or as soon as they surface and keeping
management informed.
•
Effective co-ordination and relation patterns between project
stakeholders, for example, team spirit, participative decision making.
•
Minimum scope changes, no disturbance to the organization’s main
flow of work, and no disturbance to corporate culture.
•
Completeness of the termination, absence of post project problems,
quality of post-audit analysis, identifying technical problems during
the project and solving them.
Product Success
•
Meeting the project owner’s strategic organizational objectives
(project goal).
•
Satisfaction of users’ needs (project purpose).
•
Satisfaction of stakeholders’ needs where they relate to the product.
64
General Characteristics of Success
•
One can succeed and fail.
•
Project management success is subordinate to product success.
Project management success influences product success.
•
Success has ‘hard’ and ‘soft’ dimensions.
•
Success is perceived.
•
Success criteria must be prioritized.
•
Success is affected by time.
•
Success is not always manageable.
•
Success may be partial.
Steven Wheelwright and Kim Clark (1992) created a project plan to focus on product
development. They cited the following comments regarding project management
behaviors
•
When a project ran into trouble, engineers from other projects were
reassigned or, more commonly, asked to add the crisis project to
their already long list of active projects. The more projects they
added, the more productivity dropped.
•
The reshuffling caused delays in other projects, and the effects
cascaded.
•
As deadlines slipped and development costs rose, project managers
faced pressure to cut corners and compromise quality just to keep
their projects moving forward.
A comparison of individual project Vs set of projects was set as
•
In most organizations, a management directs all its attention to
individual projects - it micromanages project development.
64
•
But no single project defines a company’s future or its market
growth over time; the set of projects does.
•
Most companies should start the reformation process by eliminating
or postponing the lion’s share of their existing projects, eventually
supplanting them with new set of projects that fits the business
strategy and capacity constraints.
•
The aggregate project plan provides a framework for addressing this
difficult task.
Furthermore, their study identified the project types as shown below and in figure 3.1.
Project Types
•
Derivative projects.
•
Breakthrough projects.
•
Platform projects.
•
R&D projects.
•
Alliances and partnerships.
‘Success’ is a complex concept (Hart, 1996). Extant studies have defined success in
different ways, measures of success include profitability, market share, staying on
schedule, technical success. Further differences exist over the level at which the analysis
should be performed, that is should it be performed at the project, business or corporate
level. Ernst, (2002) concludes that at the project level previous research shows ‘that the
existence of a formal New Product Development (NPD) process, which is
comprehensive and characterized by professionalism throughout the process …, has a
positive effect on the success of new products’. A variety of models for the NPD process
are presented within management literature. Trott (2002) cites Saren (1984) who divides
the models into seven distinct categories:
•
Departmental stage models.
•
Activity stage models and concurrent engineering.
64
R&D
Projects
More
Product Change
New Core Next Generation
Product
Product
Addition to
Product Family
Less
Derivatives &
Enhancements
Breakthrough
Projects
Platform Projects
Derivative
Projects
Process Change
R&D
Projects
Breakthrough
Projects
Platform Projects
Derivative
Projects
Figure 3.1 Project Types.
Source: Ozataly, 1992.
64
•
Cross-functional models (teams).
•
Decision stage models.
•
Conversion process models.
•
Response models.
•
Network models.
R&D Project Evaluation
Background
Over the last five years, a number of research organizations, such as National Scientific
Foundation (NSF, 1996) have updated their evaluation systems, criterion and Merit
Review System. In the State of Kuwait, KISR, KFAS, EPA, and KU all reconsidered
their method of evaluation. Until recently, KISR took two steps in order to evaluate a
project, first through their Technical Review Meeting (TRM) and the Management
Review Meeting (MRM). They have now consolidated the process into one, dubbed a
Proposal Review Meeting (PRM).
Evaluation is the analysis and assessment of goals, instruments and impacts (Georghiou
and Meyer-Krahmer, 1992).
The focus of evaluation essentially depends on the underlying theoretical approach;
fundamental approaches for the impact evaluation of R&D programs are (Matt and Petit,
2001):
Inputs
Black Box
Outputs
1. Evaluation of INPUTS/ OUTPUTS (Neoclassical approach).
Inputs: R&D budgets, staff, investments (indicators).
64
Outputs: publications, patents, artifacts, market shares…etc.
2. Evaluation of HOW INPUTS PRODUCE OUTPUTS (Monography).
History, analysis of decision processes, interactions between actors…etc.
3. Evaluation of the EFFECTS ON the Black Box (Evolutionary approach).
Creation of knowledge, technical options, networks, change in the
organizational structures, new modes of co-ordination.
The effectiveness, however, of the various approaches to the evaluation of research has
not been critically assessed (OECD, 1997). Regardless to the R&D activities of the
organization, the evaluation system typically includes the use of quantitative indicators
and/or the use of qualitative peer judgment. Clearly, there are many factors which shape
the way in which an R&D project is evaluated such as type of R&D organization, level of
evaluation, and S&T concerned in specific areas.
Evaluation and Portfolio Analysis (Souder, 1980) is usually the most expensive and timeconsuming model to use. It determines the best way to allocate the available resources (or
budget) among all the alternative projects.
Sometimes the result may indicate that a new project is so good that one or more old
projects are terminated. Alternatively, it could result in the old projects becoming
temporarily halted or backlogged in favour of the new project. Or the new project may be
even rejected. Backlogged projects will normally be retrieved at some later date, yet
rejected projects will not. However, new information or changed circumstances may
suddenly make a previously rejected project more attractive, or may cause a previously
backlogged project to be rejected.
The most increasing attention mainly in the 1990’s, or the ‘evaluation development’, are
closely linked to various circumstances such as deep economic recession, cuts in R&D
funding and the strengthening of new market-oriented policy of competitiveness.
Therefore, research evaluation is gradually becoming a legitimate tool to re-allocate
resources so that the research community itself is more or less involved in the process.
64
On the other hand, different potential interests become apparent when the analysis is
thought at the background factors and objectives of doing research evaluation. These
objectives are not usually pronounced in the public and they may even serve conflicting
interests.
Traditionally, however, evaluation of R&D project by peers has always been an inherent
part of scientific activity. The new collective and institutional trend of evaluation has
resulted from external influences and science policy aspirations. The overall aim has been
to redirect the limited resources, and even decrease research resources on the basis of
science policy objectives and the demands on accountability in science. The evaluation
development has raised active public discussion, which has mainly concentrated on the
technical implementation of evaluation and the indicators used in ranking R&D projects
disciplines.
Another issue concerned with evaluation is the increasing interaction between S&T and
society accompanied by financial constraints in research funding (National Academy of
Engineering, 1993) have placed unprecedented demands on R&D organizations to prove
their legitimacy and contribution to socio-economic development and better systematic
methods and approaches for decision making.
These demands are frequently mediated and formulated through S&T policy guidelines
and priority settings. The pressure towards accountability and efficiency are reflected in
the recent ‘evaluation development’ that includes research proposals, projects, and
activities, which have emphasized the need to develop new methods of institutional and
external evaluations more effective than the traditional ways of self- evaluation by the
scientific community. To this end, it may be argued that the conceptual basis, adequacy
and compatibility of different evaluation practices and criteria are the most complex issue
arising in this regard.
64
The Importance of R&D Project Evaluation
R&D project evaluation is significantly important and is being considered as an essential
issue in many international R&D organizations. Evaluation could not be accounted as an
end in itself. Rather, it should be continuously developed and up-graded and utilized as a
pointer to key policy issues and treated as a major question that need to be addressed.
R&D project evaluation becomes useful to the extent that it helps in clarifying policy and
decision debates and moves the right selection of R&D projects and evaluation process
forward on more rational and quantifiable grounds. In addition, evaluation should be
compatible and used as a primary tool for managing different levels of research activities
and systems, rather than only as a strict method and instrument of assessment and
judgement.
The evaluation should also provide the basis of better R&D project screening and
selection by exploring problems and reaching useful recommendations. These
considerations will also help the evaluators not only to replace evaluation output process,
but will strengthen the final right decision about which projects are to be selected and
implemented.
There is therefore a need for multiple approaches to evaluation analysis and refining
where both positive features and negative deficiencies could be highlighted. In order to
improve the existing evaluation systems, limits of the available approaches should also be
clearly recognized.
The management and the practice of R&D have become more complex, as they involve
many parties and have a wide range of, often interrelated, technological, market and
organizational options to choose from under constrained conditions. At the same time, it
has been observed that senior management's attention to R&D is rapidly increasing
(Wood, 1998), holding R&D accountable for directly contributing to business results not
just in the long term but also in the short term (Gupta and Wilemon, 1996). Together, the
complexity, the growing importance of effective and efficient R&D for company success,
64
and the increased pressure on R&D to become accountable for its actual contribution to
company success, have aroused a need to implement performance measurement and
control tools in R&D. This need becomes even clearer when one observes that in several
best-practice studies (e.g. Cooper and Kleinschmidt, 1995; Griffin, 1997; Pittiglio, et al.
1995) 'measuring performance' was found to be one of the discriminating factors between
'the best' and the rest, and that on average the capabilities of R&D groups in this area are
low (Gupta and Wilemon, 1996). Shareholders are also demanding more information
about the precise contribution of R&D to corporate performance; industrial investors, in
particular, have made it quite clear that they cannot attribute a proper value to a
company's reported R&D expenditure unless information is also provided that indicates
its effectiveness (Nixon, 1998). Fortunately, over recent decades general control and
measurement theories have matured and have gradually been elaborated for R&D
environment, resulting in a growing stream of publications on this subject.
The term 'fifth generation' R&D management (Amidon Rogers, 1996) endeavours to
capture the changes in R&D management since 1991, most notably the move from crossfunctional NPD teams to collaborative, networked groups in virtual organizations, from
R&D portfolios to integrated technology platforms, from accelerated product
development to seamless innovation through concept to customer and from technology
creation to technology selection (Chiesa and Manzini, 1998; Iansiti and West, 1997;
Klein, 1998; Lewis, 1998; Robb, 1991).
Therefore, selecting the right R&D projects through a well-designed evaluation model
and practice that includes the appropriate evaluation and selection criteria. Implementing
the right tailored model of evaluation will lead the R&D organizations to increase their
R&D research quality and their R&D performance measures and indicators.
Taking into considerations the local level and status of different regions and countries, it
is inevitable that a tailored evaluation system is needed to reduce the negative
deficiencies and enhance the positive features.
64
Evaluation has frequently been viewed as an adversarial process. Its main use has been to
provide ‘thumbs-up’ or ‘thumbs down’ about a program or project (EHR/NSF, 1993).
The purpose of R&D project evaluation and selection (Twiss, 1987; Martino, 1995) is to:
•
Ensure the fit of projects with organization's strategies and the
requirements of their operations;
•
Identify successful projects as early as possible and focus the scarce
resources on them;
•
Maximize the value of projects for an organization; and
•
Balance the set of development projects and new products.
Types of Project Evaluation
Evaluations should be conducted for action-related reasons, and the information provided
facilitate deciding a course of action.
Within National Scientific Foundation (NSF), there are two basic levels of evaluation:
Program Evaluation and Project Evaluation. Project evaluation is sometimes further
subdivided into specific project components as shown in figure 3.2.
A program is a coordinated approach to exploring a specific area related to NSF’s
mission of strengthening science, mathematics, engineering, and technology.
A project is a particular investigative or developmental activity funded by that program.
Thus, a program consists of a collection of projects that seek to meet a defined set of
goals and objectives.
A program evaluation determines the value of this collection of projects. It looks across
projects, examining the utility of the activities and strategies employed, in light of
64
PROGRAM
PROJECT
Component
Component
PROJECT
Component
PROJECT
Component
Component
Component
Figure 3.2 Project Evaluation Components.
Source: EHR/ NSF, 1993.
64
the initial policy goal. In contrast, project evaluation focuses on an individual project
executed under the umbrella of the program.
The evaluation provides information to improve the project as it develops and progresses.
Information is collected to help determine whether it is proceeding as planned; whether it
is meeting its stated program goals and project objectives according to the proposed
timeline. Frequently these evaluation findings are also used to assess whether the
particular project merits continued funding as it is currently operating, or if it needs
modifications.
An evaluation of a component frequently looks to see the extent to which its goals have
been met, and to clarify the extent to which the component contributes to the success or
failure of the overall project.
There are three general types of evaluation studies:
1. Planning Evaluation;
2. Formative Evaluation; and
3.
Summative Evaluation (EHR/NSF, 1993).
Planning Evaluation
The purpose of planning evaluation is to assess understanding of the project's goals,
objectives, and timelines. In fact, most project proposals typically mention only
‘Formative’ and ‘Summative’ Evaluation, defining these as activities to be performed
once a project has been designed, written up, and funded. The evaluator enters the scene
after the project has been put in place. A strong argument can be made for a different
approach.
Rossi and Freeman (1993) argue strongly for the involvement of evaluators in diagnosing
and defining the condition that a given project is designed to address, in stating clearly
64
and precisely the goals of the project, and in reviewing the proposed procedures for
accuracy of information and soundness of methods.
The planning evaluation is typically designed to address questions, such as:
•
Why was the project developed? What is the problem or need it is
attempting to address?
•
Who are the stakeholders (those who have credibility, power, or other
capital involved in the project)? Who are the people interested in the
project and who may not be involved?
•
What do the stakeholders want to know? What questions are most
important to which stakeholders? What questions are secondary in
importance? Where do their concerns coincide? Where are the conflicts?
•
Who are the participants to be served?
•
What are the activities and strategies that will address the problem or need
which was identified? What is the intervention? How will participants
benefit? What are the expected outcomes?
•
Where will the program be located (organizational level, geographical
area)?
•
How many months of the calendar year will the program operate? When
will the program begin and end?
•
How much does it cost? What is the budget for the program?
•
What human, material, and institutional resources are needed?
•
How much is needed for evaluation? How much is needed for
dissemination?
These questions can become a checklist to determine if all relevant elements are included
in the description of the project or program. These questions also provide the basis for the
formative and summative evaluation inquiries about the project.
64
Formative Evaluation
The purpose of formative evaluation is to assess on-going project activities. Formative
evaluation begins at project start-up and continues throughout the life of the project. Its
intent is to provide information to improve the project. According to evaluation theorist
Bob Stake; formative evaluation, when contrasted with summative evaluation, is:
“When the cook tastes the soup, that’s formative; when the guests taste the soup, that’s
summative” (EHR/ NSF, 1993).
For most NSF projects, formative evaluation consists of two segments: Implementation
Evaluation and Progress Evaluation.
Implementation Evaluation
The purpose of implementation evaluation is to assess whether the project is being
conducted as planned. Before evaluating the outcomes of a project, it must be sure that
the project is really operating according to its plan or description.
Implementation evaluation collects information to determine if the program or project is
being delivered as planned. A series of implementation questions is needed to guide the
implementation evaluation. Examples of these questions are:
•
Were the appropriate participants selected and involved in the planned
activities?
•
Do the activities and strategies match those described in the plan?
If not, are the changes in activities justified and described?
•
Were the appropriate staff members hired, trained, and are they working in
accordance with the proposed plan? Were the appropriate materials and
equipment obtained?
•
Were activities conducted according to the proposed timeline?
64
•
Were activities conducted by appropriate personnel?
•
Was a management plan developed and followed?
Progress Evaluation
The purpose of progress evaluation is to assess progress in meeting the project's goals.
Progress evaluation is also formative. It involves collecting information to learn whether
or not the benchmarks of participant's progress were attained and to point out unexpected
developments.
Progress evaluation collects information to determine what impact of the activities and
strategies is on the participants at various stages of the intervention. If the data collected
as part of the progress evaluation fail to show expected changes, this information can be
used to 'fine-tune' or terminate the project. The data collected as part of a progress
evaluation can also contribute to, or form the basis for, a summative evaluation study
conducted at some further date. In a progress evaluation, the following questions could
be asked:
•
Are the participants moving toward the anticipated goals of the project or
program?
•
Which of the activities and strategies are aiding the participants to move
toward the goals?
Summative Evaluation
The purpose of a summative evaluation is to assess the project’s success. Summative
evaluation takes place after ultimate modifications are being made, after the project is
stabilized and after the impact of the project has had a chance to be realized. (Another
term frequently used interchangeably with ‘Summative Evaluation’ is ‘Impact
Evaluation’).
64
Summative evaluation answers these basic questions:
•
Was the project successful? What were its strengths and weakness?
•
To what extent did the project or program meet the overall goal(s)?
•
Did the participants benefit from the project? In what ways?
•
What components were the most effective?
•
Were the results worth the project’s cost?
•
Is this project replicable and transportable?
Summative evaluation collects information about processes and outcomes.
The evaluation is an external appraisal of worth, value or merit. Usually this type of
evaluation is needed for decision making. The decision alternatives may include the
following: disseminate the intervention to other sites or agencies; continue funding;
increase the funding; continue on probationary status; or discontinue.
R&D Project Selection
Background
The term ‘project selection’ has many different meanings (Souder, 1975). In its narrowest
sense, project selection means determining which project will be approved. In its broadest
sense, project selection may be viewed as a sequential portfolio determination process. In
this broader view, the decision maker wishes to determine the best allocation of the
available resources among the alternative projects (Souder, 1980).
Each project can be funded at several different levels, with each level yielding a different
contribution to the individual or organizational goals.
R&D project selection becomes even more important. Choosing the right projects can
mean the difference between remaining competitive and falling behind (Martino, 1995).
64
Interest in project selection methods, among both academic scholars and industrial R&D
managers, is no longer a matter of minor importance. It has become literally a matter of
survival for an industry.
The selection of projects from a list of proposals for inclusion in the R&D program is a
matter of ranking them in order of benefit to the organization. The criteria for doing so
will be specific to the organization, but usually includes a financial measurement, such as
return on investment (ROI), as well as non-financial criteria.
The selection of the project(s) is a complex task. A project must meet some specific
market needs, be capable of being handled efficiently by concerned departments in the
organization, be a ‘good fit’ with the organization's objectives and optimize the use of the
available resources (Twiss, 1982). Due to the problematic characteristics and complexity
of R&D, the application of the selection criteria in the assessment of the organization’s
R&D process is very challenging (Bellary and Murphy, 1999; Boyer, 1991; Fisher and
Heywood, 1992; Kiella and Golhar, 1997; Lovett, 1992; Patino, 1997).
The first requirement is to have a large number of potential projects from which to select
and then to decide on those, which will produce the best returns on the resources
available. The most efficient way of utilizing R&D resources is to place the proposals in
order of priority and then to allocate funds to them, in that order, all the resources they
can usefully employ until most allocated resources are committed.
Research projects are high leverage types of projects, in the sense that their outputs or
lack of outputs can have a dramatic impact on the organization’s future. For instance, a
successful project can lead to profits that are many times larger than its original
development costs, while an unsuccessful project can mushroom into equally dramatic
losses (Cooper, 1993).
64
Thus, in the case of research projects, it is especially important to select only the very
best projects, and to identify and screen out inferior candidates before committing
significant resources to them (Souder, 1984).
Methods of Project Selection
A large number of techniques and models have been developed to aid managers;
examples are Screening Models, Evaluation Models, and Portfolio Models. The choice of
one type of method over another will depend on the nature of the projects being assessed
and the decision problem at hand.
A variety of outcomes may occur as a result of each kind of decision, as illustrated in
figure 3.3. As time passes, these three types of decisions may be repeated many times in
response to changing information states, changes in the available resources and funds,
changes in project achievements or the arrival of new project proposals. If a project is
evaluated within an R&D institute, it is vital and important to have the decision of the
end user, i.e. beneficiaries, and investors. This process undoubtedly will strengthen the
final decision and improve the quality of the proposed project.
Screening Models
Screening Models are very useful for weeding out those projects, which are the least
desirable. Since screening models are quick and inexpensive to use, they can economize
on the total evaluation efforts by reducing the number of projects to be further evaluated.
Because they require a relatively small amount of input data, they can be used where the
projects are not well understood or where there is a minimum of data available on them.
However screening models do not provide much depth of information, and they usually
are not sensitive to many of the finer distinctions between the projects. Rather, screening
models are like a coarse sieve that provides a partial separation but permits some
undesirables to pass through. Thus, screening models can be useful for some applications.
64
New Project
Proposals
Screening
Model
Backlog of
Project Proposals.
Awaiting Action
Evaluation
Model
Rejected
Proposal
On-going Projects
(Portfolio)
Terminated
Projects
Portfolio
Analysis
Model
Completed
Projects
Figure 3.3 Illustration of a Project Selection Decision Process.
Source: Souder, 1980.
64
But the decision maker should not expect them to provide a comprehensive or complete
analysis.
“Screening is a term which applies to different concepts of evaluating new products,
depending on which writer, researcher, consultant or company is using the term.” (Baker
and Hart, 1999). The evaluation of an NPD occurs at a number of points throughout the
NPD process, the difficulty with the concept of screening is that different authors use the
term to describe evaluation at different points within the development process. Cooper
(2001) applies the term screening to each gate throughout the NPD process. Other authors
reserve the term screening for the initial idea of evaluation (Rochford, 1991; Baker and
Hart, 1999).
Four types of screening models are available: profile models, checklists, scoring models,
and frontier models.
Profile Models
Table 3.1 shows an example of a profile model. The ratings are qualitative in nature. No
numerical assessments are made.
Rather, the project proposals are compared on the basis of a subjective evaluation of their
attributes.
These evaluations could be done by one individual or by group consensus. Alternatively,
the profiles developed by several informed individuals could be compared. Profile
models are simple and easy to use. They display the project characteristics and ratings in
such a way that are easily communicated and readily visualized (Souder, 1978a).
Checklists
Table 3.2 shows an example of a checklist. This type of model assumes that the decision
maker can distinguish between several finite levels of the criteria or requirements. Each
64
Table 3.1 Example of a Profile Model.
EXTENT TO WHICH PROJECTS
X AND Y MEET THE CRITERIA
CRITERIA OR
REQUIREMENTS
HIGH
Reliability
X
Maintainability
Y
Safety
MEDIUM
Y
X
X
Cost-Effectiveness
X
Durability
X
LOW
Y
Y
Y
X = project X’s score.
Y = project Y’s score.
Source: Souder, 1980.
64
Table 3.2 Example of a Checklist.
CRITERION SCORES*
CRITERIA OR
REQUIREMENT
TOTAL
SCORE
-2
-1
0
+1
+2
+5
Project X
Reliability………………………………………………………………………x.
Maintainability………………………………………………… ...x…… ……….
Safety………………………………………………………… ….x.……………..
Cost- Effectiveness……………………………………………………………..x.
Durability…………………………………………………………….… x……….
-2
Project Y
Reliability………………………………… ………….x…………………………..
Maintainability……………………………………………..……………………x.
Safety…………………………………………………… …x………..……………
Cost- Effectiveness……………………………………………… .x………………
Durability……………………………………………………x……………………..
+5
Project Z
Reliability………………………………………………………………………..x..
Maintainability…………………………………………………………………..x..
Safety……… ……………………………………… .x.………………………....
Cost- Effectiveness……………………………………………………….x….……
Durability…………………………………………………………………x….…...
* Scoring Scale:
+2 = Best possible performance.
+1 = Above average performance.
0 = Average performance.
-1 = Below average performance.
-2 = Worst possible performance.
Source: Souder, 1980.
64
candidate proposal or project is then subjectively evaluated by the decision maker and
assigned a criteria score on each requirement. The criterion score is ascertained from a
predestinated scoring scale that translates subjective evaluations into numerical score. A
total score is obtained for each project by summing its criterion scores (Souder, 1980;
Souder and Mandakovic, 1986). In general, for a checklist model:
Tj = ∑ sij ,
i
Where Tj is the total score for the ith project and sij is the score for project j on the ith
requirement or criterion.
Scoring Models
It is a short step from Checklist Models to Scoring Models. In a scoring model, each of j
= 1,…,n candidate projects are scored on each of i = 1,…, m performance requirements
or criteria. The criterion scores sij for each project are then combined with their respective
criterion importance weights wi to achieve a total score Tj for each project. Projects may
then be ranked according to their Tj values (Moore and Baker, 1969). For example, a
simple additive scoring model would be:
Tj = ∑ wi sij ,
i
Where sij is the score for project j on the ith criterion, and wi is the criterion weight. This
model is illustrated in table 3.3.
Frontier Models
Frontier Models are often very useful for examining return/risk trade-offs within
organizational objectives. Frontier models are used to indicate the need for greater
diversification in idea generation and project proposals. For example, figure 3.4 shows
that the acceptable projects are primarily of the medium to high-risk variety. Whether or
64
Table 3.3 Example of an Additive Scoring Model.
CRITERION
CRITERION, i
Project X:
Reliability
Maintainability
Safety
Cost-Effectiveness
Durability
WEIGHT, wi
CRITERION
X
SCORE,* s ij
4
2
3
5
1
5
3
3
5
4
WEIGHTED
=
SCORE
20
6
9
25
4
T1 = 64
Project Y:
Reliability
Maintainability
Safety
Cost-Effectiveness
Durability
4
2
3
5
1
1
5
2
3
2
4
10
6
15
2
T2 = 37
Project Z:
Reliability
Maintainability
Safety
Cost-Effectiveness
Durability
4
2
3
5
1
5
5
2
4
4
20
10
6
20
4
T3 = 60
* Scale: 5 = Excellent,…,Poor = 1.
Source: Souder, 1980.
64
Maximum
Risk Desired
Xi = the ith project
X4
Efficient Frontier
X7
X3
X6
Return
X5
X2
Minimum
Return
Desired
X1
Risk
Figure 3.4 Illustration of a Frontier Model.
Source: Souder, 1980.
64
not the portfolio ought to be more diversified must be resolved on the basis of the
organization’s goals and objectives. The frontier model can only point out trends and
situations for further analysis (Markowitz, 1960; Sharpe, 1963).
Evaluation Models
Evaluation Models require a greater volume and depth of input data than screening
models. But in return, they provide a more comprehensive and accurate analysis.
Some evaluation models require finite numbers for life cycle sales volumes, probabilities
of success and other parameters that may be very difficult to estimate.
Thus, by using the evaluation model as a guideline, the decision maker will be prompted
to more carefully search and analyze the proper information. In many cases, using an
evaluation model with only approximate data and rough estimates can be revealing and
helpful to the decision maker. There are four types of evaluation models: Economic index
models, decision theory models, risk analysis models, and value-contribution models.
Economic Index Models
An Economic Index Model is simply a ratio between two variables, and the index is their
quotient. Changing the values of the variables changes the value of their quotient, or the
index. An example of a commonly used index model is the return on investment (ROI)
index model:
ROI Index = ∑ (Ri / [1+r]i ) / ∑ (Ii / [1+r]i ) ,
i
i
Where Ri is the net returns expected from the project in the ith year, Ii is the investment
expected to be made in the ith year, and r is an interest rate (Souder, 1978b). Some
examples of index models are shown in table 3.4.
64
Table 3.4 Examples of Index Models.
Ansoff’s Index
rdp (T + B) E
Project Figure of Merit =
Total Investment
Olsen’s Index
Project Value Index =
rdbSPn
Project Cost
Viller’s Index
Project Index = rdp (E – R / Total Investment)
Disman’s Index
Project Return = rp (E - R)
Key: r = the probability o f research success, d = the probability of development success, p = the
probability of market success, T and B are respective indexes of technical and business merit, E
= the present worth of all future earnings from the project, S = annual sales volume in units, P =
unit profit, n = number of years of product life, R = present worth cost of research and
development activities to complete the project.
Source: Souder, 1980.
64
Economic models are the most popular project selection tool. Cooper, et al. (2001) found
that 77.3% of businesses use financial approaches in portfolio management and project
selection, Liberatore and Titus (1983) obtained a similar result, 80%, in their study of
fortune 500 firms’ project decision making. Yet it is generally agreed that at the idea
evaluation stage, economic models are not the most effective project selection tool
(Cooper, et al. 2001; Baker and Hart, 1999; Dingle, 1997).
Decision Theory Models
Decision Theory Models are based on the economics and strategies approaches to
statistical decision making. These models are based on the idea that a rational decision
maker will only adopt those policies, which maximize the expected value of the outcomes
(Souder, 1980). An illustration of a decision theory model is shown in table 3.5 for two
candidate projects. Decision theory models are attractive because they help to clarify the
available strategies and the potential risks, regrets the trade-offs. They have thus been
found to be particularly useful for interdepartmental design, where natural differences in
the risk propensities of the departments may get in the way of consensus. These models
have been used to help clarify and display these differences in such a way that the parties
are more able to work out an acceptable joint decision (Souder, 1975a, 1975b, and 1977).
Risk Analysis Models
A Risk Analysis Model provides a complete picture of the distribution of outcomes for
each alternative project. An illustration of a risk analysis approach to the comparison of
two candidate projects is shown in figure 3.5. Project one has a most likely lifetime profit
of US$100 million while the second project has a most likely lifetime profit of US$150
million.
However, there is only a .4 probability that the second project will in fact achieve the
US$150 million level, while there is a.8 probability that project one will achieve the
US$100 million level (Hertz, 1964; Souder and Mandakovic, 1986).
64
Table 3.5 Illustration of a Decision Theory Model.
PAYOFF MATRIX OF
RELATIVE WORTHS ($000,000)
MOTHER NATURE IS:
ADVERSE (A)
BENIGN (B)
Project 1
100
300
Project 2
50
500
Source: Souder, 1980.
64
Probability of Achieving the Anticipated Lifetime
Profits
1
Project 1
0.8
0.6
Project 2
0.4
0.2
0
0
50
Dow nside Risk
Difference Area
100
150
200
Aniticipated Lifetime Profits
$(000,000)
Figure 3.5 Illustration of Risk Analysis Model.
Source: Souder, 1980.
64
Value Contribution Models (V-C)
Value Contribution Models are a combination of several methods. An example is given in
table 3.6. Value contribution models permit the decision maker to examine the degree of
contribution, which a project makes to the organization’s hierarchy of goals. Thus, to
develop a V-C model, one must first list the goals (Souder, 1980). Several techniques
may be used for this. In most cases, the goals may be listed as a ‘nested hierarchy’. For
instance, as shown in table 3.6, there are two super goals: short range and long range.
Within each of these two super goals, the organization desires to achieve new product
dominance, a profitability target and to reduce their present environmental impacts.
Within the long-range super goal, the organization desires to maintain their technological
state-of-art and market share. The second step in developing a V-C model is valueweighing the goals. The techniques presented in the ‘Scoring Models’ may be used for
this. In the model illustrated in table 3.6, the long range and short range supergoals are
respectively value-weighted as V-60 and V-40, where these values must sum to 100. That
is, the value-weights are determined by allocating a total of 100 points among the super
goals according to their relative importance.
Portfolio Models
Portfolio models attempt to find those ideas that ‘fit’ with the business strategy and
attempt to balance the product portfolio. They consider a businesses entire set of projects
rather than viewing new product projects in isolation, (Cooper, 2001).
Many portfolio models have been developed to handle a wide variety of decision
problems (Souder, 1978b; Rosen, 1965; Dean, 1970).
Though these models were
developed to find the one best portfolio in a particular situation; this is not the most
productive way to use them (Souder, 1973a and 1973b).
64
Table 3.6 Value-Contribution Model.
Table 3.6 Value-Contribution Model.*
SHORT RANGE
LONG RANGE
ORGANIZATIONAL GOALS
(V = 60)
PROJECT
COSTS
($000)
ACHIEVE
NEW
PRODUCT
DOMINANCE
(V = 30)
ACHIEVE THE
PROFIT
ABILITY
TARGET
(V = 20)
ORGANIZATIONAL GOALS
(V = 40)
REDUCE
ENVIRONMENTAL
IMPACTS
(V = 10)
MAINTAIN
THE TECHNOLOGICAL
STATE-OF-ART
(V = 25)
MAINTAIN
MARKET
SHARE
(V = 15)
TOTAL
VALUECONTRIBUTION
SCORE
Scores
Project A
Project B
Project C
$100
200
150
30
15
25
20
10
10
5
10
5
Normalized Value-Contribution:
Project A : 75÷$100,000 = $75.0 x 10-5
Project B : 65÷$200,000 = $32.5 x 10-5
Project C : 65÷$150,000 = $43.3 x 10-5
* V = the goal value-weight.
15
20
15
5
10
10
75
65
65
Rankings:
Project A : First
Project C : Second
Project B : Third
Source: Souder, 1980.
64
Rather, portfolio models are best used to help the decision maker discover new solution
alternatives, to examine the rational for funding some projects, or to answer “what if”
questions about various alternatives (Souder, 1967; Bobis, et al. 1971). In this way, a
portfolio model can be used as a management decision laboratory, to test various
alternatives and discover others.
Table 3.7 illustrates the use of a portfolio model when the objective is to determine the
best allocation of the available funds among alternative candidate projects.
For example, projects A, B and C each have four alternative funding levels: US$0,
US$100,000, US$200,000 and US$300,000. The expected profits from the projects vary
with these funding levels, as shown in table 3.7. The higher funding levels result in
improved products, which yield higher expected profits.
Several alternative allocations of the available $300,000 are possible. For instance, the
funds can all be allocated to project C, for an expected profit return of $350 million. In
this case, the other two project would be zeroed-out- no money would be spent on them.
The available funds could also be spread evenly across the three projects. This would
yield an expected profit return of US$100 million + US$120 million + US$10 million =
US$230 million. This is inferior to the above alternative of funding only project C at its
upper limit.
Continued searching will show that the optimum allocation is to fund project A at its
US$100,000 level, project B at its US$200,000 level, and to zero-out project C.
This portfolio yields the largest possible total expected profits, as shown in table 3.7.
There is no other allocation of the available funds that will achieve a higher total
expected profit.
64
Table 3.7 Illustration of a Portfolio Model.
AVAILABLE FUNDS = $300,000
ALTERNATIVE
FUNDING LEVELS
EXPECTED PROFITS ($M)
FOR EACH PROJECT
PROJECT A
$ 0
100,000
200,000
300,000
PROJECT B
$
0
100
250
310
Optimum Portfolio
Project A
Project B
$100,000
200,000
$300,000
PROJECT C
$
0
120
285
335
$ 0
10
215
350
Expected Profit
$100M
285M
$385M
Source: Souder, 1980.
64
Evaluation and Selection Criteria
Supporting the vast amount of literature regarding methods of NPD selection is the many
recommended selection criteria. Broadly speaking, the recommended criteria have
changed little since O’Meara (1961) suggested four screening criteria Marketability,
Durability, Productive Ability and Potential. Cooper and Brentani (1984) from their
study of industrial product firms identified four similar broad criteria used by managers,
Magnitude of Product Opportunity, Market Opportunity, Synergy and Product Strategy.
Carbonell-Foulquie, et al. (2003) add a fifth criteria customer acceptance providing a
customer perspective within the criteria. These criteria correlate strongly with the success
factors identified from NPD methods and processes.
Most of R&D organizations evaluate and judge proposals on a competitive basis mostly
by peer review.
Proposals are screened to determine responsiveness to the specific requirements. For
example, the first step taken by NSF is the administrative screening. In this step, NSF
reviews proposals to determine that they satisfy all administrative requirements (NSF,
1997). Proposers are advised that failure to satisfy any one of administrative requirements
will be declined. Non responsive proposals will be returned to the proposer without
further consideration.
The second step taken by NSF is the technical screening. In this step, technical screening
criteria are applied to proposals. If the answer to any of the questions below is ‘no’, the
proposal will be returned to the proposer without further consideration.
•
Does the proposal provide sufficient technical substance to enable review?
•
Does the proposal fall within the scope of the topic / subtopic as
delineated in the topic/subtopic description?
•
Is appropriate research proposed in science, engineering or education?
64
When an award or declination is made, verbatim copies of reviews, excluding the names
of the reviewers, summaries of review panel deliberations, if any, a description of the
process by which the proposal was reviewed, and the context of the decision (such as the
number of proposals and award recommendations, and information about budget
availability) are mailed to the principal investigator. The company officer/organization
representative is also notified, but that individual only receives information as to whether
an award or a declination was made.
In order to proceed with the research program, further action must convert the research
results into innovative, competitive technology. A senior management team is obviously
a necessary requirement for this part of the process. The commercialization plan and the
follow-on funding commitment play a key role in NSF’s evaluation.
Therefore, NSF is considering certain criteria to be applied in the case of funding
commitment and commercialization plan as follow:
•
The company’s financial plan for commercialization.
•
The company’s planned commercialization activities.
•
What does the innovation provide to the nation in the form of
competitive advantages?
•
How
realistic
is
the
marketing
plan
to
impact
the
market
within five years of completion of the project.
•
Has the small company mapped out a strategic plan for maintaining its
intellectual property?
•
Has the company fully investigated a sound method of producing
and bringing the innovation to market?
•
Has the small company successfully converted government supported
research projects into commercial products?
The proposal will also be returned if it is principally for demonstration, technical
assistance, literature survey or market research. Proposals that are found to be responsive
64
administratively and technically are competitively evaluated in a process of external
merit review by scientists, engineers or educators knowledgeable in the appropriate fields
and by individuals familiar with commercial product development.
Proposal Review Process
Most reviewers are employed by universities or by the federal government. Others may
be employees of non-profit research laboratories, recent retirees from industrial firms,
and, on occasion, employees of industrial organizations, including small business
concerns. In all instances, proposals are handled on a confidential basis and care taken to
avoid conflicts of interest. Evaluations will be confidential to NSF, to the proposed
principal investigator.
Normally, more proposals will be found technically meritorious than can be supported.
Evaluations by external reviewers are advisory to the cognizant program officer for the
topic or subtopic, who makes recommendations on each proposal. Other factors that may
enter into consideration include the following: the balance among NSF programs; past
commercialization efforts by the firm, where previous awards exist; excessive
concentration of awards in one firm or with one principal investigator; participation by
women-owned and socially and economically disadvantaged small business concerns;
distribution of awards across the states; importance to science or society; and critical
technology areas.
In the merit review process, reviewers will consider two criteria:
Intellectual Merit
The intellectual merit of the proposal activity addresses the overall quality of the
proposed activity to advance science and engineering through research and education, it
is considered with the following issues:
64
•
How important is the proposed innovation to advancing scientific and
engineering knowledge and understanding?
•
Is the proposed plan a sound approach for establishing technical and
commercial feasibility?
•
To what extent does the proposal suggest and explore unique or
ingenious concepts or applications?
•
What will be the significant contribution of the project to the research
and knowledge base of the field?
•
Is there sufficient access to resources (equipment, facilities, etc.)?
•
How well qualified is the team (the principal investigator, other key
staff, consultants, and sub awardees) to conduct the proposed activity?
Broader Impacts
The broader impacts of the proposed activity addresses the overall impact of the proposed
activity, it is considered with the following issues:
•
What is the potential use of the product or process developed in the
project to advance NSF goals in research and education?
•
What may be the commercial and societal benefits of the proposed
activity?
•
Does the proposal lead to enabling technologies (instrumentation,
software, etc.) for further discoveries?
•
How well does the proposed activity broaden the participation of under
represented groups (e.g. gender, ethnicity, disability, geography, etc.)?
•
Does the outcome of the proposed activity lead to a marketable product
or process?
•
Evaluate the competitive advantage of this technology vs. alternate
technologies that can meet the market needs.
•
How well is the proposed activity positioned to attract further funding?
64
Views of Evaluation Methods in S&T Organizations in Kuwait
Background
Kuwait S&T entities had given a great deal of interest and attention to the evaluation
methods for their R&D projects. Unfortunately, almost all the attention has been given to
the evaluation of the project prior to approval. This means that no serious consideration
had been given to projects within other status of its review cycle. Moreover, the
evaluation of the under consideration projects is dealt with individual projects only and
not as a set of competing projects. Therefore, the proceeding subsections will glance on
the evaluation methods that S&T entities in Kuwait are practicing. This section will
highlight the current practice in order to enhance and analyze the existing status and
broaden the research tasks for further academic investigations in the field of R&D project
management evaluation and assessment.
Kuwait Foundation for the Advancement of Sciences
The Kuwait Foundation for the Advancement of Sciences (KFAS) was established in
1976 by HH the late Amir of the State of Kuwait, Sheikh Jaber Al-Ahmad Al-Sabah.
KFAS is fulfilling the Amir’s vision of a private, non-profit making entity that supports
Kuwait’s social and economic development through science.
KFAS sponsors research into many of Kuwait’s most important national issues. Research
topics include: oil and petrochemicals, health sciences, water resources, fisheries and
mariculture, the environment, social sciences and the aftermath of the Iraqi invasion.
While some grants support basic science, the program emphasizes existing problems and
current issues in the State of Kuwait.
The Research Projects Directorate administers the Foundation’s grant-award program.
Awards are made to Kuwait’s research institutions, government ministries, industrial and
commercial companies, and a wide array of public service organizations.
64
A number of projects are collaborative efforts among Kuwait’s academic, government,
and private sectors. Most awards are extended to Kuwaiti institutions, but limited support
is provided for collaborative work at the regional and international levels.
The grant is competitive and involves an internal review by the foundation and peer
reviews by Kuwaiti researchers and researchers at prestigious universities and institutions
overseas.
KFAS evaluates the R&D projects through either individual evaluations or peer
reviewers. The main evaluation criteria are:
Scientific Worth
Is the project new or, does it overlap an existing or past project?
Alternatively, does it help conserve beneficial programs that might otherwise
atrophy or be lost? What is the probability of success?
Potential Application
Is there a demonstrable need for the project on a priority basis?
Methodology
Does the plan clearly identify and describe the different phases of the project
and are the methodology and analysis techniques suitable?
Peer recommendations to streamline methodology would give added value to
end-project results.
Specifics on Feasibility
Which areas are weak and how can they be better addressed to enhance the
potentials, objectives and end-results of the project?
Commitment of the Proposers
Has sufficient time been devoted to planning the project and is it one of the
project team's primary interests or a major professional pre-occupation?
64
Relevance of the Proposal to Broader Concerns
Will the project produce significant changes in a wide circle?
Will the results of the project be transferable to other projects?
Overseas Scientific Input & Monitoring
Is the project staff in contact with international peers?
Will contact be maintained to monitor progress of the project?
Training Component
Does the project include plans for training young Kuwaitis and also potential
users of the research results?
Soundness of the Budget
Is it justified? Is it adequate, but not so generous as to be wasteful?
After reviewing and evaluating the proposal, the evaluators fill and submit reviewer’s
report.
Environment Public Authority
The Environment Public Authority (EPA) is responsible for addressing a number of
different environmental issues. Not only for limiting the environmental pollution, but also
to safeguard the different natural resources and administer the same with more rational
way to achieve sustainable development and involve the environmental elements in the
planning structures in order to test the environmental options and environmental impacts.
The EPA has formulated its own evaluation system consisting of several application
forms to be completed by the requesters. The main perspectives that are included in the
evaluation forms address:
•
Research value and importance.
•
Proposal preparation and presentation.
64
•
Research background.
•
Research methodology.
•
Research relevant on the socio-economic.
•
Research relevant on the environmental problem.
•
Expected research output.
•
The researcher abilities and background.
•
Budget, research and timing of the research.
Kuwait University
Kuwait University (KU) was established in 1966, setting its main goal to promote higher
education in the State of Kuwait. Further to this goal, the teaching staff had conducted
several scientific and technical papers mainly for career development and promotions.
More recently, the need for basic and applied research has increased and that has led the
KU to establish a dedicated organizational unit (Research Administration (RA)) to
broaden the scope of contractual research. To cope with the new development, the RA
established an evaluation system for submitting research proposals in order to facilitate
the selection and approach of the submitted proposals. The RA promotes the scientific
and technological concerns of the university by supporting programs to advance research
investigations of the faculties, and to apply technologies in response to the needs of the
society.
A Research Support System (RSS) has also been established to administer university
resources for these purposes. Research activities are supported as projects by the
mechanism of Grants and Contracts competitively awarded to individual faculty members
on the basis of technical proposals evaluated by peer review.
In general, the evaluation perspectives include the followings:
•
To what extent were the proposed objectives met for the evaluation period?
•
Has the project been executed according to the proposed research plan?
64
•
Is the project generating publications at an acceptable rate?
•
Has the teaching and research capability of the university been advanced
as a result of this project?
•
Has a basis been established for continuing research support in this field?
•
Does this project help in generating knowledge in Kuwait and in the region?
Review of Evaluation Process in Kuwait Organizations
After reviewing several evaluation systems of S&T organizations in Kuwait, it seems that
many criteria are implemented in each of these organizations in order to fit their own
goals and objectives.
In general, the new era of R&D management required well-defined criteria that could
enhance the research output and added values.
As a preliminary finding of this review, R&D projects within the State of Kuwait should
satisfy main criteria that include:
•
National, social, economic and developmental needs.
•
Relevance to the strategic goals and objectives of the organization.
•
Possibility of external funding by beneficiaries.
•
Possibility of high chances of the commercialization of the results on
acceptable returns.
•
Possibility to achieve center of excellence in certain research areas.
Among all of these, the evaluation system has to satisfy the needs of the R&D
organization as well as the needs of the beneficiary sectors, where the system should
enhance the chance of transferring the R&D results to the public and private sectors, in a
way, to promote their business development and to invest in R&D for long term returns
and advantages.
64
Organization Problems towards R&D Project Evaluation and Selection
Project evaluation and selection decision makers frequently have much less information
to evaluate candidate projects than they would wish. Uncertainties often surround the
success likelihood of a project, the ultimate market value of the project and its total cost
to completion. This lack of an adequate information base often leads to another difficulty:
the lack of a systematic approach to project evaluation and selection. Consensus criteria
and methods for assessing each candidate project against these criteria are essential for
rational decision making. Though most organizations have established organizational
goals and objectives, these are usually not detailed enough to be used as criteria for
project selection decision making. However, they are an essential starting point. Project
evaluation and selection decisions are often confounded by several behavioral and
organizational factors. Departmental loyalties, conflicts in desires, differences in
perspectives, and an unwillingness to openly share information can stymie the project
evaluation and selection process.
Much project evaluation data and information is necessarily subjective in nature. Thus,
the willingness of the parties to openly share and put trust in each other's opinions
becomes an important factor (Souder, 1984).
The rapidly changing environment often brings about a chaotic picture of innovation
management. Too many new challenges to innovation management and project selection
processes have arisen, including flexibility (Cooper, 1994), ability to respond to changes
quickly (Miller and Morris, 1999), and a more explicit management of uncertainties
(Mullins and Sutherland, 1998).
Views on Integrated Evaluation Model
The management of a research project from inception to completion is a complex
process, monitoring the project from the concept phase, preparation of the research
64
proposal, evaluation and selection process, approval, reporting and finally to the
completion stage.
To follow-up on a research project, a systematic integrated model is needed. This
suggested model has to deal with the project in the whole review cycle of the project life.
This means that several sub-phases should be built and be integrated in a way that
evaluation criteria should be in a harmony with the project progress.
Therefore, the first phase should deal with proposals under consideration. The second
phase should deal with a set of approved proposals. The third phase should deal with ongoing projects. The last phase should deal with completed projects. It is argued that this
approach encourages the conclusion to be taken in a well-defined decision support system
for the R&D project management. Archer and Ghasemzadeh (1999) have simplified the
project portfolio selection process by developing an integrated framework, which
separates the project selection into distinct stages. Each stage accomplishes a particular
objective and creates inputs to the next stage. This framework has not introduced any
evaluation techniques to be used in each stage.
The integrated model will reduce the degree of uncertainty concerning resources
utilization and the successful completion of R&D activities. The main focus of the
integrated model (Ellis, 1997) could be based on a multi-criteria management support
model that illustrates all the essential aspects exists in the R&D project management,
taking into account the effects of related factors, such as manpower, budget,
timing...beside S&T policy, objectives and others. Ellis stated that innovation
management needs to be able to use measures to control the R&D management process
not only for effectiveness but also efficiency. He argued that measures, not just in
financial terms, but also in qualitative terms.
Therefore, the model needs to posses good project management procedures and
regulations that will ease its implementation and facilitate its utilization.
64
Literature Limitations
From the previous literature review in this chapter, it is been found that the available
literature proved the lack of detailed overall integrated study addressing the R&D project
evaluation and selection techniques to be utilized in research centers mainly in the GCC
area, where this region is specified by its own local economic, political, and social
environment.
It is also observed in the literature that most of the R&D project evaluation and selection
methods are concerned only with the first stage of the R&D project, where the project is
evaluated in its proposal form. Therefore, in this research the aim is to overcome these
limitations and to broaden the methods of the evaluation to include all other project
phases in the proposed model.
Finally, as this research is implemented in one of the GCC countries, which is Kuwait, it
will investigate the unique status of the GCC countries in relation to R&D project
evaluation to overcome the literature constraints and limitations that would be an obstacle
to utilize the available knowledge in the cited literature.
Technical Discussion
It could be concluded from this efforts that R&D management has faced several changes
in the 1990's, where there is no single ‘best’ way to supervise R&D activities. Each R&D
organization has its own unique environment and culture to meet the continuous changes.
In 1950's the R&D management was described as a common R&D management method
that deals with strategy of hope (Roussel, 1991). This strategy was linked only with the
right skilled staff and financial resources, and not linked with business requirements and
needs. Afterwards, the R&D started with another management style that deals with more
systematic practice and links with business needs.
64
This R&D management style recognizes the discrete, project nature of research and
development and seeks to quantify the cost and benefits of individual projects and to
monitor progress against project objectives.
Recently the R&D management philosophy trends are specified by its interaction with the
business sectors and end users when they could implement successfully the R&D results
and outputs in a creative way (Steele, 1991). Upon these factors, evaluation and selection
of research projects should go in parallel with the new R&D management paradigm.
It is noted that most R&D evaluation process deals with the old R&D management
mainly the project management approach. Sometimes it is not clear what criteria should
be used in an evaluation and evaluators need to carefully articulate the basis for their
analysis. Evaluation does not always find a single ‘right’ answer to complex questions for
consideration by decision makers. The implementation of an evaluation system requires
criteria and priorities to be established. The choice of who is to carry out the evaluation is
significant, where peer review is used; considerable skill is required in blending the
required characteristics of objectivity and sufficient knowledge of the subject and its
practitioners.
To evaluate the projects, there are various criteria to be considered, such as: determining
the appropriate standards or criteria, against which to compare a program's performance,
manpower, budget, duration, and others.
The proposed project should conform to the strategic goals and the concerned division
objectives. Then, the research will investigate on three main propositions.
The evaluation mechanism is significant to many companies, R&D institutions, and all
concerned S&T entities who are dealing with R&D activities. These entities are: R&D
organizations, educational and academic institutions, funding firms, national industries
and public sectors (R&D beneficiaries).
64
One of the most important and difficult issues facing today’s organizations is the problem
of innovation. Only innovation increases the size of the pie, which means that only
innovation leads to improve standards of living. Innovation is many things, it is
inspiration and creation; it is renewal; it is ambiguity and the tension of change in the
learning process; and, at its best, it is also a journey of discovery (Miller and Morris,
1999).
Baker and Hart (1999) cite Von Hippel’s typology of innovation, he suggests three broad
categories:
1. Known need - the new product is instantly recognizable as ‘just what I’ve
always wanted’.
2. Customer active or need pull - frequently the end result of an approach by
a user to a prospective supplier and the outcome of joint development
work.
3. Supplier active or technology push - conforms with the stereotype of the
lone inventor single-mindedly pursuing their goal, oblivious to the world
outside.
The world is always developing and changing, it is important to follow this advancement
by learning about the new techniques and models and how to implement the best methods
that cope with this continual development and innovation.
Summary
Although the literature addresses the issue of evaluation, there is little reference to the
integration of project evaluation and selection (Tassey, 1992), mainly if the subject
evaluation aspect has to deal with limited resources, scattered objectives, conflict of
various interests and decisions, rapid changes of technology, new concerns about
64
knowledge and technology management, economic based industries and knowledge
economy, etc.
According to several references, (Hellings, 1985; Pike, 1982; Lopes and Flavell, 1998),
project appraisal has systematically concentrated on establishing the financial and
technical viability of a project. Occasionally, non-financial aspects such as social,
environmental, political or legal factors are also deemed to be important and to be
assessed through some form of subjective judgment.
In order to achieve the research aims and objectives, care should be exercised to narrow
the existing literature gap by formulating an evaluation model that identifies and analyses
the different aspects of the R&D projects, the ways in which they can be assessed, their
relative importance to the success of the project, and how they can be incorporated in the
evaluation and selection procedure.
64
CHAPTER IV
RESEARCH DESIGN AND METHODOLOGY
Introduction
This chapter is the start point to the research design and methodology approach. The
research studies and propositions are initiated to pave the road for the aimed R&D
Evaluation Model which will be proposed, designed and tested in the proceeding chapters.
Thus, this chapter will outline the research methodology approach taken into consideration
the main factors that are related to the R&D project evaluation theme, such as integrated
evaluation models, S&T and decision making, projects management procedures and their
effects on R&D project evaluation.
Research Studies and Propositions
The focus of the R&D project management methodology is to provide a proper design,
development delivery, management and control of all projects. In addition, it is to control
the project's scope, providing timely delivery and assuring that the product quality meets
organizational requirements.
As discussed earlier, there are a number of techniques for managing and selecting R&D
proposals
and
projects.
The
two
main
types
of
methodologies
are
the
managerial/judgmental methodology and the mathematical/tangible methodology.
The research investigates on the first type to enhance the management skills and decision
making process without the need for complex mathematical programming models, as it is
considered not to be used by managers with different backgrounds and experiences.
The main elements of the research process are as follows:
•
Literature review.
•
Case study – based on 30 key staff interviews.
82
•
Analysis – to develop an integrated project evaluation conceptual model:
proposed model will be designed to improve the evaluation process of project
selection and implementation in the research area.
The research presented will therefore focus on three studies critical to effective R&D
project management and evaluation. These are:
•
R&D Project Review Cycle.
•
S&T Policies.
•
R&D Project Processing Management.
These three studies will lead to three propositions:
•
Analysis of Project Evaluation Systems.
•
Evaluation of R&D Projects in respect to S&T and Decision Making.
•
Improvement of R&D Project Management Procedures.
Figure 4.1 illustrates the research studies and propositions.
Research Design
R&D Project Review Cycle
The first priority in formulating a project is defining the goals and the objectives. Each
project goes through several steps and phases; figure 4.2 shows R&D projects processing
practice at KISR.
Following the idea, the project is evaluated by the divisional evaluators. Once approved,
the project will be designed and specified to execute the proposal. Following the proposal
stage, it will consider four main phases/issues as shown in the figure of Chapter II, namely:
83
MANAGEMENT MODEL FOR R&D
PROJECT EVALUATION
R&D Project Review Cycle
S&T Policies
R&D Project Processing Management
Analysis of Project Evaluation Systems
Evaluation of R&D Projects in respect to S&T and
Decision Making
Improvement of R&D Project Management
Procedures
R&D PROJECT EVALUATION
AND SELECTION MODEL
Figure 4.1 Research Studies and Propositions.
84
KISR
Initiated
PROPOSAL
PREPARATION
Client's
Request
PRM
NO
EVALUATION
Termination
YES
Approval
KISR/CLIENT
PROPOSAL
KISR Funded
Client Funded
APPROVAL PROPOSAL
DOCUMENT
PROJECT
IMPLEMENTATION
MONITORING &
FOLLOW -UP
FINAL REPORT
Figure 4.2 R&D Project Processing Practice in KISR.
85
•
Individual Project Evaluation – includes developing the research
proposal along with criteria and methodology to evaluate the proposed
project.
•
Portfolio Analysis – includes developing the criteria and methodology in
order to evaluate and select the best projects.
•
On-going Project – using a set of criteria to determine if a project should
continue or terminate.
•
Completed Project – focuses on the importance of the project results
implementation and commercialization.
Figure 4.3 illustrates the R&D project review cycle to be integrated in a systematic manner.
This part of the research will enable a better understanding of the evaluation system
through a case study of an R&D organization in Kuwait, i.e. KISR. The existing constraints
and milestones will be investigated, analyzed and the needed modifications and
opportunities will be sought to develop the system and to go online with institution overall
goals and objectives to meet a better output from the evaluation practice.
S&T Policies
In an effort to improve technology, R&D organizations in developing countries are striving
to increase their expenditure on R&D activities. It is also important to recognize that R&D
is one of the major functions in a continuously innovative chain linking scientific research,
market research, development design, first production and market acceptance. Hence, this
leads to a point that in order to ensure economic growth, a right decision must be taken
about a proper co-ordination of S&T and production, in order to enhance S&T application
to enhance economic growth through selecting the proper R&D projects, which obviously
could be gained through a proper R&D project evaluation system. Investment of resources
in S&T could bring substantial economic returns. Therefore, it must not be regarded as
isolated activities, but as components of a dynamic system to connect R&D projects into
new technology wealth and social amenities.
86
EVALUATION
PREPARATION
Individual
Project
Idea
Division
S&T Goals
&
Objectives
Commercialization
Evaluatio
Completed
Project
Criteria/
R i
Portfolio
Analysis
Project
Selection
Proposal
Implementation
On-going
Project
PROCESS
Figure 4.3 R&D Project Review Cycle.
87
82
The effective evaluation and selection of R&D projects requires the removal of all
obstacles of political, social and economic character and to be integrated into the overall
economic and social development plan. Evidence shows that the relation between R&D
expenditure and national economics is complex, and several countries have spent a great
deal of funds on R&D without achieving spectacular economic growth. Decision making
on the evaluation and selection of R&D projects has a great influence in this respect
(Qasem, 1995).
R&D Project Processing Management
R&D project management practice is a vital tool in an R&D organization.
A good
procedure will lead to a better utilization of the evaluation system, saves time and helps the
researchers and the clients to cooperate in presenting a well-defined proposal for
implementation.
Project management is as much about attitudes as it is about skills, techniques and
procedures. At its simplest, a project team is simply an ad hoc group brought together to
achieve a given task. It is likely to be supported by a number of tools and techniques as
well as follow-up and control procedures.
The temporary nature of projects is an exciting feature of the leaders and teams that work
on them, because projects are finite and measurable, success or failure is readily apparent.
Projects are also challenging, especially when focused on the organization itself in order to
improving processes, effecting organizational change, and forming the pathway between
evaluation method and the relevant implementation systems.
On the other hand, a good processing system will insure that follow-up actions are taken
and result in clear benefits on completion. The R&D processing system or procedures are
concerned with project management issues, such as managing the totality of projects in a
cohesive manner, allocating the required resources, such as financial funding and staff
assignment.
82
The question if a project selection is so important, what type of processing system should
be available to aid the evaluation system? This therefore led to the third proposition of the
research that deals with the improvement of the R&D project management procedures.
Research Methodology
The research presented will propose an evaluation system that is transparent in terms of
progress and responsibilities and parallel to the new era of R&D project management. It
will recommend strategies and guidelines for implementation.
•
The first task is to understand the project review cycle, selecting KISR
as a case study.
•
The second proposition will lead to integrate the S&T policy with the
R&D evaluation system to insure the desired decision making support
system regarding the evaluation and selection of the attractive projects.
•
The third proposition will lead to an exploration of the best management
practice to support and insure the proper utilization of the evaluation
system.
The upcoming table illustrates the research timetable.
Analysis of Project Evaluation Systems
Several programming and judgmental models have been developed to evaluate R&D
projects. Yet, there is little reference to an integrated system that takes into account the
overall project review cycle, including on-going and completed projects alongside
proposed projects. The research work will, therefore, investigate this area through a case
study approach. It will review the evaluation and approval system operated by KISR.
This part of the research will enable a better understanding of the evaluation system
through the life cycle of the project. A case study to be made on a selective proposals and
projects of an R&D organization in Kuwait, which is KISR. This will be proceed with
83
Research Timetable.
Phase
1
2
3
4
Activity
July – Oct – Jan – Apr – July – Oct –
Sept
Dec
Mar
Jun
Sept
Dec
2001
2001
2002
2002
2002
2002
Analysis
of
Project
Evaluation Systems
Review and analysis of
KISR material
Synthesize early findings
Fieldwork preparation
KISR's Staff interviews
Second
review
and
analysis of KISR material
Synthesize findings
Evaluation of R&D
Projects in respect to
S&T
and
Decision
Making
Formulate
‘new’
integrated
evaluation
model
Investigate of the ways in
which
S&T
goals,
objectives and policies
may be inter-linked with
the
proposed
R&D
evaluation system
Improvement of R&D
Project
Management
Procedures
Investigate the proposition
that a management system
that includes the powerful
policies, procedures, rules,
and regulations that could
facilitate the R&D project
evaluation
practice
effectively enhances the
outcome of the overall
system
Synthesize
conclusions
into
proposed
R&D
evaluation system
Write-up
84
proposing criteria of evaluation and selection of R&D projects by several interviews and
analyses of KISR's staff (Research Managers, Project Leaders and other key staff). This
task will identify the needed modifications on the existing evaluation systems to enhance
the development of the new integrated model.
The aim is to reason a systematic approach with interrelated criteria of evaluation
throughout the project review life cycle. The finding of this proposition would contribute to
the research output in the later stages of the research.
The output of this phase will result in a set of findings and conclusions that should enhance
the R&D activities and implement the completed project results for the benefit of the
development and economic returns mainly in the process of the enhancement of technology
transfer and technology commercialization. Figure 4.3 illustrates the R&D project review
cycle to be integrated in a systematic manner.
Evaluation of R&D Projects in respect to S&T and Decision Making
S&T policies have a great relevance and impact in the R&D project evaluation system. It
has many concerns regarding S&T strategies, co-ordination among S&T entities in
executing R&D projects, funding and resource allocation, results utilization and
commercialization. This part of the research will investigate the ways in which S&T goals,
objectives, and policies may be inter-linked with the proposed R&D evaluation system.
Linking a stringent evaluation system that could fit the S&T infrastructure will be a great
tool to ease the process of decision making and to correlate different factors effecting the
S&T development by implementing the right R&D projects that geared to the national
development. Developing countries R&D's institutions usually depend heavily on
government funding rather than a broad base of researchers competing nationally for their
R&D funding. The future outlook of government funding for R&D is dependent on
political and economic factors that mostly do not directly relate to research. Therefore, this
research is meant to contribute and emphasize on the decision making efforts of evaluation
and selection of R&D projects with respect to S&T policy issues that will broaden the
R&D aspects to the policy makers and opinion leaders as well as members of the research
community.
85
Thus, this task will concentrate on literature review analysis and views of concerned R&D
senior managers and researchers that will lead to gather relative information on factors
effecting the selection of the right criteria to build up the integrated evaluation model
within the S&T policy framework. This task will take six months.
Improvement of R&D Project Management Procedures
R&D project management practice is a vital tool in an R&D organization and it will
support the evaluation system, so are evaluation system should be implemented with a
proper management practice.
A good procedure will lead to a better utilization of the evaluation system, which takes less
time, helps the researchers and the clients to cooperate in presenting a well-defined
proposal for implementation.
The R&D project management is an integrated process between the organization and the
external organizations that have to implement the project findings. Clearly, there is a great
deal of give and take between these organizations and should be borne in mind in the
management practice. Certain vocabulary should be established among the different
participants concerned with the evaluation system itself. Such expression may include
certain terms and concepts, for example, resource allocation, project prioritization, funding
levels, risk, uncertainty, technological competitive strength, and so on. Furthermore, there
are many different players that have to participate in the project evaluation and selection
process.
On the other hand, a good processing system will ensure the follow-up actions to be taken
toward controlling the project during its life cycle and also to have a tangible benefit out of
its results after completion. The R&D project processing system is to be concerned with the
project management procedures such as directing resources, managing the totality of
projects in a cohesive manner, allocating the required resources such as funding money and
staff assignment. The question arises here is: if project selection is so important, what type
of processing system should be available to aid the evaluation system? Therefore, this led
to the third proposition of the research that deals with the improvement of the R&D project
management procedure to insure the harmony between the evaluation system and the
86
project management practice. This is sought to be achieved through trial efforts to integrate
both project evaluation and project management
To this end, even if a reliable evaluation system is developed, it could not be used in
isolation of a parallel effective procedural management system. Thus, this research will
investigate the proposition that a management system includes the powerful policies,
procedures, rules, and regulations that could facilitate the R&D project evaluation practice
effectively and enhance the outcome of the overall system.
Summary
The previous chapters have defined the evaluation and selection phenomenon of R&D
projects, setting out the importance and the relevant issues. It also addressed several
criterion regarding types and methods of project evaluation and selection.
Thus, the proposed research area will initially investigate the evaluation process,
considering the technical and management evaluation in order to determine whether the
applicant’s proposed project has technical merit, meets program objectives, and is realistic
in term of costs and timeframe.
The authors will establish an advanced system, which is easy to use and depends on welldefined evaluation criteria. It will include an up-to-date evaluation and selection process. It
will demonstrate a tailored evaluation model that matches the unique circumstances,
objectives and goals of developing countries.
In addition, the research presented will investigate the ways in which the evaluation and
selection process can be linked in a single integrated model throughout a project's life cycle
in respect to S&T policies, and by improving project management procedures and
regulations.
A case study will be developed taking KISR as an example of an R&D institution in the
region in order to facilitate the research requirements, such as data and information
collection, interviews and related analyses.
87
CHAPTER V
R&D PROJECT REVIEW CYCLE
Introduction
This chapter presents the first proposed research area that will take a closer look at the
project review cycle within some research organizations. The R&D project evaluation
efforts and activities of the Advanced Technology Program (ATP) Committee will be
presented and discussed. European Commission (EC) proposal preparation and
evaluation guidelines will be presented. Also this chapter will discuss in detail the project
review cycle in KISR in order to point out the current management practice systems and
procedures within the life cycle of the research projects.
Views on ATP's Efforts in R&D Activities
Project selection is the process of evaluating individual projects, to choose the right
project based on an analysis so that the objectives of the company will be achieved. It
involves a thorough analysis including the most important financial aspect to determine
the most optimum project among all the alternatives. Some projects have high
uncertainty, and therefore simulation based project selection decision analysis could
evaluate the projects with a greater confidence. The process of “project selection” is
normally based on a set of criteria determined by the organization, which may range from
purely economic project drivers to subjective issues, such as production, marketing,
personnel, administrative, etc. Decision factors and prioritization of potential projects are
often based on subjective uncertain criteria. A few project selection model attempts have
been presented, such as a graphical Monte-Carlo based probability assessment of the
decision influences (Arsham, 2002) to provide a “bridge” over this gap.
Edwin Mansfield prepared a background paper for the ATP. (ATP is a scientific
committee within the National Institute of Standards and Technology (NIST) in USA,
94
where its mission is to accelerate the development of innovative technologies for broad
national benefit through partnerships with the private sector).
It focused on estimating the social and private returns from innovations, preparatory to
performing a set of project case studies (Mansfield, 1996). Professor Adam Jaffe
prepared a background report, now in ATP review, on spillover effects (Jaff, 1996). The
report models spillover effects and identifies factors that influence whether a given
project is more or less likely to lead to large spillovers. ATP's staff has also published
general methodological pieces on economic evaluation (Ruegg, 1996).
ATP partners with U.S. businesses in high-risk research to develop enabling technologies
with strong potential for economic benefits to the nation. Universities, nonprofit research
laboratories, and firms of all sizes participate in ATP-funded projects. ATP's evaluation
effort seeks not only to measure the impacts of the technology development projects it
funds, but also to understand the underlying relationships among research, technological
change, and economic impact, and to provide feedback to the program to increase its
broad-based benefits. Evaluation activities encompass developing models, conducting
surveys, compiling databases, conducting micro- and macro-economic case studies, and
performing statistical and econometric analyses. Program metrics include private rates of
return, social rates of return, and public rates of return the social-rate-of-return
component attributable to the ATP. Topics of special interest, in addition to performance
metrics, include spillover pathways, benefits and costs of collaboration, financing issues,
and new models of impact assessment (Ruegg, 1997).
Therefore, the authors will explore in more depth the ATP’s project evaluation plan and
implementation in order to highlight the recent achievement in the era of project review
cycle, where other project evaluation practices are overviewed in their respected sections.
95
ATP's R&D Evaluation Process
Peer review of proposed projects against project selection criteria is applied up-front in
ATP's rigorous awards competitions (Ruegg, 1996). The goal is to select projects that are
likely to achieve the ATP's mission. It is sometimes overlooked that this step is a form of
project evaluation, because the assessment occurs ex ante, with the emphasis on
achieving the program's mission as opposed to measuring performance towards achieving
the mission. At this initial awards stage of the ATP project cycle, ‘expert’ evaluators are
charged with using their resident knowledge and judgment to evaluate proposals against
published selection criteria and to identify those that appear to have strong potential for
achieving a high ‘Q’ value as illustrated schematically in figure 5.1.
Scientists, engineers, business people, and economists agree to abide by non-disclosure
and avoidance of conflict-of-interest rules and sign statements to that effect. These
experts serve as peer reviewers.
The opportunity exists to feedback, as appropriate, the results of performance evaluation
to the peer-reviewers, with the objective of informing the initial project selection stage so
that future success conditions are enhanced. A step in this direction would be a more
explicit evaluation of the spillover potential of proposed projects by comparing project /
proposer attributes against an identified set of factors that are found likely to increase or
decrease spillover effects.
Real-time project monitoring is used by ATP to determine the progress of funded
projects currently underway against technical, business, and economic goals. This also
brings ATP staff face-to-face with the performing entities and provides the opportunity
for staff to become familiar with the projects, particularly the technical work. The project
monitoring staff are then in a better position to serve as informational sources for project
evaluation case studies.
96
Figure 5.1 The Three Dimensional Model of ATP Program.
Source: Spender, 1996.
97
Data collection and analysis is used by the ATP to track project progress, understand
the overall project portfolio statistically, assess results, and ultimately contribute to
measuring long-run outcomes. Data is sourced mainly from third-party surveys, ATP
special studies, and ATP's internally administered ‘business reporting system.’
Case studies of ATP-funded projects and groups of related projects have been performed
at various stages of the project's life cycle to capture progress to date, measure short-tomedium term impacts at the level of the firm, and, in some cases, provide the information
needed to interface with macroeconomic models for projecting national impacts.
Surveys of ATP project participants have been used to capture and report statistically
on short-run results -- particularly early business-related progress, as well as to gather
feedback from program participants on their satisfaction with working with the ATP.
Econometrics and other statistical analysis techniques are being used by the ATP to
shed light on underlying cause-and-effect relationships, such as spillover mechanisms,
and project impacts from the firm level across the entire economy using large-scale
macroeconomic models.
Modeling has been used to provide frameworks for better understanding and assessing
the program.
During the formative years of ATP, Professor Albert Link has carried out a number of
case studies focused on research efficiency in research joint ventures. These have since
been updated (Link, 1996a and 1996b).
The ATP is experimenting with combining microeconomic case study with the use of
macroeconomic models in order to develop national economic impact projections for
projects. Two studies are taking two different approaches to using the Regional
Economic Models, Inc. (REMI) (CONSAD, 1996; Robles, 1996).
98
A study based on participant interview data was conducted of ATP's impact on
accelerating technology development by cutting the time for starting projects and by
compressing research cycle time (Laidlaw, 1996). Cycle-time reduction is of keen
interest to the ATP because its authorizing legislation calls for it to accelerate R&D and
the subsequent commercialization of technologies.
The ATP wishes to improve and extend its evaluation capabilities, tools, and metrics. The
areas listed below are of special interest:
•
Extending impact assessment to include the long-run economic effects of
organizational and ‘cultural’ changes that may result from the ATP.
•
Assessment of ATP's impact on acceleration of technology development
and commercialization in different technologies and industry sectors.
•
Measurement of direct and indirect impacts of ATP-funded technology
development on productivity, output, jobs, and earnings.
•
Measurement of spillover impacts of ATP-funded projects, including
impacts on customers, competitors, and other firms inside and outside the
innovating industry sector.
•
Case studies of groups of interrelated ATP projects, including assessment
of possible synergistic effects among them, and the estimation of private,
social, and public rates of return.
•
Project failure analysis.
Although the ATP does not consider the above topics to be an exhaustive list, it is
intended to convey to the evaluation community a general idea of the current evaluation
interests of the ATP. It is expected that experts working in the area will likely suggest
research topics not explicitly referenced here that nevertheless might be of great value to
evaluating the ATP.
99
Proposals and Projects Preparation
Development of projects is organizationally complex though the extent, degree and
sources of complexity vary from one project to another (Smyly, 1993), and from country
to country. Complexity is often attributable to uncertainties and risks which are
unavoidable. Development appraisal techniques ultimately aim at forecasting the level of
uncertainty associated with development of projects. In all development projects, there
are uncertainties regarding costs, duration, objectives, and technology. These variations
are most vividly observed in developing countries which are typically dependent on
political stability, prices, availability of foreign exchange, general economic policies of
the government
and
perceptions
and
inspirations
of
developers and financial
institutions backing a particular project. All the foregoing variables are subject to
considerable uncertainty in all countries, but the degree of uncertainty is exacerbated by
the
ambiguities
of
a
turbulent
developing
country
environment.
A well-prepared proposal should be clear and concise. Although the format and style of
presentation of the contents may differ from one institute to another in the international
scientific community, the basic contents of an R&D proposal should generally include
the following (Ghosn, 1997):
•
Cover page.
•
Abstract.
•
Table of contents.
•
Introduction and literature review.
•
Justification and benefits.
•
Objectives.
•
Scope.
•
Methodology and work plan.
•
Expected outputs and applications.
•
Project requirements.
•
Organization plan.
100
•
Funding plan.
•
Training plan.
•
References.
•
Curriculum vitae (CV) of project leader and key staff.
•
Appendixes.
Desired Proposal Characteristics
It is expected that successful proposals of economic research to the ATP will exhibit the
following characteristics:
•
The proposed research will be directed by researchers with previously
demonstrated capability in undertaking economic research of the type
proposed, as evidenced by publications and citations in economics
periodicals and professionally refereed journals. Qualified graduate students
and other qualified researchers may be involved in a study.
•
The research will pertain directly to evaluation issues of concern to the ATP.
•
The researcher will bridge the gap between a purely theoretical or general
treatment of a topic and the practical implications of the research for the
ATP.
Desired Proposal Contents
Proposers of economic research are asked at a minimum to include the following
elements:
•
Executive summary explaining in lay terms the study objective and why it is
important (a) for the ATP, and (b) for the general field of economic
evaluation of technology. Other requirements are to explain what will likely
be learned and how the ATP is likely to benefit from it. In addition, this
101
section must indicate what new methods, tools, or data are likely to result,
and their significance to the field and the ATP.
•
Background synopsis of the relevant literature and description of current
state of the art or situation.
•
A description of the proposed evaluation research, indicating the
methodology, scope, data requirements, analytical techniques to be
employed, specific research tasks, hypothesis to be tested, and relevance to
the ATP.
•
Anticipated opportunities for future research based on the work proposed.
•
Schedule, outputs, and tentative costs, showing a breakdown by categories.
•
Résumés of principal researchers, including publications, education, and
work experience.
•
Attachments of particularly relevant supporting materials, at the proposer's
discretion.
Criteria against which Evaluation Proposals will be Judged
Proposals will be judged by the Source Evaluation Board (SEB) against the following
criteria:
•
Merits of the proposed objectives and soundness of approach.
•
Relevancy to ATP's evaluation concerns.
•
Qualifications of the researchers, and
•
Cost and timelines.
Minimum Requirements for Contractors
Contractors of economic research for the ATP will at a minimum be subject to the
following requirements:
102
•
Adherence to terms regarding time, cost, and milestones established in the
contract/ task order.
•
Delivery of draft reports according to schedule.
•
Delivery of final report according to schedule, subject to approval by the
ATP, and, if task-ordered through the National Bureau of Economic
Research (NBER), by the president of the NBER or NBER's ATP Project
Director.
Requirements for Evaluation Reports
Typically, researchers should aim to submit several drafts of their reports in sequence,
with a feedback loop for comments and revision, prior to the final report submittal.
Reports would generally be expected to include the following elements:
•
Abstract.
•
Keywords list.
•
Executive summary explaining the study objective and its importance,
providing a brief overview of the research approach, a summary of the
principal findings listed preferably in bullet style, and a brief explanation
of any important limitations or caveats of which the reader should be
aware.
•
Problem statement and background information.
•
Methodology section.
•
Discussion of data and assumptions if used.
•
For some studies, a review of the literature.
•
Detailed results section.
•
Summary and conclusions.
•
References/bibliography.
•
Tables and exhibits if used.
103
Researchers are usually asked to submit both a hard copy of their report and an electronic
file copy. Figures and tables should either be integrated electronically in the text or hard
copy should be provided.
Proposal Preparation and Evaluation Guidelines
EC's staff will verify that proposals meet eligibility criteria referred to in the call for
proposals. These criteria will be rigorously applied and any proposal found to be
ineligible will be excluded from evaluation (EC, 2003).
An eligibility checklist will be filled out for each proposal on the basis of the information
contained in the proposal form. If it becomes clear during or after the evaluation phase,
that one or more of the eligibility criteria have not been fulfilled by a proposal, it will be
declared ineligible and withdrawn from any further examination.
The following criteria will be checked for all proposals. Only proposals that fulfill all of
these criteria will be retained for evaluation:
•
Date of dispatch of electronic validation file or sealed electronic proposal
before deadline for dispatching;
•
Date of reception of proposal on or before deadline for reception;
•
For proposals submitted electronically, agreement between the unique
identifier code sent with the validation file and that calculated from the
proposal file;
•
Signature of coordinating legal entity (or appropriate electronic
‘signature’);
•
Signatures of the partners who would contribute to the funding of a
project (i.e. potential contractors, assistant contractors and members) or a
signed declaration of the proposal coordinator that he/she is authorized to
send the proposal and that the proposal is agreed to by the partners;
104
•
Minimum number of eligible, independent partners, as referred to in the
call for proposals;
•
Completeness of the proposal, i.e. the presence of all relevant
administrative forms and the proposal description (the completeness of
the information contained in the proposal description will be for the
expert evaluator to judge; the eligibility checks only apply to the presence
of the appropriate parts of the proposal).
All proposals that fulfill the eligibility criteria will be evaluated to determine their
quality. In order to advise the commission in evaluating proposals, programs will enlist
the support of independent external experts. As a general rule, a minimum of three
independent experts will examine each eligible proposal submitted to the commission.
In general, experts will be expected to have appropriate competence in the areas of
activities; they must also possess a high level of professional experience in the public or
private sector. Experts must also maintain appropriate language skills within the area of
research. Each application for selection to serve as an expert comprises a form
incorporating a CV and appropriate keywords, on the basis of which the selection will be
made. The list of experts to be used for any particular evaluation session will be decided
by the relevant director(s). Experts participating in the evaluation will be required to sign
a contract with the commission, binding them to confidentiality and impartiality
regarding the proposals that they examine.
A number of evaluation criteria are common to all programs; each eligible proposal will
be examined against these criteria by the independent experts.
For the detailed examination of proposals against the criteria set out in the rules for
participation, the expert will generally provide marks and comments. In addition, the
experts will be asked to examine certain evaluation criteria by answering a set of
questions relevant to the specifications referred to in the call.
105
The following questions will be addressed at an appropriate point in the evaluation:
•
Does the proposal address the parts of the work program, including policy
issues, open for the particular call? If the proposal is only partially in line
with the call, does it have sufficient merit to be considered in its entirety
or partially?
•
Have relevant ethical issues been adequately taken into account in the
preparation of the proposal; is the proposed research compliant with
fundamental ethical principles, if relevant? Is the research proposed in line
with community policies, if relevant; have appropriate safeguards/ impacts
assessment regarding community policies (e.g. environment) been taken
into account, where necessary?
•
Does the proposal follow the requirements for preparation?
If the answer to any of the above questions is negative, the experts will be required to
provide comments with blocks of evaluation criteria to justify their answers. They will
then be required to examine the proposals assigned to them individually, filling in an
individual evaluation sheet with their comments and marks.
Each evaluation criterion will be marked by the experts on a scale from 0 to 5. In this
scheme, the scores indicate the following with respect to the criterion under examination:
0- The proposal fails to address the issue under examination or can not be
judged against the criterion due to missing or incomplete information.
1- Poor.
2- Fair.
3- Good.
4- Very good.
5- Excellent.
106
In addition, taking into account their marks for each of the individual evaluation criteria,
experts will give a mark from 0 to 5 to each of the blocks of criteria. Only these marks for
each block of criteria will be taken into account for the final mark for the proposal.
Following the examination of the individual proposals by the experts and the preparation
of evaluation summary reports by the panels, it will be the task of the experts to examine
and compare the reports of those proposals which pass the various thresholds, if any in
order to check on the consistency of the marks applied by the individual panels and,
where necessary, have these corrected. In addition, those proposals receiving the same
marks following the initial examination will be re-examined with a view to placing them
in priority order, if possible and relevant (EC, 2003).
Engineering and Physical Sciences Research Council (EPSRC) proposals will be
evaluated according to the following criteria:
•
Quality of proposal and work plan – the extent to which the proposal
addresses the issues and demands outlined in the call, and shows
innovation as appropriate; the quality of the proposal will be assessed on
the basis of the deliverables identified and the evidence provided of how
these will be achieved (30%).
•
Impact – the extent to which the project outcomes will be of overall value
to the FE / HE and e-Science communities; included in the assessment
under this criterion will be the need for sustainability of the work at the
end of the project funding period (30%).
•
Partnership and dissemination – the degree to which the proposal
demonstrates an openness and willingness to work in partnership with
JISC in forward planning, dissemination and evaluation, and the potential
for extended partnership beyond the funding period (10%).
•
Value for money – the value of the expected project outcomes vis-à-vis
the level of funding requested, taking into account the level of innovation,
chance of success and relevance to the target communities (15%).
107
•
Previous experience of the project team – evidence of the project team's
understanding of the technical and/or management issues involved, and of
its ability to manage and deliver a successful project, for example through
work done to date in the area or in related fields (15%).
The Council's Research Evaluation Committee (REC) is responsible for assessing the
successful achievement of the Economic and Social Research Council's (ESRC) strategy.
The REC conducts reviews of key areas of ESRC policy and research strategy and
evaluates individual projects, research programs and research centers. The backbone of
evaluation is peer review and almost all of our evaluation work involves selecting,
briefing,
supporting
and
in
other
ways
dealing
with
peer
reviewers.
The Evaluation Team within the Communications and Information Directorate is
responsible for the management of this activity within the office.
The three main purposes of evaluation are:
•
To provide an assessment of accountability (i.e. whether public funds were
spent as agreed).
•
To assess whether the project has been conducted effectively, whether it
has met its objectives and to make an early assessment of the quality and
impact of the research.
•
To provide award holders with some feedback about the management,
quality and rigor of the research, and to provide comments on user or
potential uses of the research.
The report should be given one of four grades:
•
Outstanding.
•
Good.
•
Problematic.
•
Unacceptable.
108
An Outstanding grade indicates that a project has fully met its objectives or has provided
an exceptional research contribution well above average or very high in relation to the
level of award.
A Good grade indicates a project whose research activities and contribution is fully
commensurate with the level of award, approach and subject area, and which has
addressed or successfully adapted its major objectives.
A Problematic grade indicates a project which has failed to address one or more of its
major objectives, has encountered significant difficulties in its execution, has incomplete
work, or has achieved substantially less than expected for the level of the award, the
approach or the subject area.
An Unacceptable grade indicates a project which has failed to conduct the work as
agreed at the time of the award (and any subsequent agreed changes to the work plan); for
example, failure to conduct agreed surveys or analyses, or failure to address most of the
major objectives.
Project Review Cycle in KISR
Despite the specialist nature of the work undertaken by KISR, it remains essentially a
service business, which shares a need for management systems and procedures.
Given that the main ‘product’ of the organization is the creation and successful
management of research projects.
The overall management efficiency of the institute is inadequate with an excessive
administrative overhead, slow management processes and a distinct lack of ‘hands-on
management’ (as opposed to administration). The various support divisions provide little
109
obvious support to the research divisions, and, if anything, actually restrict their progress
rather than facilitate it.
Scientific staff appears to be spending too much of their time on non-research activities.
It is very easy in a scientific environment to promote excellent scientists into managerial
roles where they then become hampered by bureaucracy and their valuable scientific
knowledge is lost under a sea of paper. It is vital that bureaucracy is minimized at this
level and that key members of staff have appropriate administrative support so that they
may manage actual research, rather than mundane bureaucratic functions. Good scientists
do not necessarily make good administrators; in fact the reverse is often the case. The
situation is particularly noticeable at KISR, where a heavy bureaucratic burden falls on
all levels of scientific supervision and management.
There are in general five key stages in the life of a project:
•
‘Lead’ generation - the initial client inquiry or contact that eventually
should lead to a proposal;
•
Proposal; the specification and pricing of project bids;
•
Project management;
•
Project delivery and reporting;
•
Post-project follow-up.
Lead System
No formal lead system currently appears to exist at KISR. As leads provide the basis for
proposals and eventually for the mainstay of the business, projects, it is essential that they
will be tracked. If such a system were introduced, regular analysis (in conjunction with
proposal records and project management) would show:
110
•
The number of leads necessary (and their associated value) to generate
sufficient proposals and projects to achieve plan. For example, if an
income from new projects of KD 5m is required to achieve the annual plan
and the success rate of proposals is 1 in 4 issued, then on average
proposals to the value of 4 x KD 5m will be needed each year. If the
success rate of proposals to leads is found to be 1 in 10, then leads to the
value of 10 x 5m will be needed.
This basic monitoring system, if
reviewed on a monthly or weekly basis, can therefore provide early
warnings about income predictions.
•
The success rates with particular customers, type and size of customer,
sectors, and project topics.
•
If KISR expands to international markets, the success of these versus the
home market. This would also allow the overseas investment to be
monitored against direct and indirect investment costs.
•
Average time taken between the various project-bidding stages –
important for planning and monitoring purposes.
•
Success or otherwise of those responsible for generating business.
A lead system could act as one of the principal sources of information for discussion at the
regular meetings of senior managers. It is also important that information such as this,
along with project ‘wins’ and the general financial health of the institute are made more
widely known to all staff. This will engender a team spirit and common cause.
Proposals
Proposals at KISR are in general prepared by the member of staff responsible for the key
area concerned, in conjunction with his/her colleagues. Each proposal is then reviewed by
senior management before issue.
It would seem that there is a disproportionate amount of time spent on scrutiny of project
proposals, particularly in light of the fact that proposal success is most likely to be
111
determined by the client rather than KISR senior management. The approval process as
currently constituted takes too long and seems to be unnecessarily complicated.
It
appears that the system is reasonably effective in ensuring that projects comply with
KISR’s general strategic objectives, but this should be possible within a much-reduced
time frame and with a far less involved procedure.
Issues to be considered in generally improving the standard of proposals include:
•
A clear specification of the house style and mandatory content of
proposals, ideally supported by a library of different examples. There
could be scope for different types/complexity of proposal depending on
the type of work being bid for. For example, it would be a waste of time
to issue a lengthy, ‘full’ proposal for a simple data gathering exercise. In
this case a short letter may suffice.
•
Training or ‘work shopping’ on the preparation and content of proposals
will help to produce more consistent quality.
•
Similarly, as mentioned earlier, training for appropriate staff in general
presentation techniques and proposals in particular would be beneficial.
•
A set of KISR’s standard conditions should accompany all proposals.
Although some clients place contracts, which incorporate their own
conditions, some do not. The necessary disclaimers need to be present to
cover the work done. Where clients impose their own conditions, these
should always be formally checked and approved by someone in legal
authority at the institute as they could involve substantial penalties if they
should ever be invoked.
•
To safeguard against potential liabilities, the principle that a limited
number of staff be authorized to sign proposals and accept contracts on
behalf of KISR should continue. In doing so it is however vital that the
system does not cause a ‘bottleneck’ in issuing the documents as at
present. The argument that many of the clients also have approval delays
112
is not an acceptable defense. KISR should be setting an example to others
in the professional conduct of business.
•
The progress of proposals issued and accepted should be monitored on a
regular basis for the purposes of planning described earlier.
•
KISR should ensure that central and working files should be established
for all projects. Central files should house contractual material, such as
letters relating to contractual obligations, invoice copies, proposal
acceptance, reports, minutes of key meetings and visit reports. Working
files should contain copies of the essential project documents, such as the
proposal, along with working papers and correspondence. All central files
should be kept in the office of Marketing and Project Management
(OMPM), the working files with the principal members of staff concerned.
•
A project should not be activated (no time bookings accepted) until a letter
of formal authorization, or a contract, is received from the client and
accepted by the legally responsible staff at KISR.
Management of Projects
•
Proposals will have specified hours/days needed, costs for the work and
associated expenses, invoicing, key stage dates and a final completion date.
It is important that the project management system subsequently uses this
data to provide feedback on the progress of the work and the staff who are
undertaking it.
•
Based on this data, the system can then provide important information such
as work in progress over time, order book value according to key work areas
or for the organization as a whole, staff utilization and organizational
efficiency (hours sold versus hours employed).
•
Project monitoring will also give an early warning when projects are under
or over spent against plan. Similarly, projects, which are running late at key
stages or on final delivery, will also be highlighted.
113
•
In managing projects it is important that contracted projects linked closely
with clients in order to ensure that they are satisfied with the progress of the
work.
•
At the end of a project, a quality check should be made on the client’s view
of the project, its conduct and outcome, staff concerned and any general
views on the organization. A project completion report should then be
prepared and discussed with the staff. These reports may then be used as
part of the organization's staff appraisal system and perhaps to generate
publicity in the form of case study material or editorial comment.
•
A final financial analysis of costs and expenses on the project will show
profitability levels against both notional and actual costs.
There appears to be little in the way of monitoring client satisfaction as a project
progresses and no contact with senior management in this respect. An important role of
senior managers in an applied research and consultancy environment is to act as ‘Project
Directors’, in other words, the interface between the client and the organization.
Obstacles which limit the ability of the research projects to run smoothly need to be
removed, wherever possible by giving higher priority to overcoming problems as soon as
they become apparent, or ideally to sense issues before they become real problems.
Project Delivery and Reporting
KISR conducts projects, services, and activities under the project management system.
Project management will ensure that the research is conducted according to the research
policies in KISR from the initiation to the completion. During the initiation,
implementation, and completion of research, the research team has to show the progress
of the work and the results in the form of deliverables/ reports. Deliverables/reports are
distributed to internal (KISR) and external parties according to the distribution list.
114
The distribution list shows the type of the deliverable/report, name of receiver, and
number of copies. It is designed to suit classified types of reports generated in KISR and
for their clients. Each type of deliverables has different type of distribution list,
classification, and numbers according to KISR’s policies.
Types of Deliverables/Reports: Proposal, Amendment, Extension, Progress, Interim,
Technical, and Final.
Types of Classification: Confidential, Restricted, and General.
Parties: are the one who receives the deliverables and reports whether internally (KISR)
or externally.
Proposal: is a detailed plan of proposed work, includes research and management plan.
Amendment/Extension: are the changes in approved plan, whether in budget or
duration, etc.
Progress/Interim Report: report, which provides information on progress and technical
accomplishments for a particular period of time.
Technical Report: is a report containing information on scientific or technical aspect of
a project.
Final Report: is a report providing final results of a project.
Confidential/Restricted: Deliverables/Reports cannot disclose to third party without the
prior written consent of management and client.
General: Deliverables/Reports can be disclosed to third party.
Internal delays to report delivery need to be minimized or ideally removed, with targets
for internal approval of project reports reduced to days rather than months. In some
cases, clients had placed work outside Kuwait rather than suffer the slow progress
through KISR. However, parallel steps need to be taken to raise the quality of the final
reports, which were generally seen to be lacking in presentation and not always
appropriately targeted to the particular needs of the customer. Many of the clients
commented that reports were too academic and theoretical, prepared with meaningless
calculations and charts rather than solid recommendations based on experience.
115
At present, the report approval process is not as effective as it could be since it fails to
address presentational issues and cannot easily address issues relating to content.
Researchers need to be encouraged to take much more care in how they package and
present their reports to clients, and internal checks need to be streamlined to ensure that
deadlines are not compromised due to management delays. In effect they are currently
gearing their activities more to serve the internal system standards and requirements of
KISR than their clients. Whilst internal quality standards must be maintained, they
appear to be an imbalance in this respect.
It is important that reports are actually presented to clients wherever possible. This not
only helps to resolve any queries that clients may have about the content, but also gives
an opportunity to meet others within the company and perhaps to sell on other work.
Post-Project Follow-Up
At present, post-project follow-up is almost non-existent and this has a detrimental
impact on both internal development and future sales. Unless the institute has a welldeveloped understanding of how the research results have been utilized it is missing out
on perhaps the most significant mechanism by which it can raise the effectiveness of its
activities. Research is not done for the sake of the process; it is done so that some
economic, social or scientific benefits may result. It is important therefore that such
benefits are monitored and publicized, for the status of KISR, justification of investments
and staff morale. KISR needs to understand, on a systematic basis:
•
The client’s immediate reaction to the project results, including the
graphical and verbal presentations; and
•
What actions the client intends to take in light of the research results.
At some subsequent and appropriate point, KISR needs to re-contact the client in order to
understand the actual impacts of the project, ideally as part of a larger on-going impact
assessment study.
116
Better post-project follow-up will help KISR understand the effectiveness of its activities
and staff, which will, in turn, help subsequent marketing both to existing and new clients
as well as boost staff morale. Furthermore, where measurable impacts can be identified,
efforts should be made to quantify these and the resulting information used to justify the
on-going core funding of the institute.
New Development of Project Monitoring System at KISR
The main objective of this proposed monitoring system, entitled Project Management
Information System (PMIS), is to design and develop an application for KISR research
activities, to manage and track the documents follow-up in OMPM. More specifically,
the objectives are:
•
To define the needed requirements of comprehensive project management
information system for the Office of OMPM; and
•
To design a friendly useful interactive project management information
system for the OMPM and concerned organizational units within KISR.
The expected outputs of PMIS are as follows:
•
A complete set of input data fields, related to projects, services, general
activities, scientific papers, agreements, clients, etc.
•
All OMPM monthly, quarterly, semi annual and annual research activity
reports will be generated by the PMIS system.
•
General statistics and performance indicators reports.
•
Under consideration projects and client reports.
•
Follow-up on transactions of KISR’s research documents.
•
Access to the system by some KISR organizational units (read only).
117
This proposed system will enhance the overall project review cycle in KISR, and will
work online with the expected finding and contribution of this research.
Commercialization of KISR's Projects
Doyle (2000) identifies five determinants of market attractiveness, the size of the market,
market growth rate, the competitive structure of the market as determined using Porter's
five forces, the cyclicality of the market and risk factors.
There are currently three main markets for KISR activities; private companies, public
companies, and national agencies/ministries. Combinations of these are involved on
some projects, for example KFAS funded research for publicly owned companies.
There is a very little evidence of private company projects being conducted by KISR,
mainly because of the privatization of the economy is still at an early stage and those
companies who are in the market place are either too small or unsophisticated to need
KISR’s services. It is important, however, to consider the fact that there will be growth
in private sector companies, given the government's commitment to this aim. KISR
should begin to gear its services, and the ways in which they are delivered and paid for,
with private sector clients in mind.
Public agencies and publicly owned companies already provide a steady stream of longer
term funding but they are typically susceptible to political change in the content,
operation and funding of their programs. It is therefore essential that organizations such
as KISR are close to the decision makers in these agencies in order to have early warning
of any changes and, if possible, to steer programs in directions which will be beneficial to
both the industrial sector which will benefit from the work and KISR itself. Failure to
work in this way could not only jeopardize a substantial amount of income but also make
the organization appear to be passive and reactive, rather than dynamic and proactive.
Although publicly funded work is beneficial in terms of contacts and longer term funding
118
it can make organizations over dependent on what are rather low paid and time
consuming projects. These results in lower financial returns for the time employed and
staff being out of touch with commercial markets and marketing techniques.
It is
therefore important to maintain a focus on commercial approaches to undertaking such
work, even though the bulk of work currently available to KISR is from public sources.
There are a number of other constraining factors, which also hamper the ability of the
institute to successfully market proposals and obtain funding directly from project
customers. Each of these constraints is discussed briefly below:
•
Lack of S&T policy within Kuwait: The absence of any clear national
governmental policy in relation to S&T is a major constraint when seeking
long-term financial commitments from clients.
•
Lack of R&D budgets: The lack of a coordinated S&T policy reveals itself in
the absence of dedicated R&D budgets within the major governmental agencies
that form the bulk of KISR’s client base. In addition, there appears to be a
general disinterest in government circles on the role and importance of R&D,
reflected in the nation’s low expenditure on R&D as a proportion of GDP (0.3%
compared with international figures of 2 to 3%).
•
Limited S&T capability within the customer base: It is claimed that certain
customers do not have the necessary capability to absorb and utilize the outputs
of R&D projects conducted by KISR.
•
Lack of understanding of the potential benefits of R&D: There are also
claims that many customers do not understand the full potential of R&D’s
contribution towards the realization of social and economic benefits within
Kuwait. In some cases this is blamed on a lack of direct pressure within
governmental agencies to minimize costs and maximize operational
effectiveness.
119
•
The perception of R&D as a cost rather than investment: Many of KISR’s
customers view R&D as a cost rather than as an investment and fail to justify the
expenditure on the basis of potential or actual savings made. This situation could
reasonably be blamed on KISR itself since it fails to properly establish the
effectiveness and the impact of its activities, both at the proposal and report
stages.
•
The perception that the work conducted by KISR is expensive: There is a
widespread perception within the client base that KISR projects are expensive.
Key issues for KISR in future will be how it can work to change these poor public and
industrial perceptions of R&D, how R&D can be portrayed in sound economic and social
terms and the lobbying of government and associated agencies to put R&D firmly on the
national agenda. These are not problems which are unique to Kuwait - other countries,
notably Ireland and Portugal, have made a concerted effort to move R&D further up the
public agenda as external pressures on their economies have made the need for
innovation and scientific investment a higher priority (EC, 1993/1994).
There appears to be relatively little attempt to direct activities towards other customers
outside of Kuwait, despite this potential pool, even within the Gulf area, being
considerably larger (and possibly more favourably endowed with R&D budgets)
than the existing national one. If there is a restraint on marketing outside of Kuwait
imposed by national authorities, the case may be made that the scientific and technical
advancement of KISR is being constrained by the current structure and availability of
research funding in the home market.
In developing an evaluation and selection system, it is important that views are taken in
the criteria settings of both internally within Kuwait on the needs of public and private
sectors, and externally to S&T developments in the global context. Only when this is
done, KISR will be able to prioritize the sectors, sciences and technologies, which will be
of future importance.
120
Although KISR has an extensive network of overseas contacts, there appears to be very
little in the way of real, active, working-level networking with other Kuwaiti
organizations. For example, although connections exists between KISR and KU whereby
equipment is occasionally shared and information facilities are linked, much more could
be done in the way of shared staff resources and joint projects. In a country where such
scientific resources and intellectual capital are scarce, it is important that the composite
organizations in the national network are both formally and informally linked. KISR as
the leading national scientific organization has an important role to play in this respect.
More could be done by all of the organizations in the network to share resources, staff
and projects with others, so that the infrastructure would be strengthened in the national
interest.
A vital example in this respect is considered within ATP's evaluation system, where
ATP's mission is economic in nature, its evaluation plan emphasizes the economic
impacts of the program. However, there are a number of legislated requirements and
constraints to the program that influence what is actually measured. For example, only
projects that entail high-risk research that presses the state of the art and adds
significantly to the scientific knowledge base are considered acceptable mechanisms for
seeking the desired economic growth via the ATP. Hence, one aspect of the ATP's
evaluation plan concerns the strength of purely scientific/technical contributions of
funded research. Because the ATP is charged with promoting the formation of research
joint ventures, this is similarly a topic of keen interest for ATP's evaluation plan. In
addition, from the standpoint of designing and operating such a program, there is much to
learn about underlying cause-and-effect relationships that collectively determine long-run
outcomes. ATP's evaluation plan seeks to advance that understanding and to develop
better models for capturing the diverse and complex effects of the program.
With respect to measuring economic impacts, ATP distinguishes between initial impacts
resulting from the commercial activities of the awardees and subsequent impacts
resulting from spillover effects. Since it is legislated that U.S. businesses propose, lead,
and cost-share the research, and that development and commercialization of technology
121
be accelerated, one aspect of the ATP's evaluation plan concerns the commercial progress
of award recipients in bringing goods and services based on the technologies to market.
Moreover, since the program is focused on providing national benefit, the evaluation plan
is concerned with spillover effects - including market spillovers, knowledge spillovers,
and network spillovers - those effects that extend beyond the direct awardees innovators.
Spillovers cause social returns and private returns to diverge.
Figure 5.2 provides a conceptual illustration of private and social returns to R&D, taking
into account pure market spillovers, pure knowledge spillovers, and an interaction of the
two (Jaffe, 1996).
Summary
This chapter has addressed the first research proposition. A stringent evaluation and
selection system should be implemented and utilized within the project review cycle.
Thus, the project review cycle is the supportive catalyst of the project evaluation models
and both should be implemented in parallel within the project management system.
Therefore, in practice, a well-designed project evaluation system may not achieve its
objective if implemented in narrow base within the overall project management practice.
This chapter contributes to the research finding in chapter eight by recommending the
right evaluation criteria, techniques, and models to be proposed within the project's life
cycle.
The next chapter addresses the second research proposition that deals with the role of
S&T policies in designing the evaluation and selection system.
122
Figure 5.2 Private and Social Returns to R&D.
Source: Jaffe, 1996.
123
CHAPTER VI
SCIENCE AND TECHNOLOGY POLICIES
Introduction
Chapter Six presents the second proposed research activity area by demonstrating the
linkage criteria of S&T to the proposed project evaluation model. This chapter will
consider S&T factors to be contributed to the evaluation goals and objectives. This will
be carried out by reviewing S&T policies.
This chapter also will illustrate the advantage of linking R&D project evaluation and
selection to the national policy. This will be achieved through demonstrating the linkage
mechanisms, criteria, and to examine the Kuwait National Development Plan in view of
its goals and objectives in order to investigate and suggest linkage’s criteria related to the
S&T policy research directions and outcomes.
Views on Science and Technology
S&T projects have become increasingly popular innovative responses to a broad
spectrum of national problems. To effectively evaluate and select R&D projects in certain
research areas, it is vital to develop a set of realistic criteria to enhance the R&D projects
outcome. The criteria should be linked to S&T goals and objectives in order to present
and permit analysis of R&D projects in many policy contexts. Furthermore a guideline
should be set to help the evaluation process and link it to the national policy. These
guidelines along with the criteria are based on lessons learned from analyzing best
practice.
In general, linking S&T policy to R&D project evaluation and selection process will
generate useful information to enhance the decision making process toward implementing
the right R&D projects. On the other hand there are several R&D projects that
124
demonstrate the policy formulation where government decision makers are its principle
players. R&D projects that provide the necessary technical or economic information for
setting regulations and standards are an example for a type of policy formulation and
implementation.
Therefore, there is a need for a systematic accumulation of knowledge as a base for
establishing a sound national policy, but there is also increasingly awareness of the limits
of R&D projects and programs to generate this knowledge. In this regard, linking S&T
policy to research activities will be the right step to move research results into a better
utilization scheme and operational use.
To a large extent, these depend on a number of individual decisions of scientists and
engineers, natural and socio-economic conditions, corporate strategy, management
decisions, government policy and pure chance. Innovations depend on current demands,
indicated by market prices, future feasible needs and new contributions to the stock of
scientific and technological knowledge. It depends on the inherent uncertainty of chance
and behaviour of multiple actors. This means that any policy contains a gambling
element. Therefore it might be rational to spread the risk rather than place everything on
one card.
A simple classification for policy instruments in S&T promotion is the distinction
between S&T push and demand-pull. S&T push finds place via e.g. mechanisms such as
research projects, institutes, and personnel and public procurement of products. In this
scheme, for example, it is not always easy to decide whether or not the budget of a
research institute or the salaries of university professors must be considered as S&T input
or as the reward for their scientist output. Another scheme is to classify instruments for
R&D policy as:
•
R&D
Infrastructure:
research
organizations,
research
personnel,
universities, education;
125
•
Financing: budgets, procurement, financing, investments, loans, subsidies,
tax incentives, legislation;
•
Regulation and Persuasion: import and export regulation, indicative
planning, administrative guidance, white papers, information exchange,
public opinion, policy research.
The reasons for conducting S&T policy research are multi fold and diverse:
•
Understanding the relationships between policy and science (policy for
science and science for policy);
•
Understanding how governments can use S&T for the furthering of public
goals;
•
Understanding how S&T develops;
•
Understanding how policy system works;
•
Understanding which S&T can further societal public goals, i.e. which
S&T should be selected among alternatives;
•
Understanding which S&T policies should be followed-striving for these
societal goals, by selecting the most effective and adequate policy
instruments;
•
Providing public decision makers with this information about alternatives
or with policy recommendations.
Linking S&T Policy to Project Evaluation
In traditional R&D, the idea of lasting solutions on science has been somewhat suspect
because the scientific knowledge of natural law, and the technologies and solution based
on that knowledge, have been partial and fragmented, yielding short-term benefits with
long range side-effects. Evaluating, selecting and implementing the right R&D projects
will, undoubtedly, help to overcome such issues. This section will investigate the
evaluation and selection of R&D projects taking into consideration public policy in order
126
to enhance R&D activities that contribute to the nation’s welfare and to offer practical
solutions to the problems faced by the nation.
An R&D organization should have clear goals before its priorities can be set and its
efforts organized (Grace Report, 1983). Therefore, a strategic framework must be defined
in line with the country’s S&T policies. This approach will allow priorities to be set with
open access to debate on overall levels of R&D spending. Linking S&T policies to the
evaluation and selection process will enhance and assist the policy makers to decide on
what is desirable, rather than what is possible. A clear example is conducting research in
genetic engineering towards planting vegetables to be grown using salty water in arid
lands area. If this sort of project is approved, it will contribute to the strategic goals that
have been abstracted from the S&T policy to emphasize ‘using advance technologies to
reduce consumption of fresh water and to reduce the amount of money that is budgeted to
produce fresh water for irrigation.’
Researchers at R&D laboratories are in the best position to judge what is technically
possible, but it is difficult for them to judge which project should be approved from a
given set of proposed projects.
A traditional pure bottom-up approach would collect all the proposals for research
projects from various laboratory departments and let then decide which one to fund. This
approach is similar to the ‘Peer Review’ system used in most R&D organizations.
Although this approach is still used, it generally lacks focus and is therefore contentious.
Most likely, there are more projects than available resources to be allocated for
implementing the proposed projects. In this situation, a set of criteria should be based on
project priorities that are related to a number of factors, not purely technical. In this
regard, S&T policy is one of the main factors that has played a major role in the R&D
project evaluation and selection. For instance, the peer committee could be presented by
S&T policy authority representatives and also the evaluation forms should include
criteria that are linked to the achievement of the S&T policy goals and objectives towards
its mission and future outlook.
127
Along with the rapid progress recently achieved in S&T, the resulting advantages and
disadvantages have enormous effects on both society at large and the life style of an
individual person. On the other hand, society itself has pressured S&T to meet
requirements and come under policy regulations and perspectives.
Government policy is essentially indirect participation of society, via political
representatives, in economic matters, which transcend the competence, interests, vision,
and efforts of individual economic actors and production organizations.
Freeman (1987) uses the term national system of innovation to denote a collection of
institutions, which generate resources and allocate them to specific problems and
approaches to technology and their modes of utilization. The national system of
innovation is defined by him as ‘the network of institutions in the public and private
actors, whose activities and interactions initiate, import, modify and diffuse new
technology’.
The characteristics of such a system are:
A: horizontal integration of R&D, design, production, and process
engineering and marketing,
B:
integration of process design with multi-skill training,
C:
computer networking and collaborative research,
D:
state
support
for
generic
technologies
and
university-industry
collaboration,
E:
new types of proprietary regime for software and biotechnology.
Kuwait’s National Development Plan
At the turn of the century and as the world enters the globalization era, Kuwait remains
dependent on oil and foreign labour. The main challenge for Kuwait is how to translate
128
its oil resources into a sustainable productive capacity and develop its competitive edge
in the regional and global economy, which would guarantee and maintain a high standard
of living for the future generation of Kuwait. Kuwait experienced various exercises to
formulate its socio-economic plan. The overall development objectives of Kuwait are
focused on sustaining the well-being of its people through steady diversification and
efficient management of the national economy, nationalization of public expenditures,
promotion of private entrepreneurial activities and enhancement of the capabilities of the
country's citizens. The government’s plan openly discusses the problems faced and
identifies a series of measures to redress them; it then puts forward a new vision to take
Kuwait into the twenty-first century, encompassing fundamental changes in the attitudes
and behaviour patterns on individuals and for the entire society. The national plan also
takes into account the government’s views on how it can best contribute to addressing the
development challenges facing Kuwait and those threatening the achievement of the past
and the sustainability of the future.
The latest national developmental plan concentrates on the following issues (CCF, 2000):
•
•
•
•
Enabling environment for sustainable human development,
ƒ
Advocacy for sustainable human development,
ƒ
Long term development prospective studies,
Capacity building for sustainable human development,
ƒ
Strengthening the role of the private sector,
ƒ
Supporting the role of civil society institutions,
ƒ
Redressing imbalances in the national labour force,
ƒ
Improving education and training system,
Strengthening and developing mechanisms and tools for implementation,
ƒ
Strengthening in the national planning system,
ƒ
National information management,
Management arrangements.
129
Kuwait – Reconstruction after the Gulf War
In the course of the occupation, more than half of the population, foreigners and citizens
alike, fled Kuwait. After the reestablishment of Kuwait sovereignty in February 1991,
and the restoration of basic services soon afterward, the population began to return. In
May 1991, the government opened the doors to all Kuwaiti citizens who wished to return.
The government was far more reluctant to readmit non nationals, whom it considered a
security risk and whom it regarded as not needed in prewar numbers owing to the
postwar constriction of the economy. Consequently, relatively fewer non nationals were
allowed to return. A national bank of Kuwait (NBK) report estimated the total population
of Kuwait in March 1992 at 1,175,000 people, 53% of whom were Kuwaitis, compared
with an estimated 27% Kuwaitis of the 2,155,000 population on the eve of the Iraqi
invasion in 1990.
One of the first policy decisions the government made on returning to Kuwait was to
reduce Kuwait’s dependence on foreign labour in an effort to ensure that Kuwaitis would
henceforth remain a majority in their country. Former foreign workers are unhappy with
this policy, but there is little they can do. Divided between those who oppose Iraq and
those who do not, they pose no unified threat. Their energy has been dissipated by
individual efforts to arrange to stay. The government and population alike remain deeply
suspicious of the non-national population.
The new policy of limiting the number of foreign workers has had serious economic
consequences. Foreigners represent many of Kuwait’s top technical and managerial
workers. The exodus of most of the non-national population has created special problems
for an education system that in 1990 was still heavily dependent on foreign teachers. The
direct damage inflicted on school property and looting by Iraqi forces aggravated the
education problem. Nonetheless, in September 1991 the university and vocational schools
reopened for the first time since the occupation.
130
The Public Authority for Civil Information (PACI) estimates the total population of
Kuwait at 2.75 million at the end of 2004. Population growth accelerated to an annual
rate of 8% as the number of expatriates rose sharply in 2004. Nearly 1.79 million or 66%
are expatriates, while 956 thousand are Kuwaiti nationals. The number of non-Kuwaiti
residents increased by 11% from a year ago vs. 6.4% in 2003. Growth of the Kuwaiti
segment slowed down slightly to 3.1%. In absolute terms, the number of expatriates
living in Kuwait increased by more than 178 thousand. This is attributable to the general
pick-up in economic activity in both the private and the public sectors. Growth in
Kuwait’s labour force accelerated in 2004 reaching 11.2% driven by strong growth in the
expatriate segment. The labour force reached 1.61 million at the end of 2004, with nonKuwaitis representing 82.2% of the total. Growth in the expatriate labour force rose to
12% from 8% a year ago. In the Kuwaiti segment, the increase was fastest among
executive government employees and clerks (12.4 thousand or 9%), while in the nonKuwaiti segment the increase was fastest among production and operation workers (65
thousand, or 14%) followed by workers engaged in services (48 thousand, or 12%) and
executive government employees and clerks (22 thousand, or 10%). The number of
working Kuwaitis increased by 17.8 thousand during 2004, a 6.6% increase from the
previous year. This was the fastest rate of growth in 7 years (Economic Brief, Feb.,2005).
Economic Reconstruction
Despite the devastation of the Kuwaiti economy during the invasion and occupation,
recovery has proceeded with surprising speed. This was partly because some damage,
particularly of the infrastructure, was not as serious as first feared and partly because the
government, anxious to restore the population’s weakened confidence in its ability to
administer, has given reconstruction and recovery of basic services a high priority. The
oil industry, which was badly damaged, has been a top priority because it is the source of
revenues to sustain other government spending programs. The most dramatic economic
reconstruction effort went toward capping the more than 700 oil wells set alight by
retreating Iraqi forces. In addition to an estimated 2% of the country’s 100 billion barrels
of reserves lost in the oil fires, Kuwait had to pay for extinguishing fires, repairing
131
damaged refineries, pipelines, and other oil infrastructure. By January 1992, oil output
had risen to 550,000 bpd. By June 1992, it was back to nearly 1 million bpd. Nineteen
new wells were drilled to replace those damaged by the occupation.
The government hoped to raise production to 2 million bpd by the end of 1993. During
the invasion, Iraq destroyed or incapacitated Kuwait’s entire 700,000 bpd refining
capacity at its three refineries. But by April 1992, production levels rose to 300,000 bpd.
Nonetheless, there was concern that the rapid return to production might have damaged
Kuwait’s oil reservoirs beyond the damage done by retreating Iraqi forces, lowering its
total future reserves. Accordingly, Kuwait Oil Company (KOC) contracted with several
international companies to assess reservoir damage. However, the government also has
been under tremendous pressure to increase oil production quickly to pay for war and
postwar expenses. In the mid-1980's, overseas investments outstripped oil as the primary
source of revenues. The expenses of war, postwar reconstruction, and investment
irregularities that were being uncovered in late 1992 have forced the government to use
substantial portions of its investment principal, and in the 1990's oil is again expected to
be the major revenue source.
With respect to Kuwait’s current fiscal year ending March 2006, NBK is expected that
the government revenues to come in between KD 13.6 billion and KD 14.4 billion.
Assuming that actual expenditures cover between 94% and 97% of budget projections,
Kuwait could reap a surplus between KD 6.6 billion and KD 7.6 billion before the
allocation of 10% of revenues to the Reserve Fund for Future Generations (RFFG)
(Economic Brief, Oct., 2005).
The only sector of the economy to prosper in the immediate postwar period is trade
because of the need to replace inventory emptied during the occupation. Returning
Kuwaitis and the government have created a small boom for investors. By mid-1992,
however, the return demand largely had been met, and many goods, notably automobiles
and consumer durables, were available in excess supply. In an effort to boost the private
132
sector, the government approved an offset program in July 1992 requiring foreign
companies to reinvest part of their government-awarded contracts locally.
Despite the apparently dire economic situation, the government has felt politically
obliged to sustain insofar as possible the prewar standard of living. Some of the largest
domestic postwar government expenditures have gone directly to Kuwaiti households.
The banking debt buyout was one of a series of measures taken by the government to
help nationals hurt by the invasion. The government decided to pay all government
employees (the majority of working nationals) their wages for the period of the
occupation. In March 1992, the government raised state salaries. The government also
agreed to write off about US$1.2 billion in consumer loans, a measure benefiting more
than 120,000 Kuwaitis. It wrote off US$3.4 billion worth of property and housing loans
made before the invasion. Each Kuwaiti family that stayed in Kuwait through the
occupation received US$1,750. In July 1992, the government exempted Kuwaitis from
charges for public services due as a result of the occupation, such as bills for electricity,
utilities, and telephone service and for rents on housing.
Political Background
Kuwait is an Arab-Islamic, sovereign monarchy, with a democratic regime based
separation of powers and collaboration between the executive legislative branches. The
Amir of Kuwait, HH Sheikh Sabah Al-Ahmad Al-Jaber Al-Sabah, acceded in 2006 and is
the Head of the State. The Crown Prince, HH Sheikh Nawwaf Al-Ahmad Al-Jaber AlSabah, was designated by Amiri Decree as Prime Minister in 2006. Kuwait has an elected
National Assembly of 50 members plus appointed cabinet members. Elections are held
every four years by selective male suffrage, and it is approved by next election that will
take place in 2007 to include female suffrage.
The future vision of the plan goes online with Kuwait’s strategic vision that is gradually
to transfer dependent of oil development into human resources development. This will be
achieved by introducing basic changes in the style of production employment and
consumption in order to grant sustainable development. Furthermore, the development
133
plan for the years 2001/2002- 2005/2006 has focused on the following issues (MOP,
2000):
•
Support the National Security.
•
Direct the National Economy towards the new trends, such as
Globalization and Science-Based Economy.
•
Enhance the Role of R&D in National Development.
•
Promote Technical and Service Sectors.
•
Improve the State Administrative Structure towards Development of
Management.
•
More Involvement of Society in the Development Aspects.
KISR's Mission
Objectives of the Kuwait National Developmental Plan tend to focus on alleviation of
imposed restrictive measures and amelioration of mobilization and allocation of resources.
This is to be accomplished through relevant administrative reformation and crystallization
of the concept of human development through: rectification of imbalances in employment
market, development of education and training systems, and achievement of social
development. Whereas, KISR’s goal is to maximize the contribution of S&T for the social
and economic benefit of the industrial, agricultural, and service sectors by engaging in
R&D activities aimed at sectorial economic development in priority areas, protection of
the environment, preservation and optimal utilization of natural resources, adaptation of
technology and research results to enhance and diversify national economic resources.
At this end, KISR’s mission is to conduct scientific research, technology development,
technical consultancy, and human resources development that serve the national needs
and support the economic and social development of the State of Kuwait. On the other
hand, KISR’s vision is stated as to make, within its capability, a focused yet significant
134
contribution to the social, economical, and technological advancement of the country.
Through its research, consultancy and services efforts, to be recognized as:
•
A highly efficient and respected organization contributing to the nations
needs.
•
A focal center for the transfer into society of new technological
development and utilization of the vast S&T information resources
available through global computer networks.
Linking Criteria to S&T Policy
Linking criteria of R&D project evaluation to S&T policy is rarely easy, where success of
a project is uncertain, where return on investment takes years in the future, and where
mostly the goals and objectives of the S&T policy are vague. Several evaluation systems
include criteria for certain objectives to be achieved. For example, criteria could be
tailored for human resources development, coping with advance technology, supporting
the economy, utilization of local resources, etc. As an example to support defense
projects, the USA congress has approved the following criteria (Evaluating Defense
Department Research, 1990):
•
The length of lead time before the technology will produce results;
•
Likelihood of technical success;
•
Number and importance of the technology's military applications;
•
The time required to develop countering technologies or tactics;
•
Number and difficulty of required ancillary technologies;
•
The risk of being overtaken by parallel technical developments;
•
The extent of civilian spin-off (or unintended civilian costs);
•
Alternatives to U.S. government support including industry and allies; and
•
The overall threat posed by potential adversaries.
135
In the UK, the office of S&T has established LINK, where its scheme is the
government’s principal mechanism for promoting partnership in pre-competitive research
between industry and the research base. It aims to stimulate innovation, wealth creation,
and improve the quality of life. The scheme offers an opportunity to engage with some of
the best and most creative minds in the country’s R&D institutions, to tackle new
scientific and technological challenges so that industry can go on to develop innovative
and commercially successful products, processes and services.
Throughout the LINK managerial system, several guidelines were identified to enhance
the evaluation and selection process. LINK had issued a set of criteria that each proposed
R&D project must satisfy fundamental selection criteria in order to qualify under a LINK
program.
These criteria are (LINK, 1999):
•
Defined market: The research should be within a defined LINK
program area, aligned to one of the priority areas identified by the
program sponsors.
•
Collaboration: The research must involve collaboration between at
least one industrial and one research base partner (e.g. universities and
other centers of excellence, such as Research Council Institutes,
Government
Research
Agencies
and
independent
research
organizations).
•
Pre-competitive research: The research must be pre-competitive and
involve a real element of risk, but offer good potential for eventual
commercial exploitation.
•
Innovative: The project must be innovative, high-quality and generate
knowledge in a particular industrial or scientific field. Ultimately the
research should aim to create wealth and improved quality of life.
•
Adding value: The public sector LINK funding must provide a catalyst
for the project partners to do something, which would not happen
136
otherwise. For example because the characteristics of the industry have
previously deterred partnerships; or there are particular technological or
financial issues associated with the project.
•
Appropriate partners: The project should bring together a wellbalanced partnership, for example including manufacturers and end
users from the supply chain, different science and engineering
disciplines and other industrial sectors.
•
Synergy with other initiatives: The project should complement, rather
than duplicate, any other UK or European project in the same area.
LINK also had identified several questions to test the eligibility for a LINK project, as
follows:
•
Is the proposed research pre-competitive?
•
Does the project fit within one of the open LINK programs?
(Note: Some LINK projects are supported outside existing programs.
Contact LINK directorate for advice)
•
Does the project involve at least one industrial and one academic
partner?
•
Is government funding needed for the project to go ahead?
•
Is the project original and innovative, coupled with an element of risk?
•
Is the scientific or technical content high?
•
Are there likely to be significant economic or social benefits from the
research?
•
Is the project relevant to industry?
•
Will the project complement rather than duplicate any other UK or
European research?
•
Is there a clear exploitation route?
•
Do the partners have the skills and resources to carry the project
through?
137
An Example of S&T Goals and Objectives in Austria
Investigating in other worldwide research centers, the Austrian Research Centers (ARCs),
strategic focus is rooted in technological, social, and political developments, and in the
needs of the market (ARCs, 1999). The ARCs philosophy is to maintain in the future
certain broad scope in selected research topics. This is mainly meant to keep abreast with
the organization’s corporate policy. With their support from public sectors, contact
research institutes must meet an important commitment to research and technology
policy. A certain variety must be reserved of the innovation efforts of the economy and
society are to be supported. Furthermore, ARCs supports the Austrian federal
government actively by formulation opinions on issues involving research and
technology policy, it represents Austria on scientific and technical boards and it helps
with the development of policies, strategies, and measures. This undoubtedly, will assist
ARCs to gain a better understanding of the actions and interactions of the various players
in the context of technology, society and the environment.
It should be understood that research centers could contribute to the S&T policy, but the
question that arises here is: how could the research institutes be selective within their
project evaluation system in order to choose the right project that would contribute to
achieve the National Developmental Plan?
To approach this issue, a set of criteria within the evaluation system should be set in
order to enhance the R&D project evaluation system. The following section aims to
investigate the criteria that could be developed within the evaluation system of the R&D
projects in regard of S&T policies.
K plus is an enterprise that has been established in Austria by the ARCs, where its aims
are to improve long-term cooperation between science and industry, to stimulate precompetitive research and multi-firm cooperation, it improves transfer of know-how, to
use public funding to trigger additional private expenditures, to define new areas of
138
research through bottom-up approaches and among others to ensure quality during
selection (procedure, criteria) and operation (management and evaluation).
The above mentioned aims give an outline of the general orientation of the research
programs. They form the basis for funding guidelines and selection criteria for K plus
centers as well as for the evaluation of the K plus research program.
Linking Criteria in USA
An example of linking criteria to the National Goals is suggested for USA as follows
(National Critical Technologies Report, 1995):
Economic Prosperity
•
Directly and substantially supports major S&T goal(s) of the
administration as documented in the Memorandum on 1996 Research
and Development (R&D) Priorities, dated May 6, 1994.
•
Directly and substantially contributes to the S&T base essential for
maintaining or promoting a globally competitive position for one or
more U.S. industries.
•
Meets tests of potential economic importance in the near-term for
technologies of incremental change, and in the longer term for
breakthrough technologies.
•
Has a high rate of discovery (i.e., will impact fast-moving technology
intensive industries, such as telecommunications infrastructure and
devices).
•
Meets a test that despite recognition of an industry need, sufficient
R&D investments by the private sector will not occur without federal
support due to the magnitude or protracted pay back period for the
required investment, riskiness of the technological development, or
139
generic nature of a technology in which no single company could
expect to recover its R&D investment (the latter is a ‘commons’ test).
National Security
•
Makes an essential contribution to enabling or advancing the future war
fighting requirements.
•
Makes an essential contribution to mission areas under the
administration national security priority as stated in the Memorandum
on 1996 Research and Development (R&D) Priorities, dated May 6,
1994 (Goal 6, Enhancing National Security).
•
Is essential to meeting other defense requirements that are traceable
through the 1994 Defense Science and Technology plan.
Enhancement of Economic Growth
Technology is the single most important determining factor in sustained economic growth.
The performance of an individual firm is highly linked to their use of technology.
Technology is transforming the very basic of competition, enabling small business to
perform high-quality design and manufacturing work that previously required the
resources of big business, while allowing big businesses to achieve the speed, flexibility,
and proximity to customers that were once the sole domain of smaller firms.
Technology provides the tools for creating a spectacular array of new products and new
services.
It
is
creating
new
industries-advanced
materials,
mobile
cellular
communications, electronic commerce, and revitalizing old ones like steel, automobiles,
and textiles. In today’s highly competitive global marketplace, technological leadership
often means the difference between success and failure for companies and countries alike.
140
New technologies are improving the quality of life. Medical research in pharmaceuticals,
biotechnology, and medical devices helps us lead healthier lives and offers new hope for
the sick. Environmental research brings better monitoring, prevention, and remediation
technologies. Advanced monitoring and forecasting technologies - from satellites to
simulation - are helping to save lives and minimize property damage caused by hurricanes,
blizzards, micro-bursts, and other severe weather. Sophisticated traffic management
systems for land, sea, and air transportation enable the smooth and timely movement of
more people and goods. Agricultural research is producing safer, healthier, and tastier
food products. Automobile research is providing safer, cleaner energy, efficient, and more
intelligent vehicles. Aeronautical technology is making air travel safer, less costly, and
more environmentally compatible. Energy research is helping to deliver cleaner,
renewable, and less expensive fuels. And information and telecommunications
technologies have enabled instantaneous communications around the globe.
R&D laboratories played a major role in the advancement of new technologies in their
early stages. Today, however, competitive pressures have driven many companies to
emphasize near-term product development and process improvements that support their
market strategies and profitability of their business units. This kind of R&D focus has
proven successful for many companies in the short term, however, it comes at the expense
of basic and applied research, and threatens to reduce the pool of enabling and emerging
technologies from which must draw in the future to remain competitive.
Economic Growth in USA
In February 1993, President Clinton set forth his vision for a national technology policy in
Technology for America’s Economic Growth: A New Direction to Build Economic
Strength. This policy- a core element of the Administration’s strategy for long-term
economic growth-outlines measures to ensure America’s global technological leadership
into the next century.
141
The technology policy set forth by the Administration is guided by these principles:
•
We must retain a long-term commitment to research, education, and
innovation even in this period of budgetary constraint.
•
The primary role of the Federal government in technology policy is to
create a business environment in which the innovative and competitive
efforts of the private sector can flourish.
•
The
Federal
government
must
encourage
the
development,
commercialization, and the use of civilian technology.
•
The Federal government must help create a world-class infrastructure for
the twenty-first century to support U.S. industry and promote commerce.
•
The United States must develop a world-class workforce capable of
participating in a rapidly changing, knowledge-based economy.
To advance America’s interests in science, mathematics, and engineering, the
Administration set forth the following goals in its 1994 science policy statement, Science
in the National Interest:
•
To sustain leadership across the frontiers of scientific knowledge.
•
To enhance connections between fundamental research and broad national
goals.
•
To stimulate partnerships that promote investments in fundamental science
and engineering and effective use of physical, human, and financial
resources.
•
To produce the finest scientists and engineers and effective use of
physical, human, and financial resources.
•
To produce the finest scientists and engineers for the twenty-first century.
•
To raise the scientific and technological literacy of all Americans.
Achieving these goals will ensure that USA has the specialized human resources as well
as the modern infrastructure needed for cutting-edge science and technology. The S&T
142
enterprise weaves a vast and variegated fabric of knowledge, ideas, devices, and
questions that covers a broad range of human curiosity and innovation.
Vital investment agenda in scientific research, technological innovation, and a healthy
business environment, coupled with a strong commitment to education and human
resources development, will promote the continuing beneficial application to S&T
towards USA's overarching national goals of economic growth and prosperity, personal
health, national security and global stability, and environmental stewardship.
Governments must encourage the development, commercialization, and use of
technology. They must invest in nascent technologies that offer large economic and
social returns to the nation. National policy must ensure that the fruits of research extend
beyond government and help. Firms create high-wage jobs and national economic
growth. And the government-in partnership with local firms, the academic community,
and the private sector should continue to cultivate a range of mechanisms that encourage
widespread deployment and use of technology.
Economic Growth in Japan
Japan will be more open to the world community, will try to avoid frictions and play a
responsible role in the development of the world economy.
In July 1990, the Council for Industrial Structure of Ministry of International Trade and
Industry (MITI) produced a report on international trade and industrial policy in the
1990’s (Council for Industrial Structure, 1990).
The analysis ascertained rapid political and economic global reforms, changing
international order and increasing uncertainty about the future. It highlighted reduced
military tensions between the global powers, growing populations and improved living
standards of developing countries, limited resources and environment deterioration. It
considered that Japan could and should use its present economic and technological
strength to contribute to solutions for these global problems.
143
It expresses the belief in a free market economy, democratic systems and international
fairness as culturally universal and essential for global peace and wealth. It stresses the
need to respond to rising international criticism, not only for the sake of reduced criticism
but also for the nation’s enlightened long term interest, to strive for a better harmony with
the rest of the world by an international contribution and by making its systems,
procedures and customs more transparent and understandable for the world. Japan
formulated three overall objectives for Japan’s international trade and industrial policy in
the 1990's:
1. The contribution to the international society and the promotion of
domestic reforms;
2. The improvement of quality of life for the Japanese people;
3. The securance of the fundament for long–term economic growth.
These goals are perceived as supporting, complementing and amplifying each other.
Japanese industry is investing heavily and increasingly in R&D. This is the main factor in
the fast growth of the internationally high level of Japan’s R&D investment, which by
then had reached almost 3% (Japan Center for Economic Research, 1990).
According to a survey by science and technology agency (STA), enforcement of R&D is
seen as the most important task for the corporate management. The motives are: first, the
self-support in technological development and second, independent discovery of
technological sources for future growth. The industry’s share of R&D funding is more
than 80% and this is financed mainly from its own means.
Industrial R&D focuses especially on marketable products, although it increasingly tends
to conduct research directed at finding fundamental knowledge for later technological
development.
144
Although the ratio of basic research is increasing, it remains directed at the area of
interest of the companies. The research in advanced technology is considered to be
necessary for the industries’ transition to produce goods with high added value as a must
in the maintenance and fostering international competitiveness, especially under the
condition of a strong Yen. The research tends to be concentrated on the fields of
communications, electronics and electrical instruments.
By the end of 1989, Japan registered its 49th straight month of economic growth due to
continuing domestic personal consumption and high investments in plants. In fiscal 1989,
Japan’s General Net Present (GNP) grew at an inflation-adjusted rate of 5%. Japan’s
GNP exceeded \400 trln in fiscal 1989, which placed it as second after the U.S.A (Japan
Institute of International Affairs, 1991). The Japanese business cycle is in a growth phase
for a long time and is expected to stay there. The international economic situation is
favourable, the domestic demand is expanding and the external frictions tend to be
relaxed as a result of growing imports. The labour markets are tightening and the
employment level is high. The prices, wages and currency rates are relatively stable. The
expectation thus is that this favourable economic situation will continue in the
foreseeable future.
The Japanese economy is in its best shape in a decade. Output has increased at an annual
rate of more than 2% since 2002 and 3¼% excluding the negative contribution from the
public sector. The pace of growth has been sufficient to boost employment during the
past year and reduce the unemployment rate from its record high. Profit margins, as well
as confidence in the business and household sectors, are at their highest levels since the
early 1990's. These positive developments raise hopes that Japan is emerging from a
decade of economic stagnation, although there are a number of risks to the current
expansion (OECD, 2005).
145
Relationship between S&T and Society
The relation between science and society has become stronger and more complicated as it
is not easy to predict an exact description of these dynamics. S&T permeates a greater
part of daily life. Societies cannot provide for their livelihood, nor survive any longer
without S&T. S&T in Japan has been evaluated quite high according to a public opinion
poll (NISTEP, 1992).
The Japanese national government has already addressed the issue of the relationship
between S&T and people and society, and has declared in the fundamental principals of
S&T policies to achieve harmony in this relationship (Cabinet Meeting, 1992). Since its
foundation in 1988, the USA National Institute of Science and Technology Policy
(NISTEP) has also considered the relationship between S&T and people and society as
one of the key themes that should be addressed in the research activities and surveys.
NISTEP has conducted a variety of studies, including public opinion surveys on this
relationship, followed by analysis of data obtained on public opinion towards S&T for
improving quality of life (NISTEP, 1989). An international comparison in this regard also
was studied (NISTEP, 1992).
Changes of people's acknowledgements influence enterprises strongly. Many of
enterprises are seeking new ‘Better Corporate Citizenship’ And ‘Corporative culture’ as
they set their sights on the ‘Environment’. Not only enterprises, but also public
‘Organizations’, such as international organizations, governmental organizations, and
local governmental organizations, stress the importance of the ‘Environment’.
It is important to know specifically what those organizations think of ‘Scientists and
Engineers’ and of the ‘Public’.
The maturity of the modern industrialized society, which the development of S&T has
brought about, is now ready to help ‘Scientists and Engineers’ and the ‘Public’
communicate with each other.
146
It is a challenge of ‘Scientists and Engineers’ to consider how and where they could
manage communication with the ‘Public’. The ‘Public’ must join in developing a
consensus regarding S&T because arriving at a consensus of S&T is one of the most
important matters societies must deal with when serious social concerns are entangled
with S&T. To achieve this, participants should receive ‘up-to-date S&T information’. In
present situations, it is ‘Scientists and Engineers’ who have ‘up-to-date S&T
information’, and it is ‘organizations’ that use it most effectively.
It is a challenge for the ‘public’ to gain access to ‘up-to-date S&T information’ as a
participant in the making of consensus.
Therefore, the category and the impotents of R&D are aimed to grasp comprehensive
relationship between S&T and society to improve the communication between S&T and
society, and to correctly reflect societal concerns to research and technology
development, and to the measures to orient technology to economic and societal needs,
where vital research projects should include measures to strengthen bridges among S&T
and societies.
Summary
It is clear that S&T has a great influence on the R&D project evaluation and selection; a
proper evaluation system should consider within its evaluation’s criteria the role of R&D
activities in supporting the achievement of S&T goals and objectives of a country
national developmental plan. Therefore, the proposition of this chapter is meant to
contribute to the main findings of the research work in linking S&T to the development
of the R&D project evaluation and selection model.
Chapter seven will address the third proposition of the research, investigating proper
management practice to go on line with the implementation and usage of the proposed
R&D project evaluation and selection model.
147
CHAPTER VII
R&D PROJECT PROCESSING MANAGEMENT
Introduction
Chapter Seven presents the third proposed research activity area aiming to develop a
proper management procedural practice that will improve the implementation of the
expected R&D project evaluation and selection model. This task will be carried out by
conducting a case study of project processing by interviewing KISR's staff. Finally, the
views will be analyzed and findings utilized to develop the project evaluation model.
Views on R&D Scheme
Technological progress plays a central role in the modern economy. It is an important
contributor to economic growth and a crucial factor in determining the competitiveness of
firms in the marketplace, nationally and internationally. R&D is widely recognized to be
the linchpin of technological advance and innovation capacity.
Traditionally, firms have supported R&D because the technical advances made possible
by innovation allowed them to improve productivity, succeed in competitive markets, and
meet environmental and regulatory requirements. R&D has also contributed to the
development of new products and, in many cases, the creation of new markets. Although
business have traditionally developed research capabilities in house, they have also
established collaborative links with other organizations, such as universities, and acquired
the results of innovation from other enterprises through licensing of takeovers.
Within firms, decisions about the magnitude and nature of R&D performance are mainly
guided by consideration of economic returns (though other returns such as the public
relations benefits of high-profile research breakthroughs are also deemed important). A
number of economic studies have shown that rates of return of R&D to firms, although
148
difficult to measure precisely, are high and that returns to society, from lower cost,
improved, or new products and services, are even higher. Of course, firms will usually
engage in R&D only when the results are appropriable and offer rates of return exceeding
those of other available investment options, (such as acquisition of new machinery,
advertising, or speculative asset purchase) Boskin and Lau, 1992.
There are, however, many R&D activities that do not offer enough of an incentive for the
private sector, but whose results can yield significant benefit to the nation as a whole. In
these cases, there are often good reasons for government to step in and support R&D
efforts. Boskin and Lau (1992) added that rationales for government participation in
R&D in general-and in energy R&D in particular-include the following:
•
Some kinds of innovations that would lower costs for all consumers, and
hence are in society’s interest, are not purchased by individual firms because
the resulting gains are judged unlikely to be appropriable. Therefore, the
firm that does the R&D may obtain little advantage over competitors who
can utilize the results nearly as fast as the first firm, but without paying for
them. This ‘free rider’ problem can be, and is, overcome to some extent by
creating research consortia, such as the Gas Research Institute (GRI) and the
Electric Power Research Institute (EPRI). However, even in consortia,
industry tends to eschew basic research, and even much applied research, in
favor of shorter-term product development.
•
Some kinds of innovations are not pursued by the private sector because
they relate to production or preservation of public goods-national security,
for example-that are not reflected in the profit-and–loss reduction of
environmental and other externalities. There is little incentive for firms to
invest in such innovations unless regulations, emission charges, or other
policy instruments internalize these externalities into the private sector’s
economic calculus.
•
Research that is costly and has a high chance of failure may exceed the risk
threshold of the private sector, even though, from a societal point of view,
149
having a certain number of such projects in the national R&D portfolio is
worthwhile because occasional successes can bring very high gains. Further,
research that will take a long time to complete is likely to fall short of the
private sector’s requirement for a rate of return attractive to investors, even
if confidence of success is high. Fusion energy R&D provides an example
where the chance of failure is substantial and the time scale would probably
be too long for the private sector even if success were assured, but where the
potential benefits of the technology are so large and the prospects of other
very long-term energy options are so uncertain that government investment
is clearly in society’s interest.
Organization for Economic Cooperation and Development (OECD) countries spend
significant amounts on R&D activities. Annual public and private R&D investments
within the OECD have, on an average, exceeded 2% of GDP during the last two decades.
These activities are funded and performed by many organizations, including firms,
universities, and government laboratories. Although the roles of various institutions
involved in the national R&D enterprise vary from country to country, the main funder
and performer of R&D in industrial economies is generally the private sector. More than
one-half of all OECD R&D expenditure is financed by companies, and they perform twothirds of all R&D activities (OECD, 1997).
The Concept of R&D Project Management
Almost any human activity that involves carrying out a non-repetitive task can be a
project. So we are all project managers! We all practice project management, but there is
a big difference between carrying out a very simple project involving one or two people
and one involving a complex mix of people, organizations and tasks.
150
This has been true for millennia, but large-scale projects like the pyramids often used
rather simple control and resource techniques including brute force to ‘motivate’ the
workforce!
The art of planning for the future has always been a human trait. In essence, a project can
be captured on paper with a few simple elements: a start date, an end date, the tasks that
have to be carried out and when they should be finished, and some idea of the resources
(people, machines, etc.) that will be needed during the course of the project.
When the plan starts to involve different things happening at different times, some of
which are dependent on each other, plus resources required at different times and in
different quantities and perhaps working at different rates, the paper plan could start to
cover a vast area and be unreadable.
This was a problem facing the US Navy in the development of the Polaris missile system.
There were so many aspects to the project that a new technique had to be invented to
cope with it: the PERT (Program Evaluation and Review Technique) technique (NNH
Enterprise, 1996). This and later developments led to mathematical techniques that can be
used to find the critical path through a series of planned tasks that interconnect during the
life of a project.
Although it may be argued that the story of modern project management dates back to
this period, that would be unfair as project management is not only about planning but
also about human attributes like leadership and motivation.
Nevertheless, the idea that complex plans could be analyzed by a computer to allow
someone to control a project is the basis of much of the development in technology that
now allows projects of any size and complexity not only to be planned but also modeled
to answer ‘what if?’ questions.
151
The original programs and computers tended to produce answers long after an event had
taken place. Now, there are many project planning and scheduling programs that can
provide real time information, as well as linking to risk analysis, time recording, costing,
estimating, and other aspects of project control. But computer programs are not project
management: they are tools for project managers to use. Project management is all that
mix of components of control, leadership, teamwork, and resource management that goes
into a successful project.
Project managers can be found in all industries. Their numbers have grown rapidly as
industry and commerce have realized that much of what it does is project work. And as
project-based organizations have started to emerge, project management is becoming
established as both a professional career path and a way of controlling business. So
opportunities in project management now exist not only in being a project manager, but
also as part of the support team in a project or program office or as a team leader for part
of a project. There are also qualifications that can be attained through the professional
associations.
One reason for the rapid growth is the need to understand how to look after complex
projects, often in high tech areas, which are critical to business success but also, have to
be efficient. Most people still want their projects to be on time, meet quality objectives,
and not cost more than the budget. These form the classic time, quality, cost triangle. In
fact if you have an unlimited budget and unlimited time, project management becomes
rather easy. For most people, however, time and money are critical and that is what
makes project management so important today.
Project management has a great role in promoting scientific and technical research by
developing and enabling technologies with strong potential for producing broad
economic benefits. It also has a main role in evaluating research activities to support
increasing and measuring the short- and the long-term impacts of technology
development projects in order to optimize the funding and monitoring the entire research
programs. Project management lies within the R&D organizational strategy to promote
152
the overall economic well-being. Excellent project management procedures will,
undoubtedly, accelerate the facilitation and commercialization of enabling technologies
that are expected to yield large economic benefits.
It is a catalyst to enhance and improve R&D research projects and to help in overcoming
challenges and barriers that research entities may face. Furthermore, it should house staff
in planning, modeling, developing databases, conducting surveys, case studies, statistical
and economic analysis, and in carrying out other studies that effectively could enhance
and monitor the research program performance and contribute to the quality and
reliability of the performance metrics.
A Case Study for Project Processing Practice
Introduction
Prior of investigating in linking project processing to project evaluation and its related
criteria, the research work in this section, will implement a case study, in order to
diagnose the project processing practice in KISR, where project processing practice in
KISR is to be considered due to the easy access to interview research and other staff as
will as other needed information.
The primary objective of this case study is to assess the strengths and weaknesses of the
project processing practice in order to suggest and develop a project processing system
guideline in order to quote a reasonable image and recommendation that will be used to
be set within the evaluation criteria that have to improve both the processing system as
well as the evaluation prospectus. This case study will seek to investigate internal
management and administrative procedures and to look for ways and means in which the
R&D institutions could enhance their internal functions in respect to their project
management philosophies.
153
Case Study Approach
In attempting to meet the case study objectives, the approach will depend on interviewing
30 of KISR’s management and research staff, which is considered to be a reasonable
number as the total managerial and senior researcher posts are around 50, in order to
determine how KISR manages its research activities and the extent to which it efficiently
uses the available resources and techniques. This information will be gathered through a
number of structured interviews with a representative sample of key KISR’s staff.
KISR’s Organizational Structure
KISR’s organizational structure is shown in figure 7.1. It consists of divisions, offices,
departments and groups by which its employees and activities are organized.
At the top of the organization chart is the Board of Trustees, the DG, the Office of Senior
Advisors, the Assistant DG, the Division of Policy and Planning, the Office of Internal
Auditing, and DDGs.
These collectively represent the senior policy and advisory
functions of the institute. On the bottom left hand side of the chart are the five scientific
research divisions and the various groups and departments of which they are comprised.
These are the operational research performing units of the institute. On the bottom right
hand side of the chart are the various support divisions such as administration, public
relation, finance, information, training, etc.
KISR’s Projects
KISR has the following three types of projects:
•
Contractual Research Projects- these are conducted on behalf of specific
customers and are considered to have significant research components. They
were initiated either in responses to a direct request from the customers, on the
basis of an idea developed within KISR and then marketed to the customer, or
154
BOARD OF TRUSTEES
DIRECTOR GENERAL
OFFICE OF
SENIOR ADVISORS
DIVISION OF
POLICY &PLANNING
Office of Marketing &
ASSISTANT
DIRECTOR GENERAL
OFFICE OF
INTERNAL AUDITING
Office of Planning
Project Management
Deputy Director General/
Research
Deputy Director
General/ Information
Deputy Director
General/Administration &Finance
Central Analytical
Laboratory
Petroleum
Researches &
Studies Center
Environment
&Urban
Petroleum
Production
Department
Environmental
Management
Petroleum
Refining
Department
Petrochemicals
& Materials
Department
155
Development
Division
Coastal
Management &
Atmospheric
Pollution All
Pollution Dept.
Urban
Infrastructure
Development
Advanced System
Department
Food Resources
& Marine
Sciences
Division
Water
Resources
Division
TechnoEconomics
Division
Public Relation &
Publication
Agriculture
Technology &
Biodiversity
Water
Resources
Management
Department
Department of
Economic
Studies
Public Relations
& Information
Department
Quantitative
Methods &
Modeling
Department
Publications &
Editing
Department
Aquaculture,
Fisheries &
Oceanography
Biotechnology
department
Water
Technologies
Department
National Scientific
& Technical Info
Center
Office of
Legal Affairs
Information
Services Dept.
Division of
Finance
Technical
Services Dept.
Accounting
Department
Technical &
General Services
& Physical
Projects
Department
Division of
Human
Resources
Systems
Development Dept.
Computer
Technology &
Communication
Department
Figure 7.1 KISR’s Organizational Structure.
Budget &
Costing
Department
Purchase &
Stores
Department
Personnel
Department
Manpower
Development
Department
some combination of these two. The customer typically paid for the operational
costs of the project and for any capital items required but was not normally
required to contribute towards the salaries of those researchers employed on the
project. Increasingly, however, customers have been paying a higher percentage
of the costs and this has come to include a contribution towards salaries.
•
Service Projects- these were also conducted on behalf of specific customers,
but with less of a research component to them. They were more likely to have
been conducted in response to direct requests from customers and were typically
funded at a higher level by the customer (often 100%). It was not unusual for
service projects to produce confidential projects.
•
In-house Research Projects- these were projects initiated by KISR researchers
or management for which no external customers had been identified. They were
fully funded by KISR itself and tended to include a strong internal development
perspective or be seen as strategically important for Kuwait despite the lack of
external funding.
Project Approval Process
All proposed projects equal or exceed KD 15000, are subjected to a one-stage internal
evaluation process, which is conducted by KISR management. The review process is the
PRM, including technical and management evaluation criteria, as well as various
processing steps conducted by the OMPM.
At first, project leaders develop proposals, which are then reviewed within the concerned
research division. Suitably qualified staff within KISR, score the proposal in the PRM
meetings against criteria organized under the following main headings (Appendix. I):
•
Scientific/ R&D Merits Perspective;
•
Technical Competence and Compatibility Perspective;
•
Project Management Perspective;
•
KISR Management and National Perspectives.
148
Even though PRM approval may be waived/ exempted for small projects (<KD 25,000)
by the authority of the DDG for research. Whereas, projects less than KD 15,000 have no
PRM.
A rather involved methodology for scoring proposals against the specified criteria,
followed by subsequent weighting of these scores based on the relative importance of
each, is determined within PRM. The weighting of scores attempts to include both
technical and management evaluation criteria. However, the way in which the scores are
used to inform the decision making process, and the precise mechanism by which
proposals are deemed to have passed or failed at each stage is not clear. Whatever the
actual mechanism, proposals are either terminated or approved on the basis of the
outcome of the PRM.
Successful proposals undergo further review and processing by OMPM before being
submitted to the client for approval. If final client approval is obtained, contracts and
other formal documents are prepared before the project is launched.
Interview Structure and Findings
A representative group of relevant KISR’s research staff including managers, project
leaders and researchers were interviewed regarding the project processing practice in
KISR and the research activities in general. A total of 30 members of staff were
interviewed.
The following are the main questions and contributions of the interviews:
Q: Indicate whether or not KISR’s projects had utilized work or results from
earlier projects;
A: KISR’s projects are highly related to previous work, due to the fact that
most of research programs and elements are set in the strategic plan for a
long period.
149
Q: Indicate strategic importance of KISR’s research projects;
A: KISR’s projects are highly relevant to the strategic importance as this is a
criterion in the project evaluation system.
Q: Classify projects along a number of criteria such as: cost, risk, necessity,
etc;
A: The importance of KISR projects rests on securing fund, highly technical
with low risk.
Q: Rate the scientific and technological relevance of KISR’s projects and their
relevance to KISR’s strategic goals;
A: KISR’s projects are mission oriented and technically challenging with high
scientific and technological relevance.
Q: Views concerning KISR’s procedures for granting approval for research
projects;
A: It is difficult to obtain project approval even though most of the reviewed
proposals receive high scores. The majority of staff interviewed indicated
that
project review and approval process is too slow and usually takes
several months for the proposal to be implemented even after its
preliminary approval that is usually set by the proposal review
committee.
Q: Rate the performance and quality of project management, such as work
conducted, adequacy of resources, quality of output, etc;
A: On-going and completed projects evaluation is almost neglected. No
serious efforts have been applied nor practiced as the reviewers indicated
that they do not view the performance and quality of work in such a
professional way.
Q: Indicate factors affecting project progress, both positive and negative;
150
A: The projects are well progressed in general, even though there are some
factors that contributed to slow the project progress, such as lack of
funds, expert researchers, research equipment and facilities, managerial
practices, where it is common to face day-to-day problems.
Q: Indicate inputs received from KISR management at various stages of the
life cycle of the project;
A: Management contributions to the projects are very helpful prior to project
approval and during proposal submission. There was no effective input
from the management during project implementation nor after project
completion.
Q: Indicate whether project output is being successfully delivered;
A: Project output seems to be achieved normally, several publications are
published but few patents are registered. Projects results, technology
development and technology commercialization are not implemented.
Q: Indicate views on project resources utilization and execution;
A: Capital operational budget is well utilized but the manpower is less utilized
than as planned. Project execution usually takes more time than the
planned and approval schedule. Several projects go through amendment
and extension due to improper project preparation and planning.
Q: Suggest criteria for project evaluation and selection;
A: It is difficult to suggest a specific criterion, but it is important to revise the
current criteria and to develop criteria that suit KISR's goals and
objectives.
Q: Comparison of proposal evaluation process to the project management
practice;
151
A: The PRM evaluation system in some cases is miss utilized due to improper
staff attending the evaluation meeting or improper timing schedule or
duration or even not chaired by the right person or absence of the key staff
as they present a stronger management practice to be implemented.
Q: Impact of KISR’s project output on general scientific and technological
terms, KISR’s itself, users and beneficiaries and the socio economic
significant returns;
A: The completed projects have socio-economic returns in Kuwait, with a
significant contribution towards the development and the reputation of
KISR beside a great impact in training KISR’s junior research staff.
Q: Views on future research policy and S&T policy in general.
A: It is strongly recommended that KISR re-prioritize its research programs
and elements, considering room for new emerging research areas.
Linking Project Evaluation to Project Management Practice
An R&D Project Evaluation System should go hand-in-hand with Project Management
Practice. In this respect, the Evaluation Peer Committee Members have a major role to
play in enhancing the R&D project evaluation and selection mechanism. Therefore, this
research section will demonstrate the formation, roles, and responsibilities for the
evaluation peer committee members.
Kostoff and Schaller 2001, ‘Science and Technology Roadmaps’ is a first attempt to
bring some common definition to road mapping practices. S&T roadmaps are used in
industry, government, and academia to portray the structural relationships among science,
technology, and applications. Roadmaps are employed as decision aids to improve coordination of activities and resources in increasingly complex and uncertain
environments.
152
Specific uses of roadmaps include S&T management including strategy, planning,
executing, reviewing, and transitioning; S&T marketing; enhancing communications
among researchers, technologists, product managers, suppliers, users, and other
stakeholders; identifying gaps and opportunities in S&T programs; and identifying
obstacles to rapid and low-cost product development. S&T managers also use roadmaps
to help identify those S&T areas that have high potential promise, and to accelerate the
transfer of the S&T to eventual products.
The R&D Project Evaluation Peer Committee
The choice of members in the decision making group can help to avoid problems. Leifer
et al. (2000) stresses the importance of getting the right people to conduct the evaluation,
recommending a diverse team made up of senior corporate managers and business
leaders, business development managers, senior technologists and veterans of radical
projects.
The proposed committee is an advisory structure intended to guide R&D activities and
progress. It should serve to promote discussion among the founding members. The
purpose of the advisory structure is to provide the R&D institute with the technical and
operational activities of the research programs. Its main purpose is to ensure that the
R&D institute provides full value to its resources to carry out potential research projects
in order to fulfill its vision, mission and objectives, and to develop the S&T that will meet
the required needs of the social and economic development.
The committee members should be a drawn from management, technical, and business
from the research institute as well as public and private organizations and sectors. These
groups will provide general guidance on the technical directions of the overall programs;
as well as they will control the specific content of the institute R&D portfolio.
153
It is proposed that the peer committee will oversee the management of the activities and
processes and will provide technical guidance on the approach, execution, and
technology transfer of in-house and funded R&D projects.
The peer committee will provide oversight to the operational and business activities,
funding priorities, and technology development directions of the research activities by
reviewing single projects as well as a set of projects within the same research program.
Peer Committee Structure
•
A precise number of key management and technical staff representing
the research institute.
•
Representatives of potential funding partners are to be invited.
•
A representative from concerned public and private sectors.
•
When necessary, international experts from outside that could be
invented to participate in meetings.
The research institute will be responsible for notifying partners of upcoming meetings,
arranging meeting facilities, and distributing proposed materials in a sufficient time
before the meeting takes place, preparing an agenda and distributing minutes of the
meeting.
Responsibilities
•
Decide how often and how long it will hold regular meetings.
•
Review the research institute strategic objectives and continually reevaluate these objectives in light of the evolving structure and
priorities for research infrastructure.
•
Provide technical program planning, including the identification of
R&D projects to support the needs of the strategic goals and
objectives.
154
•
Prioritize R&D projects and aligns on resulting deliverables.
•
Evaluate the progress of R&D projects at milestones, including sunset
reviews, and provide guidance for technology transfer.
•
Establish
policies
and
practices
on
awarding
contracts,
commercialization, and licensing.
•
Establish performance goals for the institute management and evaluate
results regularly.
•
Assist in identifying and recruiting new peer committee members.
•
Review effectiveness of the peer committee members.
Approval Process

The peer committee will take action by implementing an evaluation considering the
followings:
•
Prioritization of R&D projects.
•
Performance of goals and objectives.
•
Members that are funding a specific research project/program can evaluate.
•
Every effort will be made to reach unanimous alignment on decisions.
•
A quorum will be reached when 3/4s of the steering committee is present.
•
Motions will be passed if 3/4s of the group approves.
•
Policies and practices regarding the awarding of contracts.
Types of Participants
Participants should be empowered to represent their organizations and their individual
views, be available to participate in meetings, and have long-term strategic visions.
Membership would typically include:
•
Corporate executives;
•
R&D executives;
•
Manufacturing executives;
155
•
Product development executives;
•
Government agency senior representatives; and
•
Non-profit local society representatives.
On-Going Project Advisory Groups
A Project Monitoring Committee (PMC) to be formed for each group of on-going R&D
research program’s projects, identified and approved by the peer committee. The PMC
will provide technical oversight to ensure that the objectives of the projects, as set forth
by the peer committee, are met and that the planning and execution of the projects are
done in a professional and timely manner. The PMC will also facilitate the transfer of
technology to the institute’s clients.
Project Monitoring Committee Structure
•
Participants in a PMC will be technical experts from both the research
institute and the interested organizations and sectors.
•
Each PMC will meet on a regular basis.
•
A representative from the research institute will moderate the meetings.
The research institute will be responsible for notifying members of upcoming
PMC meetings, preparing an agenda and distributing minutes of the meeting.
Responsibilities
The project leader will conduct the day-to-day technical, financial, and administrative
management of the projects. His responsibilities will include the development of project
plans, preparation of requests for proposals, co-ordinate the evaluation of proposals,
contract negotiations, oversight of contractors, reporting of progress to the institution
management, the PMC and peer committee and the timely delivery of results. The
responsibilities of the PMC are to:
156
•
Assist in the development of a project plan that describes the
objectives of the project, technical approach, value, deliverables, and
approach for implementation.
•
Assist in determining budget and schedules.
•
Assist in tracking the technical progress of the project.
•
Assist the project manager and the concerned division/ department in
technical and administrative matters to enhance the project executions.
Types of Participants
PMC members should have experiences within the research proposal area. Participants
should be empowered to represent their entity, be available to participate in PMC
meetings, and respond to requests for information and review project documents in a
timely manner.
Project Termination
R&D is risky, with heavy odds against commercial success; only one in ten ideas reach
the market (Raelin and Balachandra, 1985). It is thus helpful to recognize when a
foundering project has reached the point where funds should be spent on more promising
work.
As expected, there is considerable literature in both the academic and popular business
press on how to manage R&D projects; most of this, however, pertains to the on-going
management issues of viable projects, with relatively few papers on the matter of
recognizing failed R&D projects (Crowley, 1999).
In reviewing the literature on recognizing failed projects, somewhat of an evolution is
identified. The earliest works consist of developing common-sense general guidelines, or
‘rules of thumb’. Improving on this is the later application of more sophisticated survey
and analysis techniques to develop better general guidelines. Most recent are
157
sophisticated quantitative decisions models to determine the best termination point of a
troubled R&D project.
The general-guidelines works contain qualitative techniques that can be used to help
decide whether to terminate an R&D project. Often, these seem too general. For example,
Baker, et al. (1986), described research of 211 companies and concluded with the
following four tests that should be met in order to predict that a project will succeed:
1. A relevant business need, problem, or opportunity has been clearly identified.
2. An appropriate scientific or technical approach has been matched with the
need, problem, or opportunity.
3. The project result can be transferred to an internal user.
4. The internal user can produce, market, distribute, and sell the resulting project.
A study conducted in 1985, based on a survey of 51 companies with the data analyzed
using discriminant analysis, identified 16 factors that are said to discriminate ‘very well’
in the decision to continue or terminate an R&D project (Crowley, 1999). The author
claims that ‘R&D managers can immediately benefit from the analysis by considering the
factors and applying them to their current project portfolios’ as a decision making aid.
The most promising model, however, with respect to feasibility of practical application
and usefulness today, appears to be a recent study that analyzes company survey results
using multiple logistic regression techniques (Kumar, et al. 1996) applied to a data set
from 60 successful and unsuccessful projects of Canadian firms, eight major factors (as
shown in the upcoming table) are developed to help distinguish between successful and
unsuccessful R&D projects.
158
Summary of Project Termination Decision Variables.
Stage
Variable
1. Initial Screening
CORPFIT Fit of project with corporate goals.
2. Commercial Evaluation
SCITEK Availability of related science and technology.
REACTION Support for project from others in firm.
USES Applications for innovation not previously
available using similar or substitute technology.
TEKCAPAB The adequacy of a firm’s technical
capability to support the project’s complexity.
3. Development
DEVPROC Efficiency of development process.
COMTEKFT Association between project’s commercial
and technological aspects, such as the extent to which the
end product matched the user’s requirements, and the
extent to which the firm can sell the product within
acceptable markup levels.
4. Manufacturing / Marketing
TEKCAPAB (see above)
Launch
MKTEFORT Level of effort invested in
promotion/selling activities.
These Variables are assigned a value on a scale of 1-5 by personnel familiar with
the R&D project.
Source: Kumar, et al. 1996.
159
As part of the decision model, the R&D process is broken down into five stages entailing
four decision points. The five project stages are:
1. Initial Screening,
2. Commercial Evaluation,
3. Development,
4. Manufacturing/Marketing launch, and
5. Initial Commercialization.
The four decision points are between the stages. This model is diagrammed in figure 7.2.
A probability function is derived for each stage that measures the probability of success
at that stage.
Completed Projects
Completed projects differ in many respects. They vary in terms of costs, duration, form
(single applicant or joint venture), industry, size of company, public/private ownership
status, type of participating organizations, research problems addressed, technology
developed and the degree of progress made toward meeting technical and business goals.
A completed project is defined as one for which a final report has been filled and
submitted, the financial and other paperwork required for close-out has been done. The
Economic Assessment Office (EAO) of the ATP, is providing an interim assessment of
the status of all completed ATP projects, it had set the selection criteria as follows:
•
30% for scientific and technical merit;
•
20% for potential net broad-based economic benefits;
•
20% for adequacy of plans for eventual commercialization;
•
20% for level of commitment and organizational structure; and
•
10% for experience and qualifications.
160
Ideas
Initial Screening
Stage 1
- Preliminary Market Assessment
- Preliminary Financial Assessment
- Preliminary Technical Assessment
Decision 1
Commercial Evaluation
Stage 2
- Detailed Market Assessment
- Detailed Financial Assessment
- Detailed Technical Assessment
Decision 2
Development
- Specifications and Design
- Prototype Construction
- Prototype Testing
Stage 3
Decision 3
Manufacturing/Marketing Launch
Stage 4
-
Tooling and Documentation
Pre-Production
Testing and Redesign
Initial Market Testing
Market Strategy
Distribution and Sales
Failure Tracking
Decision 4
Stage 5
Initial
Commercialization
Figure 7.2 Activity-decision Stage Model.
161
On the other hand, ATP considers other major activities and impacts of the completed
projects such as:
Gains in Technical Knowledge
A major goal of the ATP is to build the nation's scientific and technical knowledge base.
Each completed project targeted a number of specific technical knowledge discovery
goals. This assessment briefly describes, in non-technical language, the technical
advances of the completed projects, where detailed descriptions of the research are
embedded within the scientific papers and patent applications generated by the projects.
Dissemination of New Knowledge
The pursuit of the tasks in a project usually produces a number of distinguishable
outcomes. Projects generate new knowledge about how to apply underlying scientific
principles to develop products or processes. If the technology is commercially successful,
it provides the basis for products or processes that can be marketed. And with
commercial success and dissemination of the newly gained knowledge comes the
possibility of benefiting the economy in ways that go beyond the benefits received
directly by the innovating firm.
Commercialization of the New Technology
New technical knowledge must be put to good use if economic benefits are going to
accrue to the nation. In most instances, the use will be through the introduction into the
market of a new product or process by the inventing firm or other companies. The new
knowledge may be used by outside researchers before it makes its way into the
marketplace. But the new knowledge must eventually result in new products or processes
in the marketplace for there to be real-world benefits to the economy. In competitive
markets, the producer is typically unable to capture all the benefits of a new product, and
the consumer reaps part of the benefits.
162
Broad-Based Economic Benefits
The actual use of new products and processes that results from a new technology
generates a variety of benefits for the economy. These benefits may result from lower
costs or higher quality relative to products and processes they replace. Or they may stem
from unprecedented performance capabilities, such as a novel treatment for cancer. Those
who receive these ‘incremental’ benefits typically do not pay for their full value.
Technical Discussion and Summary
This chapter is concerned with key issues in R&D project process management. This
chapter and the previous two chapters addressed the whole process of project evaluationfrom project selection to process management and then to an evaluation of proposed, ongoing and completed R&D projects. Furthermore, project termination decision is
discussed for R&D projects.
The next chapter proposes an integrated evaluation model for R&D project evaluation
that combines the whole life cycle of the R&D project as well as the concerned process
management practice. In today's complex and competitive global economy, knowledge
based-economy, knowledge based-industries, and information based market; so much
depends on making the right R&D project choice. Thus, an evaluation and selection
system, offers a comprehensive coverage of state-of-the-art selection techniques to be
utilized and used by decision makers in R&D centers, industry, government, and
universities in particular. R&D project selection is an indispensable working resource for
R&D managers, department heads, administrators, policymakers, consulting engineers,
and all those involved in the project selection and management process. Designing the
most suitable evaluation and selection model will:
•
Provide an in-depth analysis of the best evaluation and selection models.
•
Explores the strengths and weaknesses of each method and shows how
they relate to one another.
163
•
Features extensive coverage of the data requirements for each method.
Supplies R&D managers with clear guidelines on how to effectively
tailor the various selection models to meet the demands of their
organization's unique situations and goals.
•
Offer R&D center a support of proposal-writing tips that can
dramatically increase their chances of getting approval for their projects.
This research began by reviewing the R&D project evaluation methods, selection
techniques, and overall R&D management literature. It provides vital and fruitful
information and in-depth analysis of main evaluation and selection theories. The research
also explored the strengths and weaknesses of each method, as well as comparison of
how they are related to a certain applications.
Just as importantly, this research has presented three major propositions related to; R&D
Project Review Cycle; S&T Policy; and R&D Project Processing Management.
These unique propositions will guide to the main contributions of the research work in
the following chapter.
Thus, it will provide a new tailored guidelines and framework to the R&D’s management
on how effectively the suited evaluation and selection model will meet the demands of
their organization's unique situations and goals. At this stage, KISR has been chosen in
the research, where a thorough review and analysis took place within the aims of this
research.
Finally, it is expected that the research will provide R&D project management with clear,
step-by-step guidelines on effectively selecting projects so as not to detract from work
already in progress or to overcommit and exceed available resources. At the same time, it
will provide the R&D management with respected information on how best to tailor
project proposals to increase the likelihood of contributing to national technological,
social and economical development.
164
CHAPTER VIII
CONCLUSIONS AND CONTRIBUTIONS OF THE RESEARCH
Introduction
This final chapter evaluates the research project findings in the light of the original aims,
setting out the research proposed model to R&D project evaluation that is meant to
enhance the evaluation and selection process for a research institute within the GCC
region. This will be demonstrated by implementing an evaluation model which consists
of evaluation model and analysis model. The research will illustrate and test the model by
implementing some of KISR's existing projects.
Evaluation of Findings
Introduction
In formulating the research propositions, the following constraints of R&D project
evaluation and selection procedures were found:
•
Decisions are made project by project.
•
An individual's opinions or single criteria have too much influence on the
selection.
•
The selection criteria are unclear and implicit, and not aligned with
organization strategies.
•
There are a lack of practical tools for prioritization and selection.
•
Proposal preparation is not reflecting the actual financial and
administrative part of the proposal.
•
Proposed projects face high risk of technical approach and methodology as
well as the uncertainty of marketing and commercialization of end results.
•
Lack of inter-disciplinary R&D projects among the research programs.
•
High rate of unhealthy competition among research programs.
•
The evaluation process usually takes a long time.
173
A New Approach to R&D Project Evaluation
Consequently, this research presents a new approach for R&D project evaluation and
selection that overcomes the above constraints. It aims to reach a compromise that will
satisfy most of the parties participating in the evaluation process. More specifically, the
evaluation model consists of two models to cover the whole R&D project evaluation and
selection process, namely:
•
Evaluation Model.
•
Analysis Model.
The proposed model has a number of advantages, including:
•
Covers the whole R&D project evaluation and selection process from
generated idea gathering to the final evaluation of completed project.
•
In-depth analysis and continuous appraisal of generated idea.
•
Makes it possible to produce different decision hierarchies for different
kinds of projects.
•
Makes it possible to take into account business strategies in project
evaluation and selection
•
Group discussion of ideas
•
Supports project evaluation and decision making through evaluation and
analysis models, which include comprehensive analysis of R&D projects.
•
Utilizes both qualitative and quantitative criteria in project evaluation.
•
The evaluation model will include coherence criteria that will deal with
the several stages of the project and be relevant to the evaluation
perspectives.
•
Views of peer members with different disciplinary experiences and
backgrounds to be assigned for each research program.
•
Reduces the degree of uncertainties concerning resources utilization
through a basic mechanism for financial and administrative planning.
174
•
Integration of the research programs within the R&D institutions and
within the end users.
•
Evaluation will be implemented with a set of proposed projects from
different research programs at the same time.
The following section demonstrates the proposed evaluation model for R&D research
project evaluation and selection.
Proposed New Model for R&D Project Evaluation
Introduction
The new model adopts an integrated approach, enabling it to cope with the various phases
of R&D projects from inception to completion. It encompasses four project phases,
namely: proposal, on-going, terminated and completed project. The approach of
designing the model is based on both Mental and Analytical Modeling (Arsham, 2003).
The research cited and reviewed several evaluation and selection theories within the
available literature, in order to prepare for the design of a new proposed evaluation model
that will lead to a better result and overcome the obstacles, mainly faced by KISR
management. Among these articles, Martino, Souder, Cooper, Liberatore, Twiss, NSF,
Baker, and Hart, etc., they were included and discussed in the literature review as
mentioned in chapter three.
Two other articles have been cited in the literature, which are strongly and closely related
to the theme of the research, where mainly lead the authors of the research to abstract and
quote the idea of building both the evaluation and the analysis models. In this regard, the
research had reviewed the following articles (Arsham, 2003) and (Bach, 2002), which
precisely open the channels towards designing the research proposed model. The
evaluation model is applying a mental and judgmental approach, whereas the analysis
model is applying a simplified mathematical approach, where each type of modeling is
specified as follow (Arsham, 2003):
175
•
‘Mental Models shape the firm's actions because they affect what decision
makers see and pay attention to. In other words, mental models determine
which information receives the attention of decision makers and which is
ignored. Decisions are the result of applying a decision rule or policy to
information about the world, as we perceive it. The policies are themselves
conditioned by institutional structures, organizational strategies, and cultural
norms. Therefore, an appreciation of the hygiene of mental models is
important for the decision maker.’
•
‘Analytical Models are mathematical models aimed at simplification,
abstraction of real systems in order to provide insight, and understanding of
some interested aspect of the reality. However, modeling the reality by
abstraction should be connected with real problems, domains, and practice by
means of verification and or validation. An abstraction may be looked at from
one side as a compression of many instances into one generality or from the
other side as a special purpose power tool that yields the solution for many
problems. These types of models are applied mostly to the static and/or
deterministic systems.’
In other words, qualitative and quantitative models are designed as follows (Bach, 2002):
•
Qualitative Model: effects crossed with participants and/ or projects
characteristics.
•
Quantitative Model: collection of effects (different effects of different
types for each participant), which are added for the sample of participants.
Model Design and Structure
The R&D Project Evaluation and Selection Model consists of two models. The model
should verify the meanings and content’s hierarchy. The model's main features comprise
the following issues:
176
•
Setting goals.
•
Defining criteria and sub-criteria.
•
Defining evaluation perspectives categories.
•
Defining weights for criteria and categories.
The defined criteria and weights should reflect the organization’s strategies and goals.
The Model is divided into, Evaluation Model and Analysis Model.
The Evaluation Model is based on the following:
•
Selecting the right perspective category for certain criteria,
•
Experts evaluate the criteria they know best,
•
Integration of distributed knowledge of the experts,
•
Entering comments for evaluations.
The Analysis Model, contents and meanings, depends on the goal that to find out the best
combination of R&D projects that;
•
Is aligned with the organization’s strategies,
•
Achieves a desired balance in terms of defined parameters, and
•
Yields a maximum value in terms of the organization’s objectives.
R&D Project Evaluation and Selection Process
The Project Evaluation and Selection Process consists of six steps regarding the
establishment of the model that shows the overall parameters, including the various
concepts of the model. Figure 8.1 explains the main steps that the model should start
with, as follows:
•
Generated Proposal: when the idea of the proposal is generated, it
investigates the local interests that are concerned with the R&D institutes,
177
Local
d
Generated
Global
Interest
Proposal
Outlook
Public /
Pre-Evaluation
Clients/End
Organizatio-
Discussions
users Views
Evaluation
Evaluation
nal Views
Criteria
Model
Analysis
Rejections
Forum
Approval
Model
Post
Evaluation
Project
Termination
Project Planning
&
Implementation
Project
Final
Continuation
Evaluation
Reporting
Figure 8.1 R&D Project Evaluation and Selection Process.
178
interested organizations, clients, societies and the whole country. On the other
side proposed generated ideas should investigate the related global research
and development that are taking place regarding the proposal idea.
•
Pre-Evaluation Discussions: this stage will include the outline and the first
draft of the proposed idea to be pre evaluated and discussed within the
concerned division. This step also has to collect and investigate both other
organizational divisions within the R&D institutes and public views. It is also
important at this stage that the clients and end users views are consulted.
•
Evaluation Model: The evaluation model mainly consists of evaluation
criteria and evaluation forum that contribute to the mechanism and
functionality of the model which is considered to be the main part of the R&D
project evaluation and selection process. Respectively, the model will be
designed in a separate section within this chapter.
•
Analysis Model: The analysis model is the tool that in one hand would
analyze the comments and the score rates that are been given by the
evaluation members and on the other hand, it would rank a portfolio of
projects that have been submitted for evaluation at the same time. The
analysis model will assist the decision maker to either approve or reject a
project, as it will analyze to the outcome of the evaluation model. The model
should also be fed by the standard requirements of the R&D management that
to be considered as essential criteria for evaluation (e.g. matrix should be
applied between divisions, budget not to exceed $…, duration not to exceed
certain limited time, etc.). Respectively, the model will be designed in a
separate section within this chapter.
•
Project Planning and Implementation: this step starts after the approval of
acertain project which consists of reporting and post evaluations, the periodic
reporting should be submitted to indicate the progress of the project, the post
evaluation should be applied to the on-going projects in order to improve the
tasks and the progress of the projects and to take a decision whether the
project should be continued or terminated.
179
•
Final Evaluation: upon the completion of the project, a final evaluation
should be proceeded in order to avoid short comings for future projects and
also to investigate in further research in the same area of the project.
The Evaluation Model
There are several factors affecting the evaluation model, in order to design the evaluation
model, the research will look in detail to these factors as shown in figure 8.2, as follows:
•
Objectives: the evaluation model should consider the main objectives of the
R&D institute, whereas the project to be submitted to the evaluation should be
set within the organizational goals and objectives, as stated in the Institute
Strategic Plan and the National Developmental Plan of the country. These
objectives vary from scientific, technological, social, economic, etc.
Therefore, if a project is to be approved it should show the desired
achievements that contribute to the strategic goals and objectives.
•
Evaluation Criteria: the evaluation criteria should start with identifying
perspective categories that go online with the above-mentioned objectives.
Main perspectives should include the following:
ƒ
Proposal Preparation and Formal Presentation;
ƒ
Contribution to Objectives;
ƒ
Technology Assessment and Appraisal/Methodology Approaches;
ƒ
Expected Contributions;
ƒ
Financial/Administrative Issues;
ƒ
Manpower/ Human Resources Development.
Each of the above mentioned perspectives should have its own evaluation and
selection criteria in order to give the right approaches to evaluate the proposed
project; these criteria will assist the evaluators (peer experts) to give their right
views and concerns regarding the acceptance and approval of the proposed
180
Objectives
-
Organizational Goals &
Objectives.
Strategic Objectives.
Social, Economic,
Political…Objectives.
Scientific & Technological
Objectives.
-
The
Evaluation
Model
-
Evaluation
Evaluation
Criteria
Forum
Perspective Categories.
Evaluation & Selection
Criteria.
Coherences with
Objectives.
-
Committee Members.
Methodology.
Evaluation Forms &
Design.
Scoring Techniques.
Portfolio
Evaluation
Figure 8.2 Evaluation Model Design.
181
project. These criteria should be coherent to each other; and they should be
also coherent with the other perspectives criteria of the evaluation model.
•
Evaluation Forum: in order to achieve a proper evaluation and most
appropriate usage of the evaluation criteria, a unique evaluation forum should
be formulated with the right procedures to be followed to reach the desired
goals in which the right projects will be selected. The evaluation forum is
considered to be the core of the evaluation model, where the decision will be
taken by the peer committee members.
First of all a selective committee member should be chosen for each project
with different disciplinary members from certain aspects. For example a peer
committee for an environmental project should consist of environmental
engineers, scientists (biologists, chemists), lawyers, and socio economists,
government members as a decision key makers, end users, and investors from
private sectors, etc. The committee should also consist of senior management
as well as divisional and departmental scientists and administrative staff. Each
research program should formulate its own peer committee members and be
fed with special experts as needed.
The procedure of the methodology should be set in a way to recover the
obstacles that had been faced in the conventional methods of the evaluation.
Given that the objective of the model is to choose a number of projects within
a portfolio of proposed projects, the committee should evaluate a set of
projects in a certain time. That means each research program division should
submit their projects for per sé twice a year.
This will enable the peer
members to choose the right proposed projects with the institute resources as
well as compare the proposed projects among
the
alternative research
programs within the institute. This will leapfrog the first come, first approved
project.
182
It is more advisable that the special forum be designed for each research
program, where a certain criteria would be useful for oil research program and
other criteria could be more appropriate for food research program.
It is been recommended in this research that scoring techniques be
implemented as mentioned in the previous chapters (Chapter III).
It is
important that a precise weight criterion and score criterion are assigned to
each perspective and related criteria. A good example to explain this can be
shown in appendix I, where KISR as a leading research institute in GCC and
has used an equal weight criterion (four points) for each of criteria of certain
perspective. On the other hand the first three perspectives have total weight of
20, where the fourth perspective has a total weight of 40. The question that
arises here is: would this allocation of the set weight be applied for all
research programs as it is more preferred that each research program has its
own perspective categories, evaluation and selection criteria, weight and score
criteria? This evaluation model will deal with several projects from different
research programs in a certain time period. This will present to the decision
makers several evaluated proposed projects with different weighted scores. In
practice, several projects will be ended with almost similar results within the
same range of total score. This will create an obstacle for the decision makers
to choose the best projects and therefore this research will investigate more
in the next section by proposing the Analysis Model to support the R&D
management in prioritizing the most suitable projects within the institute
capabilities and resources and taking into consideration the vital contribution
to meet the overall goals and objectives of the R&D organizations and the
national development concerns.
183
The Analysis Model
The analysis model is meant to act as the operational unit of the evaluation process. It is
the heart of the evaluation, where it pumps the outputs of the evaluation in a final form
and distributes all the results to the decision makers after diagnosing all the inputs, beside
other built-in criteria that were not included in the evaluation model. It is mostly a
simplified quantitative methodology that leads the decision maker to take the best choice
in selecting among the proposed R&D projects. It depends on forming a triangle that
consists of input, methodology, and final output results. A simple computerized program
could be designed to integrate the various input factors in a way to present the pros and
cons of the evaluated R&D projects. Figure 8.3 explains the main steps that the analysis
model includes, as follows:
•
Inputs: The analysis model will be fed with several information and data to
be further analyzed. The information and the data will include the scoring
results and individual comments that have been obtained from the evaluation
model. Other data related to the proposal such as technology appraisal and
methodology , resources utilization (cost, manpower, duration) and other
relevant information to be built in the analysis model which mainly should
include strategic objectives, program objectives, matrixing among programs.
The subjective data will be viewed and screened in a way that they could be
referred and compared to the outputs results of the model. The data should be
screened prior to entering them into the analysis model. In order to
demonstrate this issue, the individual comments could be taken as an example,
where they should be arranged in a way that they could be compared to the
final results of the model. Undoubtedly, this will enhance the overall
evaluation procedures where the subjective results of the evaluation model
will be compared and double checked with the analysis model. The
information and data will include guidelines that would assist the model with
the opportunities and risks for each proposed R&D project. For example, if
the analysis model predicts a high rate of manpower utilization and end users
interests, the model would give a prediction of good opportunities that the
184
Analysis
Inputs
-
-
Model
Methodology
-
Scoring Results.
Individual’s
Comments.
Built-in Criteria.
Opportunities.
Risks.
-
Results
Integrations.
Certainty Level.
Credibility.
Prioritization.
Outputs
-
Results
Utilizations.
Prioritization List
of Projects.
Advantages.
Disadvantages.
Overall Results.
Figure 8.3 Analysis Model Design.
185
project is likely to succeed. Otherwise, if the analysis model predicts for a
certain proposed project, for example, the project is dealing with mature
technology, high budget, and less interests from the end users, respectively
this model will give a prediction of high risk and waste of resource utilization.
•
Methodology: The methodology will be based on a simplified and
computerized program that could integrate the input data and rank the
proposed R&D projects depending on the designed built-in criteria of the
analysis model. The following example will explain how the model would
operate, taking into consideration a certain number of proposed projects that
have been approved for the evaluation model. For example, if a research
program consists of ten proposed projects and the aim of the analysis model is
to prioritize those ten projects as there are limited resources to implement all
of them at the same time. This step is considered to be the first level of the
analysis model, where this will be applied to all other research programs
independently. The second level of the analysis is considered with a portfolio
of projects among all the research programs, where the chance will be to rank
set of projects in priority order within the organization as a whole. This gives
the decision maker of the senior management to select projects that are more
useful to the implementation of the organizational goals and objectives. The
analysis model methodology is considered to be the final evaluation step
toward results integration, and certainty/ credibility of the proposed projects.
Furthermore, the methodology process will present the final outcome of the
proposed R&D projects in a prioritization list to facilitate the selection
process.
•
Outputs: The outcome of the analysis model will present the evaluated first
and second level of the evaluated proposed R&D projects along with the
prioritization list; the output would also include the advantages and
disadvantages for each project. This is considered to be the final presentation
of the decision to be taken in respect of selecting the right projects for each
program and for the organization within a certain time limit.
186
The set of the projects that have been chosen from the overall results will be implemented
and further evaluation will take place in the upcoming project phases that include post
and final evaluation. In order to examine the proposed model, next section will
demonstrate hypothetical projects evaluation case utilizing the new proposed model that
consists of both evaluation and analysis models. This will give the chance to discuss the
usage and the advantages of the model and leave room for final modifications and fine
tuning of the evaluation model.
A Hypothetical Case Study for R&D Project Evaluation
Case Study Definition
Using the new model, this section sets out a hypothetical case study for an R&D institute
that has five research programs, where each of the research divisions has to submit their
R&D research projects in a certain time within the fiscal year to be evaluated. By the
time of the evaluation, the research divisions (A, B, C, D, and E) have submitted several
research projects as follows:
Research Division
Number of Projects
A
5
B
3
C
4
D
6
E
2
Total
20
The above mentioned 20 research projects will be first evaluated individually by both the
evaluation and the analysis models. This means that the above 20 proposed projects
would be evaluated simultaneously, taking into account that each research program will
have its own criteria of evaluation and peer committee.
187
Through the evaluation model, the outputs will be a set of score ratings and peer
committee comments where special evaluation forms will be distributed to the peer
members for their score ratings and comments.
Prior to the evaluation, a rules and regulations manual will be set in order to manage the
overall system and procedures of the evaluation, such as advanced time for proposal
distribution to the committee members, maximum time of the evaluation meeting, time of
proposal presentation, time of submitting the evaluation forms, forming the peer
committee members, and so on. This manual should be strictly implemented to assure
that the evaluation model will be successfully utilized. After this phase of evaluation, the
selection decision will not reflect the right decision to be taken due to that proposed
projects with high scores will be more than 50% of the evaluated proposed projects as
been experienced in most of the R&D institutions and on the other hand, the available
resources will not accommodate all of the high rate scoring projects. To overcome this
issue, this research is proposing a second phase of a quantitative analysis model.
For example, if we consider the same R&D institute that has resources to accommodate
from seven to ten projects and the outcome of the evaluation model is 12 projects out of
the evaluated 20 (as shown below) has successfully scored an acceptable rate of 65% and
over, which is the limit for the project to be approved.
Research Division
Number of Projects
A
3
B
2
C
3
D
3
E
1
Total
12
These 12 projects will be re-evaluated and prioritized in the second phase through the
R&D project evaluation and selection process as these are considered to be the portfolio
188
projects for the institute where the right number within the institute resources will be
selected.
The output of this second phase of evaluation will be the prioritization of the portfolio
projects (as shown below), which will facilitate the final decision to be under taken.
Research Division
Number of Prioritization
A
1
B
2
C
3
A
4
D
5
A
6
C
7
C
8
B
9
D
10
D
11
E
12
This prioritization list may exclude all the projects within one of the research program
division (as the result of E research division) or it would approve all the projects (as the
result of A research division) within any other research program division. This issue
could be overcome by several means and decisions to be taken within the institute’s
senior management, as there are several managerial factors within the institute which will
affect the final decision. An example of the research division (E) that all its proposed
projects is not been approved in the final stage and at the same time this division has very
few on-going projects that would be completed shortly. In this case, the decision maker
in the institute would waive the low ranked project #12 from E research division to be
replaced with the high ranked project #6 from A research division. Several other issues
could be treated in the same manner.
189
Model Structure and Evaluation Trial and Testing
This section will examine the evaluation and analysis models, a hypothetical proposed
project to be evaluated using the new proposed model of this research. The model
structure including criteria and methodology for both evaluation and analysis models is
set out below.
Evaluation Model Structure
An evaluation form is designed including the following perspectives:
•
Proposal Preparation and Formal Presentation,
•
Contribution to Objectives,
•
Technology Assessment and Appraisal/Methodology Approaches,
•
Expected Contributions,
•
Financial/Administrative Issues,
•
Manpower/ Human Resources Development.
Each of these perspectives will have a set of criteria, upon which various questions will
be put to the peer committee members. Scoring and rating techniques will be applied to
these questions. Specific and general comments will also be gathered in the same form.
Technology assessment and appraisal / methodology approaches perspective is taking an
example to demonstrate the evaluation model and mechanism as follows:
•
Rating of the scientific problem and the technology status addressed,
•
Rating of the methodology approach to the problem,
•
Rating of research team skills and qualifications,
•
Clustering of existing scientific and economic competence.
This will be done for all other perspectives shown above, as the evaluation form will
include criteria for all of them. The proposed project at the end of the evaluation model
process will have a final score along with a certain comments. This evaluation will be
190
done for all other proposed projects within each research programs individually. As an
example, if research division (A) proposed five projects and the final evaluation results
are as follows:
P1= 60%
P2= 75%
P3= 90%
P4= 55%
P5= 85%
For this case, three projects will be approved as long as they scored 65% or over, which
they are: P2, P3, and P5. These three projects and other projects from different research
divisions (B, C, D, E) that they have scored 65% and over will be included for the next
level of evaluation, which is the analysis model, where they will be prioritized and ranked
in a high score order.
Analysis Model Structure
This model is based on quantitative analysis for the evaluation model results and the
built-in standards criteria within the model. These criteria will be determined by the
research institute in order to fine tune the evaluated portfolio projects according to the
rules, regulations, objectives, resource limitations, and all other factors that affect the
final decision. Following are some examples of the built-in criteria:
•
Projects with duration of more than three years will be scored less than others.
•
Projects with budget exceeding $1,000,000 will be scored less than others.
•
Matrixing projects will be scored more than others.
•
Projects within the strategic and research programs will have a higher score.
•
Projects with more utilization national manpower will have a higher score.
•
In house projects will have a lower score than contractual projects.
191
The analysis model also will analyze the results of the evaluation model, to explain this
issue; the model will compare the total results of the perspectives rather than the total
score. If two projects from different research divisions have scored the same final results
which came to be 80%, the model will compare the results of the perspectives and if for
example the institute will have more influence in the expected contribution perspective,
then if project one scored 70% and project two scored 85% in the same perspective then
the model will prefer project two on project one.
The final output of this analysis model will be a list of high order ranking showing the
priorities of projects to be selected; furthermore, the decision maker could refer to the
peer members’ comments for final decision.
Assessment and Appraisal of the Proposed R&D Project Evaluation Model
Investment in R&D is a critical ingredient for economic growth, where the development
of an appropriate project evaluation and selection model is an important element in this
regard. R&D project evaluation and selection is an important task for organizations with
R&D project management. It is a complicated multi-stage decision making process,
which involves groups of decision makers. R&D project selection is a crucial task for
organizations like government funding agencies, universities, research institutes, and
technical intensive companies. It is a complicated decision making process with features
of multiple stages, multiple groups of decision makers, multiple and often-conflicting
objectives, and high risk and uncertainty in predicting the future success and impacts
(Ghasemzadeh and Archer, 2000).
Current research on R&D project evaluation and selection mainly focused on one single
decision model and its applications, but ignored the prioritisation of a set of projects.
This research has gone further by introducing a linked analysis model that could
extend the evaluation strategy, where more organizational aspects of the decision
making process towards the evaluation models have been introduced.
192
The proposed model aims at modelling the process of R&D project evaluation and
selection at organizational level. It also supports the whole life cycle of the decision
making process. Considerable efforts have been made in the past four decades to help
organizations make better decisions in R&D project selection (Martino, 1995; Henriksen
and Traynor, 1999). Most of them focus on building decision models and developing
decision making methods. Henriksen and Traynor (1999) review literature and classify
current decision models and methods into the following categories: unstructured peer
review, scoring, mathematical programming, economic model, decision analysis,
interactive method, artificial intelligence, and portfolio optimization.
To improve the usability of these decision models and methods, current research efforts
are to employ computer-aided decision support systems to support the R&D project
selection tasks (Bard, 1988; Liberatore, 1988a, 1988b, 1995; Iyigun, 1993; Kocaoglu, and
Iyigun, 1994; Ghasemzadeh and Archer, 2000). Also, the decision models proposed are
usually effective to facilitate single decision making task with limited participation of
decision makers. In practical applications, there is an urgent need to integrate the decision
models, methods, and decision support systems to facilitate the whole life cycle of the
project selection process.
This research argues that R&D project evaluation and selection is a decision making
task of organization-wide rather than any single task with limited procedures that
evaluate project by project and gives the approval chance as first come first go. It is very
different from the traditional single approach methodology where the major
enhancements of this research are as follows:
•
First, a new design of two linked models (Evaluation and Analysis
models) for R&D project evaluation and selection is proposed with the
goal to extend the current literature of R&D Project Evaluation field.
The linked model includes a subjective (qualitative) and an objective
(quantitative) approach in a linked-based modelling method that facilitates
193
R&D project evaluation and selection process, and a corresponding
architecture that supports and co-ordinates the work of decision making
groups.
•
Second, this research has eliminated most of the main shortcoming
obstacles of existing evaluation models by proposing new strategy
application of a real R&D project appraisal system. This overcomes
certain constraints such as, selecting the right projects, optimum utilization
of operational/ financial/ human resources, meeting the end user
requirements, contributing to the national developmental plan, integrating
the several phases of the projects stages, achieving the overall strategic
objectives of the organization, etc.
In conclusion, this research contributes a new architecture, which differentiates itself
from the existing ones in addressing the support to decision making and senior
management groups of organizations for R&D project evaluation and selection process.
Evaluation of Research Method and Approach
The research began with a review, diagnosis and analysis of the current R&D project
evaluation and selection methods. It reviewed the existing systems relevant to R&D
organizations in developed industrial nations and the relevant existing conditions and
circumstances in Kuwait as a part of the GCC region.
This effort highlighted the need for a comprehensive research work about S&T/ R&D
evaluation and selection systems in Kuwait.
Three major propositions/ studies (R&D Project Review Cycle, S&T Policies, R&D
Project Processing Management), were then formulated and addressed in detail in order
194
to understand the obstacles that lack the proper implementation of a rigid system that
could deal with the different phases of a proposed research project from the idea’s
generation all the way to the completion of the project.
This approach led to the investigation of an integrated evaluation and selection
model that not only considered the overall life cycle of a research project, but extended it
to include a prioritization of set of projects portfolio that considered the organizational
goals and objectives as well as the participation roles of all important interested parties.
The research presented also revealed a need to investigate the opportunities to propose
a responsive conceptual management practice for the appropriate implementation of the
suggested model. The studies included:
•
Understanding of the evaluation system of an R&D project through the
main phases of the project life cycle.
•
Understanding the main existing evaluation and selection models
through the literature reviewed.
•
Reviewing several approaches regarding the evaluation and selection
models in respect of R&D proposals and projects.
•
Spotting the main interested parties to be involved in the evaluation
committees besides investigating the needs of the end users.
•
Analyzing views of the key research staff in respect of the R&D project
evaluation and practice.
•
Understanding the decision making process and how it effects the R&D
project evaluation practice.
•
Recommending policy guidelines to support the implementation of the
proposed evaluation model.
•
Improvement of R&D project management process.
195
The methodology and the approach of the research led to the preliminary
conclusion that the research organizations in the developing countries are lacked behind
developed countries on the methodologies of evaluating and selecting the right research
projects. This is reflected by their research program implementations, as it is not quietly
geared to the objectives of the National Development Plans. Therefore, a well-defined
evaluation system should be considered to utilize the available research resources along
with allocated research funds in order to highly support the research activities by
selecting the right R&D projects that will be more focused toward the improvement of
the national development.
This research found that the evaluation approach should draw the right attention
scenarios about the best alternatives that could select the best projects and ease its
implementation. Among these scenarios several options should be considered in a way to
be tailored in order to meet the requirements of the R&D organizations in Kuwait and the
GCC region. In this regard, several issues should be answered prior to designing the
evaluation and selection system. It should consider which method ought to be utilized?
What perspectives and criteria to be introduced? How to formulate the peer committee
members? What sort of managerial practice to be involved with the proposed evaluation
methodology? How often the evaluation model should be modified and upgraded?
As the research investigated these issues through the research studies and propositions, it
was observed that Kuwait’s research organizations efforts could be further
enhanced in the field of R&D project evaluation and selection to include the
following issues:
Issue One: Evaluation Philosophy
Kuwait’s S&T organizations have approved low achievement and low efficiency in
this issue, where there is no linkage and harmony within the evaluation steps and
phases. The evaluation is mainly concerned with the early phase of the R&D
project, i.e. proposal phase. Unfortunately, other phases of the R&D project are
196
ignored. Thus the project should closely be monitored and evaluated within the
whole project’s life cycles as recommended in the research proposed model.
Issue Two: Model Design
The risks of failure of individual R&D projects for developing innovative
technological output and end results are always significant and can be very high
when a new research project is involved. A recent survey (Wadlow, 2000) indicates
that it now takes an average of 6.6 ideas to produce one successful new proposal,
while on average 50 - 60% of new proposed developmental projects fail. Using
strategy with both evaluation and analysis linked models, R&D project portfolios
can be designed and analyzed in a way that directly accounts for risks in terms of an
overall evaluation’s risk. An integrated model will provide a means for
developing customized project evaluation and selection criteria, which can
then support the portfolio design. An integrated model strategy is also
particularly applicable to situations where higher degrees of innovation and higher
degrees of growth are top ranked strategic objectives, which are mainly concerned
with R&D organizations.
This issue is considered to be the common practice of R&D project evaluation in
developed countries. This type of R&D evaluation is identified as potentially
improving existing evaluation practice. Even though there are several R&D projects
that are evaluated and selected in this manner, the final outputs and results are not
fully exploited. This is mainly due to the lack of integrated S&T/ R&D
infrastructure and the national developmental plan and interest. This supports the
fact that cooperation among R&D institutions, public firms, industrial and private
firms, and funding organizations is needed in order to strengthen the evaluation
philosophy. Hence, vital R&D results could not be achieved in isolation.
197
Issue Three: Managerial Practice
It is clear that an evaluation model applied to an R&D project needs to possess the
capability of a managerial practice that will ease its implementation process.
Kuwait’s S&T firms show that this issue is being neglected. A high tech device
needs to be operated by highly skilled and trained staff. Likewise, the evaluation
model needs a set of policies, rules, regulations, and procedures to go online with
the evaluation model in order to facilitate its usage and the utilization.
As a conclusion, Kuwait’s R&D organizations should improve their evaluation practice
in order to meet their local requirements by selecting the right projects at the right time.
This could be done by implementing the appropriate techniques that could be tailored and
well utilized to meet their unique situation. Joint cooperation among concerned
organizations should be encouraged to explore the management of Evaluation and
Selection, as it requires high cooperation among the economic, marketing, S&T aspects.
Through the fieldwork, this research shows substantial evidence that current R&D project
evaluation practice in Kuwait is geared toward approval of proposed projects with a high
score, evaluating them on a first come, first served basis, regardless of a prioritizing list
neither on the proposed projects nor on research programs. The evaluation criteria
overlooks the organization’s input regarding the acceptable project’s number and
duration, budget, manpower utilization, funding mechanisms, etc.
Therefore, this research proposes a unique approach, where two qualitative and
quantitative linked models are used to overcome the existing obstacles and
constraints as well as modify and improve the existing evaluation practice.
198
Implementation on KISR’s Existing Projects; Evaluation Model
The proposed evaluation model is implementing the scoring technique, which is
implemented in KISR with several enhanced modifications. These modifications include
the following:
•
Evaluation criteria to be set for each research program.
•
Designated group of peer committee members to be appointed for each
research program.
•
The committee should evaluate a set of projects in a certain time. For
example, each research program division should submit their projects twice
a year. This will leapfrog the first come, first approved project.
•
To set rules and regulations to activate the projects submission and
evaluation.
When the above mentioned modifications will be implemented, it is expected to have
more accurate results and to overcome the obstacles of the current evaluation practice.
As it is not within the research theme to evaluate projects through peer committees due to
the confidentiality of KISR’s projects and it is almost impossible to get authorization for
the arrangement of formulating evaluation committee to evaluate KISR’s projects, but it
is inevitable that the new modifications of this research will be utilized by KISR
management when it will be proposed by the authors.
Therefore, this section will demonstrate hypothetically the implementation of scoring
technique with the modifications set above. This will be an assessment to the evaluation
model in one hand and to understand the pros and cons of the evaluation model on the
other hand. For this reason, six existing evaluated projects were chosen to represent two
research programs within KISR (Appendix. II). Hypothetically, assuming these projects
are evaluated using the modified research model. As a result of this modification,
199
different final results will be achieved and compared to KISR’s evaluation results, as
shown in table 8.1.
•
Project EA021P was gained a total score of 52% as evaluated using KISR’s
model, where perspective A got the lowest score (46%), and perspective D got
the highest score (56%). When this project was re-evaluated using the
research model, all of the four perspectives were changed with new scores,
where perspectives A and C got higher scores, and perspectives B and D got
lower scores than what have been scored with KISR’s model, and got a higher
total score (60%) with the research model.
•
Project EC037K was gained a total score of 65% as evaluated using KISR’s
model, where perspective A got the highest score (70%), and perspective B
got the lowest score (60%). When this project was re-evaluated using the
research model, all of the four perspectives were changed with new scores,
where perspectives A and B got higher scores, and perspectives C and D got
lower scores than what have been scored with KISR’s model. Even though,
each model scored differently for each perspective, the total score remains the
same (65%).
•
Project EM018K was gained a total score of 71% as evaluated using KISR’s
model, where perspective C got the lowest score (70%), and perspective D got
the highest score (73%). When this project was re-evaluated using the
research model, all of the four perspectives were changed with new scores,
where perspectives A and C got lower scores, and perspectives B and D got
higher scores than what have been scored with KISR’s model, and got a lower
total score (65%) with the research model.
•
Project FA043K was gained a total score of 76% as evaluated using KISR’s
model, where perspective A got the highest score (78%), and perspective C
got the lowest score (74%). When this project was re-evaluated using the
research model, all of the four perspectives were changed with new scores,
they all got higher scores than what have been scored when evaluated with
KISR’s model, and got a higher total score (84%) with the research model.
200
Table 8.1 Results Comparison between KISR and Research Evaluation Models.
PROJECT
CODE
KISR EVALUATION RESULTS (%)
STUDY EVALUATION RESULTS (%)
A
B
C
D
TOTAL SCORE
A
B
C
D
TOTAL SCORE
EA021P
46
49
52
56
52
70
45
75
50
60
EC037K
70
60
63
66
65
77
70
56
58
65
EM018K
71
71
70
73
71
53
73
60
75
65
FA043K
78
76
74
76
76
90
79
80
85
84
FB046K
71
71
78
72
73
58
62
70
68
65
FM043K
74
75
78
69
73
60
60
85
84
73
201
•
Project FB046K was gained a total score of 73% as evaluated using KISR’s
model, where perspectives A and B got the lowest scores (71%), and
perspective C got the highest score (78%). When this project was
re-
evaluated using the research model, all of the four perspectives were changed
with new scores, they all got lower scores than what have been scored when
evaluated with KISR’s model, and got a lower total score (65%) with the
research model.
•
Project FM043K was gained a total score of 73% as evaluated using KISR’s
model, where perspective C got the highest score (78%), and perspective D
got the lowest score (69%). When this project was re-evaluated using the
research model, all of the four perspectives were changed with new scores,
where perspectives A and B got lower scores, and perspectives C and D got
higher scores than what have been scored with KISR’s model. Even though,
each model scored differently for each perspective, the total score remains the
same (73%).
This comparison indicates that KISR’s evaluation model has certain constraints as shown
in the above assessment. One example is that projects FM043K and EC037K got the
same total score from both models, even though the perspectives were scored differently.
Another example is shown clearly with the changes appeared in the total scores for each
model to the rest of the evaluated projects. Therefore, the research is proposing a linked
and a matrixing model, which is the analysis model that will close the gap between KISR
and the research model.
Implementation on KISR’S Existing Projects; Analysis Model
A simplified computerized program (Appendix.III) is designed to be the matrixed model
to the evaluation model and to re-prioritize the six projects. The idea behind this model is
to introduce built-in criteria that are mandatory guidelines and conditions within KISR’s
rules and regulations. For example, KISR is favouring contractual research projects with
173
securing external funding to recover budget deficiency, as well as favouring matrixed
projects within KISR’s research divisions for better manpower utilization. Other research
institutions may have different guidelines and conditions that will go online with their
rules, regulations, and objectives. For example, favouring a new high tech research in a
certain research area for their national economic growth.
Therefore, the research is considering the following built-in criteria that are mostly
required by KISR's R&D management:
1. Total Budget,
2. Project Duration,
3. Manpower Utilization,
4. Client Contribution,
5. Project Matrixing.
The first three built-in criteria (Budget, Duration, and Manpower) are integrated in the
model as shown in the triangle below.
Budget
Duration
Manpower
A certain score for each criterion of the triangle will be set accordingly to the
organizational guidelines. For example, if the management is favouring a highest score
for a budget of KD 50,000 per year, the related score will be highest, which is 100%.
Otherwise if the budget is below or over KD 50,000, then the score will be less than
100%. Scores for the other built-in criteria will be calculated taking into consideration the
management guidelines.
174
Furthermore, next subsections will explain in detail how the scores will be calculated and
set in the computerized model for each built-in criterion.
Total Budget
The total budget is considered to be one of the major criteria involved in the decision of
evaluating and selecting the research project. Investing in R&D activities is a very crucial
and sensitive issue, where the right allocated budget will enhance the financial planning
within the R&D institution and leads to maintain the financial resources. In KISR, as it is
ruled by government financial regulations, the management should be aware of allocating
the right amount to be spent on its R&D activities and at the same time it should not
exceed the overall budget of the institute to avoid any future financial debts. As
mentioned earlier, an assumption of KD 50, 000 per year is considered to be a reasonable
budget, where it will be scored 100%, other results for different budgets are manipulated
in the computerized analytical program.
Project Duration
Project Duration is playing a major rule in achieving the results of the R&D project,
considering the right timing to start and end the project will enhance the success of the
project. A short duration will not lead mostly to an accurate results, on the other hand,
along extended duration may not lead to a better utilization of resources and will delay
the results and the outcomes of the project, where mainly nowadays technology is being
enhanced enormously where it will reduce the chance of the competition with other R&D
institutions, therefore the model has set a duration of 24 months as a reasonable average
time, for example, a two years project will be scored 100%, where a manipulation
formula is being set to calculate the other results of different projects duration in the
computerized analytical program.
175
Manpower Utilization
Manpower utilization is an important criterion to be decided by the management in order
to control and utilize its human resources. For example, if the optimum manpower
utilization is set to be 12 man month (MM) per year, therefore a project with 12 MM per
year will get the highest score of 100%. Otherwise, if a project utilizing more or less than
12 MM per year, then the score will be manipulated in the computerized analytical
program.
Client Contribution
R&D institutes are depending on generating funds from public and private sectors, in
order to maintain their financial resources. Therefore, if a project is fully funded by an
external client, then it will get the highest score of 100%. Otherwise the score will be
given according to the amount of the client contribution. For example, if a project will be
funded by a client with a 50% of the total budget, then it will be scored 50%.
Project Matrixing
KISR is in favor of utilizing its all human resources effectively and simultaneously,
matrixing with other divisions, local firms, and regional/ international organizations will
give the chance for KISR not only to fully utilize its manpower, but also to gain on hand
experiences from other skilled research professionals. Therefore, this built-in criterion is
suggested to be scored as follows:
No Matrix
20%
Matrix w/ 1 Div.
30%
Matrix w/ more than 1 Div.
50%
Matrix w/ reg./Int. Org.
50%
,where the matrixing total score will not exceed 100%.
176
After demonstrating the five built-in criteria, the next task is to test the analysis model on
the six KISR’s research projects that had been evaluated previously (table 8.1) using both
KISR and research evaluation models.
The data of the six projects will be entered into the computerized analytical program and
the outputs are determined as follows:
•
Table 8.2 scored the six projects according to the values of the five built-in
criteria for the prioritization purpose, where project EM018K obtained the highest
score (57.13%), while project EA021P obtained the lowest score (30.67%).
Hence, the low scores indicate that these projects do not fully meet KISR’s
organizational objectives.
•
Table 8.3 illustrates the total results obtained for both evaluation and analysis
models. The analytical results were added to both KISR and research evaluation
models in order to obtain the total average score for each project. As it is shown
in the table, project EM018K obtained the highest average KISR total result
(64.065%), whereas project FA043K obtained the highest average research total
result (63.325%). Project EA021P obtained the lowest average result from KISR
(41.335%) and from the research (45.335%).
Therefore, matrixing the analysis model to the evaluation model has improved the
projects prioritization list as several organizational factors were accounted in the
evaluation technique.
•
Finally Table 8.4 illustrates the prioritization list of the six KISR’s projects in
relation to KISR’s evaluation results, research evaluation results, analytical
results, average KISR total results, and average research total results. Therefore,
implementing the proposed research model (evaluation and analysis models) is
lead to the optimum prioritization list that took into consideration the research
findings.
177
Table 8.2 Built-in Criteria Values and Projects Results.
207
Project
Code
Total
Budget
Project
Duration
Manpower
Utilization
Client
Contribution
Project
Matrixing
Analytical Total
Score (%)
EA021P
Score (%)
200,000
0
8
33.33
16
0
200,000
100
No Matrix
20
30.67
EC037K
Score (%)
99,720
99.72
24
100
37
45.83
0
0
No Matrix
20
53.11
EM018K
Score (%)
127,540
85.03
36
50
43
80.56
51,100
40.07
Matrix w/1Div
30
57.13
FA043K
Score (%)
37,220
37.22
24
100
8
33.33
8,440
22.68
No Matrix
20
42.56
FB046K
Score (%)
79,020
63.22
30
75
41
63.33
19,380
24.53
No Matrix
20
49.21
FM043K
Score (%)
88,860
88.86
24
100
34
58.33
0
0
No Matrix
20
53.44
Table 8.3 Evaluation Model Results.
Project
Code
208
EA021P
KISR
Evaluation Results
(A)
52
Research
Evaluation Results
(B)
60
Analytical
Results
(C)
30.67
Average KISR
Total Results
{(A+C)/2}
41.335
Average Research
Total Results
{(B+C)/2}
45.335
EC037K
65
65
53.11
59.55
59.55
EM018K
71
65
57.13
64.065
61.065
FA043K
76
84
42.65
59.325
63.325
FB046K
73
65
49.21
61.105
57.105
FM043K
73
73
53.44
63.22
63.22
Table 8.4 Evaluation Model Prioritization List.
209
KISR
Evaluation Results
Research
Evaluation Results
Analytical
Results
Average KISR
Total Results
Average Research
Total Results
FA043K
FA043K
EM018K
EM018K
FA043K
FB046K
FM043K
FM043K
FM043K
FM043K
FM043K
FB046K
EC037K
FB046K
EM018K
EM018K
EC037K
FB046K
EC037K
EC037K
EC037K
EM018K
FA043K
FA043K
FB046K
EA021P
EA021P
EA021P
EA021P
EA021P
REFERENCES
Amidon Rogers, D.M. The Challenge of Fifth Generation R&D. Research-Technology
Management, Vol. 39, No. 4, PP. 33-41, 1996.
Anderton, A. Economic for GCSE. Second Edition, Collins Educational, Scotland, 1993.
Archer, N.P. and Ghasemzadeh, F. ‘An Integrated Framework for Project Portfolio
Selection.’ International Journal of Project Management, Vol. 17, No. 4, 1999.
ARCS: Austrian Research Centers, Intellectual Capital Report, Seibersdoff, Austria 1999.
Arsham, H. ‘Probabilistic Modeling: Decision Analysis, Higher Level Systems Ltd.’
(http://www.assistum.com/2002/solutions/knowledge_management/knowledge_manage
ment.htm). 2002.
Arsham, H. ‘Applied Management Science: Making Good Strategic Decisions.’
University of Baltimore, USA, 2003.
Bacarini, D. ‘The Logic Framework Method for Defining Project Success.’ Project
Management Journal 30, No. 4, PP. 25-32, December 1999.
Bach, Laurent ‘The Assessment of the Socioeconomic Impact of Public R&D in France:
Perspectives on Practice and Research,’ BETA method-results, Presentation of results.
BETA- L. Pasteur University, Strasbourg, France, Tokyo Conference, April 2002.
Baker, N., Green, S., and Bean, A. ‘Why R&D Projects Succeed or Fail.’ Research
Management, PP. 29-34, November-December 1986.
Baker, Michael and Hart, Susan. Product Strategy and Management: Hemel Hempstead,
Prentice Hall. 1999.
210
Bard, J.F. ‘An Interactive Approach to R&D Project Selection and Termination,’ IEEE
Transactions on Engineering Management, 1988.
Bellary, A. and Murthy, D.N.P. New Product Development Process and Total Quality
Management, Processings of Portland International Conference on Management of
Engineering and Technology, Portland, Oregon, USA, 1999.
Bobis, A.H.; Cooke, T.F.; and Paden, J.H. ‘A Funds Allocation Method to Improve the
Odds for Research Success.’ Research Management, Vol. 14, No. 2, PP. 34-39, March,
1971.
Bordley, R.F. R&D Project Selection Versus R&D Project Generation. IEEE
Transactions on Engineering Management, Vol. 45, No. 4, 1998.
Boskin, M. and Lau, L. Capital, Technology, and Economic Growth. In: Technology and
Wealth of Nations. Editors: N. Rosenberg and D. Mowery. Stansford University Press,
1992.
Boyer, S.M. Total Quality Management and New Product Development, Total Quality
Management, Vol. 2, No. 3, 1991.
Brenner, M. ‘Practical R&D Project Prioritization.’ Research Technology Management,
Vol. 37, No. 5, PP. 38-42, 1994.
Cabinet Meeting. Fundamental Principals of Science and Technology Policy. Japan,
April 1992.
Carbonell-Foulquie, P., Manuueara-Aleman, J., L., Rodriquez-Escudero, A., I. Criteria
employed for go/no-go decisions when developing successful highly innovative products.
Industrial Marketing Management, June 2003.
CCF: Country Cooperation Framework. United Nations Development Programmes,
2000.
211
Chiesa, V. and Manzini, R. Organizing for Technological Collaborations: a Managerial
Perspective. R&D Management, Vol. 28, No. 3, PP. 199-212, 1998.
CONSAD Research Corporation. The Application of a Macroeconomic Interindustry
Model to an ATP Joint Venture Project; A Case Study of the Development of Advanced
Technologies and Systems for Controlling Dimensional Variation in Automobile Body
Manufacturing (the 2 millimeter Project), (Report prepared for the National Institute of
Standards and Technology), draft report, July 1996.
Cooper, R. G. and Brentani, Ulrike de. Criteria for Screening New Industrial Products.
Industrial Marketing Management Vol. 13, PP. 149-156, 1984.
Cooper, R.G. ‘Winning at New Products: Accelerating the Process from Idea to Launch,’
2nd edition, Reading, USA: Addison-Wesley, 1993.
Cooper, R.G. ‘Third Generation New Product Processes.’ Journal of Product Innovation
Management, Vol. 11, No. 3, 1994.
Cooper, R.G. and Kleinschmidt, E.J. Benchmarking the Firm's Critical Success Factors in
New Product Development. Journal of Product Innovation Management, Vol. 12, No. 5,
PP. 374-391,1995.
Cooper, R.G., Edgett, S. J. and Kleinschmidt, E. J. ‘Portfolio Management for New
Products,’ Reading, USA: Addison-Wesley, 1998.
Cooper R. G., Edgett, S. J., and Kleinschmidt, E. J. Portfolio Management for New
Products Second Edition. Cambridge MA, Perseus Publishing. 2001.
Cooper, Robert G. Winning at New Products 3rd Edition, Cambridge MA, Perseus
Publishing. 2001.
Council for Industrial Structure, MITI; International Trade and Industrial Policy in the
1990’s, Toward creating human value in the global age. Tokyo, 1990.
212
Crowley, Steven J., P.E. Communications Engineering and Economics, Failed R&D
Project Identification (a working paper) Rev.1.4, December 1999.
Dean, B.V.
Project Evaluation: Methods and Procedures. New York: American
Management Association, 1970.
Dingle. Project Management Orientation for Decision Makers. Arnold, London. 1997.
Doyle, Peter. Value-Based Marketing: Marketing Strategies for Corporate Growth and
Shareholder Value. Chichester: John Wiley & Sons Ltd. 2000.
EC: European Commission’s, SPRINT Programme of Technology Transfer,CU, 1993/94.
EC: European Commission. The Sixth Framework Programme, Guidelines on Proposal
Evaluation and Selection Procedures, 2003.
Economic Brief, Kuwait’s Populations and Labor Force, Economic Research Division
NBK, Vol. 5, No. 6, Kuwait, February 16, 2005.
Economic Brief, Oil Market and Budget Development, Economic Research Division,
NBK, Vol. 5, No. 34, Kuwait, October 10, 2005.
EHR/NSF. User-Friendly Handbook for Project Evaluation: Science, Mathematics,
Engineering and Technology Education. The National Science Foundation. Publications
of the Directorate for Education and Human Resources. 1993.
Ellis, L. Evaluation of R&D Processes: Effectiveness through measurements. Artech
House Boston & London, 1997.
Ernst, Holger. Success Factors of New Product Development: A Review of the Empirical
Literature. International Journal of Management Reviews, Vol. 4, No. 1, PP. 1-40, 2002.
213
EUREKA and COST. A process for continuous evaluation was introduced by the
EUREKA initiative and is being considered by COST and the Framework Programme.
1998.
Evaluating Defense Department Research, background paper, prepared by the Office of
Technology Assessment, International Security and Commerce Program, June 1990.
Fisher, J. and Heywood, C. ‘Total Quality Management of Canadian R&D Activities,
CMA Magazine.’ Vol. 66, No. 7, 1992.
Freeman, C. Technology Policy and Economic Performance, Lessons from Japan,
London, New York, 1987.
Georghiou, L. and Meyer-Krahmer, F. ‘Evaluation of Socio-Economic Effects of
European Community R&D Programmes in the SPEAR Network.’ Vol.2, No.1, PP.5-15,
England, April 1992.
Ghasemzadeh, F. and Archer, N.P. ‘Project Portfolio Selection through Decision
Support.’ Decision Support System, 2000.
Ghosn,
A.
‘Preparation
of
R&D
Proposals:
Fundamental
Guidelines
and
Recommendations.’ KISR, Kuwait, 1997.
Grace Report, Task Force Report on Research and Development, Washington, DC.
President’s Private Sector Survey on Cost Controls, 1983.
Griffin, A. PDMA Research on New Product Development Practices: Updating Trends
and Benchmarking best Practices. Journal of Product Innovation Management, Vol.14,
No. 6, PP. 429-459, 1997.
Gupta, A.K. and Wilemon, D. Changing Patterns in Industrial R&D Management.
Journal of Product Innovation Management, Vol.13, No.6, PP.497-511, 1996.
Hart, Susan. New Product Development: A Reader, London, The Dryden Press. 1996.
214
Hellings, J. Capital Budgeting in the Real World. Management Accounting, 1985.
Henriksen, A.D. and Traynor, A. J. ‘Practical R&D Project Selection Scoring Tool.’
IEEE Transactions on Engineering Management, 1999.
Hertz, D.B. ‘Risk Analysis in Capital Investment.’ Harvard Business Review, Vol. 42,
No.1, PP. 95-106, January 1964.
Iansiti, M. and West, J. Technology Integration: Turning Great Research into Great
Products. Harvard Business Review, Vol. 75, No.3, PP. 69-79, 1997.
Iyigun, M.G. ‘A Decision Support System for R&D Project Selection and Resource
Allocation under Uncertainty.’ Project Management, 1993.
Jaffe, Adam B. Economic Analysis of Research Spillovers; Implications for the
Advanced Technology Program, (Report prepared for the National Institute of Standards
and Technology by Prof. Jaffe of Brandeis University and the National Bureau of
Economic Research), 1996.
Japan Center for Economic Research; Five-Year Economic Forecast. Tokyo, 1990.
Japan Institute of International Affairs; White papers of Japan 1989-90, annual abstracts
of official reports and statistics of the Japanese government, P.12, Tokyo, 1991.
Joint Committee on Standards for Educational Evaluation. Standards for Evaluation of
Educational Programs, Projects, and Materials. New York, NY: McGraw-Hill, 1981.
Kiella, M.L. and Golhar, D.Y. ‘Total Quality Management in an R&D Environment,’
International Journal of Operations & Production Management, Vol. 17, No. 2, 1997.
Klein, D.A. (ed.). The Strategic Management of Intellectual Capital. Boston, MA:
Butterworth-Heinemann. 1998.
215
Kocaoglu, D.F. and Iyigun, M.G. ‘Strategic R&D Project Selection and Resource
Allocation with a Decision Support System Application.’ In: Proc. IEEE International
Engineering Management Conference, 1994.
Kostoff, R.N. and Schaller, R.R.
‘Science and Technology Roadmaps.’ IEEE
Transactions of Engineering Management, Vol. 48, No. 2, PP. 132-143, May 2001.
Kumar, V., Persaud, A., and Kumar, U. ‘To terminate or not an ongoing R&D project: a
managerial dilemma.’ IEEE Transaction on Engineering Management, Vol. 43, No. 3,
PP. 273-284, 1996.
Laidlaw, F. ‘Acceleration of Technology Development by the Advanced Technology
Program: The Experience of 28 Projects Funded in 1991,’ National Institute of Standards
and Technology, draft report, October 1996.
Lapiere R.A. A Theory of Social Control. New York, 1954.
Leifer, Richard, Peter, Lois S., and O'connor, Gina C. Radical Innovation: How Mature
Companies Can Outsmart Upstarts. Boston Ma. : Harvard Business School Press. 2000.
Lemert, E.M. Human Deviance: Social Problems and Social Control. New York.1967.
Lewis, R. Membership and Management of a 'Virtual' team: The Perspective of a
Research Manager. R&D Management, Vol. 28, No. 1, PP. 5-12, 1998.
Liberatore M. J. and Titus G. J. The Practice of Management Science in R&D Project
Management. Management Science. August. 1983.
Liberatore, M. ‘A Decision Support System Linking Research and Development Project
Selection with Business Strategy,’ Project Management Journal, Vol. 19, No. 5, PP. 1421, 1988a.
Liberatore, M. ‘An Expert System for R&D Project Selection.’ Math Computer
Modeling, 1988b.
216
Liberatore, M. and Stylianou, A. ‘Expert Support System for New Product Development
Decision-Making: A Modeling Framework and Applications,’ Management Science, Vol.
41, No. 8, PP. 1296-1316, 1995.
Lim, C.S. and Zain Mohamed, M. ‘Criteria of Project Success: An Exploratory ReExamination,’ International Journal of Project Management, Vol. 17, No. 4, PP. 243248, 1999.
Link, A.N. ‘Evaluating Public Sector Research and Development,’ Praeger. London,
1996a.
Link, A.N. Economic Analysis of the Printed Wiring Board Research Joint Venture
(Report prepared for the National Institute of Standards and Technology by Prof. Link of
the University of North Carolina at Greensboro), September 1996b.
LINK, Building Innovative Partnerships, 1999.
Loch, C. and Bode-Greuel, K. Expansion Options: Evaluating Strategic Opportunities
from Research Projects. 2000.
Lockyer, K. and Gordon, J. Project Management and Project Network Techniques,
(Sixth edition), Pitman, London, 1996.
Lopes, M.D.S. and Flavell, R. Project Appraisal- A Framework to Assess Non-Financial
Aspects of Projects during the Project Life Cycle. International of Project Management,
Vol. 16, No. 4, 1998.
Lovett, J.R. ‘Doing the Right Things, All the Time,’ Research Technology Management,
Vol. 35, No. 5, 1992.
Man and Development, The Cooperation Council for the Arab States of the Gulf
Secretarial General. 1995.
217
Mansfield, E. Estimating Social and Private Returns from Innovations Based on the
Advanced Technology Program: Problems and Opportunities (Report prepared for the
National Institute of Standards and Technology by Prof. Mansfield of the University of
Pennsylvania), 1996.
Markides, C. ‘Strategic Innovation: The Leader's Delimma,’ European Management
Journal, Vol. 17, No. 1, PP. 33-36, February 1999.
Markowitz, H. Portfolio Selection. New York: John Wiley & Sons Inc., 1960.
Martino, J.P. ‘R&D Project Selection.’ New York, USA: John Wiley & Sons, 1995.
Matheson, David and Jim. ‘The Smart Organization: Creating Value through Strategic
R&D.’ Harvard Business School press, Boston, Massachusetts, 1998.
Matt, Mireille and Petit, Serge, ‘Evaluation of R&D Programmes: General Guidelines
and Evolutionary Methodologies,’ ESSID, 2001.
Miller, W.L. and Morris, L. ‘Fourth Generation R&D: Managing Knowledge,
Technology, and Innovation.’ John Wiley & Sons, 1999.
Moore, J.R. and Baker, N.R. ‘An Analytical approach to Scoring Model Design:
Application to Research and Development Project Selection.’ IEEE Transactions on
Engineering Management, Vol. EM-16, No. 3, PP. 90-98, August 1969.
MOP: Ministry of Planning. Kuwait Development Plan 2001/2002- 2005/2006, Kuwait,
2000.
Mullins, J.W. and Sutherland, D.J. New Product Development in Rapidly Changing
Markets: An Exploratory Study. Journal of Product Innovation Management, Vol. 15,
No. 3, 1998.
National Academy of Engineering, Preparing in a Global Economy-Mastering a New
Role, National Academy Press, Washington, DC., 1993.
218
National Critical Technologies Report, Appendix B, Technology Selection and Analysis,
President's Committee of Advisors on Science and Technology, VA, USA. 1995.
Nedeva, M., Cunningham, P., and COST: Toward a System for Continuous Evaluation
and Monitoring, Report to the COST Secretariat. October 1998.
NIST: National Institute of Standards and Technology. The Advanced Technology
Program: A Progress Report on the Impacts of an Industry-Government Technology
Partnership (NIST Report to Congress), 1996.
NISTEP: National Institute of Science and Technology Policy. ‘Social Awareness to
Science and Technology.’ USA, 1989.
NISTEP: National Institute of Science and Technology Policy. ‘Comparison between
Japan, the USA, and Europe in Social Awareness to Science and Technology.’ Report
No. 17, 1992.
Nixon, B. Research and Development Performance Measurement: A Case Study.
Management Accounting Research, Vol. 9, No. 3, PP. 329-355, 1998.
NNH Enterprise, Program Evaluation and Review Technique (PERT), (or Problems
Eventually Resolve Themselves!), Management and Information Systems. Milford, New
T
Hampshire, U.S.A., 1996.
NSF: National Scientific Foundation. Report on the National Scientific Foundation Merit
Review System. 1996.
NSF: National Scientific Foundation. Methods of Selection and Evaluation Criteria.
Washington, DC. 1997.
OECD: Organization for Economic Cooperation and Development. ‘The Evaluation of
Scientific Research: Selected Experiences.’ Paris, 1997.
219
OECD: Organization for Economic Cooperation and Development. Economic Survey of
Japan 2005: Key economic challenges facing Japan, Paris, France, 2005.
O’Meara, J. T. Selecting Profitable Products. Harvard Business Review. Jan/Feb, 83-89.
1961.
Ozataly, Savas, ‘Project Management Network Models.’ MGT 680- Project
Management- Part I, Widener University PA, USA, 1992.
Parker, K. ‘Structuring Technology Management Projects.’ Economic and Social
Commission for Western Asia. Beirut, Sept. 1999.
Patino, H. ‘Applying Total Quality to R&D at Coors Brewing Company,’ Research
Technology Management, Vol. 40, No. 5, 1997.
Piippo, P. and Tuominen, M. ‘Promoting Innovation Management by Decision Support
System: Facilitating New Products’ Relevance to the Corporate Objectives,’ in
Consulting in Innovation, Jurgen Allesch, Ed. Amsterdam, Holland: Elsevier Science
Publishers, PP. 267-292, 1990.
Pike, R. Capital Budgeting in the 80’s. In Occasional Paper ICMA, Institute of Cost and
Management Accountants. London, 1982.
Pittiglio, Rabin, Todd and McGarth. Product Development Leadership for TechnologyBased Companies: Measurement and Management A Prelude to Action. Weston, MA:
Pittiglio, Rabin, Todd & McGarth. 1995.
Qasem, S. Science and Technology Situation and its Environment in the Arab World.
Amman ,Arab Thought Forum. 1995.
Raelin, J. and Balachandra, R. ‘R&D Project Termination in High Tech Industries.’ IEEE
Transactions on Engineering Management, Vol. EM-32, No. 1, PP. 16-23, 1985.
220
Robb, W.L. How Good is Your Research? Research-Technology Management, Vol. 34,
No. 2, PP. 16-21, 1991.
Robinson, Lesley. Greater than the Sum of its Parts. Knowledge Management, Vol. 5,
Issue 10, July 2002.
Robles, E. ‘Using the REMI Model to Estimate the National Economic Impacts of the 2
millimeter Project,’ draft report, October 1996.
Rochford, Linda. Generating and screening new product ideas. In Hart, Susan (Ed.)
(1996): New Product Development A Reader (PP. 254-267). London, The Dryden Press,
1991.
Rosen, E.M. and Souder, W.E. ‘A Method for Allocating R&D Expenditures.’ IEEE
Transactions on Engineering Management, Vol. EM-12, No. 4, PP. 87-93, September
1965.
Rossi, P.H. and Freeman, H.E. ‘Evaluation – A Systematic Approach (5th Edition).’
Newbury Park, CA: Sage, 1993.
Roussel, P. A.; Saad, K.N.; and Erickson, T.J. ‘Third Generation R&D.’ Managing the
Link to Corporate Strategy. Harvard Business School Press, Boston, Mass, 1991.
Ruegg, R.T. ‘Economic Evaluation Methods.’ Handbook on Energy Economics. The
Advanced Technology Program’s Evaluation Plan and Progress. National Institute of
Standards and Technology, 1996.
Ruegg, R.T. Journal of Technology Transfer. The Advanced Technology Program, Its
Evaluation Plan, and Progress Implementation. Advanced Technology Program, National
Institute of Standards and Technology, Technology Administration, U.S. Department of
Commerce Gaithersburg, MD 20899. Vol.23, No.2, PP. 5-9, 1997.
Sand, F. and Nedeva, M. ‘The EUREKA Continuous and Systematic Evaluation: An
Assessment of the Socio-Economic Impact of the International Support Given by the
221
EUREKA initiative to Industrial R&D Cooperation,’ in proceedings of the APEC
Symposium on the Evaluation of S&T Programmes among APEC Member Economies,
National Centre for Science and Technology Evaluation, Ministry of Science and
Technology of China, 1998.
Schmidt, R. L. and Freeland, J. R. ‘Recent Progress in Modeling R&D Project –
Selection Processes.’ IEEE Transactions on Engineering Management, Vol. 39, No. 2,
PP. 189-201, May 1992.
Sharpe, W.F. ‘A Simplified Model for Portfolio Analysis.’ Management Science, Vol.9,
No.1, PP. 227-293, January 1963.
Smyly, B. Framework for Improved Development Appraisals in Developing Countries.
Journal of Property Research, Vol. 10, PP. 135-145, 1993.
Souder, W.E. ‘Selecting and Staffing R&D Projects via Operations Research.’ Chemical
Engineering Progress, Vol. 63, No. 11, PP. 27-37, November 1967.
Souder, W.E. ‘Analytical Effectiveness of Mathematical Programming Models for
Project Selection.’ Management Science, Vol. 19, No. 8, PP. 907-923, April 1973a.
Souder, W.E. ‘Utility and Perceived Acceptability of R&D Project Selection Models.’
Management Science, Vol. 19, No. 12, PP. 1384-1394, August 1973b.
Souder, W.E. ‘Achieving Organizational Consensus With Respect to R&D Project
Selection Criteria.’ Management Science, Vol. 21, No. 6, PP. 669-691, February 1975a.
Souder, W.E. ‘Field Studies with a Q-Sorts/Nominal Group Process for Selecting R&D
Projects.’ Research Policy, Vol. 5, No. 4, PP. 172-188, 1975b.
Souder, W.E. ‘Effectiveness of Nominal and Interacting Group Decision Processes for
Integrating R&D and Marketing.’ Management Science, Vol. 23, No. 6, PP. 595-605,
1977.
222
Souder, W.E. ‘A System for Using R&D Project Evaluation Methods.’ Research
Management, Vol. 21, No. 5, PP. 29-37, September 1978a.
Souder, W.E. ‘Project Selection, Planning and Control.’ In handbook of Operations
Research: Models and Applications, New York, Chapter 10, Vol. 2, PP. 301-344, 1978b.
Souder, W. E.
‘Management Decision Methods for Managers of Engineering and
Research,’ (Chapter Nine, Project Screening, Evaluation and Selection). New York,
PP.137-190, 1980.
Souder, W.E. ‘Project Selection and Economic Appraisal.’ New York, PP. 60-85, 1984.
Souder, W.E. and Mandakovic, T.
‘R&D Project Selection Models,’ Research
Management, Vol. 29, No. 4, PP. 36-42, July-August, 1986.
Spender, J.-C. The Three Dimensional Model of the Advanced Technology Program,
NIST Report in draft, 1996.
Steele, Lowell, W. ‘New Paradigms for R&D,’ Research –Technology Management,
1991.
Steven, C. Wheelwright and Kim, B. Clark. ‘Revolutionizing Product Development,
Quantum Leaps in Speed, Efficiency, and Quality.’ New York: Free Press, 1992.
Szakonyi, R. ‘How to successfully keep R&D projects on Track!’ Lomond, USA, 1988.
Tassey, G. ‘Technology Infrastructure and Competitive Position.’ Norwell, MA: Kluwer
Publications, 1992.
Trott, Paul. Innovation Management and New Product Development Second Edition:
Harlow. Pearson Education Limited, 2002.
Twiss, B. ‘Management of Technology Innovation.’ Longman, 1982.
223
Twiss, B. ‘Management Technological Innovation’ 3rd ed, London, Great Britain, Pitman
Publishing, 1987.
Vonortas, N.S. and Hertzfeld, H.R. R&D Project Selection in the Public Sector, Journal
of Policy Analysis and Management, Vol. 17, No. 4, PP. 621-638, 1995.
Vonortas, N.S. ‘Prioritizing Long-Term, Strategic R&D Projects in the Public Sector.’
Center for International Science and Technology Policy, George Washington University,
Washington, DC, 1998.
Wadlow, David. The Role of risk in the design, evaluation and management of corporate
R&D project portfolios for new products, Sensors Research Consulting, Inc. 5 Lyons
Mall, Ste.800 Basking Ridge, NJ 07920, 2000.
Wood, R. Industrial Research Institute's R&D Trends Forecast for 1998. ResearchTechnology Management, Vol. 41, No. 1, PP. 16-20, 1998.
224