Evaluable

FUNDAMENTAL RESEARCH
GRANT APPLICATION &
EVALUATION
24 April 2015
Muhammad Fauzi Mohd Zain
CRIM, UKM
1
Introduction
FRGS
TRGS
Fundamental Research Grant Scheme
Trans-Disciplinary Research Grant Scheme
- Penyelidikan yang menghasilkan teori, konsep,
dan idea baru
- Menjawab persoalan “WHY?” dan “HOW?”.
- Hasil penemuan boleh dikembangkan kepada
projek bersifat gunaan
- Menjawab persoalan “WHAT?” dan “WHERE?”.
LRGS
PRGS
Long-term Research Grant Scheme
Prototype Development Research Grant Scheme
- Penyelidikan fundamental yang memerlukan
tempoh pelaksanaan melebihi 3 tahun
- Penghasilan produk penyelidikan, tetapi belum
sampai ke peringkat pengkomersilan
1
Introduction
Fundamental Research Grant Scheme (FRGS)
-permohonan projek penyelidikan melibatkan satu (1) bidang penyelidikan.
-siling peruntukan adalah RM250,000.00 sepanjang tempoh penyelidikan.
Trans Disciplinary Research Grant Scheme (TRGS)
Program
Penyelidikan
Fundamental
2015
- permohonan projek penyelidikan melibatkan sekurang-kurangnya tiga (3) bidang
penyelidikan (trans disciplinary) melangkaui jabatan / fakulti dari institusi yang sama.
- sasaran hasil penyelidikan yang sama.
- siling peruntukan adalah RM1,500,000.00 sepanjang tempoh penyelidikan.
Research Acculturation Grant Scheme (RAGS)
-dana tunas yang bertujuan untuk membudayakan penyelidikan di kalangan penyelidik
muda di IPTA bukan RU sebagai persediaan untuk membangunkan prestasi
penyelidikan supaya dapat berdaya saing di peringkat kebangsaan dan antarabangsa.
Research Acculturation Collaborative Effort (RACE)*
Membantu Non-RU meningkatkan budaya penyelidikan dan seterusnya dapat
meningkatkan output hasil penyelidikan. Usaha ini dapat mempercepatkan Non-RU
untuk mencapai tahap setanding dengan RU yang lain.
1
Introduction
BIL
PERKARA
FRGS
TRGS
RAGS
LRGS
RACE*
1
Siling
Permohonan
RM250,000 RM1,500,000
RM50,000 –
RM80,000
RM3
RM50,000 –
RM500,000
juta/tahun RM80,000
2
Tempoh
Penyelidikan
1 hingga 3
tahun
1 hingga 3
tahun
1 hingga 2
tahun
3 hingga 5
tahun
3
KPI
• 1 PhD
• 3 papers in
index link
journal (2
years)
•4 PhD or 8
sarjana
• 8 jurnal
terindeks (2
Q1)
• 1 paten
• 1 PhD
• 3 papers in
index link
journal (2
years)
2 tahun
PRGS
2 tahun
• 10 PhD (3 • 1 PhD
•1 IP/project
years)
• 3 papers in
• 50 papers index journal
(3 years)
• 1 IP (filed)
• 3 IP (per
program) number of
researchers
with Citation
Index of 100)
CARTA ALIR PROSES KERJA RMC DI PERINGKAT UNIVERSITI
Pemberitahuan pembukaan geran dan garis panduan kepada
penyelidik
RMC terima proposal
Pelantikan Panel Penilai peringkat Universiti mengikut Kluster
Penilaian dilakukan
Ketua Penyelidik membuat pembetulan dan penambahbaikan
mengikut saranan Panel Penilai
CARTA ALIR KERJA RMC DI PERINGKAT UNIVERSITI
Semakan semula proposal oleh pihak yang dipertanggungjawabkan
oleh IPT
Proposal yang muktamad dan lengkap diperakui oleh RMC
RMC sediakan ‘masterlist’ proposal mengikut kod rujukan KPT
Penilaian dan perakuan oleh Panel KPT
Borang permohonan yang lengkap dihantar ke KPT
Senarai Dokumen Yang Perlu Disemak
1. Proposal asal
2. Borang penilaian yang telah dilengkapkan oleh Penilai
3. Ringkasan pembetulan dan penambaikan yang dilakukan
(disediakan oleh Ketua Penyelidik)
4. Bukti pengesahan dan perakuan pembetulan
5. Proposal muktamad
1
Introduction
Characteristics of good governance
www.unescap.org/pdd/prs/ProjectActivities/Ongoing/gg/governance.asp
2
Fundamental Research-FRGS
Fundamental Research (FRGS)
Basic
research
Pure research
Fundamental research
generates new
knowledge and
technologies to deal
with unresolved
problems.
2
Fundamental Research-ERGS
Exploratory Research (ERGS)
Relies on secondary research
To gather preliminary information
Research intended only to provide greater
familiarity with the phenomena that researcher
wants to investigate so that he can formulate
more precise research questions and perhaps
develop hypotheses. Such studies can be
essential when researcher is investigating new
phenomena or phenomena that have not been
studied before.
2
F. Research-Cutting Edge
2
F. Research-High Impact Journal
2
F. Research-Int. Collaboration
4
Grant Application
Most common reasons for grant writers
(GWs) not receiving funds
1. Not new or lack of original ideas
2. Diffuse, superficial or unfocused research plan
3. Lack of knowledge of published relevant work
4. Lack of experience in the essential methodology
5. Uncertainty concerning the future directions
6. Questionable reasoning in experimental approach
7. Unacceptable scientific rationale
8. Unrealistically large amount of work
9. Insufficient experimental detail
10. Uncritical approach
4
Grant Application
Quality
of the Proposal
The measures for a good quality proposal are:
Informative title;
Convincing executive summary;
Clear problem statement and objective;
Scientific background and rationale;
Good selection of research methods;
Ethical considerations; and
Realistic budget and schedule.
4
Grant Application
Characteristics of a good proposal:
A document that is neat, well organized and easy to
read;
Responsiveness to the program need, with specific
references showing how the proposed project will
achieve program goals and objectives;
Fresh insight into an important problem;
Writing that communicates the enthusiasm and
commitment of the researcher;
Evidence that the PI knows the field;
Convincing preliminary data; and
A feasible work plan that is supported by an
appropriate budget.
5
Evaluation: Evaluator
Internal evaluation
Department/Faculty/Institute level
University level
Internal evaluators improve the quality of grant
submissions
How to
“improve”
5
Evaluation: Evaluator
External evaluation
Sponsor level
Many constraints
Number of proposals, amount of
money available, etc.
To find mistakes/weaknesses, etc.
(quality…quality…quality)
To find ways on…
How to “reject”
5
Evaluation: Evaluator
Selecting An Evaluator
The selection of an evaluator for F/TRGS project based on
the following criteria:
Credentials/Reputation- The evaluator affiliated
with an academic institution which has an
experience evaluated research grants, particularly
fundamental focused researches.
Education- The evaluator have a certificate or
doctoral degree related to evaluation program?
Experience- The evaluator have formal or informal
experience with evaluation in fundamental
research grant.
Sensitivity- The evaluator have experience working
with the target population.
Integrity- The evaluator aware of any conflicts of
interest during evaluating the project.
5
Evaluation: Evaluator
Selecting An Evaluator
The selection of an evaluator for F/TRGS project based on
the following criteria:
Communication Skills- The evaluator able to
explain technical concepts in
understandable language and demonstrate
clear verbal and written expression.
Availability- The evaluator available to
meet with the program timelines and be
flexible if timelines need to be modified.
Contract/Scope of Service-… KPM???
Cost- ……KPM???
5
Evaluation: Evaluator
Conflict of Interest
Evaluators are required to declare any personal interests
according to the following criteria. Evaluators must disqualify
themselves if they can in any way benefit from the approval or
rejection of the proposal. They must also disqualify themselves
in the following circumstances:
Evaluators have close collaboration with the GW (e.g. have coauthored and published an article with the GW during the past three
years, have been involved in the preparation of the application, or are
involved in the publication or application of the results)
Evaluators have been a superior, subordinate or instructor of the
GW during the past three years
Evaluators are currently applying for the same post as the GW
Evaluators are currently applying for funding from the same funding
instrument on the same research area.
The GW is a close person to evaluator.
5
Evaluation: Process
Fundamental Criteria of Evaluation
CARE: Are GWs tackling an important problem? If they can make progress on it,
will anyone care?
NOW: Why now? If this problem is so important, why has it not been addressed
before?
IDEAS: Do GWs have concrete ideas for starting an attack on the problem and
a vision for proceeding further? Is initial progress likely and subsequent progress
possible?
RESULTS: Do GWs have some preliminary results? Do they demonstrate a good
understanding of the problem and the methods needed attack it further?
PLAN: Do GWs have sensible plans and methods (e.g., concrete steps and ways of
decoupling risks)?
CAN-DO: Why these GWs? Why are their qualifications and infrastructure
appropriate?
LEGAL: Have GWs followed the rules of the solicitation (e.g., compelling broader
impacts for FRGS/TRGS)?
.
5
Evaluation: Process
Main Criteria of Evaluation
All proposals must be evaluated based on the criteria outlined below.
Relevance
Degree to which the proposal was related to KPM (F/T/L/PRGS)
Degree to which the proposed research results would be relevant to
policy-makers
Extent and appropriateness of dissemination plans
Degree to which the proposed research would facilitate the goals of KPM
Degree to which the proposed research represented an innovative
approach and develop new knowledge in the field of engineering or other
disciplines
Research Team
Quality of the research team and their research
The applicant and the research team are among the leading in their field.
The publications are at a remarkable international level. The articles are
published in the best peer-reviewed journals, or proceedings, which are
indexed in the leading databases of the field.
The impact of the applicant (number of citation; the scientific level of
the journals, where the articles are published) is, in the respective field,
at a remarkable international level)
5
Evaluation: Process
Main Criteria of Evaluation
All proposals must be evaluated based on the criteria outlined
below.
Proposal Quality
Degree to which proposal demonstrated general criteria
of excellence
Innovation
Quality of the research objectives and linkage to literature
review
Clarity of the research questions
Appropriateness of methodology
Quality/clarity and detail of proposed research method
Potential for peer-review publication, presentation, exhibition
Budgetary appropriateness.
Potential for disseminating research to broader academic
community.
Value for professional development of faculty member
etc
5
Evaluation: Process
Evaluators must ask these questions as they
develop the evaluation section of GWs proposal:
What is the evaluation's purpose?
How will GWs use the findings?
What will GWs know after the evaluation that
they didn't know before?
What will GWs do as a result of the
evaluation that they couldn't do before
because they lacked the relevant information?
Evaluating what worked and what didn't will be crucial
for grant sponsor and for GWs. What impact do
evaluators expect to achieve and how will they
evaluate it?
5
Evaluation: Process
Main Criteria of Evaluation (detail)
Review questions include, but are not
limited to:
Does the proposal communicate the importance of
the work?
Is the importance of the project within its field
made clear?
Will successful completion of the project have an
impact upon the field?
Is the project significant to the development of a
program of scholarly activity by the lead GW?
Are the objectives clearly defined, and is the basic
question to be answered clearly identified?
Are all necessary facilities available?
5
Evaluation: Process
Main Criteria of Evaluation (detail)
Review questions include, but are not limited to:
Does the lead GW have the necessary background and
expertise?
Are project activities well planned, and do they
realistically fall within an appropriate timetable?
Have all items requested been justified?
Is the amount requested reasonable and consistent with
the total funding available to this grant program?
If equipment is requested, has the possibility that it is
already available elsewhere on campus been addressed?
If support for students is requested, is it clear that
their activities are essential to the research program?
5
Evaluation: Process
External evaluation
Sponsor level
Evaluators focus on the Four Cs
Clarity. How GWs do cross-reference current literature in
laying out their premises.
Content. How GWs organize their ideas around aims linked
to their hypothesis.
Coherence of concepts. How GWs present coherent set of
ideas predicated by previous work.
Cutting edge. Are GWs ready to take legitimate risks.
5
Evaluation: Process
Research Proposal VS Research evaluation
Module of FRGS/TRGS Proposal
What Ext. Evaluators Look For:
Title
Details of Researcher
Research Information
Executive Summary
Research Background
Problem Statement
Hypotheses
Literature Review
Research Objectives
Methodology/Research Design
Timeline/Schedule
Expected Results
Facilities and Special Resources
Budget
Resume/Brief CV
Appendices
Title (1)
Details of Researcher
Research Information
Executive Summary (2)
Research Background
Problem Statement (3)
Hypotheses
Literature Review
Research Objectives (4)
Methodology/Research Design (5)
Timeline/Schedule
Expected Results (6)
Facilities and Special Resources
Budget (7)
Resume/Brief CV
Appendices
5
Evaluation: F/TRGS Guideline
5
Evaluation: New Approach
5
Evaluation: Title
A good title should:
Indicate the type of study.
Address the main problem.
Be concise, short, and descriptive.
Convey to the evaluator the main focus of
the research.
Use the correct terms in the title.
Should be intelligible to non-specialists.
Limit the title to a single sentence.
Relevant in 2 years time?
5
Evaluation: Title
Selection of research topic should be
based on…..
Magnitude of the problem and its impact
Urgency of the need for a solution.
Relevance to the aim of the funding agency.
Amenability of the problem to investigation.
Feasibility of the approach.
The proposed research topic is very timely and relevant
both internationally and locally.
The prospective results make a substantial contribution
to the development of science, technology, and/or
society
5
Evaluation: Executive Summary
An informative abstract, giving evaluators
the chance to grasp the essentials of the
proposal without having to read the details
GWs must present their project
Concisely
State significance Clearly
State Hypotheses, Research Problem,
Solution
Methods and Rationale
Expected output.
5
Evaluation: Research Background
Research Background (Grants Fail…)
Problem: The grant proposal lacks
direction and key details
GW does not organize his/her thoughts in
a clear and consistent manner
GW includes miscellaneous details and
leaves out details on essential grant
components
Solution: GWs must keep their proposal
organized and include the right details
5
Evaluation: Research Background
Research Background (RB)
1. Title
2. Problem statements
3. Objectives
Flows naturally from Title,
Problem Statement to
Research Objectives
5
Evaluation: Research Background
The RB serves several important functions: Evaluators must ensure
that…
GWs are not "reinventing the wheel".
GWs demonstrate their knowledge of the research
problem.
GWs demonstrate their understanding of the theoretical
and research issues related to their research question.
GWs show their ability to critically evaluate relevant
literature information.
GWs indicate their ability to integrate and synthesize the
existing literature.
GWs provide new theoretical insights or develops a new
model as the conceptual framework for their research.
The proposal will make a significant and substantial
contribution to the literature (i.e., resolving an important
theoretical issue or filling a major gap in the literature).
5
Evaluation: Research Background
Problem Statements
The most important aspect of a research proposal is
the clarity of the research problem
The problem statement is the focal point of the research
Evaluators must ensure that…
GWs give a short summary of the research problem
that have been identified.
The research proposal may not acceptable or credible
if GWs not clearly identify the problem.
GWs present the persuasive arguments as to why the
problem is important enough to study or include the
opinions of others (politicians, futurists, other
professionals)
5
Evaluation: Research Background
References-Most resent
Up-to-date
Highly relevant with the problem
Original source
First Order : High Impact Journals and
Books
Second Order : Indexed Proceeding
Publications
Third Order : Reputable Technical Report
5
Evaluation: Objectives
F/ERGS 2013: Objectives (Grants Fail…)
Problem: The grant proposal does not use
clear, measureable goals and objectives
The proposed goal is vague or
overreaching
The goals and objectives do not align
The objectives are not SMART
Solution: GWs should define their goals
and objectives clearly and ensure that
they are attainable in the given
timeframe
5
Evaluation: Objectives
Objectives specify
of the project,
the end
Goalsthe outcome VS
Objectives
product(s). GWs must state the objectives clearly and keep them
“S-M-A-R-T”
or “S-I-M-P-L-E.“
Broad terms, may
be vague
Narrow
Specific
to change
Give
direction - what GWs intend
Clearly
definethrough
scope of their
goal
project.
General
intentions; overall
Very
precise;
specificproblem
Immediate-time
frame during
which
a current
purpose
objectives
will be addressed.
Measurable-what GWs would accept as proof of
Intangible
Tangible
project success.
Practical - how each objective
is a real solution to a
Abstract
Concrete
real problem.
Cannot
be validated
is objective
Can be
Logical
- how as
each
canvalidated
contribute to
achieving GWs overall goal(s).
Evaluable
- how much change
has to occur
for the
Cannot
be measurable
Measurable,
observable
project to be effective.
5
Evaluation: Methodology
F/ERGS 2013: Methodology (Grants Fail…)
Problem: The methods section lacks clear
measurable objectives. Many proposals are
turned down because the methodology is
unsound.
The GW does not outline the tasks with a
given timeframe, staff responsible, or how
the task will be measured
No timeline is included
Solution: GWs must explain how they plan to
carry out and measure each objective
5
Evaluation: Methodology
Basically,
GWs state
GWs
it must
explicitly
provide
answers
to the following
Walk
the
reader
through
GW project
questions:
GWs give an
overall
summary
the research
Describe
the
activities
asofthey
relate todesign
the and
methodological approach.
objectives
GWs
Whatprovide
activities
theneed
methodology
to take place
for each
in order
specific
to meet the
Develop
a time line and/or and organizational
objective.
objectives?
chart
Whatdescribe
are the start and finish dates for the activities?
GWs
will
the
activities
be they
conducted?
How
Who
has
responsibility
for completing
each
the
specific
design
(what
will
do
andactivity?
how, number
How
ofwill
replicates,
participants
etc.),be selected? (Check…!?)
When?
What
thefactors
materials
and techniques
the suitability
that willofbeGWs
used, and
How
long?determine
methodology?
the feasibility of these techniques.
Who?
Does
usethis
literature
projectto
build
support
on models
design,
already
materials
in existence?
&
if
Where?
not,
techniques
how is it superior?
facilities?
What
facilities
and equipment will be required to conduct
What
the activities?
5
Evaluation: Methodology
Milestones
The milestones are the results which
the project seeks to achieve.
The milestones should, as much as
possible relate to ‘tangible products’
(quantifiable, qualitative or verifiable)
from conduct of the research.
They indicate viable achievements.
5
Evaluation: Methodology
Gantt’s Chart /Flow Chart
GWs must clearly show the research
activities and milestones (•/M)
Reflection of the project objectives,
methodologies, outputs, etc.
Very important
10 Evaluation: Expect Output
What are the expected outcomes and what do GWs wish to
achieve, e.g.:
A Scientific
new theory Outcomes
A prototype
A newNew
model methodology
Patents
An artefact
A newPublications
plant process
(High Impact Journal)
A solution to a practical problem
Discoveries
A specific
aid to practitioners in a particular field
AnSocio-Economic
instrument of use inoutcomes/impact
the manufacturing industry, etc.
Betterment of Society
What contribution will this research make to the body of
Development
Economy
knowledge
in the particular of
fieldthe
of study?
Improving
ofdefined,
Peopleas well as
The expected
outcomes Livelihood
must be clearly
the likelihood that the research will achieve the expected
results within the stated timeframe.
5
Evaluation: Track Record
Track Record (CV)
Experience, Qualifications and Availability
of Research Team
This section should begin with the principal
investigator, and then provide similar information on
all individuals involved with the project. Two elements
are critical:
Professional
research
competence
(relevant
research experience, the highest academic degree
held, and technical societies).
Relevant management experience (if any).
5
Evaluation: Quality of Proposal
F/ERGS 2013: Quality of Proposal (Grants Fail…)
Problem: The planning process is not well organized,
resulting in a poorly written proposal
The grant proposal is difficult to read or is not
concise
The GW uses incorrect grammar or incorrect
terms
The flow of the proposal is not logical and is hard
for reviewers to follow
GW does not collect the relevant information for
planning
GW does not delegate tasks
GW does not develop a timeline
Solution: Develop a work plan
5
Evaluation: Quality of Proposal
Style:
Use most recent form
Follow guidelines (font, size, margins, etc.)
Spell check, correct grammar
Highlight signposts (italic, bold, underlining)
One main idea per paragraph
Use topic sentences
Use transitions (e.g., in contrast, however,
likewise, etc)
Use graphics in methodology and needs sections
End paragraph with closing sentence
Evaluator-friendly application
GWs must give evaluators enough time !
5
Evaluation: Elements of FRGS
Novelty, Cutting Edge, High Impact
Does the research use novel techniques,
tools, and procedures?
Is new data required?
Is data gathered in a new way?
Is existing data utilised in a new way?
Can an existing application be used in a
new way?
Is the proposed research potentially
patentable and publishable?
5
Evaluation: Budget
Budget:(Grants
Evaluators
must ensure that…
Budget
Fail…)
Problem:
budget
exceeds
thebased
available
GWs The
present
the
budget
on amount
the sponsor
The
budgeted
items
are not reasonable for the
Guideline)
requests.
(Read
work proposed
The
budget must be reasonable, acceptable, and
The cost of the program is greater than benefit
appropriate (GWs must not inflate…)
The budget justification is not included
GWs
must
strictly
theactivities
Guidelines
There
is afollow
mismatch
between
and budget
Detail
justifications
ondoes
each
item explain
must be
The budget
justification
not clearly
provided
the
budget (Vote
item 35000)
Inappropriate
use of funds
Itemized
Budget
Solution:
Outline
budget items carefully and use
Budget
Narrative
standard amounts for expenses
5
Evaluation: Facility and Support
Infrastructure/Facilities
GWs must use whatever available in campus
(related to proposed project)
Reduce to a minimum any call upon outside
facilities and expertise
The requirements of infra will vary from
study to study. GWs must carefully list the
relevant facilities and resources that will
be used.
The costs for such facility use should be
detailed in GWs budget.
5
Evaluation: Evaluator Comments
Sample Comments
Proposal is poorly written (confusing, not logical,
poorly organized, typos, etc.) - Overall
Too big a leap from preliminary data to the
proposed hypothesis; failure to provide sound
scientific data for the support of the hypothesis.
– Conceptual Analysis
Using sub-optimal techniques: i.e., using out-of-date
techniques, or conversely, using unnecessary new
techniques when standard techniques will work
just as well. Equipment is ill-suited
for
the
proposed projects. - Methodology
6
Conclusion
Make Life Easy for Evaluators
Evaluators are knowledgeable, experienced scientists, but
they can’t know everything.
Problem: evaluators may not get the significance of the
proposed research.
Solution: GWs write a compelling argument.
Problem: evaluators may not be familiar with all the research
methods.
Solution: GWs write to the non-expert in the field.
Problem: evaluators may not be familiar with the research
lab.
Solution: GWs show to evaluators that they can do the job.
Problem: evaluators may get worn out by having to read 10 to
20 applications in detail.
Solution: GWs write clearly and concisely, and make sure the
application is neat, well organized, and visually
appealing.
6
Conclusion
Ethical Statement
Researchers undertaking any form of
fundamental research using animals or
people have to submit a proposal to
either the animal ethics committee or
the human ethics committee for
approval before the data gathering can
begin.
6
Conclusion
Joel Orosz, of the W.K. Kellogg Foundation,
has sagely observed that there are really only
four kinds of proposals
Bad
idea, bad proposal
Bad idea, good proposal
Good idea, bad proposal
Good idea, good proposal
6
Conclusion
Good Proposal
Grant Proposal (Failed/Success)-Sample
Good
Good
Good
Good
Good
Ideas
Grantsmanship
Presentation
Review
Luck
7
Bibliography
Kevin C. Chung, MD, Melissa J. Shauver
Cheryl Anne Boyce, Ph.D
Xander HT Wehrens, M.D. Ph.D
Gitlin, L. N., Lyons, K. J.
Simon Peyton Jones
Baharuddin Salleh
8
Ketua Panel Kluster Geran KPT
1. Sains Tulin - Prof. Emeritus Dato' Dr. Muhamad bin Yahaya, UKM
2. Sains Gunaan - Prof Emeritus Dato' Dr Md Ikram Bin Mohd Said,
UKM
3. Sains Sosial - Prof. Dr. Samsudin bin A. Rahim, UKM
4. Sains Tabii dan Warisan Negara - Prof. Dato’ Dr. Nik Muhamad bin
Nik Ab. Majid, UPM
5. Sains Kesihatan dan Klinikal - Prof Dato‘ Dr. Amin bin Jalaluddin, UM
6. Sastera dan Sastera Iktisas - Dato‘ Prof. Salleh bin Yaapar, USM
7. ICT - Prof Dr. Ku Ruhana binti Ku Mahamud, UUM
8. Teknologi dan Kejuruteraan - Prof Dr Muhammad Fauzi Mohd Zain,
UKM
Thank you
[email protected]