Bibliometrics in higher education: the impact on libraries Keith Webster

Bibliometrics in higher education:
the impact on libraries
Keith Webster
University of Queensland, Australia
Dr Berenika M. Webster
CIBER, Univ. College London
Outline
• Bibliometrics and university libraries
– Structure of disciplines
– Collection management
– Research evaluation
• Bibliometric techniques
• Sample projects
Structure of disciplines
• Domain analysis
• Patterns of scholarship
Collection management
• Patterns of use
• Core journals in discipline
• International comparison
Research evaluation
• Approaches to research assessment: experiences from
UK, NZ and Australia
• Peer assessment and metrics
• Bibliometrics
– Assessment tools
– Challenges of bibliometric approach
– How to increase the quality and impact of published outputs
What do governments want for
their money?
• Economic outcomes
– increase wealth creation & prosperity
– improve nation’s health, environment &
quality of life
• Innovation
• R&D from private sector
• Improved competitiveness
• Less “curiosity-driven” activity
Research Assessment Exercise (UK)
• Started in mid-1980s to determine size of research funding from HEFCEs to
universities. Early 1990s brought shift from quantity to quality
• 60 subject panels examine, for quality, selected outputs of UK academics.
Also, previous funding, evidence of esteem, PhD completions and res.
environment are taken into consideration
• Grades for submission units; correlates with bibliometrics measures (Charles
Oppenheim’s research)
• It will be scraped in 2008. New systems will be a hybrid of a “basket of
metrics” and peer-review (???)
• Changed behaviour of scholars
– Increased volume of publications (salami slicing?)
– Increased quality of outputs (numbers of 5 and 5* departments increase)
Performance-Based Research Fund
(PBRF) New Zealand
• Introduced in early 2003
• 12 panels grades individuals in 41 subject categories
• Portfolio has three components
– Publications (4 best and the rest)
– Peer esteem
– Contribution to research environment
• 2003 round produced lower scores than expected and great
variability between subjects (from 4.7 in philosophy to 0.3 in
nursing). 40% of submitted staff were judged research inactive
(grade R)
• Scores for institutions are calculated from these and used together
with PhD completion and external funding metrics to calculate
awards to institutions
• 2006 round incorporates “breaks” for new researchers
Research Quality Framework (RQF)
- Australia
• To be introduced in 2007 (?)
• Quality includes the intrinsic merit of original research and academic
impact. This relates to recognition of the originality of research by peers
and its impact on the development of the same or related discipline areas
within community of peers.
Assessment: peer review and metrics
• Impact or use of the original research outside of the peer community that
will typically not be reported in traditional peer review literature. Relates to
the recognition by qualified end-users that quality research has been
successfully applied to achieve social, cultural, economic and/or
environmental outcomes.
Assessment: expert opinion and narratives or metrics?
RQF evidence portfolio
• Four “best” outputs (with rationale for selection)
• List of all published outputs in the last six years
• Statement of “early impact” as assessed by qualified
end-users (description of outcome; identification of
beneficiaries; metrics illustrating benefits; linkage
between claimant and beneficiary)
• Context statement for the research group
Evaluation: peer vs. metrics
• Evaluate/reward past performance
• Measures short to medium term impacts
• Established use for the evaluation of academic
impact (within the group of peers)
• No established/accepted procedures for
evaluation of impact outside academe
Peer review
• “Traditional” assessment is an outputs-based peer-review
–
–
–
–
–
–
–
–
–
–
a panel of experts “reads” submissions or outputs
not always transparent (selection; evaluation criteria)
subjective (concept of quality is very difficult to objectivise)
relies on proxies (e.g. ranking/prestige of publishing journal; inst.
affiliation of authors, etc.)
composition of panels will affect their judgements
lacks comparability between panels
“punishes” innovative or multidisciplinary research
“old boys” club disadvantaging young researchers
assesses what already has been assessed by journals’ peer-review
processes
time and resource intensive
Metrics
• Range of unobtrusive and transparent
measures can be:
– Input-based
• Income
• Success rates
– Output-based
• Long-term impact (patents, IP, etc.)
• Bibliometric measures (volume and impact/quality)
Problems with metrics (1)
• Input-based
– Retrospective
– Creates “Matthew effect”
– Disadvantages younger researchers (without
previous track record)
– Undervalues research done “on the string”
Problems with metrics (2)
• Output based
–
–
–
–
–
–
–
–
Citations as a measure of intellectual influence?
Comparing like with like (benchmarking)
Inadequate tools
Social science and humanities (in search for new metrics)
Research on the periphery (local citation indices)
New models of communication (capturing it)
Impacts on wider community (“tangible” outcomes)
Lack of expertise in analysis and interpretation (education)
Indicators of quality as measured
using published outputs
•
Number of publications
•
Citation counts to these publications (adjusted for self-citations)
•
Citations per publication
•
Percentage of uncited papers
•
Impact factors (of publishing journals)
•
Diffusion factor (of citing journals) – profile of users of research (who, where, when and what)
•
“Impact factor” of a scholar - Hirsh index (h index)
-what “window” should be used? 4, 5, 10 years?
–
–
(numbers of papers with this number of citations).
Your h index =75 if you wrote at least 75 papers with 75 citations each.
Note: These should not be seen as “absolute” numbers but always seen in the context of the
discipline, research type, institution profile, seniority of a researcher, etc.
Compare like with like!
• Applied research attracts fewer citations than basic research.
• Differences in citation behaviour between disciplines (e.g. papers in
organisational behaviour attract 5 times as many citations as papers
in accounting).
• Highest IF journal in immunology is Ann Rev Immun (IF 47.3) Mean
for cat. 4.02; and in health care and services category is Milbank Q.
(IF of 3.8). Mean for cat. 1.09.
• Matthew effect.
Benchmarking must be done using comparable variables!
Tools available
• Publication and citation data
–
–
–
–
Web of Science
SCOPUS
Google Scholar
Local/national databases
• Other bibliometric indicators
– Journal Citation Reports (JCR)
– Other indicators databases (national, essential, university,
institutional)
– ISIHighlyCited.com
WoS and Scopus: subject coverage
(% of total records)
WoS
Social
Sciences, 14
SCOPUS
Arts &
Humanities,
9
Social
Sciences, 2
Physical
Sciences, 25
Science, 77
Biological &
Environmental
Sciences, 13
Health & Life
Sciences, 60
Google Scholar ?
Jacso, 2005
Web of Science
• Covers around 9,000 journal titles and 200 book
•
•
•
•
•
•
series divided between SCI, SSCI and A&HCI.
Electronic back files available to 1900 for SCI
and mid- 50s for SSCI and mid-70s for A&HCI.
Very good coverage of sciences; patchy on
“softer” sciences, social sciences and arts and
humanities.
US and English-language biased.
Full coverage of citations.
Name disambiguation tool.
Limited downloading options.
Scopus
• Positioning itself as an alternative to ISI
• More journals from smaller publishers and open access (13,000
•
•
•
•
•
•
•
•
journal titles; 600 open access journals; 750 conf proceedings; 600
trade publications, 2.5 mil “quality” web pages)
Source data back to 1960.
Excellent for physical and biological sciences; poor for social
sciences; does not cover humanities or arts.
Better international coverage (60% of titles are non-US)
Not much of a back file (e.g. citation data for the last decade only)
Not “cover to cover” and not up to date
Easy to use in searching for source publications; clumsy in searching
cited publications.
Citation tracker works up to 1000 records only.
Limited downloading options.
Google Scholar
• Coverage and scope?
• Inclusion criteria?
• Very limited search options
• No separate cited author search
• Free!
Taiwan in the three sources,1996-2005
70000
Scopus (1.23%)
60000
WoS (0.98%)
Google Scholar
50000
40000
30000
20000
10000
0
1996
1997
1998
1999
2000
2001
2002
2003
2004
2005
Social sciences
• Communication in social sciences
–
–
–
–
Books
National journals
International journals
Non-scholarly outputs (culture-creating role)
–
–
–
–
–
Chinese Social Sciences Citation Index
Polish Sociology Citation Index
Serbian Social Sciences Index
Polish Citation Index for Humanities
European Humanities Citation index
• “International” tools inadequate
• Creation of “national” and “regional” tools
Science on the periphery
• Poor coverage in international databases
• Intl. benchmarking difficult
• Impossible for local evaluations
• Development of national sources of data
• While volume is growing, citation rates lag
behind
Science on the periphery
Figure 1: Polish scientific papers in SCI (in Polish and foreign
journals), three-year running means and Polish SCI journals, 19801999
30
Polish SCI
journals (right
scale)
7000
Papers per year
6000
25
20
5000
4000
15
3000
10
Papers in
foreign journals
2000
5
1000
Papers in Polish journals
0
journals
1981
1983
1985
0
1987
1989
1991
1993
1995
1997
Number of Polish journals in SCI
8000
Capturing new forms of
communication
• Use of online journals (deep log analysis)
• Traffic on websites and downloads
• In-links to websites
Impact of research: example of
biomedicine
• Informing policies (citations on guidelines, govt.
policy, development of medicines)
• Building capacity (training; development)
• Relationship between research and health
outcomes and cost savings
• Healthier workforce
Citations on clinical guidelines
• 47 Clinical Guidelines and 64 Health Technology
Appraisals prepared by the UK’s National
Institute for Clinical Excellence (NICE) were
analysed for citations
– cited 25% of UK papers (2.5 more than expected)
– Majority were clinical papers
– Median lag time between publication of res. paper
and its citation was 3.3 years
Relative commitment to research and
burden of disease
0.8
Percent of biomed research
Prostate cancer
SE
NL
0.6
US
y = 0.1112x + 0.1645
R 2 = 0.087
0.4
IT
CA
UK
DE
ES FR
JP
0.2
AU
CH
0.0
0.0
0.5
1.0
1.5
Percent of all deaths
2.0
2.5
3.0
How to increase “quality” of your
publications?
•
Publish in the right journals (prestige; importance to the discipline; impact factor vs. diffusion
factor)
•
Publish in English
•
Write review articles
•
Engage in basic research
•
Become a journal editor (Lange, 1997)
•
Acquire a co-author (preferably from US or UK)
•
Get external funding (from different sources)
•
Make your outputs available in open access (own website, institutional and subject repositories)
(Antelman, 2004; Harnard various)
•
“Advertise” your publications on listservs and discussion groups
•
Make and maintain professional/social contacts with others in your research area (Rowlands,
2000)