Making Sense out of Metrics ISDA / PRMIA 17

Making Sense out of
Metrics
ISDA / PRMIA
17th August 2004
In 2002………

A global survey of 76 banks on the
existence of formal KRI programs:
Acknowledgement Raft International Limited
What is a KRI?
The number of fails has increased by 2%
Is this a KRI?
Some definitions……
Metric – something observed or calculated that is used
to show the presence or state of a condition or trend;
an instrument or gauge that measures something and
registers the measurement; something such as a light,
sign, or pointer that gives information, for example
about which direction to follow
 KRI – Key Risk Indicator
 KPI – Key Performance Indicator
 KCI – Key Control Indicator
 KMI – Key Management Indicator

Overall, prefer the general term “indicator”
What is a Risk Indicator?

Risk indicators are usually monitored over
time
 Late trade processing in Bank X in May 2004:
London 9%, New York 10%, Singapore 9%
Late Trade Processing
 In which branch
15%
do we have a
10%
London
problem?
New York
5%
0%
Singapore
Feb
Mar
Apr
May
What is a Risk Indicator?

So, Singapore is the problem!
 Is it?
Late Trade Processing
absolute numbers
3500
15%
3000
2500
10%
2000
1500
London
New York
5%
1000
Singapore
500
0
Feb
Mar
Apr
May
0%
Feb
Mar
Apr
May
Data without context may not expose the entire problem!
The Choice from the Multitude

A typical operation can
identify hundreds of
indicators
 Some are risk, others
performance indicators
 Indicators’ relevance
and weight change over
time
 Some indicators are
meaningless on their
own
Graphics adapted from Reason, J.: “Managing the Risks of Organizational
Accidents", Aldershot: Ashgate, 1997
Other factors
If a ratings agency is to rate operational
exposure, how will they compare
different organisations?
 How will regulators evaluate the
effectiveness of different organisation’s
operational risk capabilities?
 How does a business unit provide
senior management quality information?
 Can the organisation use operational
metrics to provide stakeholder
information?

But, KRIs have been disappointing…
No means of consistently relate the
occurrence of loss events and the
location of problems/situations
 No means of classifying types of KRIs
 Plenty of data but no idea of its relevance
 No way to determine relevance
 No observable best practice
 No means of comparison, either internally
or externally

The solution?

An organisation needs a common
language or framework which identifies
areas of exposure and then allows…..
– Metrics to be identified to measure, monitor
and manage those exposures
– Data on losses, near misses and control
failures to be recorded
– Ongoing assessment of the exposure
– Performance measurement around the
exposure, including use of capital
KRI Study - Background
Recognising the need, RiskBusiness
developed a strawman framework for
identifying risk points within an
institution
 RMA and RiskBusiness co-sponsored
Part I of the Study to ascertain the
feasibility of using the framework as a
KRI framework
 Seven institutions tested the framework
in a risk mapping exercise

Initial Participants

7 banks participated in Part I:
– Citigroup
– Deutsche Bank
– Dresdner Kleinwort Wasserstein
– JP Morgan Chase
– KeyCorp
– Royal Bank of Canada
– State Street
 ANZ and Abbey then joined to form the Study
Steering Group for Part II
The Study
Part I – Proof of Concept
 Part II – 3 primary activities :

– Broaden participation, transfer experience,
build industry risk profile
– Define KRI Library
– Develop detailed specifications

Future :
– Industry benchmarking
– Extend participation further
The KRI Framework
Validation
Initial risk maps were evaluated to
establish “most risky” risk categories
and business functions
 Compared these to a similar evaluation
of the QIS 3 data and to the complete
Fitch Risk First database
 Broad correlation, taking into account
the nature of the two data sources

Participant risk map deviations

Based on risk category:
RMA/RiskBusiness Risk
Categories
Data Management
Improper Practices
Infrastructure & Systems
Transaction Management
Internal Theft & Fraud
Stage 1
Rankings
(additive
High
ratings)
1
2
3
4
5
Participant Participant Participant Participant Participant Participant Participant
A
B
C
D
E
F
G
2
1
5
7
3
5
1
7
6
2
3
1
2
6
5
2
1
3
6
5
1
4
2
6
3
2
1
3
5
4
2
7
5
1
9
Risk
Business
3
2
1
4
5
Participant risk map deviations

Based on business function:
RMA/RiskBusiness Business
Functions
Payment/Settlement/Collection
(cash/securities)
Instruction or Order Management
Custody and Actions (including
assets)
Transaction/Fees Capture and
Record Update
Relationship Management
Confirm/Affirm/Matching and
Documentation
Transaction Maintenance and
Administration
Infrastructure, Networks &
Maintenance
Pricing and Quotations
IT Security
Stage 1
Rankings
(additive
High
ratings)
Participant Participant Participant Participant Participant Participant Participant
A
B
C
D
E
F
G
Risk
Business
1
2
3
1
1
1
3
3
2
2
3
1
6
1
2
3
4
2
3
1
3
1
8
-
1
3
4
21
23
2
5
5
5
2
11
5
6
3
5
4
9
19
19
11
4
14
4
18
2
1
-
14
4
7
21
7
13
13
10
5
5
15
8
3
36
21
6
7
7
-
8
9
10
13
10
13
5
28
14
9
5
7
13
3
9
-
8
12
Participants in Part II

























Abbey†
ABN Amro
ABSA
Acleda Bank
Alliance and Leicester
ANZ†
Bank Austria Creditanstaldt
Bank Julius Bäer
Bank of America
Bank Rakvat Indonesia
Bank Vontobel
BNP Paribas
Byblos Bank
Capital One
Citigroup†
Commerzbank
De Lage Landen (sub of Rabobank)
Deutsche Bank†
Dresdner Kleinwort Wasserstein†
Erste Bank
Euroclear
Federation de Caisses Desjardins du Quebec
GMAC
Halifax Bank of Scotland
HSBC
† = Lead Participant
























Huntington National Bank
Investec
JP Morgan Chase†
KeyCorp†
Kookmin Bank
Macquarie Bank
Mizuho International
National Australia Group
National Bank of Canada
Nomura International
Northern Trust
People’s Bank
Royal Bank of Canada†
Royal Bank of Scotland
RZB
San Paolo IMI
SE Banken
Southwest Bank of Texas
Standard Bank of South Africa
State Street†
Sumitomo Mitsui
TD Financial Group
Woori Bank
Washington Mutual
Defining Indicators
Identify candidates for each risk point
 Evaluate candidates against qualifying
criteria (effectiveness, comparability, ease of
use/collection)
 Agree descriptions for each qualifier
 Prioritise nominated indicators
 Participant comment and review
 Generate detailed specifications, stored
in KRI Library at www.KRIeX.org

Considering each risk point…..
Before the problem has been generated
Look for ‘Early
Warning Flags’
After the problem has been generated
Identify ‘Problems
in Progress’
Potential loss events, include
events that could lead this or another
institution to loss
Near misses, loss = 0
Research
‘Historical Events’
0 < Losses < threshold
Losses > threshold
Issues for consideration

What is a KRI...KPI…KCI…or MIS?
– An indicator can perform multiple roles
depending on who is using it

What about scaling and aggregation?
– Do we scale then aggregate, or vice versa?

How many indicators should a firm be
monitoring?
– The quest for the “Magic 10” – the KRI
Library has 1,600 indicators
Issues for consideration

Top-down versus Bottom-Up
– Operations develop metrics for ongoing
use, Management want information

Combinations and clusters of indicators
– Experience has demonstrated that in many
cases, it is groups of indicators which will
provide the best management information
– Staff Turnover and Transaction Volumes
and Error Rates and ……
Next Steps - Benchmarking

Selected indicators will be driven totally by
broad participant agreement
 Consists of participants delivering KRI data to
centralised function :
– Data will be anonymous
– Data will be collated, analysed and benchmark
values calculated

Participants have access to benchmarks for
comparative purposes
 First submission expected in Q2, 2005
Next Steps – KRI Library
Next intake of participants into KRI
Library starts in October 2004 –
currently have 68 additional firms
wishing to join
 Insurance KRI Study starts during Q4
2004
 Ongoing maintenance and extension to
the Library

In Summary…..
A well-developed and structured
indicator program can deliver quality
management information and could
possibly be used as an adjustor to
capital…or at least as a measure of
efficient and effective use
 Common language and standardisation
is imperative
 The indicator program must deliver
value at all levels

Contact details
 RiskBusiness
International Limited
– URL : www.riskbusiness.com
– Study URL : www.kriex.org

Mike Finlay, Managing Director –
Europe, Asia, Australia and Africa
– Telephone : +44 7721 969 224
– E-mail : [email protected]