Forecast Evaluation Overview Why Evaluate Point Forecasts vs

Transcription

Overview
Evaluating forecast distributions
Combining forecast distributions
Disseminating results
The DIME website
Further perspectives

Forecast Evaluation
LSE CATS Operational Weather
Risk Meeting @ GaTech
Point Forecasts vs
Ensemble Forecasts
Why Evaluate
Ensemble weather forecasts appear to be
invaluable to weather dependent business
Methods for handling and valuing ensemble
forecasts are subject of ongoing research

An ensemble forecast is a set of numbers:
London temperatures tomorrow:
23°
23°, 26°
26°, 27°
27°, 29°
29°, 31°
31°
3
Temperature at London
Heathrow – Point Forecast
A point forecast is a single number:
London temperature tomorrow: 25°
25°
The ultimate criterion is their usefulness to
end users
2
4
Evaluating Point Forecasts
5

For classical point forecasts we can evaluate
the error:
London temperature tomorrow: 25°
25°
Reality turns out to be: 28°
28°
Error: 3°
3

A point forecast is considered good if the
error is small on average
6
Temperature at London
Heathrow – Ensemble Forecast
Enhancing Point Forecasts –
Method of Dressing

Dressing point forecasts adds uncertainty
information
Probability
Point Forecast
7
Temperature
Kernel width needs adjustment –
Many possible ways to do that!
8
Dressed Ensemble Forecast
Dressing Ensemble Forecasts
Dressing ensemble forecasts
Probability

Kernel Function
Temperature
www.dime.lse.ac.uk
How do we combine the probability densities?
9
Combining Forecast Distributions

To quantify the potential usefulness of
different forecast distributions we need to
evaluate their skill
Take into account that the different
forecasts have different skills
Determine optimal combination by
looking at the skill, e.g. Ignorance
10
Evaluating Forecast Distributions
There exist many different methods to
combine forecast distributions

How do we compare reality (a single number)
with the forecast (a distribution)?
11
12
Different Problems Require
Different Skill Scores!
What DIME does
DIME aims to investigate the skill of NWP’s
and dressing techniques using various skill
scores
DIME disseminates background information

A forecast can be
Specific (electricity output from a wind farm)
General (wind speed)

What is the best dressing method
for your application?
These require different measures of
skill
Evaluating various skill scores of
operational NWP’s over a period of time
MODELS
MODEL 1

Ignorance Skill Brier Skill
Score
Score
4.5
0.25

Bet on the outcome of tomorrow’s weather
Tomorrow’s Temperature
…
Temperature
…
4 5 6 7 8 9 10 11 12
Spread wealth for safety
What do these skill scores mean?
Restrict spread to make money
15
When Is a Forecast Distribution
Good in Weather Roulette?
Skill Scores for Binary Event
Forecasts – The Brier Score
A good forecast distribution balances
between spread and accuracy
Criterion is:

A forecast distribution is better the more
money it yields in weather roulette
Forecast:
It freezes with probability p
It doesn’t with probability (1(1-p)
The ignorance reflects the expected rate of
wealth growth.
16
Binary (yes/no) events are e.g.: Will it
freeze? Will precipitation exceed a
threshold? etc.
A forecast for a binary event:

14
Ignorance And Weather Roulette
Example of an Assessment of Skill

13

17
Brier score reflects the quality of binary
event forecasts
18
Comparing Forecasts

Skill evaluation done properly
Skill is a statistical quantity and needs
errorbars
Two different forecasts can be compared by
means of their ignorance
NWP
schemes
Score
Score
NWP
schemes
…
MODEL 1
4.5
0.25
…
MODEL 2
3.8
0.3
…
Score
Score
…
MODEL 1
4.5 ± 0.2
0.25 ± 0.03
…
MODEL 2
3.8 ± 0.1
0.3 ± 0.02
…
(errorbars will be suppressed from now on)
Comparing Forecasts
Ignorance
Skill Score
20
Combining Forecasts
Dressing allows us to compare point
forecasts and ensemble forecasts
MODELS
19
Brier Skill
Score

MODELS
…
MODEL 1 (Ens
.)
(Ens.)
4.5
0.25
…
MODEL 2 (Ens
.)
(Ens.)
3.8
0.3
…
MODEL 3
(Point Forecast)
5
0.6
…
The table grows again…
MODEL 1 (Ens
.)
(Ens.)
MODEL 2 (Ens
.)
(Ens.)
Ignorance
Skill Score
4.5
3.8
Brier Skill
Score
0.25
0.3
…
MODEL 3
(Point Forecast)
5.0
0.6
…
MODEL 1 and
MODEL 2
3.2
0.2
…
21
DIME Objectives
Dissemination of Results
Objectively compare operational NWP
model ensemble forecasts by their skill
Suggest schemes to combine ensemble
forecasts and evaluate the skill of these
schemes
Provide actual weather forecasts for specific
locations using a selection of our methods

23
…
…
22
The medium to disseminate DIME results is
the internet
The web allows users to interactively
request the products they are most
interested in
www.dime.lse
.ac.uk
uk
www.dime.lse.ac.
24