Indian bobsledder Shiv Keshavan

XXII Olympic Winter Games
Feb - Mar 2014
Finnish Meteorological Institute
Indian
bobsledder
Shiv Keshavan
[email protected]
Acknowledgement:
Raghu
FROST-2014 Verification Framework
XXII Olympic Winter Games
Feb - Mar 2014
Finnish Meteorological Institute
Forecast Verification Framework of
The Sochi 2014 Winter Olympics
( FROST-2014 ~ Forecast and Research in the Olympic Sochi Testbed )
Pertti Nurmi
WMO WWRP JWGFVR aka Joint Working Group on Forecast Verification Research
Acknowledgements : FMI verification system development team & WMO FROST-2014 Expert Team
[email protected]
[email protected]
FROST-2014 Verification Framework
XXII Olympic Winter Games
Feb - Mar 2014
Finnish Meteorological Institute
Sochi
[email protected]
FROST-2014 Verification Framework
3
XXII Olympic Winter Games
Feb - Mar 2014
Finnish Meteorological Institute
[email protected]
FROST-2014 Verification Framework
4
XXII Olympic Winter Games
Feb - Mar 2014
Finnish Meteorological Institute
Laura
Rosa Khutor
Mountain
Cluster
~2,3 km
~650m
[email protected]
FROST-2014 Verification Framework
5
XXII Olympic Winter Games
Feb - Mar 2014
Finnish Meteorological Institute
Nanette (WMO)
[email protected]
FROST-2014 Verification Framework
6
XXII Olympic Winter Games
Feb - Mar 2014
Finnish Meteorological Institute
Aibga ridge ~2300 m
Rosa Khutor
Sanki
RusSki Gorki
Laura
Olympics Mountain Cluster sports venues
[email protected]
FROST-2014 Verification Framework
7
XXII Olympic Winter Games
Feb - Mar 2014
Finnish Meteorological Institute
Issues to consider + take into account … #1
 All FROST-2014 contributors verify their own output  General practice !
… BUT can be biased  NOT sufficient / adequate
 Also centralized (Roshydromet, Russia) AND ”external” verification ( = FMI )
… to validate verification results
 Two complementary approaches (like Vancouver Olympics SNOW-V10)
a)
b)
High temporal resolution Point Verification  User needs : FDP*

Users require thresholds for their decisions

Sports activities and events take place at, or along, points
Research-oriented verification, Detailed model comparisons : RDP*
* FDP = Forecast Demonstration Project / RDP = Research Demonstration Project (WMO terminology)
[email protected]
FROST-2014 Verification Framework
9
XXII Olympic Winter Games
Feb - Mar 2014
Finnish Meteorological Institute
Issues to consider + take into account … #2
 Details of all available models/systems + observations

Enhanced observation network with N=31 stations used in verification

Hi-fidelity site-specific (point) observations highly important
 On-line forecast production from all contributors

Common data base structure: ftp site - for storage

”Freeze” the FDP system at certain stage
“ No changes are allowed in the last 3-4 months before the Olympics ”
 Standardized XML formats for fc & obs delivery + storage !
Only some did
1. Close to real-time verification  Soon as data(sets) become available
Hmm…
2. Retrospective verification  Comprehensive post-event verification possible
only after all fc/obs archived and QC’d  2 differing sets of results (?)
[email protected]
FROST-2014 Verification Framework
10
XXII Olympic Winter Games
Feb - Mar 2014
Finnish Meteorological Institute
Issues to consider + take into account … #3
 Pointwise verification of all forecasts, tuned at all forecast = decision points
a)
”Baseline quality” from a coarse model as reference
b)
Various hi-res models from contributors, various (time/space) resolutions
c)
“Consensus” products, if available (?)
d)
End-Products by the Olympics forecaster team !
e)
Inter-comparisons between, and added value, (a) to (d)  IMPACTs
 Define each sports site/event relevant observing station(s), observations and
forecast variables and their thresholds  Not trivial !
 Use of verification software, R, MET, etc …

Roshydromet : MET, adapted for FROST output (COSMO-RU forecasts)
 FMI Verification package “tuned” for Sochi

[email protected]
Input (fc + obs) provided at common ftp site
FROST-2014 Verification Framework
11
XXII Olympic Winter Games
Feb - Mar 2014
Finnish Meteorological Institute
Issues to consider + take into account … #4
 Formal evaluation period : 15 January – 15 March = 2 months
 High-impact  What is hi-impact ?  Thresholds
 Sample size issue (statistical inference; confidence intervals)
 Timing (error) of events – onset, duration, cessation
 Stratification vs. Aggregation  Need both !
 Case studies  Hard to generalize results  RDP
 FMI speciality : Running Road Weather Forecast Model for Sochi roads
 Verification later

[email protected]
Under externally funded “CoMoSeF” project
FROST-2014 Verification Framework
12
XXII Olympic Winter Games
Feb - Mar 2014
Finnish Meteorological Institute
FROST-2014 : Models - Contributors
Model
Analysis / FC hours
Contributor
FMI Verification
00, 06, 12, 18 UTC / + 78 hr
00, 06, 12, 18 UTC / + 42 hr
23 UTC / + 27 hr
21 UTC / + 25 hr
00 UTC / + 24 hr
00, 06, 12, 18 UTC / + 36 hr
00, 12 UTC / + 24 hr
00, 12 UTC / + 72 hr
Hourly / + 48 hr
HMC,Russia
”
”
Env. Canada
”
”
FMI, Finland
NOAA,USA
KMA, S-Korea
ARPA SIMC, Italy
ZAMG, Austria
“consensus”
***
***
06, 18 UTC / + 54 hr
Hourly / + 48 hr
06, 18 UTC / + 30 hr
-
ZAMG,Austria
HIRLAM, Norway
”
”
HMC,Russia
ARPA SIMC, Italy
Notes
Deterministic Forecasts
COSMO-RU - 7 km
COSMO-RU - 2 km
COSMO-RU - 1 km
GEM - 2.5 km
GEM - 1 km
GEM - 0.25 km
HARMONIE-Sochi - 1 km
NMMB - 1 km
KMA
ARPA (Sochi-mini)
INCA
Joint
***
***
***
***
***
***
to be considered as RDP
under processing as of early-March
to be considered as RDP
***
Ensemble Forecasts
Aladin-LAEF-EPS - 7 km
GLAMEPS - 11 km
GLAMEPS - updated T
HarmonEPS - 2.5 km
COSMO-RU-EPS - 2 km
COSMO-S14-EPS - 7 km
NMMB-EPS - 7 km
-
***
***
***
NOAA, USA
Nowcasting
MeteoExpert - single station
Joint
INCA
NTW
CARDS
ABOM
RW Model, Harmonie-driven
[email protected]
under processing as of early-March
under processing as of early-March
to be considered as RDP
to be considered as RDP
10 minutes / + 4 hr
Hourly / + 48 hr
IRAM,Russia
“consensus”
ZAMG,Austria
Env.Canada
Env.Canada
Env.Canada
FMI, Finland
***
***
to be considered as RDP
to be considered as RDP
to be considered as RDP
Verification within CoMoSeF project
FROST-2014 Verification Framework
13
XXII Olympic Winter Games
Feb - Mar 2014
Finnish Meteorological Institute
FROST-2014 : Weather parameter thresholds
Temperature ( OC )
T < -20
-20 ≤ T < -5
-5 ≤ T < -2
-2 ≤ T < 0
0≤T<2
2≤T<5
T≥5
Wind speed ( m/s )
WS ≥ 3
WS ≥ 4
WS ≥ 5
WS ≥ 7
WS ≥ 11
WS ≥ 15
WS ≥ 19
Horizontal visibility ( m )
V < 100
V < 300
V < 1000
V < 10 000
Precipitation amount
RR < 0.3
RR ≥ 0.3
RR ≥ 1.0
RR ≥ 5.0
RR ≥ 10.0
RR ≥ 15.0
1-hr and 24-hr (mm)
Photo © johnny9s on Flickr
[email protected]
Photo © Prentice Korea on Flickr
FROST-2014 Verification Framework
14
XXII Olympic Winter Games
Feb - Mar 2014
Finnish Meteorological Institute
FMI Operational Verification System
Web User Interface
[email protected]
FROST-2014 Verification Framework
Finnish Meteorological Institute
Verification User Interface
Adapted for Sochi 2014 Olympics
[email protected]
FROST-2014 Verification Framework
17
Finnish Meteorological Institute
[email protected]
FROST-2014 Verification Framework
18
Finnish Meteorological Institute
Ski
Biathlon
Ski Jump
Valley
Alpine Ski
[email protected]
FROST-2014 Verification Framework
19
XXII Olympic Winter Games
Feb - Mar 2014
Finnish Meteorological Institute
Bias
Biathlon
Alpine
Biathlon
Alpine
Biathlon
Alpine
RMSE
Biathlon
Alpine
[email protected]
Biathlon
Alpine
FROST-2014 Verification Framework
Biathlon
Alpine
20
XXII Olympic Winter Games
Feb - Mar 2014
Finnish Meteorological Institute
SEDI
-2 < T < 0
o
C
Biathlon
Alpine
[email protected]
Biathlon
Alpine
FROST-2014 Verification Framework
Biathlon
Alpine
21
XXII Olympic Winter Games
Feb - Mar 2014
Finnish Meteorological Institute
MAE
All stations
[email protected]
All models > +72 hr
FROST-2014 Verification Framework
22
XXII Olympic Winter Games
Feb - Mar 2014
Finnish Meteorological Institute
MAE
All stations
[email protected]
All models > +48 hr
FROST-2014 Verification Framework
23
XXII Olympic Winter Games
Feb - Mar 2014
Finnish Meteorological Institute
PoP
ROC
[email protected]
FROST-2014 Verification Framework
24
XXII Olympic Winter Games
Feb - Mar 2014
Finnish Meteorological Institute
PoT
ROC
[email protected]
FROST-2014 Verification Framework
25
XXII Olympic Winter Games
Feb - Mar 2014
Finnish Meteorological Institute
The
Impact
Issue
Sochi 2014 Road Weather Pilot: End-user Feedback
1 = Fully disagree / 2 = Disagree / 3 = Neutral / 4 = Agree / 5 = Fully agree
Friction forecast
Storage terms Road weather
gave a good
(water, snow, ice) forecast was
estimate of
are realistic
useful
slipperiness
Date
16.2.2014
20.2.2014
21.2.2014
5
23.2.2014
5
Notes (free text)
5
4
Precipitation, however road surface dried towards Adler
5
5
2
3
Dry road surface, i.e. No real need for RW forecast
5
2
Dry road surface, i.e. No real need for RW forecast
Variable road surface humidity
End-user ( = Finnish skiing service team ) comments:
• ”HARMONIE was superior”
• ”HARMONIE visibility forecasts were excellent”
• ”HARMONIE precipitation and cloud height forecasts were
highly useful”
• ”HARMONIE five-panel user interface was really good”
[email protected]
FROST-2014 Verification Framework
26
XXII Olympic Winter Games
Feb - Mar 2014
Finnish Meteorological Institute
Future actions and activities
 Comprehensive diagnostic verification
 Re-run of statistics after full data sets available
 Compare with others’ verification results
 Joint reporting/ publishing with WMO FROST-2014 expert group
 Consider societal aspects  The Impact Issue  WMO SERA ?
 Presentation of results:
 WWOSC, Montreal; FROST Fall Meeting; EMS; AMS etc…
 Do same or similar at 2018 Winter Olympics in S-Korea:
[email protected]
FROST-2014 Verification Framework
27
XXII Olympic Winter Games
Feb - Mar 2014
Finnish Meteorological Institute
Thank You
[email protected]
FROST-2014 Verification Framework
(Sochi, 2013; Photos © Nikolai Bocharnikov)
28