An operational R-based interpolation facility for climate and meteo

An operational R-based
interpolation facility for
climate and meteo data
From science to operations
DailyMeteo2014
Belgrade 27 June 2014
Dr. Raymond Sluiter
Researcher Geo-ICT
ESA-DOSTAG Delegate
Overview
!   Context
!   Interpolation: research, production environment, users.
!   Data distribution
!   International projects
!   Issues, challenges &
lessons learned.
2
DailyMeteo2014
KNMI
!   KNMI: “The national institute for weather, climate research and
seismology in The Netherlands”...
!   KNMI is an agency of the Ministry of Infrastructure and
Environment.
3
DailyMeteo2014
KNMI - main activities
• 
Operational services
! Weather forecasts (public & aviation)
! Weather alerts
• 
Observations
!   Meteorological / climatological observation network
! Seismological observation network
!   Meteorological satellites & climate satellites
! 
• 
4
Data processing and distribution
Research
! Improve weather forecasting
! Climate research including Climate change models (numerical
computation)
!   Sensor / IT research
DailyMeteo2014
Interpolation: 2000 - 2008
!   One Method: spline
! Arcview 3.2, GMT
!   Manual “intervention”
!   .Png only, no real data available for end-users.
5
DailyMeteo2014
Network
6
DailyMeteo2014
Research
!  Data: temperature, precipitation, radiation, evaporation, wind… .
!  Daily to 30 year averages.
!  1951 - present
!  5 - ~300 measurements.
!  1*1 km resolution.
!  Methods: Kriging, KED, IDW, splines, regression, … .
!  R-libraries: methods, sp, gstat, automap, fields
!  Quality/uncertainty: Kriging variance, cross validation, visual.
!  Metadata generation.
!  Gridded datasets provided through OGC web services.
7
Research
!  Interpolation methods for climate data literature review / R. Sluiter 2009
http://www.knmi.nl/bibliotheek/knmipubIR/IR2009-04.pdf
!  Het interpoleren van temperatuurgegevens / F.W.J. Salet 2009
http://www.knmi.nl/bibliotheek/stageverslagen/stageverslag_Salet.pdf
!  Optimization of Rainfall Interpolation / I. Soenario, R. Sluiter 2010
http://www.knmi.nl/bibliotheek/knmipubIR/IR2010-01.pdf
!  Interpolation of Makkink evaporation in the Netherlands / P. Hiemstra and R.
Sluiter 2011
http://www.knmi.nl/bibliotheek/knmipubTR/TR327.pdf
!  Interpolating wind speed normals from the sparse Dutch network to a high
resolution grid using local roughness from land use maps / A. Stepek and I. L.
Wijnant 2011
http://www.knmi.nl/bibliotheek/knmipubTR/TR321.pdf
!  Interpolation Methods for the Climate Atlas / R. Sluiter 2012
http://www.knmi.nl/bibliotheek/knmipubTR/TR335.pdf
!  Assimilation of satellite data and in-situ data for the improvement
of global radiation maps in the Netherlands / J. van Tiggelen 2014
8
Interpolation facility
! GeoSpatial Interpolation environment (GSIE)
!  Recipe editor
!   Input files
!   Metadata
!   R-scripts
!   Database queries
!   Legend descriptions
! WebGIS
!  Wiki
!  File access
9
Interpolation facility
10
GSIE: Parallel computing
11
For who?
12
Climate atlas:
“Bosatlas van het Klimaat”
For who?
13
Climate atlas:
“Bosatlas van het Klimaat”
For who?
14
Climate atlas:
“Bosatlas van het Klimaat”
For who?
15
Climate atlas:
“Bosatlas van het Klimaat”
For who?
16
Climate atlas:
“Bosatlas van het Klimaat”
For who?
17
Climate atlas:
“Bosatlas van het Klimaat”
For who?
!  Climate atlas, general public & professional users
18
Climate Atlas
http://www.klimaatatlas.nl
19
NCG Commissie Geovisualisatie 23 juni 2014
For who?
!  Climate atlas scenarios, general public & professional users
! http://www.climatescenarios.nl/
20
For who?
!  Climate atlas scenarios, general public & professional users
21
For who?
!  Netherlands Hydrological Instrument, professional users
22
What exactly?
Netherlands Hydrological Instrument, professional users
!   Daily precipitation 1961-present, 1*1km, ~300 observations.
!   using ordinary kriging.
!   Daily Makkink evaporation 1961-present, 1*1km, ~5-36
observations.
!   using spline interpolation.
!   Transformed scenarios time series
23
NHI
24
NHI
Groundwater recharge (mm/year)
25
Evaporation (mm/year)
LCW: (National Coordination Committee Water
Distribution)
Cumulative precipitation deficit (mm) region South-East NL
26
What exactly? – quality/uncertainty
!   Leave One Out Cross Validation (LOOCV): RMSE, R2, ... .
!   Radar daily precipitation; visual comparison:
Interpolation
27
Radar
What exactly? - quality
!   Daily precipitation; Kriging variance:
Interpolation
28
Kriging variance
Present & future research
!   High resolution climatology & data quality
!   Temp, Wind, Radiation
!   Data assimilation (model reanalysis, satellite images)
29
30
Data processing and distribution at KNMI
Keyword:
KDC = KNMI Data Centre
31
DailyMeteo2014
Why KDC?
KNMI data:
!  Large diversity of themes (weather, climate and
seismology)
!  Historical, real-time and forecast data (model data)
!  Research & operational data
!  Applicable in many domains
However:
!  Difficult to find
!  Difficult to use
!  Limited standardization
!  Many different portals
!  High maintenance costs
32
DailyMeteo2014
Why KDC?
Earth Observation context:
•  Atmospheric processing systems
! OMI Data Processing System (ODPS)
! Netherlands SCIAMACHY Data Center (NL-SCIA-DC)
! Gome2 Processing System (G2PS)
! Netherlands Atmospheric Data Center (NADC)
•  Atmospheric portals
! Tropospheric Emission Monitoring Internet Service
(TEMIS)
33
DailyMeteo2014
The KDC basis:
• 
• 
• 
• 
Dataset managers can create and manage datasets
Alle data is archived and managed
Governance is in place
Alle data has metadata
•  Metadata conform NL core set, INSPIRE and WMO and “open data”
•  Suitable for all KNMI data (and more!)
•  Findable by different criteria:
–  Preview
–  Search on location, time, key-words
•  Harmonization of metadata, file formats, file content
•  Solid base for further development
34
DailyMeteo2014
What is in KDC
• 
• 
• 
• 
• 
• 
• 
• 
• 
Temperature and precipitation (daily/climate atlas)
Scatterometer (wind above sea, OSI-SAF)
NL Radar composite/volume
European station time series (EOBS ECA&D)
Multi sensor reanalysis of ozone
OMI cloud/ozone/aerosols
MSG Cloud Physical Properties (MSG-CPP)
EC-Earth model data
etc.
KDC is growing with (“open data”) datasets and
functionality
35
DailyMeteo2014
How does it look like?
http://data.knmi.nl
36
DailyMeteo2014
Find
37
DailyMeteo2014
KNMI
Data Centrum | 14 december 2012
38
DailyMeteo2014
39
DailyMeteo2014
40
DailyMeteo2014
41
DailyMeteo2014
42
DailyMeteo2014
International:
•  ECA&D
•  E-OBS
•  And many more
43
DailyMeteo2014
44
DailyMeteo2014
45
DailyMeteo2014
46
DailyMeteo2014
47
DailyMeteo2014
48
DailyMeteo2014
49
DailyMeteo2014
50
DailyMeteo2014
51
DailyMeteo2014
52
DailyMeteo2014
Issues, challenges and lessons learned
•  Large demand for (high resolution) interpolated datasets.
•  More communities are reached; quality/uncertainty information
becomes crucial:
•  Metadata is essential for including quality/uncertainty
descriptors.
•  Open issues how to communicate & visualize quality/
uncertainty information.
•  Station density.
•  International challenges:
•  (open) data availability.
•  Standardization gridded products (INSPIRE: 2020).
•  Organizational challenges:
•  Manage your internal processes (“chain”) well.
•  (Meta)data management.
•  More resources needed for (inter)national collaboration.
53
An operational R-based
interpolation facility for
climate and meteo data
From science to operations
DailyMeteo2014
Belgrade 27 June 2014
Dr. Raymond Sluiter
Researcher Geo-ICT
ESA-DOSTAG Delegate