model - ampere

Validation/Evaluation in Environmental
Modelling
AMPERE & PIAMDDI Workshop
Seville
May 28-29 2013
Tony Jakeman, Joseph Guillaume, Barry Croke,
Sondoss El Sawah, Baihua Fu and John Norton
iCAM, Fenner School of Environment and Society &
National Centre for Groundwater Research and Training
[email protected]
Socioeconomic & environmental impacts of climate
change, technology and water policy drivers in the
Namoi catchment – adaptation opportunities
Tony Jakeman, Jenifer Ticehurst, Rachel Blakers, Barry
Croke, Baihua Fu, Wendy Merritt, Darren Sinclair, Neil
Gunningham, Joseph Guillaume, Andrew Ross (ANU)
Allan Curtis and Emily Sharp (CSU)
David Pannell, Alex Gardner, Alison Wilson and Madeleine
Hartley (UWA)
Cameron Holley (UNSW)
Rebecca Kelly (iSNRM and ANU)
Steering Committee: State and local agencies, Namoi Water
(irrigators)
Integrated Model
•  Integrates the work of each of the disciplinary
sub-teams
•  Three components
–  Social Bayesian Network using results of the
social survey
–  Core integrated deterministic model
•  Simulates hydrogeological system, constraints on
extraction, farmer decision making, crop yields and
ecological impacts
•  With inputs of the possible practice changes , climate
change scenarios and water allocation policies
–  An integrated trade-off analysis
Spatial Scale
Hydrological model zones
Hydrological Model Development
•  A key challenge was the choice of
hydrological model structure, including:
–  Surface-groundwater, groundwater level and
routing sub-modules needed
–  Which hydrological processes should be
simulated?
–  The spatial resolution
–  The level of process detail – conceptual or
physics-based?
•  The driving consideration was the needs of
the Integrated Assessment Project
Fraction
CMD
Module
NONLINEAR
MODULE
Effective
Slowflow
Rainfall
Fraction
Surface
Storage
Shallow
Subsurface
Storage
Shallow
Groundwater
Storage
Surface Water
Extractions
Surface
Runoff
Shallow
Streamflow
Sub-surface
Runoff
Discharge
Recharge
Infiltration
Deep
Groundwater
Storage
Groundwater
Extractions
SHALLOW
GROUNDWATER
Quickflow
DEEP
GROUNDWATER
Temperature
Rainfall
SURFACE WATER
Model Structure
Natural Losses
and Lateral Flow Overview questions
• 
• 
• 
• 
What is the evaluation practice (and why)?
What sorts of technical tools do we use?
How useful are they?
The need for avoiding poor predictions is
obvious but what else is required:
- To address uncertainty in problem definition
- To ensure modelling is fit-for purpose
- To constantly question assumptions
•  So what philosophies, processes and methods
can we build on?
9
The underwhelming modelling practice
•  a lot of development of models of environmental
processes and integrated models with some history
matching/calibration – and modellers stubbornly
preferring their familiar paradigm and methods
•  widespread acceptance of key models – incremental
modifications the norm versus rigorous creation
•  much talk but less analysis of uncertainty, and even
sensitivity, in those models; stress testing infrequent
•  scant discussion of model assumptions, strengths and
weaknesses; very little frank reporting of uncertainties
•  model purpose and objective functions weakly argued
and matched
•  stakeholders only beginning to be involved in the whole
‘process’ - for saliency, legitimacy and new knowledge
10
Reasons for poor practice
•  Too easy to publish models with modest but
inadequate evaluation, and obvious overparameterisation masquerading as knowledge
•  Modeller s ignorance and/or complacency
•  Lack of resources
•  Too much science and technology push, not
enough customer pull
•  Sheer volume and complexity of uncertainties
11
Some high-level solutions
•  Enhance publication standards and modelling guidelines
eg Position Papers of Env Mod & Software
•  Improve education and training of modellers – including
fora like MSSANZ and iEMSs
•  Set aside resources for model evaluation
•  Link the application context/purpose(s) to the model
development and testing
•  Devise a workplan capturing a set of issues that should
be addressed and how to identify and prioritise
uncertainties including:
•  Begin/continue serious analysis, revision and associated
documentation of common models
12
Quantitative techniques and model types:
variously used
•  Error analyses of models, time variation of parameters – eg bias,
correlation with inputs and conditions (rainfall-runoff example)
•  Global and regional sensitivity analyses and model emulation to
improve model identifiability, discriminate between alternative
hypotheses, and assess what uncertainties are crucial
•  Psuedo MC (inc. Bayesian) methods to quantify model uncertainty
•  Bounding methods to propagate through models and for solving
inverse problems – non-probabilistic
•  Fuzzy logic approaches
•  Modelling families – various alternates and hybrids
•  Modelling intercomparisons
•  BUT we often forget part of the qualitative – e.g. sensibility/sanity
and explanatory testing, use of stylized facts
13
And in IAM we have some bigger problems
•  Limited data – and only a handful of parameters can be
identified in models, even when data is informative
•  Sensitive parameters can change according to driving
conditions
•  The usually unquantified uncertainty may dominate – the
neglected sources (see next slide)
Ramifications
•  Catalogue, rank and appreciate all relevant types and
sources of uncertainty
•  Analyse model components in context of their linkages
•  Evaluation will unlikely get us all the confidence we d
prefer in our uncertainty bounds – but it can help
reduce and define the margins of uncertainty
14
Unquantified uncertainty – the neglected
sources beyond errors in inputs, outputs and
parameters
•  Neglected sources in the decision pathway may
dominate the usual quantified uncertainty eg
- the conceptual model error (ie model structure
assumptions) may dominate the numerical model
- transfer from mental models to collective wisdom of
conceptual model to numerical model (involves
many judgments that beg for transparency
- we might be solving the wrong problem
- we could be iterating and solving a more
appropriate and/or tractable one
15
Uncertainties in the decision pathway
Identification
Decision prompt Scope • Boundaries of analysis Monitor & evaluate • Trea.ng emerging concerns • Iden.fying need for change Development & Evaluation
Iden=fy data Choose and knowledge methodology • Measurement error • Point of view • Representa.vity • Limita.ons • Imprecision • Assump.ons • Inaccuracy • Technical issues Implement • Adop.on • Compliance Commitment to ac+on Frame Search Deliberate Analyse • How inputs and outputs are represented in the model • ACtudes and rela.ons between stakeholders • Communica.on • Ranking of and trade-­‐offs between objec.ves • Avoiding locally op.mal solu.ons • Missed alterna.ves For integrated modelling: • Model structure • Model parameters • Calibra.on method • Valida.on method • Technical • Integra.on Action
Decision pathway step • Corresponding uncertain.es Some useful developing trends
•  Distinguishing between quantitative
evaluation of the model, and evaluation of
the process and outcomes (surveying)
•  Towards increased debate in framing the
problem and weighing aspects and
uncertainties
•  Use of many models and methods
17
Four useful concepts/techniques
•  Boundary Critique (Midgley 2011, Ulrich
2005)
•  Hypothesis testing in the broad
•  Sensitivity analysis
•  Robust decision making
Guillaume et al. (draft): Beyond quantifying uncertainty: a review of concepts for
predicting uncertain outcomes
18
Boundary Critique
•  Judgments define which models are and aren t plausible
–  inc. assumptions wrt structure, parameters and performance
measures ie feasible sets
•  Secondarily, BC determines what problem the modelling
is trying to address: as important as deciding feasibility
•  Debate and agreement where to fix the boundaries
leading to a core ensemble of agreed models (ie inputs,
structures, parameter values), and a marginal set of
contested models
•  Examining areas of dissenting opinion help to avoid
surprise failures
•  Reducing uncertainty is eliminating an unacceptable
model from the ensemble
19
Hypothesis Testing: beyond the classical
•  Gives problem definition a central, broader and more
explicit role
•  There are limits to probabilities w.r.t. epistemic
uncertainties and stakeholder participation
•  Testing whether a stated conclusion could be false;
whether the model survives an agreed testing process
•  Counter examples can be used for falsification
•  Encourages a designed approach: explicit statement of a
problem and test conditions and the need to be critical
20
Sensitivity Analysis
•  Not just (first) what values of variables make a
difference, but (second) trying to understand what
(variables) cause a change
•  Methods for the second: eg global sensitivity analysis
•  Methods for the first: eg set inversion and scenario
discovery
•  POMORE: estimates how far parameters have to
change to obtain a different conclusion
•  Link with BC: ability to discuss model scenarios within
a feasible set facilitates judgements
21
Robust Decision Making
•  Can extend RDM concept from avoiding ‘actions’ that
lead to failure to making robust decisions about
‘knowledge’
•  Expecting a problem definition to change
•  Identify model ‘scenarios’ that would cause high regret –
anticipating surprise and/or searching for breaking
‘scenarios’ (hypothesis testing links here)
•  Becomes an iterative process addressing the boundary
critique of feasible models and the problem definition
22
How do these four techniques support UA?
BC
•  Constantly questioning assumptions
•  Constantly questioning problem definition
HT
•  Testing whether uncertainty could change conclusions
•  Designing analysis with possible conclusions in mind
SA
•  Trying to understand possible causes of failure of a
prediction
RDM
•  Making changes to counter possible causes of failure of
a prediction
23
Main messages
•  Analyse models and errors for crucial weaknesses,
reducing unnecessary complexity or building it rigorously
•  Focus on purpose, make limits of knowledge explicit,
generate stress-testing model scenarios
•  Emphasize possibilities rather than probabilities
•  Evaluation can be a holistic process, not just a reporting
exercise; focus on defining boundaries, scenarios and
agreed feasibility to refine margins of uncertainty
•  Emphasise learning from evaluating scenarios
•  Uncertainty assessment will never be complete but
opportunity is there for wholesale improvements
•  Develop a workplan among the communities to address
model evaluation systematically
24
References
• 
• 
• 
• 
• 
• 
• 
• 
Bankes, S. C., R. J. Lempert and S. W. Popper (2001). "Computer-assisted reasoning."
Computing in Science & Engineering 3(2): 71-77 DOI: 10.1109/5992.909006
Guillaume J.H.A, Pierce S.A., Jakeman A.J. (2010) Managing uncertainty in determining
sustainable aquifer yield. Groundwater 2010, Canberra, Australia, 31 October – 4
November 2010, from http://www.groundwater2010.com/documents/
GuillaumeJoseph_000.pdf
Guillaume, J.H.A, Qureshi E.M. and Jakeman, A.J. (2012) A structured analysis of
uncertainty surrounding modeled impacts of groundwater extraction rules, Hydrogeology
Journal
Lempert, R. J. (2002). "A new decision sciences for complex systems." Proceedings of the
National Academy of Sciences of the United States of America 99(Suppl 3): 7309-7313,
from http://www.pnas.org/content/99/suppl.3/7309.short
Norton, J. P. (1996). "Roles for deterministic bounding in environmental modelling."
Ecological Modelling 86(2-3): 157-161 DOI: 10.1016/0304-3800(95)00045-3
Reason, J. (2000). Human error: models and management. BMJ 320(7237): 768-770 DOI:
10.1136/bmj.320.7237.768
Refsgaard, J. C., J. P. van der Sluijs, J. Brown and P. van der Keur (2006). "A framework
for dealing with uncertainty due to model structure error." Advances in Water Resources
29(11): 1586-1597 DOI: 10.1016/j.advwatres.2005.11.013
Rittel, H. W. J. and M. M. Webber (1973). "Dilemmas in a general theory of planning."
Policy Sciences 4(2): 155-169 DOI: 10.1007/bf01405730