Reliability of multimodel and structurally differ

Authors: Tokuta Yokohata James D Annan Matthew Collins Charles S Jackson Michael Tobis Mark J Webb Julia C Hargreaves

Publish Date: 2011/10/12

Volume: 39, Issue: 3-4, Pages: 599-616

PDF Link

Abstract

The performance of several stateoftheart climate model ensembles including two multimodel ensembles MMEs and four structurally different perturbed parameter single model ensembles SMEs are investigated for the first time using the rank histogram approach In this method the reliability of a model ensemble is evaluated from the point of view of whether the observations can be regarded as being sampled from the ensemble Our analysis reveals that in the MMEs the climate variables we investigated are broadly reliable on the global scale with a tendency towards overdispersion On the other hand in the SMEs the reliability differs depending on the ensemble and variable field considered In general the mean state and historical trend of surface air temperature and mean state of precipitation are reliable in the SMEs However variables such as sea level pressure or topofatmosphere clearsky shortwave radiation do not cover a sufficiently wide range in some It is not possible to assess whether this is a fundamental feature of SMEs generated with particular model or a consequence of the algorithm used to select and perturb the values of the parameters As underdispersion is a potentially more serious issue when using ensembles to make projections we recommend the application of rank histograms to assess reliability when designing and running perturbed physics SMEsIn order for society to efficiently mitigate and adapt to climate change it is necessary to have climate projections accompanied by assessments of the uncertainty in the projections Ensembles of climate models sampling uncertainties in model formulation are commonly used as the basis for generation of probabilistic projections It is therefore very important to evaluate the performance of these ensemblesThere are a large number of methods one could adopt to evaluate the performance of model ensembles and there are many examples in the literature These methods generally use one of two paradigms One paradigm sometimes called the truth centred paradigm Knutti et al 2010b assumes that the truth should be close to the centre of the ensemble members ie close to the ensemble mean Knutti et al 2010a investigated the behaviour of the stateoftheart climate model ensemble created by the World Climate Research Programme’s Coupled Model Intercomparison Project Phase 3 CMIP3 Meehl et al 2007 and found that the truth centred paradigm is incompatible with the CMIP3 ensemble the ensemble mean does not converge to observations as the number of ensemble members increases and the pairwise correlation of model errors the differences between model and observation between two ensemble members does not average to zero Knutti et al 2010a Annan and Hargreaves 2010 hereafter AH10An alternative paradigm is to consider the truth as being drawn from the distribution sampled by the ensemble In this case the model ensemble can be regarded as perfect if the ensemble members and the truth are “statistically indistinguishable” In this case the truth is not necessarily at the centre of the ensemble Predictions made with such a model ensemble are regarded as “reliable” in the technical sense that the natural probabilistic interpretation based on counting ensemble members matches the frequency of occurrence of predicted events evaluated over multiple verifications This idea of a “statistically indistinguishable” ensemble is common in the field of weather forecasting and other ensemble prediction fields and under this paradigm the reliability of model ensembles can be evaluated through the rank histogram approach Anderson 1996 whereby the distribution of the observed occurrence of an event in the prediction ensembles is evaluated Such an analysis can reveal if prediction ensembles are too narrow too broad or biased In the present paper we analyse the reliability of model ensembles in statistical terms We discuss the concept of “reliability” in more detail in Sect 23AH10 applied the rank histogram method to the evaluation of spatial fields of timeaveraged presentday variables from climate models and concluded that the CMIP3 ensemble appears reasonably reliable on large scales However AH10 only investigated the CMIP3 ensemble and the three most commonly investigated climate variables surface air temperature SAT sea level pressure SLP and precipitation PRCP They did not consider those variables which play an important role in determining the range of climate responses to increasing greenhouse gases such as radiation and/or cloud effects at the top of atmosphere TOA Here we extend the evaluation to those variables and analyse several ensembles two multimodel ensembles MMEs from CMIP3 and four structurally different single model ensembles SMEs sometimes also referred to a perturbed physics or perturbed parameter ensembles with different ranges of climate sensitivity We investigate the relationship between climate sensitivity and the reliability of the presentday climate simulation We also check the validity of the rank histogram approach by comparing the modeldata difference with the ensemble spread through calculating the root mean square modeldata difference RMSE and the standard deviation of the ensemble SDIn Sect 2 we describe the model ensembles and the application of the rank histogram approach including a description of the statistical method used to define the reliability of model ensembles from the rank histogram and a method for handling uncertainties in the observations In Sect 3 the results from the rank histogram analyses are described In this study we primarily investigate the reliability of the climatology longterm mean of model simulation of largescale features of climate model ensembles but we also consider the trend for surface air temperature where transient simulations are available that is for the coupled ocean–atmosphere models Our main result is to show that under this analysis the performance of the MME is qualitatively different from and superior to the SMEs A conventional analysis of RMSE and SD is also presented in Sect 3 which supports our results and analysis using rank histograms Finally in Sect 4 we present our conclusionsOne approach is to use an MME which consists of simulations contributed by different models of climate research institutes from around the world often referred to as an “ensemble of opportunity” Each model may be considered a social construct which embodies the beliefs of those modellers who created it as how best to represent the climate system within the computational and technological constraints at the time Thus the whole ensemble may be interpreted at least potentially as sampling our collective beliefs and uncertainties regarding the climate system although the adhoc and uncoordinated nature of the modelbuilding process around the world may raise some doubts as to the plausibility of such an assumption The current MME of stateoftheart global climate models is the CMIP3 ensemble Meehl et al 2007 While this ensemble samples uncertainties in model structure each model has one parameter set and a fixed model structure Some members of the MME may be different resolution versions of the same model structure albeit with resolutiondependent parameters adjusted and some others may share common components

Keywords: