An outofsample evaluation framework for DEA with

Authors: Jamal Ouenniche Kaoru Tone

Publish Date: 2017/02/17

Volume: 254, Issue: 1-2, Pages: 235-250

PDF Link

Abstract

Nowadays data envelopment analysis DEA is a wellestablished nonparametric methodology for performance evaluation and benchmarking DEA has witnessed a widespread use in many application areas since the publication of the seminal paper by Charnes Cooper and Rhodes in 1978 However to the best of our knowledge no published work formally addressed outofsample evaluation in DEA In this paper we fill this gap by proposing a framework for the outofsample evaluation of decision making units We tested the performance of the proposed framework in risk assessment and bankruptcy prediction of companies listed on the London Stock Exchange Numerical results demonstrate that the proposed outofsample evaluation framework for DEA is capable of delivering an outstanding performance and thus opens a new avenue for research and applications in risk modelling and analysis using DEA as a nonparametric frontierbased classifier and makes DEA a real contender in industry applications in banking and investmentSince the publication of the seminal paper by Charnes Cooper and Rhodes in 1978 Data envelopment analysis DEA has become a wellestablished nonparametric methodology for performance evaluation and benchmarking DEA has witnessed a widespread use in many application areas—see Liu et al 2013 for a recent survey and Mousavi et al 2015 and Xu and Ouenniche 2011 2012a b for a recent application area—along with many methodological contributions—see for example Banker et al 1984 Andersen and Petersen 1993 Tone 2001 2002 and Seiford and Zhu 2003 Despite the growing use of DEA to the best of our knowledge no published work formally addressed outofsample evaluation in DEA In this paper we fill this gap by proposing a framework for the outofsample evaluation of decision making unitsWe illustrate the use of the proposed framework in bankruptcy prediction of companies listed on the London Stock Exchange Note that prediction of risk class or bankruptcy is one of the major activities in auditing firms’ risks and uncertainties The design of reliable models to predict bankruptcy is crucial for many decision making processes Bankruptcy prediction models could be divided into two broad categories depending on whether they are static see for example Altman 1968 1983 Taffler 1984 Theodossiou 1991 Ohlson 1980 Zmijewski 1984 or dynamic see for example Shumway 2001 Bharath and Shumway 2008 Hillegeist et al 2004 In this paper we shall focus on the first category of models to illustrate how outofsample evaluation of companies could be performed The most popular static bankruptcy prediction models are based on statistical methodologies eg Altman 1968 1983 Taffler 1984 stochastic methodologies eg Theodossiou 1991 Ohlson 1980 Zmijewski 1984 and artificial intelligence methodologies eg Kim and Han 2003 Li and Sun 2011 Zhang et al 1999 Shin et al 2005 DEA methodologies are increasingly gaining popularity in bankruptcy prediction eg Cielen et al 2004 Paradi et al 2004 Premachandra et al 2011 Shetty et al 2012 However the issue of outofsample evaluation remains to be addressed when DEA is used as a classifierThe remainder of this paper is organised as follows In Sect 2 we propose a formal framework for performing outofsample evaluation in DEA In Sect 3 we provide information on the bankruptcy data we used along with details on the design of our experiment and present our empirical findings Finally Sect 4 concludes the paperNowadays outofsample evaluation of statistical stochastic and artificial intelligence methodologies for prediction of both continuous and discrete variables is commonly used for validating prediction models and testing their performance before actual implementation The rationale for using outofsample testing lies in the following well known facts First models or methods selected based on insample performance may not best predict postsample data Second insample errors are likely to understate prediction errors Third for continuous variables prediction intervals built on insample standard errors are likely to be too narrow The setup of the standard outofsample analysis framework requires one to split the historical data set into two subsets where the first subset often referred to as a training set an estimation set or an initialization set is used to estimate the parameters of a model whereas the second subset generally referred to as the test set or the handout set is used to test the prediction performance of the fitted model The counterpart of this testing framework is lacking in DEA In this paper we propose an outofsample evaluation framework for static DEA models The proposed framework in general in that it can be used for any classification problem or number of classes and any application Note that without loss of generality the proposed framework is customized for a bankruptcy prediction application with two risk classes eg bankrupt class and nonbankrupt class or low risk of bankruptcy class and high risk of bankruptcy class as customary in most research on bankruptcy prediction for the sake of illustrating the empirical performance of our framework Obviously this risk classification into two categories or classes could be refined if the researcher/analyst wished to do so into more than two classes when the presence of nonzero slacks is suspected or proven to be a driver of bankruptcy for example one might be interested in refining each of the above mentioned risk classes into two subclasses depending on whether the slacks of a bankrupt respectively nonbankrupt DMU sum to zero or not In other practical settings the researcher/analyst might be interested in the level or degree of distress prior to bankruptcy in which case one might also consider more than two risk or distress classes In the remaining of this paper we denote the variable on risk class belonging as Y Hereafter we describe the main steps of the proposed outofsample evaluation framework for DEAdivide the “historical” sample X into an estimation set X E – hereafter referred to as training sample I—and a test set X T – hereafter referred to as test sample I Then customize X E and X T for the implementation of a specific DEA model by only retaining the input and output information used by the DEA model which results in X EIO and X TIO – hereafter referred to as training sample II and test sample II respectivelysolve an appropriate DEA model to compute DEA efficiency scores and the slacks for DMUs in training sample X EIO and classify them according to a userspecified classification rule into for example risk or bankruptcy classes say hatY EIO Then compare the DEA based classification of DMUs in X EIO into risk classes that is the predicted risk classes hatY EIO with the observed risk classes Y E of DMUs in the training sample and compute the relevant insample performance statisticsuse an appropriate algorithm to classify DMUs in X TIO into for example risk or bankruptcy classes say hatY TIO Then compare the predicted risk classes hatY TIO with the observed risk classes Y T and compute the relevant outofsample performance statistics

Keywords: