A Robustness Study of StateoftheArt Surrogate W

Authors: Mats Danielson Love Ekenberg

Publish Date: 2016/07/19

Volume: 26, Issue: 4, Pages: 677-691

PDF Link

Abstract

A vast number of methods for solving multicriteria decision problems have been suggested for assessing criteria weights requiring more exact input data than users normally are able to provide In particular the selection of adequate criteria weights is difficult and in order to be realistic other methods must be introduced One class of such methods is to introduce so called surrogate weights where numerical weights are assigned to each criterion based on a cardinal or ordinal rank ordering assumed to represent the information extracted from the user One essential problem is the robustness of such methods In this article we compare stateoftheart methods based on surrogate weights from the literature and utilising a simulation approach discuss underlying assumptions and robustness properties This results in a quantitative measurement of these weighting methods and a methodology applicable also to forthcoming methodsIn multicriteria decision analysis MCDM the most common underlying measurement mechanism is MultiAttribute Value or Utility Theory MAVT / MAUT Within MAUT a common form of evaluation function is the additive model Vleft a right = mathop sum nolimits i=1m w i v i left a right where Vleft a right is the overall value of alternative a v i left a right is the value of the alternative under criterion i and w i is the weight of this criterion One of the problems with the additive model as well as other models is that in reallife decision making numerically precise information is seldom available and when it comes to providing reasonable weights for the criteria most decisionmakers experience difficulties due to most humans seemingly do not have the required granulation capacity and also suffer from other cognitive deficiencies pertinent to the specification of a decision problem To somewhat facilitate eliciting weights from decisionmakers some of the approaches in the literature utilise ordinal or imprecise importance information to determine criteria weights and sometimes values of alternatives Other approaches instead make use of surrogate weights which represent the most likely interpretation of the preferences expressed by a decisionmaker or a group of decisionmakers This paper deals with the latter approach to eliciting preferences or importance informationHowever it is not obvious how to determine the decision quality of a multicriteria surrogate weighting method Methods were mostly assessed in case studies until Barron and Barrett 1996a introduced a process utilising systematic simulations The basic idea is to generate surrogate weights as well as “true” reference weights from some underlying distribution and investigate how well the result of using surrogate numbers match the result of using the “true” results The idea in itself is good but the methodology is vulnerable since the validation result is heavily dependent on the distribution used for generating the weight vectors Barron and Barrett themselves 1996a argue that the elicitation of exact weights demands an exactness which does not exist in the mind of the decisionmaker and already von Winterfeldt and Edwards 1986 claim that “the precision of numbers is illusory” And for example ratio weight procedures can be difficult to accurately employ due to response errors Jia et al 1998 The common lack of reasonably complete information increases this problem significantly Several attempts have been made to resolve this issue Methods allowing for less demanding ways of ordering the criteria such as ordinal rankings or interval approaches for determining criteria weights and values of alternatives have been suggested The idea is as far as possible not to force decisionmakers to express unrealistic misleading or meaningless statements but at the same time being able to utilise the information the decisionmaker is able to supply An approach of this type is to use surrogate weights which are derived from ordinal importance information Barron and Barrett 1996a b Katsikopoulos and Fasolo 2006 In such methods the decisionmaker provides information on the rank order of the criteria ie supplies ordinal information on importance Thereafter this information is converted into numerical weights consistent with the extracted ordinal information Several proposals on how to convert the rankings into numerical weights exist in the literature eg rank sum RS weights and rank reciprocal RR weights Stillwell et al 1981 and centroid ROC weights Barron 1992 However the use of only ordinal information is often perceived as being too vague or imprecise resulting in a lack of confidence in the alternatives’ final rankingsFurthermore it is not obvious how “correct” a surrogate weight method is since the “real” weights are unknown or even inexistent in some objective sense The decision quality of a method was at first mostly assessed in case studies until Barron and Barrett 1996a introduced a process utilising systematic simulations The basic idea is to generate surrogate weights as well as “true” reference weights from some underlying distribution and investigate how well the result of using surrogate numbers match the result of using the “true” numbers The idea is good but is nevertheless vulnerable since the validation result is heavily dependent on the distribution used for generating the weight vectorsIn this article we discuss a spectrum of methods for increasing the expressive power of user statements with a particular aim at how the weight functions still can be reasonably elicited while preserving the comparative simplicity and correctness of ranking approaches Below we discuss and compare some important aspects of robustness of a set of ranking methods for weights as well as their relevance and correctness After having briefly recapitulated some ordinal ranking methods in the Sect 2 we continue with stateoftheart ranking methods taking strength into account and discuss a spectrum of interesting candidates as well as cognitive models of decisionmakers Thereafter using simulations we investigate robustness properties of the methods and conclude with pointing out according to the results a particularly attractive method for weight elicitationIn multicriteria decision making MCDM different elicitation formalisms have been proposed by which a decisionmaker can express preferences Such formalisms are sometimes based on scoring points as in point allocation PA or direct rating DR methods1 In PA the decisionmaker is given a point sum eg 100 to distribute among the criteria Sometimes it is pictured as putty with the total mass of 100 being divided and put on the criteria The more mass the larger weight on a criterion and the more important it is When the first N1 criteria have received their weights the last criterion’s weight is automatically determined as the remaining mass Thus in PA there is N1 degrees of freedom DoF for N criteria DR on the other hand puts no limit to the total number of points to be allocated The decisionmaker allocates as many points as desired to each criterion The points are subsequently normalized by dividing by the sum of points allocated When the first N1 criteria have received their weights the last criterion’s weight still has to be assigned by the decisionmaker Thus in DR there are N degrees of freedom for N criteria Regardless of elicitation method the assumption is that all elicitation is made relative to a weight distribution held by the decisionmaker2Providing ordinal rankings of criteria seems to avoid some of the difficulties associated with the elicitation of exact numbers It puts fewer demands on decisionmakers and is thus in a sense effortsaving Furthermore there are techniques such as those above for handling ordinal rankings with some success However decisionmakers might in many cases have more knowledge of the decision situation even if the information is not precise For instance importance relation information containing strengths may implicitly exist3 However these cannot be taken into account in the transformation of an ordinal rank order into weights This entails that the surrogate weights may not really reflect what the decisionmaker actually means by his/her ranking Some form of strengths often exist and this information should reasonably be used when transforming orderings into weights to utilise all the information the decisionmaker is able to supply Below we will therefore investigate whether the above ordinal methods can be successfully extended to accommodate some information regarding relational strengths as well ie to handle ordinal information together with strength relations information while still preserving the property of being less demanding and more practically useful than other types of methods The idea is that instead of using a predetermined conversion method as in eg ROC weights to obtain surrogate weights from an ordinal criteria ranking the decisionmaker will be able to express and utilise known differences in importance between the criteria

Keywords: