Criteria weights for decision making---the easy way

Introduction

At the San Francisco Department of Public Health, Population Health Division, we promote Decision Quality (DQ) [1,2]. DQ starts by knowing what a good decision looks like. A good decision is built with six quality requirements (Table 1).

Table 1: Decision quality requirements: A decision is only as strong as its weakest link
Name Quality requirements Key question
Frame Appropriate frame What are we deciding and why?
Choices Creative alternatives What choices do we have?
Results Clear values and trade-offs What consequences do we care about?
Data Relevant and reliable information What do we need to know?
Reasoning Sounding reasoning Are we thinking straight?
Commitment Commitment to action Is there commitment to action?

The DQ requirements connect in an intentional way and a causal graph (Figure 1) depicts how these requirements are causally related. Choices, results, and data are a triad called the “decision basis” (blue). For details see Reference [1] or [2].

Figure 1: Causal graph of the decision quality process. Note that the quality of a decision process does not depend on the final results.

Making high quality decisions as a group can be challenging. Our brains prefer intuitive (System 1) decision making. Deliberations are cognitively exhausting (i.e., System 2). That’s why, unfortunately, it’s easier to defer decision-making to an authority figure. It’s better to embrace team decision making but practice with simple tools and methods that promote group engagement, deliberation, and DQ.

Making single and multi-attribute decisions

Single-attribute decision

Suppose our team needs to select from among several alternatives and it will be based on a single attribute; for example, choosing one of five available colors to paint our office. That attribute—color—becomes our single criterion and we apply the ranking method illustrated below.

Multi-attribute decision

However, more commonly, based on our frame (purpose, perspective, and scope) our team needs to select from among several alternatives (choices), each of which has mutliple attributes. Selection examples include buying a smartphone, hiring an employee, or selecting a vendor. As individuals and as a group, we have preferences for these attributes based on what we know (data) and what we want (results). We need sound reasoning but will not require a complex decision analysis. Engaging the team in fair and transparent deliberative decision making builds consensus and commitment.

Our main task is to (a) develop criteria based on these attributes, (b) weight the criteria based on importance to us (preference), and (c) rate the alternatives based on the criteria and evidence (data, community, experts, analysis). Ideally, the criteria should be weighted without any knowledge of the alternatives. This is to prevent evaluators from biasing the criteria weights in favor of their favorite alternative.

Now, suppose we wish to buy a car and our choices are a Honda Civic, and Subaru Impreza, or Toyota Corolla. We have data on the following attributes: safety (S), mileage (M), reliability (R), color (C), price (P), and resale value (V). Table 2 summarizes the DQ requirements for buying our car.

Table 2: Decision quality requirements for buying a car
Name Key question Answer
Frame What are we deciding and why? Buy a car we need (vs. want)
Choices What choices do we have? Civic, Corolla, or Impreza
Results What consequences do we care about? Car with combination of attributes that best meets our needs, given trade-offs
Data What do we need to know? Data and/or expert knowledge
Reasoning Are we thinking straight? Multi-criteria decision method
Commitment Is there commitment to action? Participatory, fair, transparent, objective

Calculating criteria weights—the easy way

Group deliberative decision-making is cognitively exhausting. So, anything you can do to make the process easier will keep team members engaged. Do not let “perfection become the enemy of the good”. The easiest way to generate criteria weights from a team of evaluators is to use a rank ordinal method [3].

Give evaluators small pieces of paper with one criterion printed on on each. If we have five criteria, they get five small pieces of paper. Have them rank them from top to bottom. Once they have ranked them, tape their ranking onto an 8.5in x 11in paper and hand to the facilitator. This ranking is entered into the computer for analysis (see below).

Ratio ordinal method in R

We will demonstrate this method using the R function rank.ordinal.weights (provided in the Appendix). Suppose we have five evaluators who rank the criteria for buying a car.

In R (or RStudio) we conduct the following analysis:

eval1 <- c("M", "C", "P", "S", "R", "V")
eval2 <- c("M", "C", "S", "R", "V", "P")
eval3 <- c("C", "V", "P", "M", "R", "S")
eval4 <- c("M", "V", "C", "S", "R", "P")
eval5 <- c("S", "P", "C", "R", "M", "V")
mycritab <- rbind(eval1, eval2, eval3, eval4, eval5)
#### next two lines are optional but useful for displaying results
crit <- c("C", "M", "P", "R", "S", "V") # criteria in alphabetical order
labs <- c("Color", "Mileage", "Price", "Reliability", "Safety", "Value (resale)")
rlts <- rank.ordinal.weights(x = mycritab, criteria = crit, labels = labs)
rlts
## $x
##       [,1] [,2] [,3] [,4] [,5] [,6]
## eval1 "M"  "C"  "P"  "S"  "R"  "V" 
## eval2 "M"  "C"  "S"  "R"  "V"  "P" 
## eval3 "C"  "V"  "P"  "M"  "R"  "S" 
## eval4 "M"  "V"  "C"  "S"  "R"  "P" 
## eval5 "S"  "P"  "C"  "R"  "M"  "V" 
## 
## $criteria
## [1] "C" "M" "P" "R" "S" "V"
## 
## $labels
## [1] "Color"          "Mileage"        "Price"          "Reliability"   
## [5] "Safety"         "Value (resale)"
## 
## $results
##    Evaluator Criterion     Weight          Label
## 1      eval1         M 0.33613445        Mileage
## 2      eval1         C 0.22408964          Color
## 3      eval1         P 0.16806723          Price
## 4      eval1         S 0.12605042         Safety
## 5      eval1         R 0.08963585    Reliability
## 6      eval1         V 0.05602241 Value (resale)
## 7      eval2         M 0.33613445        Mileage
## 8      eval2         C 0.22408964          Color
## 9      eval2         S 0.16806723         Safety
## 10     eval2         R 0.12605042    Reliability
## 11     eval2         V 0.08963585 Value (resale)
## 12     eval2         P 0.05602241          Price
## 13     eval3         C 0.33613445          Color
## 14     eval3         V 0.22408964 Value (resale)
## 15     eval3         P 0.16806723          Price
## 16     eval3         M 0.12605042        Mileage
## 17     eval3         R 0.08963585    Reliability
## 18     eval3         S 0.05602241         Safety
## 19     eval4         M 0.33613445        Mileage
## 20     eval4         V 0.22408964 Value (resale)
## 21     eval4         C 0.16806723          Color
## 22     eval4         S 0.12605042         Safety
## 23     eval4         R 0.08963585    Reliability
## 24     eval4         P 0.05602241          Price
## 25     eval5         S 0.33613445         Safety
## 26     eval5         P 0.22408964          Price
## 27     eval5         C 0.16806723          Color
## 28     eval5         R 0.12605042    Reliability
## 29     eval5         M 0.08963585        Mileage
## 30     eval5         V 0.05602241 Value (resale)
## 
## $ranking
##   Criterion          Label    Weight
## M         M        Mileage 0.2448179
## C         C          Color 0.2240896
## S         S         Safety 0.1624650
## P         P          Price 0.1344538
## V         V Value (resale) 0.1299720
## R         R    Reliability 0.1042017

To print out a nice table of final results in Rmarkdown:

knitr::kable(rlts$ranking, align=c('l','l','l'), caption ="Criteria weight for buying a car")
Table 3: Criteria weight for buying a car
Criterion Label Weight
M M Mileage 0.2448179
C C Color 0.2240896
S S Safety 0.1624650
P P Price 0.1344538
V V Value (resale) 0.1299720
R R Reliability 0.1042017

Table 3 displays the final criteria weights. Using any multi-criteria decision method, (a) score each car using the criteria (e.g., analytic hierarchy process), and (b) weight the scores using the criteria weights. That’s it—you’re done!

How to use criteria weights (optional; intermediate)

We now have six criteria weights (\(w_i\)). How do we use them? (What follows is just arithmetic.) Each of five evaluators (\(E_k\)) must rate (\(R_{ijk}\)) the three cars (\(A_j\)) based on these six criteria (\(C_i\)). Let’s suppose they will rate each car based on “Level of Acceptability” using a 7-point Likert Scale.1 (We are using the rating method to keep things simple.)

Level of Acceptability

  1. Totally unacceptable
  2. Unacceptable
  3. Slightly unacceptable
  4. Neutral
  5. Slightly acceptable
  6. Acceptable
  7. Perfectly Acceptable

Therefore the rating sheet for Evaluator \(k\) might look like Table 4. Notice how the rating sheet enables evaluators to do a side-by-side comparision of each alternative by criteria. This is a critical evaluation design principle that should not be violated—but often is!

Table 4: Rating sheet for Evaluator \(k\) (for \(k=1,2,3,4,5\))
Criteria (\(C_i\)) Honda Civic (\(A_1\)) Subaru Impreza (\(A_2\)) Toyota Corolla (\(A_3\))
1. Color \(R_{111}\) \(R_{121}\) \(R_{131}\)
2. Mileage \(R_{211}\) \(R_{221}\) \(R_{231}\)
3. Price \(R_{311}\) \(R_{321}\) \(R_{331}\)
4. Reliability \(R_{411}\) \(R_{421}\) \(R_{431}\)
5. Safety \(R_{511}\) \(R_{521}\) \(R_{531}\)
6. Value \(R_{611}\) \(R_{621}\) \(R_{631}\)

Now, we just use multiplication to re-weight the rating scores and sum across criteria and evaluator to get the weighted rating for each car (Table 5).

Table 5: Rating sheet for Evaluator \(k\) (for \(k=1,2,3,4,5\))
Criteria (\(C_i\)) Honda Civic (\(A_1\)) Subaru Impreza (\(A_2\)) Toyota Corolla (\(A_3\))
1. Color \(w_1 R_{111}\) \(w_1 R_{121}\) \(w_1 R_{131}\)
2. Mileage \(w_2 R_{211}\) \(w_2 R_{221}\) \(w_2 R_{231}\)
3. Price \(w_3 R_{311}\) \(w_3 R_{321}\) \(w_3 R_{331}\)
4. Reliability \(w_4 R_{411}\) \(w_4 R_{421}\) \(w_4 R_{431}\)
5. Safety \(w_5 R_{511}\) \(w_5 R_{521}\) \(w_5 R_{531}\)
6. Value \(w_6 R_{611}\) \(w_6 R_{621}\) \(w_6 R_{631}\)

Therefore, the final weighted rating score for car \(j\) is

\[ R_j^w = \sum_{k=1}^{5} \sum_{i=1}^{6} w_i R_{ijk} \]

This type of “spreadsheet arithmetic” is trivial in R. I will simulate the rating scores and illustrate the arithmetic.

wts <- rlts$ranking$Weight # criteria weights from above
n.crit <- 6 
n.cars <- 3 
n.eval <- 5
  #### simulate evaluator scores
scores <- array(sample(x = 1:7, size = n.crit*n.cars*n.eval, replace = TRUE), 
                dim = c(n.crit, n.cars, n.eval), 
                dimnames = list (Criterion = labs, 
                Car = c("Honda Civic", "Subarua Impreza", "Toyota Corolla"), 
                  Evaluator = 1:5))
scores[,,1] # display Evaluator 1 scores
##                 Car
## Criterion        Honda Civic Subarua Impreza Toyota Corolla
##   Color                    1               7              1
##   Mileage                  5               2              1
##   Price                    6               4              3
##   Reliability              3               6              6
##   Safety                   1               6              5
##   Value (resale)           1               4              2
  #### reweight scores using the `sweep` function
scores.wtd <- sweep(scores, MARGIN = 1, STATS = wts, FUN = "*") 
scores.wtd[,,1] # display Evaluator 1 weighted scores
##                 Car
## Criterion        Honda Civic Subarua Impreza Toyota Corolla
##   Color            0.2448179       1.7137255      0.2448179
##   Mileage          1.1204482       0.4481793      0.2240896
##   Price            0.9747899       0.6498599      0.4873950
##   Reliability      0.4033613       0.8067227      0.8067227
##   Safety           0.1299720       0.7798319      0.6498599
##   Value (resale)   0.1042017       0.4168067      0.2084034
scores.wtd.cars <- apply(scores.wtd, MARGIN = 2, FUN = sum)
scores.wtd.cars # display final weighted car scores 
##     Honda Civic Subarua Impreza  Toyota Corolla 
##        17.16303        19.77703        16.14342

Based on the analysis above we decide to select the Subarua Impreza.

Radar chart (optional; intermediate)

Sometimes it is useful to visually display evaluators’ criteria weighting. This promotes discussion, clarification, and consensus building. Below is a radar (or spider) chart using the Plotly R package.

library(plotly)
r2 <- rlts$results[order(rlts$results$Evaluator,rlts$results$Criterion),]
erlab <- as.character(r2[r2$Evaluator=="eval1","Label"])
erlab <- c(erlab,erlab[1])
er1 <- r2[r2$Evaluator=="eval1","Weight"]
er1 <- c(er1,er1[1])
er2 <- r2[r2$Evaluator=="eval2","Weight"]
er2 <- c(er2,er2[1])
er3 <- r2[r2$Evaluator=="eval3","Weight"]
er3 <- c(er3,er3[1])
er4 <- r2[r2$Evaluator=="eval4","Weight"]
er4 <- c(er4,er4[1])
er5 <- r2[r2$Evaluator=="eval5","Weight"]
er5 <- c(er5,er5[1])
plot_ly(
    type = 'scatterpolar',
    fill = 'toself',
    mode = "lines+markers"
  ) %>%
  add_trace(
    r = er1,
    theta = erlab,
    name = 'Evaluator 1'
  ) %>%
  add_trace(
    r = er2,
    theta = erlab,
    name = 'Evaluator 2'
  ) %>%
  add_trace(
    r = er3,
    theta = erlab,
    name = 'Evaluator 3'
  ) %>%
  add_trace(
    r = er4,
    theta = erlab,
    name = 'Evaluator 4'
  ) %>%
  add_trace(
    r = er5,
    theta = erlab,
    name = 'Evaluator 5'
  ) %>%
  layout(
    polar = list(
      radialaxis = list(
        visible = T,
        range = c(0,0.35)
      )
    )
  )

Summary

Knowing how to quickly weight criteria with a team is very important. The rank ordinal method is very easy to deploy. The R function rank.ordinal.weights is provided below, and its use is illustrated above.

Appendix

Here is the R function. Test and report any bugs to me.

rank.ordinal.weights = function(x, criteria, labels, evaluators,
                               method=c("sr","roc"), weights){
    ## x = matrix: row is criteria ranking for each evaluator
    ## criteria = character vector of criteria factor levels (values)
    ## labels = character vector of criteria labels (long names)
    ## evaluators = evaluator names
    ## weights = full vector of customized weights (optional)
    ## NOTE: unique values in x must be subset of criteria levels
    ##       To override default factors, provide both criteria
    ##       levels and labels
    ## methods = see https://link.springer.com/chapter/10.1007/978-3-319-52624-9_2
    method <- match.arg(method)
    if(method=="sr") {
        calc.rank.wts = function(k){
            if(missing(k)){
                stop("Must provide integer value for k (number of criteria)")
            }
            wi = dd = rep(NA, k)
            for(i in 1:k){
                dd[i] = (1/i)+((k+1-i)/(k))
            }
            denom = sum(dd)
            for(i in 1:k){
                wi[i] = ((1/i)+((k+1-i)/(k)))/(denom)
            }
            return(wi)
        }
    }
    if(method=="roc"){
        calc.rank.wts = function(k){
            if(missing(k)){
                stop("Must provide integer value for k (number of criteria)")
            }
            wi = rep(NA, k)
            for(i in 1:k){
                wi[i] = (1/k)*sum(1/(i:k))
            }
            return(wi)
        }
    }

    nr = nrow(x); nc = ncol(x)
    if(missing(criteria)) criteria = levels(factor(as.vector(t(x))))
    if(missing(labels)) labels = criteria
    if(missing(evaluators)){
        if(!is.null(rownames(x))) evaluators = rownames(x)
        if(is.null(rownames(x))) evaluators = paste("Eval_", 1:nr, sep = "")
    }
    eval.vec = rep(evaluators, rep(nc, nr))
    crit.vec = as.vector(t(x))
    roc.wts = calc.rank.wts(k = nc)
    if(missing(weights)) wts.vec = rep(roc.wts, nr)
    if(missing(weights)) roc.vec = rep(roc.wts, nr)
    df = data.frame(Evaluator = eval.vec,
                    Criterion = factor(crit.vec, levels = criteria),
                    Weight = roc.vec,
                    Label = factor(crit.vec, levels = criteria, labels = labels)
                    )
    criteria.wts = rev(sort(tapply(df$Weight, df$Criterion, sum)/nr))
    criteria.wts2 = as.matrix(rev(sort(tapply(df$Weight,
                                              df$Label, sum)/nr)))
    ranktab = data.frame(Criterion = names(criteria.wts),
                         Label = rownames(criteria.wts2),
                         Weight = criteria.wts)
    list(x= x,
         criteria = criteria,
         labels = labels,
         results = df,
         ranking = ranktab
         )
}

References

1. Aragón TJ, Garcia BA. We will be the best at getting better: An introduction to population health lean [Internet]. San Francisco Department of Public Health; UC Berkeley School of Public Health; 2017. Available from: http://www.escholarship.org/uc/item/825430qn

2. Spetzler C, Winter H, Meyer J. Decision quality: Value creation from better business decisions. 1st ed. Wiley; 2016.

3. Danielson M, Ekenberg L. Trade-offs for ordinal ranking methods in multi-criteria decisions [Internet]. Bajwa D, Koeszegi ST, Vetschera R, editors. Vol. 274. Springer; 2017. pp. 16–27. Available from: https://link.springer.com/chapter/10.1007/978-3-319-52624-9_2


  1. For a good selection of Likert Scale samples visit http://www.marquette.edu/dsa/assessment/documents/Sample-Likert-Scales.pdf.

Avatar
Tomás J. Aragón
Health Officer, City & County of San Francisco; Director, Population Health Division

Related

comments powered by Disqus