## Introduction

At the San Francisco Department of Public Health, Population Health Division, we promote **Decision Quality** (DQ) [1,2]. DQ starts by knowing what a good decision looks like. A good decision is built with six quality requirements (Table 1).

Name | Quality requirements | Key question |
---|---|---|

Frame | Appropriate frame | What are we deciding and why? |

Choices | Creative alternatives | What choices do we have? |

Results | Clear values and trade-offs | What consequences do we care about? |

Data | Relevant and reliable information | What do we need to know? |

Reasoning | Sounding reasoning | Are we thinking straight? |

Commitment | Commitment to action | Is there commitment to action? |

The DQ requirements connect in an intentional way and a **causal graph** (Figure 1) depicts how these requirements are causally related. Choices, results, and data are a triad called the “decision basis” (blue). For details see Reference [1] or [2].

Making high quality decisions as a group can be challenging. Our brains prefer intuitive (System 1) decision making. Deliberations are cognitively exhausting (i.e., System 2). That’s why, unfortunately, it’s easier to defer decision-making to an authority figure. It’s better to embrace team decision making but practice with simple tools and methods that promote group engagement, deliberation, and DQ.

## Making single and multi-attribute decisions

### Single-attribute decision

Suppose our team needs to select from among several alternatives and it will be based on a **single attribute**; for example, choosing one of five available colors to paint our office. That attribute—color—becomes our single criterion and we apply the ranking method illustrated below.

### Multi-attribute decision

However, more commonly, based on our **frame** (purpose, perspective, and scope) our team needs to select from among several alternatives (**choices**), each of which has **mutliple attributes**. Selection examples include buying a smartphone, hiring an employee, or selecting a vendor. As individuals and as a group, we have preferences for these attributes based on what we know (**data**) and what we want (**results**). We need sound **reasoning** but will not require a complex decision analysis. Engaging the team in fair and transparent deliberative decision making builds consensus and **commitment**.

Our main task is to (a) **develop criteria** based on these attributes, (b) **weight the criteria** based on importance to us (preference), and (c) **rate the alternatives** based on the criteria and evidence (data, community, experts, analysis). Ideally, the criteria should be weighted without any knowledge of the alternatives. This is to prevent evaluators from biasing the criteria weights in favor of their favorite alternative.

Now, suppose we wish to buy a car and our choices are a Honda Civic, and Subaru Impreza, or Toyota Corolla. We have data on the following attributes: safety (S), mileage (M), reliability (R), color (C), price (P), and resale value (V). Table 2 summarizes the DQ requirements for buying our car.

Name | Key question | Answer |
---|---|---|

Frame | What are we deciding and why? | Buy a car we need (vs. want) |

Choices | What choices do we have? | Civic, Corolla, or Impreza |

Results | What consequences do we care about? | Car with combination of attributes that best meets our needs, given trade-offs |

Data | What do we need to know? | Data and/or expert knowledge |

Reasoning | Are we thinking straight? | Multi-criteria decision method |

Commitment | Is there commitment to action? | Participatory, fair, transparent, objective |

## Calculating criteria weights—the easy way

Group deliberative decision-making is cognitively exhausting. So, anything you can do to make the process easier will keep team members engaged. Do not let “perfection become the enemy of the good”. The easiest way to generate criteria weights from a team of evaluators is to use a rank ordinal method [3].

Give evaluators small pieces of paper with one criterion printed on on each. If we have five criteria, they get five small pieces of paper. Have them rank them from top to bottom. Once they have ranked them, tape their ranking onto an 8.5in x 11in paper and hand to the facilitator. This ranking is entered into the computer for analysis (see below).

### Ratio ordinal method in R

We will demonstrate this method using the R function `rank.ordinal.weights`

(provided in the Appendix). Suppose we have five evaluators who rank the criteria for buying a car.

In R (or RStudio) we conduct the following analysis:

```
eval1 <- c("M", "C", "P", "S", "R", "V")
eval2 <- c("M", "C", "S", "R", "V", "P")
eval3 <- c("C", "V", "P", "M", "R", "S")
eval4 <- c("M", "V", "C", "S", "R", "P")
eval5 <- c("S", "P", "C", "R", "M", "V")
mycritab <- rbind(eval1, eval2, eval3, eval4, eval5)
#### next two lines are optional but useful for displaying results
crit <- c("C", "M", "P", "R", "S", "V") # criteria in alphabetical order
labs <- c("Color", "Mileage", "Price", "Reliability", "Safety", "Value (resale)")
rlts <- rank.ordinal.weights(x = mycritab, criteria = crit, labels = labs)
rlts
## $x
## [,1] [,2] [,3] [,4] [,5] [,6]
## eval1 "M" "C" "P" "S" "R" "V"
## eval2 "M" "C" "S" "R" "V" "P"
## eval3 "C" "V" "P" "M" "R" "S"
## eval4 "M" "V" "C" "S" "R" "P"
## eval5 "S" "P" "C" "R" "M" "V"
##
## $criteria
## [1] "C" "M" "P" "R" "S" "V"
##
## $labels
## [1] "Color" "Mileage" "Price" "Reliability"
## [5] "Safety" "Value (resale)"
##
## $results
## Evaluator Criterion Weight Label
## 1 eval1 M 0.33613445 Mileage
## 2 eval1 C 0.22408964 Color
## 3 eval1 P 0.16806723 Price
## 4 eval1 S 0.12605042 Safety
## 5 eval1 R 0.08963585 Reliability
## 6 eval1 V 0.05602241 Value (resale)
## 7 eval2 M 0.33613445 Mileage
## 8 eval2 C 0.22408964 Color
## 9 eval2 S 0.16806723 Safety
## 10 eval2 R 0.12605042 Reliability
## 11 eval2 V 0.08963585 Value (resale)
## 12 eval2 P 0.05602241 Price
## 13 eval3 C 0.33613445 Color
## 14 eval3 V 0.22408964 Value (resale)
## 15 eval3 P 0.16806723 Price
## 16 eval3 M 0.12605042 Mileage
## 17 eval3 R 0.08963585 Reliability
## 18 eval3 S 0.05602241 Safety
## 19 eval4 M 0.33613445 Mileage
## 20 eval4 V 0.22408964 Value (resale)
## 21 eval4 C 0.16806723 Color
## 22 eval4 S 0.12605042 Safety
## 23 eval4 R 0.08963585 Reliability
## 24 eval4 P 0.05602241 Price
## 25 eval5 S 0.33613445 Safety
## 26 eval5 P 0.22408964 Price
## 27 eval5 C 0.16806723 Color
## 28 eval5 R 0.12605042 Reliability
## 29 eval5 M 0.08963585 Mileage
## 30 eval5 V 0.05602241 Value (resale)
##
## $ranking
## Criterion Label Weight
## M M Mileage 0.2448179
## C C Color 0.2240896
## S S Safety 0.1624650
## P P Price 0.1344538
## V V Value (resale) 0.1299720
## R R Reliability 0.1042017
```

To print out a nice table of final results in Rmarkdown:

`knitr::kable(rlts$ranking, align=c('l','l','l'), caption ="Criteria weight for buying a car")`

Criterion | Label | Weight | |
---|---|---|---|

M | M | Mileage | 0.2448179 |

C | C | Color | 0.2240896 |

S | S | Safety | 0.1624650 |

P | P | Price | 0.1344538 |

V | V | Value (resale) | 0.1299720 |

R | R | Reliability | 0.1042017 |

Table 3 displays the final criteria weights. Using any multi-criteria decision method, (a) score each car using the criteria (e.g., analytic hierarchy process), and (b) weight the scores using the criteria weights. That’s it—you’re done!

### How to use criteria weights (optional; intermediate)

We now have six criteria weights (\(w_i\)). How do we use them? (What follows is just arithmetic.) Each of five evaluators (\(E_k\)) must rate (\(R_{ijk}\)) the three cars (\(A_j\)) based on these six criteria (\(C_i\)). Let’s suppose they will rate each car based on “Level of Acceptability” using a 7-point Likert Scale.^{1} (We are using the *rating method* to keep things simple.)

**Level of Acceptability**

- Totally unacceptable
- Unacceptable
- Slightly unacceptable
- Neutral
- Slightly acceptable
- Acceptable
- Perfectly Acceptable

Therefore the rating sheet for Evaluator \(k\) might look like Table 4. *Notice how the rating sheet enables evaluators to do a side-by-side comparision of each alternative by criteria*. This is a critical evaluation design principle that should not be violated—but often is!

Criteria (\(C_i\)) | Honda Civic (\(A_1\)) | Subaru Impreza (\(A_2\)) | Toyota Corolla (\(A_3\)) |
---|---|---|---|

1. Color | \(R_{111}\) | \(R_{121}\) | \(R_{131}\) |

2. Mileage | \(R_{211}\) | \(R_{221}\) | \(R_{231}\) |

3. Price | \(R_{311}\) | \(R_{321}\) | \(R_{331}\) |

4. Reliability | \(R_{411}\) | \(R_{421}\) | \(R_{431}\) |

5. Safety | \(R_{511}\) | \(R_{521}\) | \(R_{531}\) |

6. Value | \(R_{611}\) | \(R_{621}\) | \(R_{631}\) |

Now, we just use multiplication to re-weight the rating scores and sum across criteria and evaluator to get the weighted rating for each car (Table 5).

Criteria (\(C_i\)) | Honda Civic (\(A_1\)) | Subaru Impreza (\(A_2\)) | Toyota Corolla (\(A_3\)) |
---|---|---|---|

1. Color | \(w_1 R_{111}\) | \(w_1 R_{121}\) | \(w_1 R_{131}\) |

2. Mileage | \(w_2 R_{211}\) | \(w_2 R_{221}\) | \(w_2 R_{231}\) |

3. Price | \(w_3 R_{311}\) | \(w_3 R_{321}\) | \(w_3 R_{331}\) |

4. Reliability | \(w_4 R_{411}\) | \(w_4 R_{421}\) | \(w_4 R_{431}\) |

5. Safety | \(w_5 R_{511}\) | \(w_5 R_{521}\) | \(w_5 R_{531}\) |

6. Value | \(w_6 R_{611}\) | \(w_6 R_{621}\) | \(w_6 R_{631}\) |

Therefore, the final weighted rating score for car \(j\) is

\[ R_j^w = \sum_{k=1}^{5} \sum_{i=1}^{6} w_i R_{ijk} \]

This type of “spreadsheet arithmetic” is trivial in R. I will simulate the rating scores and illustrate the arithmetic.

```
wts <- rlts$ranking$Weight # criteria weights from above
n.crit <- 6
n.cars <- 3
n.eval <- 5
#### simulate evaluator scores
scores <- array(sample(x = 1:7, size = n.crit*n.cars*n.eval, replace = TRUE),
dim = c(n.crit, n.cars, n.eval),
dimnames = list (Criterion = labs,
Car = c("Honda Civic", "Subarua Impreza", "Toyota Corolla"),
Evaluator = 1:5))
scores[,,1] # display Evaluator 1 scores
## Car
## Criterion Honda Civic Subarua Impreza Toyota Corolla
## Color 4 3 6
## Mileage 6 6 6
## Price 2 2 7
## Reliability 2 1 5
## Safety 3 7 7
## Value (resale) 2 2 7
#### reweight scores using the `sweep` function
scores.wtd <- sweep(scores, MARGIN = 1, STATS = wts, FUN = "*")
scores.wtd[,,1] # display Evaluator 1 weighted scores
## Car
## Criterion Honda Civic Subarua Impreza Toyota Corolla
## Color 0.9792717 0.7344538 1.4689076
## Mileage 1.3445378 1.3445378 1.3445378
## Price 0.3249300 0.3249300 1.1372549
## Reliability 0.2689076 0.1344538 0.6722689
## Safety 0.3899160 0.9098039 0.9098039
## Value (resale) 0.2084034 0.2084034 0.7294118
scores.wtd.cars <- apply(scores.wtd, MARGIN = 2, FUN = sum)
scores.wtd.cars # display final weighted car scores
## Honda Civic Subarua Impreza Toyota Corolla
## 21.30868 18.37927 22.99272
```

Based on the analysis above we decide to select the Toyota Corolla.

### Radar chart (optional; intermediate)

Sometimes it is useful to visually display evaluators’ criteria weighting. This promotes discussion, clarification, and consensus building. Below is a radar (or spider) chart using the Plotly R package.

```
library(plotly)
r2 <- rlts$results[order(rlts$results$Evaluator,rlts$results$Criterion),]
erlab <- as.character(r2[r2$Evaluator=="eval1","Label"])
erlab <- c(erlab,erlab[1])
er1 <- r2[r2$Evaluator=="eval1","Weight"]
er1 <- c(er1,er1[1])
er2 <- r2[r2$Evaluator=="eval2","Weight"]
er2 <- c(er2,er2[1])
er3 <- r2[r2$Evaluator=="eval3","Weight"]
er3 <- c(er3,er3[1])
er4 <- r2[r2$Evaluator=="eval4","Weight"]
er4 <- c(er4,er4[1])
er5 <- r2[r2$Evaluator=="eval5","Weight"]
er5 <- c(er5,er5[1])
plot_ly(
type = 'scatterpolar',
fill = 'toself',
mode = "lines+markers"
) %>%
add_trace(
r = er1,
theta = erlab,
name = 'Evaluator 1'
) %>%
add_trace(
r = er2,
theta = erlab,
name = 'Evaluator 2'
) %>%
add_trace(
r = er3,
theta = erlab,
name = 'Evaluator 3'
) %>%
add_trace(
r = er4,
theta = erlab,
name = 'Evaluator 4'
) %>%
add_trace(
r = er5,
theta = erlab,
name = 'Evaluator 5'
) %>%
layout(
polar = list(
radialaxis = list(
visible = T,
range = c(0,0.35)
)
)
)
```

## Summary

Knowing how to quickly weight criteria with a team is very important. The rank ordinal method is very easy to deploy. The R function `rank.ordinal.weights`

is provided below, and its use is illustrated above.

## Appendix

Here is the R function. Test and report any bugs to me.

```
rank.ordinal.weights = function(x, criteria, labels, evaluators,
method=c("sr","roc"), weights){
## x = matrix: row is criteria ranking for each evaluator
## criteria = character vector of criteria factor levels (values)
## labels = character vector of criteria labels (long names)
## evaluators = evaluator names
## weights = full vector of customized weights (optional)
## NOTE: unique values in x must be subset of criteria levels
## To override default factors, provide both criteria
## levels and labels
## methods = see https://link.springer.com/chapter/10.1007/978-3-319-52624-9_2
method <- match.arg(method)
if(method=="sr") {
calc.rank.wts = function(k){
if(missing(k)){
stop("Must provide integer value for k (number of criteria)")
}
wi = dd = rep(NA, k)
for(i in 1:k){
dd[i] = (1/i)+((k+1-i)/(k))
}
denom = sum(dd)
for(i in 1:k){
wi[i] = ((1/i)+((k+1-i)/(k)))/(denom)
}
return(wi)
}
}
if(method=="roc"){
calc.rank.wts = function(k){
if(missing(k)){
stop("Must provide integer value for k (number of criteria)")
}
wi = rep(NA, k)
for(i in 1:k){
wi[i] = (1/k)*sum(1/(i:k))
}
return(wi)
}
}
nr = nrow(x); nc = ncol(x)
if(missing(criteria)) criteria = levels(factor(as.vector(t(x))))
if(missing(labels)) labels = criteria
if(missing(evaluators)){
if(!is.null(rownames(x))) evaluators = rownames(x)
if(is.null(rownames(x))) evaluators = paste("Eval_", 1:nr, sep = "")
}
eval.vec = rep(evaluators, rep(nc, nr))
crit.vec = as.vector(t(x))
roc.wts = calc.rank.wts(k = nc)
if(missing(weights)) wts.vec = rep(roc.wts, nr)
if(missing(weights)) roc.vec = rep(roc.wts, nr)
df = data.frame(Evaluator = eval.vec,
Criterion = factor(crit.vec, levels = criteria),
Weight = roc.vec,
Label = factor(crit.vec, levels = criteria, labels = labels)
)
criteria.wts = rev(sort(tapply(df$Weight, df$Criterion, sum)/nr))
criteria.wts2 = as.matrix(rev(sort(tapply(df$Weight,
df$Label, sum)/nr)))
ranktab = data.frame(Criterion = names(criteria.wts),
Label = rownames(criteria.wts2),
Weight = criteria.wts)
list(x= x,
criteria = criteria,
labels = labels,
results = df,
ranking = ranktab
)
}
```

## References

1. Aragón TJ. We will be the best at getting better: An introduction to population health lean [Internet]. San Francisco Department of Public Health; UC Berkeley School of Public Health; 2017. Available from: http://www.escholarship.org/uc/item/825430qn

2. Spetzler C, Winter H, Meyer J. Decision quality: Value creation from better business decisions. 1st ed. Wiley; 2016.

3. Danielson M, Ekenberg L. Trade-offs for ordinal ranking methods in multi-criteria decisions [Internet]. Bajwa D, Koeszegi ST, Vetschera R, editors. Vol. 274. Springer; 2017. pp. 16–27. Available from: https://link.springer.com/chapter/10.1007/978-3-319-52624-9_2

For a good selection of Likert Scale samples visit http://www.marquette.edu/dsa/assessment/documents/Sample-Likert-Scales.pdf.↩