Weighting Indicators

In any composite indicator system, it is important to consider the relative importance, or weight, that should be given to each indicator (and sub-indicator) when they are combined into a numeric index. Giving each indicator equal and proportional weights avoids the issue of determining relative importance, but by default then the weights are determined by the number of indicators and the structure of the framework.

For this reason, we recommend carrying out a dedicated weighting exercise as part of the Freshwater Health Index. These weights are used to calculate the Major Indicator scores as well as the three Component scores. The calculated weights can always be restored to default weights for the purposes of comparison to other basins, and the FHI Desktop Tool also allows users to explore different weights and their impact on scores.

Figure: Weights applied in two FHI cases-studies derived from Stakeholder surveys

There are a variety of methods for assigning weights, but they can be roughly assigned to one of two categories—statistical methods and public/expert opinion (see OECD, 2008 Chapter 8 for a thorough review). Statistical methods such as Principal Component Analysis are desirable because they are considered objective; they are based exclusively on existing quantitative data and mathematics. However, statistical methods require a substantial amount of longitudinal data to determine what drives change in the indicators, making them less suitable for the Freshwater Health Index until several assessments have been conducted. Moreover, because each basin has unique characteristics, it would not be advisable to construct statistical weights based on FHI assessments from outside of the study area.

The alternative, then, is to engage with experts and other members of the public to derive qualitative information that can be transformed into quantitative weights. Expert elicitation is the simplest method, where a small number of selected participants with expertise in freshwater sustainability and/or water resource management provide weights. Approaches include a Budget Allocation Process (BAP) in which each expert has 100 points that s/he must budget across individual indicators; these can be done in a single round, or following some version of the Delphi method where experts are told the results from the group and then given an opportunity to revise their individual estimates (Brown 1968, http://www.rand.org/topics/delphi-method.html).

Another approach, and the main method that has been applied in FHI assessments to date, is to use the Analytic Hierarchy Process (AHP—Saaty, 1990), where experts make judgments about the tradeoffs among indicators. Participants are asked to make a series of pairwise comparisons between, for example, “Provisioning” and “Regulating” services, and then determine their preference as well as the strength of that preference. Although this approach is more cognitively demanding than, say a BAP, it has the advantage of providing a more structured, systematic and mathematically rigorous approach to deriving weights.

A template for applying the AHP to derive weights for FHI indicators is available for download from FHI website. It can be administered on paper, but data collection and processing is easier if administered online. In practice thus far, we have not derived weights for Ecosystem Vitality indicators, because we consider these less subjective than obviously, expert elicitation

In practice, the weighting exercise provides much more than just a set of quantitative weights for aggregating indicators—it provides real insights into stakeholders’ preferences and, importantly, where there may be differences of opinion. However, we do not recommend that weights be applied to the Ecosystem Vitality indicators. Weights should only be applied if there is strong evidence that some ecosystem processes or attributes play a greater role in ecosystem functioning than others. This is an empirical question rather than a subjective one, which requires a great deal of understanding of ecosystem process and pattern