Water: Recovery Potential
Step 6: Refine Your Assessment
Screening Methodology Tutorial
In Steps 1 through 5, you compiled your dataset and finished one or more runs through the screening process. You used the tools and techniques in Step 5 to compare differences revealed by your screening, and you likely developed a more specific understanding of how specific indicators and summary indices affect the scoring. Additionally, you may have generated multiple sets of screening results and now wonder which one to use, or how to use multiple results together. Step 6 is where you revisit these alternatives in light of your original screening purpose and determine the products that best meet your needs.
Revisit screening purpose and products. It is not unusual for a Recovery Potential Screening to begin with a very general goal, such as 'identify restorability differences' and refine or even change that goal as screening steps proceed. For example, an original plan such as 'identify and target the most restorable watersheds for restoration' might have evolved into 'identify the watersheds that have good RPI scores as well as moderately heavy nutrient loads and target these for TMDL implementation'. An originally single goal may also have transformed into multiple, more specific goals, such as 'target watersheds for CWA Section 319 nonpoint source projects that have impairments and a high social index score', 'target watersheds with pathogen TMDLs as well as drinking water intakes for expedited restoration', and 'target watersheds with impaired but restorable aquatic habitats for collaborative restoration with fisheries programs'. Because creating alternative screening runs is so rapid, the emergence of multiple goals is not a problem unless it is necessary to choose just one from among them; and in such a case, the outputs of screening runs designed to address each goal may help inform the selection process.
Some considerations for revisiting purpose and products include:
- Has the original screening purpose remained the same?
- Will the originally planned products still meet the intended purpose?
- Have new goals or purposes emerged during the screening process, and do indicators and screening runs exist that address them?
- Will the products identify or compare waters or watersheds by way of a transparent, reproducible process?
- Will interim products support reaching any needed decisions or strategies?
- Will specific products help in communicating decisions or strategies?
In brief, these questions should reveal whether you have the data products you need to support the actions that you need to take. If they reveal additional data or screening needs, you can repeat steps 3, 4 and 5 to fill those gaps. If they point instead to sufficient data with a need to choose among alternatives, proceed as follows.
Consolidate the screening runs. After conducting multiple screening runs and revisiting the scoring process, it is possible that you may have multiple versions that identify different sets of waters as more restorable. On the other hand, it is likely that you have observed some degree of consistency or pattern in the results - variations in indicator selection and weights often still point to many of the same waters as high-scoring. The opportunity to explore different combinations of indicators has been informative, but your challenge now is to reduce your findings into the form you will need to guide action. This is a project-specific task without a single, universal solution for all settings; nevertheless, the following approaches may suit your own circumstances:
- Select the preferred screening run. Reaching consensus on one screening run may be desirable as a decisive solution to evaluating many alternatives. During the project, your working group has likely established preferences for one (or very few) of the screening runs and indicator selections, drawing from their expert judgment and familiarity with the area being assessed. Or, your group's discussions of candidate and preferred indicators early in the assessment may have led to general agreement and interest in performing just one screening. Selection of a specific screening run should consider the source and indicator data quality, the evaluation of its component indicators against reference sites, how well the screening run addresses the assessment purposes and products as discussed above.
- Integrate multiple screening runs if necessary. Should your group feel that no single run is ideal, consider integrating the results of the best runs available. Averaging or summing the RPI scores of multiple runs are two ways to combine results, if the objective is to reach one set of final scores. Further, it is possible to combine the results from multiple screening runs with the results of an entirely independent prioritization process (see for instance Table 7 in the screening example (PDF) (12 pp, 842K, About PDF)). Yet another approach involves targeting waters that surpass a defined set of scoring threshold (e.g., all watersheds with highest quartile ecological scores, lowest quartile stressor scores and above-mean social context scores in three of five screening runs). Statistical methods may provide a systematic, repeatable way to vary the combinations of indicators in multiple screening runs and observe the waters that consistently score well.
- Maintain multiple alternatives intentionally. It is possible in some circumstances that the most useful output is, in fact, multiple sets of results rather than one. An assessment designed to assist a complex decision-making process, for example, might intentionally not select or rank-order specific waters but instead provide a few well documented alternatives to decision-makers for their final action. Different screenings could also reflect contingency plans for different budget scenarios, or different priority watershed selections for collaboration with different program partners. If your screening approach involved subsets such as separately screening lake and stream watersheds, or screening the three most common impairments separately, maintaining and using multiple sets of results is appropriate.
Match screening selections to specific actions. Reconciling several alternative screening runs may have clarified the outcome of your screening assessment while not yet providing the specifics you require - usually a subset of the screened waters or watersheds that is specifically matched to an intended action. Again, screening goals and project circumstances can vary so widely that making final selections is a project-specific task. The following are some simple options that use different types of Recovery Potential Screening products to target selected waters or watersheds:
- Select a defined portion from rank-ordered screening results. Resource-limited programs often are aware of their capacity in terms of yearly number of projects, funding available, or other measures. It is possible to match annual or long-term capacity estimates to rank-ordered screening scores in identifying priorities for restoration. Alternatively, prioritization might use rank-ordered sites more generally (such as 'within the top quartile RPI score' or 'ranking above the state mean in ecological and social index score') or in combination with other qualifying factors (such as 'also an Environmental Justice area' or 'needing three projects per sub-state region'). It is also possible to avoid selecting a priority subset at all, and use the complete set of rank-ordered recovery potential scores purely to inform day-to-day activities on the likely level of effort and related factors that may be expected if working on a given location.
- Use bubble plot quadrants to help select priorities. The 3-D bubble plotting tool automatically plots two lines on each bubble plot that represent the mean ecological and stressor index scores. As the lines bisect the plot in horizontal and vertical dimensions, they create quadrants that offer a basis for different types of prioritization strategies. Selecting the upper left quadrant, for example, would target all waters with ecological and stressor scores better than the mean. The upper right quadrant, on the other hand, might be selected if priorities lean toward high ecological scores paired with elevated and potentially threatening stressor scores. The lower right quadrant, where high stressor and low ecological scores converge, would be a place where potentially more expensive and complex restoration challenges could be found. In any of the above, prioritizing the larger 'bubbles' within the selected quadrant would specifically target better social context for restoration as well.
- Use geographic and scoring factors together. If geographic proximity-based factors like 'adjacency to green infrastructure hub or corridor' were not already included in the screening indicators, selection of priorities might also include a mapping component. Prioritization might start with rank-ordering or bubble plot quadrants as above, then target further with geographically observable factors. For example, impaired watershed priorities might include the best quartile based on RPI score but also contiguous with a patch of healthy watersheds, or extending a landscape corridor of several healthy watersheds. Another approach might prioritize restoring specific impaired watersheds to expand the downstream extent of healthy headwaters above highly significant and economically valuable waterbodies.
See also ways to use rank-ordering (PDF) (4 pp, 727K, About PDF), ways to use bubble plotting (PDF) (6 pp, 619K, About PDF), and ways to use mapping (PDF) (4 pp, 653K, About PDF) for more information.