Evaluating, comparing and reporting plans

A plan is a proposal to check, not an answer to accept. Does it meet its targets, and by how much? Which of its places are essential, and which could be swapped? How much would a more compact plan cost? Would a ranking have told the same story? In this part you will evaluate Serra Verde's 30 % plan with prioritize.evaluate, measure the importance of each of its cells four ways, run a gap analysis of the existing protected areas, calibrate the boundary penalty and draw what each budget buys with prioritize.calibrate, and put Zonation's best cells beside the optimized plan. The part ends with a checklist for reporting a prioritization in a paper.

The idea#

Evaluation works on any plan#

Every measure of this part takes a problem (features, costs, targets) and a solution: a map of 0 and 1 (or shares). The solution can come from prioritize.optimize, from Marxan, from a threshold of a Zonation ranking, from the existing reserves, or from a map drawn by hand in a workshop. The same yardstick for all is what makes plans comparable. prioritize.evaluate computes the measures described by Hanson et al. (2025).

Which places matter most#

A plan of 800 cells is a long list. Stakeholders ask which cells are essential and which could be swapped. Four importance scores rank the selected cells; they answer slightly different questions:

Trade-offs#

Every plan is a compromise: cost against compactness, money against representation, one stakeholder against another. A single map hides the compromises; a trade-off curve shows them. prioritize.calibrate builds the calibration curves described by Hanson et al. (2025):

• blended: the problem with a range of penalty weights, each giving a plan with a cost and a boundary (or a connectivity), as in part 5;
• hierarchical: the least boundary within cost caps above the least cost (1 %, 2 %, 5 % more...): a clearer question ("how compact for 2 % more?") but harder problems;
• pareto: the least total shortfall of the targets within growing budgets: what each budget buys;
• cohon: a starting penalty from the two extreme plans (Cohon, Church & Sheer, 1979).

Two reproducible rules choose a candidate: the knee of the curve, the point farthest from the line between its two ends (Satopää et al., 2011), where compactness stops being cheap; and TOPSIS (Hwang & Yoon, 1981), the candidate closest to the ideal (least cost and least boundary) and farthest from the anti-ideal, both scaled.

When to use it#

• After every solve: the coverage of the targets, the cost, the shape.
• Before a workshop: the importance of places, to know where there is room to negotiate.
• Gap analysis: what the existing reserves already hold (a solution of the protected areas).
• Choosing a penalty or a budget: a calibration curve instead of a guess.
• Comparing methods: Zonation, Marxan and an optimal plan on the same problem.

What you need#

The features and costs of the series (serra-verde.zip), a plan or any other solution map, and the targets. The template Templates ▸ Conservation and restoration ▸ Irreplaceability of a plan evaluates a plan for four random features and draws a portfolio's selection frequency.

Build the model#

Evaluate a plan#

format = "terraflux/1"

[[nodes]]
id = "features"
op = "raster.read"
params = { path = "features.tif" }

[[nodes]]
id = "cost"
op = "raster.read"
params = { path = "cost.tif" }

[[nodes]]
id = "plan"
op = "prioritize.optimize"
inputs = { features = "features.raster", cost = "cost.raster" }
params = { objective = "min_set", relative_targets = 0.3, gap = 0.001 }

[[nodes]]
id = "evaluation"
op = "prioritize.evaluate"
inputs = { features = "features.raster", cost = "cost.raster", solution = "plan.solution" }
params = { relative_targets = 0.3, replacement = true, rank = 10, extinction_z = 0.25, gap = 0.001 }

[[nodes]]
id = "ferrier_total"
op = "raster.calc"
inputs = { f = "evaluation.ferrier" }
params = { expression = "f.total" }

[outputs.coverage]
from = "evaluation.coverage"

[[plots]]
id = "show"
title = "Replacement cost"
layers = [{ from = "evaluation.replacement" }]

Calibrate#

format = "terraflux/1"

[[nodes]]
id = "features"
op = "raster.read"
params = { path = "features.tif" }

[[nodes]]
id = "cost"
op = "raster.read"
params = { path = "cost.tif" }

[[nodes]]
id = "calibration"
op = "prioritize.calibrate"
inputs = { features = "features.raster", cost = "cost.raster" }
params = { approach = "blended", penalty = "boundary", objective = "min_set", relative_targets = 0.3, values = [0.001, 0.002, 0.005, 0.01, 0.02, 0.05, 0.1], gap = 0.01 }

[outputs.table]
from = "calibration.table"

[[plots]]
id = "show"
layers = [{ from = "calibration.best" }]

prioritize.calibrate takes the problem as prioritize.optimize does, without the penalty it trades off, and its own settings: approach, penalty (boundary, connectivity, asym_connectivity or linear), values (the penalties, cost caps or budgets; without them it chooses n values around Cohon's penalty), inflation (for the hierarchical approach: the cost caps as shares above the least cost), gap and time_limit per solve (no limit by default: with a strong penalty one solve can take hours, so set one on a large problem; a candidate stopped by it is marked not optimal, with its gap). Its table has a row per candidate (its value, cost, criterion, gap, TOPSIS score, and knee and topsis set to 1 for the chosen), solutions a stack of the candidate plans, and best the plan TOPSIS chooses.

Read the results#

Coverage#

The coverage of the plan: all eight targets met, six of them within 0.2 % of 30 %, the harpy eagle at 43.7 % and the araucaria forest at 60.2 %, "for free" in the mountains the plan needs anyway. Its summary: 812 cells, 37.1 million USD, a boundary of 480.5 km. With extinction_z = 0.25, the extinction_risk table sums the features' risks \(1 - \text{held}^{z}\): 1.86 (of 8).

The importance of each place#

Enlarge

The measures agree on the essential places and disagree on the ordinary ones, which is as it should be: ordinary cells are, by definition, interchangeable. Of the fifteen cells of highest replacement cost, fourteen are on the mangrove coast and in the tamarin's fragments (the fifteenth is a river cell of the otter); the dearest to lose, 1.0 on the rescaled scale, is a mangrove cell. Replacement cost is the most direct (it is money) and the slowest; Ferrier's and rarity are instant and work on any plan; rank importance says where to start when money arrives in instalments.

A portfolio's selection frequency (part 4) is a fifth view; on Serra Verde it separates only a few interchangeable cells, since the plans near the optimum differ by a pair of cells.

A gap analysis#

Evaluating the existing reserves as a solution answers the first question of systematic conservation planning (Margules & Pressey, 2000): what do we have? Serra Verde's protected areas are in the mountains and on an island of forest; the species of the lowlands and the coast are not in them. Part 4 locked them into a plan and measured what completing the network costs.

Calibrating the boundary penalty#

The knee answers "up to where is compactness a bargain?"; TOPSIS "which plan is the best all-rounder, if cost and compactness matter equally?". Neither is the answer; both are reproducible starting points, better than a value copied from another study. Note that TOPSIS here picks the end of the range given: with wider values it could pick another point, so give a range that covers the bend and beyond.

The hierarchical approach asks the clearer question: how short can the boundary be if the plan may cost 0.5, 1, 2, 4, 7 or 10 % more than the cheapest? Its candidates are minimum penalty problems, harder to solve: with a time limit of 120 s each, four of the six stopped before proving their plan (gaps of 9 to 33 %), and the curve is ragged. Allowing 4 % more cost (38.6 million) shortens the boundary to 278 km, 58 % of the cheapest plan's; 10 % more, to 229 km. Run it on coarser planning units, or with longer time limits, before relying on it.

What each budget buys#

Every point of the front is efficient: no plan is both cheaper and closer to the targets. The number of targets met does not always rise with the budget (4 at 20 million, 3 at 25), because minimum shortfall reduces the total gap, not the count of targets fully met. Choosing a point on the front is a political decision; the front makes it an informed one. (The pareto approach takes the problem with its targets and any objective that uses them, min_set here or min_shortfall: whatever the objective, it solves the minimum shortfall problem within each budget itself.)

A ranking against a plan of the same cost#

Zonation ranks every cell (part 2); a plan meets targets. To compare them fairly, take Zonation's best cells (CAZ2, with costs) until they cost what the 30 % plan costs (37.1 million USD), and evaluate both against the same targets:

Enlarge

Neither is wrong; they answer different questions. CAZ2 keeps as much as possible of every feature at every fraction, weighted towards the rarest: it knows no targets and goes on adding the tamarin's habitat long after 30 %. The plan meets exactly what was asked, at the least cost, and nothing more. By the species-area yardstick, the ranking does better (a lower sum of risks); by the targets, the plan does. A good workflow uses both: the ranking to explore before targets are agreed, the plan once they are, and this comparison to show what the targets leave out.

How to report a prioritization#

A plan is only as useful as its documentation. Whatever the method, a paper or a report should let a reader repeat the analysis and judge its choices. A checklist, drawing on the practice of the field (Ardron et al., 2010; Hanson et al., 2025):

Data

• The study area, the planning units (cells, hexagons, polygons; their size and number) and the coordinate system.
• Every feature: what it is (species, ecosystem, service), the data (ranges, distribution models with their scores, maps), units and date; how features were weighted.
• The cost: what it measures (land price, opportunity cost, area), its source, units and year.
• Existing reserves and excluded areas (locked in and out), and why.

The problem

• The method (ranking or optimisation) and the software with its version (here Terraflux, and the model it reproduces: Zonation 5, Marxan 4, or the integer linear program and the paper it follows).
• The targets and their justification (policy, ecology, stakeholders), or the budget.
• The objective; the decisions (binary, proportion); penalties and constraints with their values and how they were calibrated (the curve, the knee or TOPSIS).
• The solver and its optimality gap (asked and reached), or for heuristics the runs, iterations and settings.

The results

• A map of the plan, and of the importance of its places.
• The cost, the area, the number of units, the boundary.
• A table of every feature: the target, the amount held, met or not (the coverage of prioritize.evaluate).
• The trade-offs explored (cost against boundary, budgets against shortfall) and the alternatives (portfolios, scenarios).
• The sensitivity of the plan to its main choices: targets, costs, weights, data.

What it is not

• The limits of the data and the assumptions (that features persist in what is selected, that costs are right), and that the plan is a support for decisions that also depend on what no map holds (Game et al., 2013).

Put the model file (the TOML, or the Terraflux project) and the data in the supplementary material: with them, anyone can run the same plan again.

Settings that change the answer#

• The targets the evaluation uses. Coverage is measured against them; evaluate every plan against the same.
• The gap of the replacement costs. Each is a difference between two optimised costs: with a gap of 1 % on a plan of 37 million USD, noise of 370 thousand swamps the small values.
• The rescaling (rescale): scores from 0.01 to 1 compare places within a plan, not across plans.
• The candidates of a calibration: the knee and TOPSIS choose among the values you give; a curve with too few points or too narrow a range can miss the bend.

Common pitfalls#

• Presenting a plan without its coverage table. It is the first thing a reader needs.
• Comparing plans evaluated against different targets or costs.
• Treating importance as a ranking of the whole landscape. It ranks the selected cells of one plan; for the whole landscape, use a Zonation ranking.
• A calibration chosen by eye on a map. Use the curve, and say how the point was chosen.
• Reporting the objective as the cost when penalties are in the objective.

Going further#

• Part 7, Multi-action planning: actions against threats.
• The template Irreplaceability of a plan for a quick start.
• The operators: prioritize.evaluate, prioritize.calibrate.

References#

• Albuquerque, F., & Beier, P. (2015). Rarity-weighted richness: a simple and reliable alternative to integer programming and heuristic algorithms for minimum set and maximum coverage problems in conservation planning. PLoS ONE, 10(3), e0119905. https://doi.org/10.1371/journal.pone.0119905
• Ardron, J. A., Possingham, H. P., & Klein, C. J. (Eds.) (2010). Marxan Good Practices Handbook, Version 2. Pacific Marine Analysis and Research Association, Victoria, Canada.
• Cabeza, M., & Moilanen, A. (2006). Replacement cost: a practical measure of site value for cost-effective reserve planning. Biological Conservation, 132(3), 336–342. https://doi.org/10.1016/j.biocon.2006.04.025
• Cohon, J. L., Church, R. L., & Sheer, D. P. (1979). Generating multiobjective trade-offs: an algorithm for bicriterion problems. Water Resources Research, 15(5), 1001–1010. https://doi.org/10.1029/WR015i005p01001
• Ferrier, S., Pressey, R. L., & Barrett, T. W. (2000). A new predictor of the irreplaceability of areas for achieving a conservation goal, its application to real-world planning, and a research agenda for further refinement. Biological Conservation, 93(3), 303–325. https://doi.org/10.1016/S0006-3207(99)00149-4
• Game, E. T., Kareiva, P., & Possingham, H. P. (2013). Six common mistakes in conservation priority setting. Conservation Biology, 27(3), 480–485. https://doi.org/10.1111/cobi.12051
• Hanson, J. O., Schuster, R., Strimas-Mackey, M., Morrell, N., Edwards, B. P. M., Arcese, P., Bennett, J. R., & Possingham, H. P. (2025). Systematic conservation prioritization with the prioritizr R package. Conservation Biology, 39(1), e14376. https://doi.org/10.1111/cobi.14376
• Hwang, C.-L., & Yoon, K. (1981). Multiple Attribute Decision Making: Methods and Applications. Springer. https://doi.org/10.1007/978-3-642-48318-9
• Jung, M., Arnell, A., de Lamo, X., García-Rangel, S., Lewis, M., Mark, J., et al. (2021). Areas of global importance for conserving terrestrial biodiversity, carbon and water. Nature Ecology & Evolution, 5(11), 1499–1509. https://doi.org/10.1038/s41559-021-01528-7
• Margules, C. R., & Pressey, R. L. (2000). Systematic conservation planning. Nature, 405(6783), 243–253. https://doi.org/10.1038/35012251
• Satopää, V., Albrecht, J., Irwin, D., & Raghavan, B. (2011). Finding a "kneedle" in a haystack: detecting knee points in system behavior. In 31^st International Conference on Distributed Computing Systems Workshops (pp. 166–171). IEEE. https://doi.org/10.1109/ICDCSW.2011.20