"But if you miss, you had better miss very well."
Tuco: "The Good, The Bad, and the Ugly" (1966)
Consider the problem faced by the U.S. Federal Government. According to Executive Order 12866, executive agencies must evaluate the benefits and costs from every economically-significant regulatory action. In fiscal year 2003/2004, seven regulatory impact analyses were completed for regulations proposed just by the Environmental Protection Agency (EPA). In just one of those analyses, for new regulations on nonroad diesel engines, the EPA assembled estimates of the benefits associated with decreases in the numbers of premature deaths, nonfatal heart attacks, chronic bronchitis, asthma attacks, hospital visits, and lost work days, in addition to estimates of benefits associated with reductions in agricultural crop damage and improvements in visibility.
Similarly, the Forest Service needs estimates of ecosystem values for use in forest planning; the USDA needs estimates of environmental benefits from the Conservation Reserve Program; and the Army Corps of Engineers needs estimates of recreation values for designing management plans for its reservoirs. These agencies need information on benefits and costs of hundreds of environmental goods and services in thousands of different locations. There has been remarkable progress in developing methods for estimating these unpriced values; these methods are discussed in the two companion articles to this one.
However, it is simply not practical to conduct an original stated or revealed preference research study for each environmental good in each location every time a new policy analysis is conducted. Nor is it really necessary. If someone has already conducted a study that valued a day spent fishing on a small lake in Tennessee, the Army Corps may be able to use that number to value a day spent fishing on a small lake in Kentucky. If someone has already estimated the value of a reduction in cancer risk from decreased exposure to dioxin, the EPA may be able to use that number to value a reduction in cancer risk from decreased exposure to benzene. These are examples of benefit transfer.
The Quick, the Dirty
The terminology used in benefit transfer studies traces back to its early use in recreation applications. In the Army Corps reservoir example given above, the Tennessee lake where a primary study was conducted is called the study site, while the Kentucky reservoir, where the information is used for policy evaluation, is called the policy site. The study site/policy site terminology is now used even when the good is not provided at a distinct site.
Benefit transfer is widely used by government agencies because it is quicker and cheaper than conducting original studies every time a benefit estimate is needed. In some cases, benefit transfer is relatively straightforward and even familiar. When the Federal Highway Administration evaluates the time-savings benefits from new highway construction, it does not conduct original research on the value of travel time for every new highway. Rather, it uses per-hour values based on previous studies. Guidelines issued in 1997 suggest a value of $11.90 per hour for intercity personal travel, for example. When the EPA values a decrease in mortality risk from an improvement in air or water quality, it does not conduct original research for each mortality risk. Rather, it uses estimates of the value of a statistical life (VSL) based on previous studies. In the nonroad diesel rule analysis, a VSL of $5.5 million was used.
These are both examples of a benefit transfer technique called unit value transfer. This approach is best suited for situations where the projected impacts of a policy can be measured in fairly homogeneous, divisible units (hours of travel time saved, premature fatalities avoided). A constant benefit value is used per unit of the unpriced good, based on one or more previous valuation studies for that good. Ideally, a unit value is used that was estimated at a study site similar to the policy site. An estimate from a study site located close to the policy site is also generally preferred, to minimize differences between the population of users at the study site and at the policy site. For example, the US Forest Service uses region-specific unit day recreation values as part of their mandated periodic Renewable Resource Planning Act (RPA) Assessment.
The use of unit values may be justified for valuing health impacts, transportation improvements, and some types of outdoor recreation. These are goods that we tend to think of as being more or less homogeneous across users and across policy contexts. Where benefit transfer becomes more difficult is where the context of the good at the policy site differs from that at the study site, either with regard to the attributes of the good being valued or the population enjoying the benefits. An acre of wildlife habitat in Utah is very different from an acre of wildlife habitat in Pennsylvania, and the values generated will likely differ as well.
Value function transfer has the potential to improve the performance of benefit transfer in situations where the good or the user population differs between the study site and the policy site in measurable ways. In this approach, a value function is first estimated at a study site or group of sites. A value function predicts the value of a good as a function of its measurable characteristics (quantity and quality), those of its users (income, etc.), and the context within which the good will be provided (availability of substitutes, etc.). In principle, the value of the good at any policy site can be determined by plugging in the relevant measures for that site. Some have argued that the form of the value function should be motivated by economic theory (Smith, Van Houtven, and Pattanayak, 2002), but more typically it is chosen in an ad hoc manner in an attempt to maximize goodness of fit.
One example of a simple value function is the Army Corps of Engineers' point system for determining user day values for recreation at Corps facilities. Points on a scale from 0 to 100 are awarded for the quality of the site, the number and types of activities enjoyed at the site, and the availability of substitutes for the site. In 2002, user day values ranged from $2.90 for general recreation with low point values to $34.41 for specialized recreation with high point values. Value functions will more often include objective measures of the quality (e.g., catch rate, reservoir size) and measures describing the population of users (e.g., income, travel distance to the site).
Value function transfer will work well only if a) there is sufficient variation at the study site in the attributes of the good, b) there is sufficient variation at the study site in the attributes of the user population, c) the attributes of the good and the population at the policy site fall within the range of the original data at the study site, and d) preferences for the good are similar at the study site and the policy site. One challenge to conducting value function transfer is that original valuation studies are often conducted in a limited geographic area, and important attributes of the good or the population may not vary within an individual study. However, by combining results from several original valuation studies, a value function can be estimated based on a richer set of goods and user populations. In a meta-analysis, value estimates are combined from several different studies. A value function is estimated with these value estimates as the dependent variable and with characteristics of the good, the population of users, and study methodology as the independent variables. For goods where a large number of source studies are available, meta-analysis has the potential to provide value functions that can be applied in a wider range of situations.
While benefit transfer typically tries to tailor value estimates to the policy site's good and population, in some situations it may not be desirable to adjust values to individual contexts. Even though there is some limited empirical evidence that willingness to pay to reduce mortality risk decreases somewhat with age, the EPA chooses to apply a constant VSL regardless of the population at risk. There are important ethical considerations when values are adjusted for age, income, or ethnicity, particularly if those values are used to set policy or to direct resources.
Benefit transfer is clearly feasible only if a study already exists that valued a good similar to the good in question. The analyst must assess the quality of the existing study or studies, and decide whether the good valued at the study site(s) is similar enough to the good at the policy site. The Office of Management and Budget, in its guidance to executive agencies on conducting regulatory analyses (OMB, 2003) provides a common-sense set of criteria that must be satisfied when choosing a source study.
The task of finding a suitable source study has been simplified for analysts with the recent creation of databases of previous valuation studies. One of the most comprehensive is the Environmental Valuation Reference Inventory (www.evri.ca) maintained by Environment Canada with support from the United States, Great Britain, and France. Even with such databases, a common problem is lack of documentation in the source studies. Often, this is due to the difference between the information that journal reviewers are looking for and the information that policy analysts need to conduct benefit transfer. For example, a researcher publishing a hedonic pricing analysis will always present the estimated house price function, but might not always report the average house price in the dataset. That kind of information is critical, however, when using that study in a meta-analysis or a benefit transfer.
...and the Ugly?
The conventional wisdom is that benefit transfer is inherently inferior to conducting original studies, but that it is a necessary evil given time and budget constraints. The concern is over transfer error, defined as the difference between the transferred value estimate and the true (unknown) value estimate at the policy site. Several studies have assessed the validity of benefit transfer by comparing value estimates between two sites, asking the question, if one of these sites had been used as a study site in a benefit transfer for the other site, how large would the transfer error have been? These studies typically test the validity of benefit transfer in three ways. First, the values estimated for the imagined study site and the imagined policy site are compared, to see if they differ statistically. Second, value functions are estimated at each site, and the validity of a pooled model is tested. Third, projected transfer error is calculated as the absolute value of the percent difference between the value transferred from the imagined study site and the value estimated at the imagined policy site.
The first two validity tests are often rejected. That is, it is common to find statistically significant differences between the unit values estimated at two sites and between value functions. While negative results for these tests are informative, they are not necessarily fatal to benefit transfer. With enough data, statistically significant differences can be found even where the values themselves are quite similar. From a policy perspective, the size of the potential transfer error is much more important than statistical convergence.
Regarding the size of the potential transfer error, these studies often find average transfer errors of 40 or 50%, but with a wide range that can span from zero percent to several hundred percent for individual transfer exercises. While generalizations are difficult, there is some evidence that transfer errors tend to be smaller when the two goods are located in the same geographic region (Rosenberger and Phipps, 2001). This may be because the goods themselves are more similar, or it may be because the user populations are more similar. Interestingly, the evidence that value function transfer outperforms unit value transfer is mixed at best. Some studies find an improvement in performance, others do not.
It is important to realize that transfer errors calculated in these validity studies are artificially inflated because the criterion (the value at the policy site) is not perfectly known. Calculated transfer error is the sum of actual transfer error and error in the criterion. Suppose a good has the same value, $100, at two different sites, and that each of the values are estimated at with a standard deviation due to sampling error of $20. A validity test of benefit transfer between these two sites will show an average transfer error of 24%. Compared to this "best case" expected transfer error, an observed transfer error of 40% is not that bad.
...Compared to What?
So does benefit transfer work or not? That question raises two more questions. First, how large of a transfer error is acceptable? Second, compared to what?
The answer to the first question depends both on the reason for doing the policy analysis and on the degree to which the value estimate is decisive. Some valuation situations require high precision and reliability. A good example is resource damage assessment, where a responsible party has to write a check based on the value estimate. In contrast, a higher level of uncertainty in the value estimates is probably acceptable when conducting a regulatory impact analysis for a regulation that is mandated by law. Further, the larger the value, both in absolute terms and as a proportion of the total benefit from the policy, the more important it is to get the right number. Finally, value estimates must be more reliable if their uncertainty could potentially tip the balance in favor of or against a proposed action.
With regards to the second question, the uncertainty introduced by benefit transfer may not be large relative to other sources of uncertainty in the value estimates. Estimated values of the same good measured using stated and revealed preference techniques can vary by an order of magnitude. Even within a given technique, research design decisions such as question format or the functional form used for data analysis can have dramatic impacts on value estimates. Indeed, meta-analyses often show that research design features are more important in explaining variation in values than attributes of the goods or the population of users.
The conventional wisdom that an original study is always preferred to a benefit transfer needs to be reexamined. While the potential exists for very large transfer errors, original studies have their own potential for problems. A thoughtful, carefully executed benefit transfer from a high quality, large-sample study conducted at a similar site, or a set of studies conducted at multiple sites, is probably preferable to a small-sample, rushed original study conducted at the policy site.
Nor should the choice between benefit transfer and conducting an original study be necessarily viewed as an either/or choice. Where information on the value of similar goods is available, but there is concern that the value at the policy site may be unique, a Bayesian perspective can be adopted. Value estimates or functions from existing studies can be used to form a prior distribution on the value of the good at the policy site. Valuation research conducted at the policy site provides new information on the value of the good. An updated distribution of the value of the good at the policy site contains information from both previous studies conducted at other sites and from the new research conducted at the policy site. A Bayesian perspective also suggests that the decision whether to conduct original research at the policy site, and if so how much, should be made based on the expected value of the information to be gained and the cost of conducting new research.
Does benefit transfer work? Should we be doing it? The answers to these questions are similar to the answers for nonmarket valuation more generally. Benefit transfer, if done carefully using appropriate sources for the transferred values, can work quite well. However, it can perform very poorly. The same can be said for nonmarket valuation in general. A more constructive discussion is over how to improve benefit transfer protocols and minimize the potential for large transfer error.
And the decision whether we should be doing it has already been made. Federal agencies routinely use benefit transfer to conduct regulatory and program analyses because they have to. If the values of most environmental goods and services are going to be included in these analyses, then they will have to come from benefit transfer. There is not enough time or resources to conduct original studies for each policy analysis. The choice is not between benefit transfer and conducting original studies. The choice, in many cases, is between conducting a benefit transfer and not including any estimate of the benefits from environmental goods and services.
How can benefit transfer be improved? First, the single most important action to improve benefit transfer is to increase the stock of high-quality original valuation studies. With the exception of some types of outdoor recreation and some types of health impacts, the set of available studies for most environmental goods is thin. Second, these studies have to be made available to analysts. Databases like EVRI can serve an important role. Third, the authors of new original studies need to report more details about the methods, data and the good valued. Academic journals tend to discourage publication of study details that are not central to the methodological or theoretical contribution of the research. There have been calls for a new publication outlet, perhaps an online journal, to serve as a repository for this kind of detail. Fourth, the analyst conducting benefit transfer has an obligation to document and justify the assumptions and protocols used. Just as original nonmarket valuation studies must be accompanied by enough documentation to allow judgment of their validity, so too must benefit transfer exercises be transparent and fully documented.
For More Information
The literature on benefit transfer begins in earnest with a collection of papers organized by David Brookshire and published in the March 1992 issue of Water Resources Research. Desvousges, Johnson, and Banzhaf (1998) provide an in-depth exploration of benefit transfer protocols, with particular attention to a case study estimating externalities from electricity generation. Rosenberger and Loomis (2003) provide more of a how-to treatment of benefit transfer. Navrud and Ready (Forthcoming) assemble several studies demonstrating current state of the art in benefit transfer.
Desvousges, W.H., Johnson, F.R., & Banzhaf, H.S. (1998). Environmental policy analysis with limited information: Principles and applications of the transfer method.
Cheltenham, UK and Northampton, MA, USA: Edward Elgar.
Navrud, S. & Ready, R.C. (Eds.). (Forthcoming). Environmental value transfer: Issues and methods. Dordrecht, The Netherlands: Kluwer Academic Publishers.
OMB (Office of Management and Budget). (2003). Circular A4: Regulatory Analysis. Washington, D.C.
Rosenberger, R.S., & Phipps, T.T. (2001). Improving the validity of benefit transfers: A site correspondence model and meta-analysis transfer. RRI Research Paper 2001-6. Morgantown, WV: Regional Research Institute, West Virginia University.
Rosenberger, R.S., & Loomis, J.B. (2003). Benefit transfer. In P.A. Champ, K.J. Boyle, & T.C. Brown (Eds.), A primer on nonmarket valuation, Dordrecht, The Netherlands: Kluwer Academic Publishers.
Smith, V.K., Van Houtven, G., & Pattanayak, S.K. (2002). Benefit transfer via preference calibration: ‘Prudential algebra' for policy. Land Economics, 78(1), 132-152.