Do Energy Efficiency Investments Deliver? Evidence from the Weatherization Assistance Program*

Abstract

A growing number of policies and programs aim to increase investment in energy efficiency, because conventional wisdom suggests that people fail to take up these investments even though they have positive private returns and generate environmental benefits. Many explanations for this energy efficiency gap have been put forward, but there has been surprisingly little field testing of whether the conventional wisdom is correct. This article reports on the results of an experimental evaluation of the nation’s largest residential energy efficiency program—the Weatherization Assistance Program—conducted on a sample of approximately 30,000 households in Michigan. The findings suggest that the upfront investment costs are about twice the actual energy savings. Furthermore, the model-projected savings are more than three times the actual savings. Although this might be attributed to the “rebound” effect—when demand for energy end uses increases as a result of greater efficiency—the article fails to find evidence of significantly higher indoor temperatures at weatherized homes. Even when accounting for the broader societal benefits derived from emissions reductions, the costs still substantially outweigh the benefits; the average rate of return is approximately −7.8% annually.

I. Introduction

Energy efficiency investments are widely believed to offer a rare win-win opportunity. Detailed engineering projections, such as those summarized by the well-known McKinsey curves (McKinsey & Company 2009), routinely project that investments pay for themselves through the energy saved alone (win 1). Moreover, by reducing the energy necessary to achieve a given level of energy services (e.g., indoor heating), these investments promise to decrease the greenhouse gas emissions causing climate change and other pollutants that compromise human health (win 2).

Despite these apparent opportunities, there is a large and persistent difference between the levels of investment in energy efficiency that are projected to save consumers money and the investments that individuals actually pursue. This has become known as the “efficiency gap.” Over the past three decades, a wide variety of explanations have been offered for this apparent failure of consumers to avail themselves of profitable investment opportunities. The most popular explanations have emphasized the possibility of market failures, such as imperfect information, capital market failures, split incentive problems, and behavioral explanations, including myopia, inattentiveness, and prospect theory and reference-point phenomena (see, for example, Allcott and Greenstone 2012; Gillingham and Palmer 2014; Gerarden, Newell, and Stavins 2015). In contrast, relatively little attention has been paid to the more pedestrian possibility that the real-world returns on energy efficiency investments are lower than the engineering models indicate.¹

Mounting concern about climate change has increased the urgency of understanding this phenomenon. Governments around the world are pursuing a wide range of policies designed to narrow or close the energy efficiency gap. For example, the International Energy Administration (IEA) has outlined a suite of policies that do not harm economic growth and limit warming to the 2°C recommended by climate scientists; in this scenario, end-use energy efficiency improvements account for 49% of the greenhouse gas emissions abatement in 2020 (IEA 2013).² U.S. electric utilities are rapidly expanding their energy efficiency programs (Barbose et al. 2013), and federal and state regulators routinely tighten energy efficiency building codes, appliance standards, and fuel economy standards for automobiles and trucks.

This article provides the first large-scale field evidence on the returns to energy efficiency investments from a randomized controlled trial. Specifically, we use experimental and quasi-experimental variation in participation in the federal Weatherization Assistance Program (WAP) to identify the returns to these investments. WAP is the nation’s largest residential energy efficiency program and has provided more than 7 million low-income households with weatherization assistance since its inception in 1976. Recipient households in our study received approximately $5,150 worth of home improvements on average, with zero out-of-pocket costs. The most common measures included furnace replacement, attic and wall insulation, and infiltration reduction. Importantly, WAP only pays for energy efficiency measures that pass a cost-benefit test, based on ex ante engineering projections, with the aim of ensuring that only beneficial investments are undertaken.

The randomized-encouragement design experiment was conducted on a sample of approximately 30,000 Michigan households that were presumptively eligible for participation in WAP. Approximately one quarter of these households were randomly assigned to a treatment group that was encouraged to apply for the program and received significant application assistance. The control households were free to apply for WAP but were not contacted or assisted in any way by our team.

There are three primary findings. First, an aggressive encouragement intervention increased WAP participation from less than 1% in the control group to about 6% in the encouraged group. The encouragement was implemented by a firm with extensive experience managing outreach campaigns and neighborhood canvassing operations, including among low-income populations. The field activities included almost 7,000 home visits, more than 32,000 phone calls, and 2,700 follow-up appointments. Ultimately, these extensive efforts only managed to increase the participation rate by 5 percentage points at a cost of more than $1,000 per weatherized household (Fowlie, Greenstone, and Wolfram 2015), revealing low demand in the eligible population for a program with considerable potential benefits.

Second, the findings suggest that the benefits of these investments are substantially less than the upfront costs. We estimate that the WAP energy efficiency investments reduce monthly energy consumption by 10–20% on average. Although this surely provides a substantial assist to participating low-income households in the form of reduced energy bills, the upfront investment costs are about twice the realized energy savings. Furthermore, the ex post estimated savings are roughly 30% of the model-projected savings.³

Third, while the modest energy savings might be attributed to the “rebound” effect, when demand for energy end uses increases as a result of greater efficiency, the article fails to find evidence of significantly higher indoor temperatures at weatherized homes. This finding comes from a novel survey of measured indoor temperatures and thermostat set points that we conducted in the study population. Across a variety of metrics, the WAP energy efficiency investments appear to be poor performers on average. Although these investments were free to participating households, we can nevertheless estimate the private returns if households had been responsible for the upfront costs, which is the case for households that do not qualify for WAP. Counting private returns—a household’s reductions in energy bills and willingness to pay for any change in indoor temperatures—the annual internal rate of return that would rationalize these efficiency investments is −2.3%. This finding of low, indeed negative, returns suggests that at least for residential home retrofits, there may not be much of an efficiency gap to explain. Rather, just like in all other sectors of the economy, investments with low returns are not taken up frequently. Importantly, the engineering model used by WAP is similar, and in some cases identical, to those used to develop recommended residential efficiency investments in the broader population (i.e., not just low-income households).

In principle these investments could still be beneficial socially, but on average, this is not the case with the measures we evaluate. In contrast to the private calculation, the social one accounts for the benefits of reduced greenhouse gas and local pollutant emissions and the fact that part of households’ energy savings is a transfer from other energy consumers, rather than genuine social savings. When we account for these factors, the annual social internal rate of return that would justify these investments is −7.8%. Finally, we also calculate the average cost per ton of avoided CO₂ under a range of assumptions. The most plausible estimates are $200/ton, which is significantly larger than the U.S. government’s estimate of the social cost of carbon of roughly $38 (Greenstone, Kopits, and Wolverton 2013).

This article makes three primary contributions to the literature on energy efficiency. First and most important, it is one of the first studies to provide causal evidence on the returns to residential energy efficiency investments from a large-scale field test. The closest paper methodologically is Dubin, Miedema, and Chandran (1986), which reports results from an experiment conducted more than three decades ago on fewer than 400 households. There have been a series of other important academic papers that have taken on issues related to residential energy efficiency using various sources of nonexperimental variation, including Metcalf and Hassett (1999) on returns to insulation improvements; Davis, Fuchs, and Gertler (2014) about appliance replacement programs in Mexico; Jacobsen and Kotchen (2013), Kotchen (2017), and Levinson (2016) on the energy savings associated with building code standards; and Allcott and Greenstone (2017), who use experimental and nonexperimental variation to estimate the welfare consequences of residential energy efficiency programs that are not means tested. There is also a large (measured in the many hundreds) “gray” literature that is often commissioned by agents with a stake in the outcome (e.g., utilities that could face regulatory sanctions if their energy efficiency programs fail to deliver promised savings). These studies generally do not meet modern standards of evidence, often relying on simulations, rather than real-world data, failing to specify clear counterfactuals/control groups, or not reporting standard statistical tests.⁴

Second, this article is the first effort to directly measure the rebound effect and develop a framework or method that allows for rebound behavior to count as a benefit or part of the returns to efficiency investments. The previous literature has generally had to rely on methods of detecting rebound that place great faith in the validity of the engineering predictions of savings. For example, Dubin, Miedema, and Chandran (1986) assume that the engineering estimates are correct and use them to model demand for energy services. Because they find elastic demand for energy services, the paper concludes that there is a rebound effect. Though the existence of the rebound effect has been the subject of much debate (Gillingham et al. 2013), our study is the first to provide a direct field test of this phenomenon across a broad spectrum of residential energy efficiency investments.

Third, the article provides a demanding and credible test of the validity of engineering models’ predictions about the returns to residential energy efficiency investments. This test is important because the case for the energy efficiency gap—the motivation for almost all energy efficiency policy—is derived from the promise of abnormally high returns to these investments; yet the promises are derived from engineering models that have largely been untested in the field. Furthermore, the article’s comparison of actual and modeled savings is conducted among the most frequent residential efficiency investments. Finally, WAP’s engineering model is used extensively in other contexts, and there is considerable similarity between its underlying assumptions that generate ex ante predictions in the returns to efficiency investments and the assumptions in competing engineering models. Thus, it seems that the pedestrian explanation for the energy efficiency gap (that low take-up is due to low returns) deserves greater consideration.

The article proceeds as follows. Section II introduces a conceptual framework useful for deriving the private value of energy efficiency investments, including any behavioral adjustments they may cause. Section III outlines key details on the WAP and describes our study design. Section IV describes the data sources and provides summary statistics. Section V reports the main results on actual savings and on observed rebound effects. Section VI develops measures of both the private and social returns to energy efficiency investments and discusses the results’ external validity to other settings. Section VII concludes.

II. Conceptual Framework for Measuring Private Returns

Private gains from an investment in energy efficiency are realized through two main channels: reduced energy consumption and increased consumption of energy services (e.g., lighting, space heating, air conditioning) due to reductions in the price of energy services. With respect to the first channel, any reduction in dollars spent on energy can be allocated to other forms of welfare-enhancing consumption. The second channel becomes important when an efficiency-induced reduction in energy end-use costs leads to an increase or “rebound” in the demand for the energy service. For utility-maximizing agents, any reoptimization of consumption that occurs in response to an efficiency improvement will be (weakly) welfare improving.

These basic ideas are illustrated in the upper quadrant of Figure I, which plots consumption of a particular energy service, home heating, on the horizontal axis and consumption of the numeraire (i.e., all other goods), X, on the vertical axis. This framework focuses on home heating because it is a particularly important end use in our empirical setting; over 90% of projected energy savings from the weatherization investments we analyze are heating related.

The two downward-sloping lines in the upper quadrant of the figure reflect budget constraints, the lower before the efficiency improvement (i.e., the status quo) and the higher after weatherization. The budget constraint pivots postweatherization because the price of heating services (e.g., the price of keeping the house at a certain indoor temperature in winter) has fallen; energy efficiency improvements reduce the cost of purchasing any given level of thermal comfort.

The figure also illustrates a family of indifference curves for a representative consumer, each of which trace out the bundles of the numeraire, X, and heating services, H, that deliver the same level of utility. The U-shape of the indifference curves reflects that households do not like to be too hot or too cold. In the status quo (i.e., absent an efficiency improvement), the representative agent will maximize utility through the choice of H₀ and X₀. The weatherization-induced expansion of the budget constraint allows the agent to move to a higher level of utility associated with the bundle of H₁ and X₁. In the figure, status quo consumption occurs below the satiation point for thermal comfort. Thus, when the price falls, demand for heating services increases by H₁ − H₀. The positive income effect also increases consumption of the numeraire by X₁ − X₀.

The article’s empirical challenge is to measure the welfare gains conferred by weatherization investments. Our empirical setting allows us to develop a measure of willingness to pay (WTP) for weatherization that accounts for both reductions in energy expenditures, and increased consumption of heating services. The effect of energy efficiency improvements on other consumption can be measured using data on monthly energy expenditures which vary one-for-one with the consumption of the numeraire. Put another way, a $1 decrease in energy expenditures allows for a $1 increase in consumption of all other goods. Measuring willingness to pay for the increase in heating services (i.e., the direct rebound effect) is more challenging because demand for energy services, such as heating, is not readily observable in household energy consumption or expenditure data.

To obtain an estimate of the efficiency-induced increase in demand for heating (H₁ − H₀ in the figure), we conduct a survey of indoor temperatures in weatherized and nonweatherized homes. With this estimate of the treatment-induced change in indoor temperatures in hand, we can construct bounds for the welfare consequences of any observed increase in warmth by imposing some structure on the relationship between energy demand and heating services.⁵ The bottom quadrant of Figure I plots a representative building-specific relationship between heating services and the energy required to achieve that temperature, E, holding constant outdoor temperatures and building characteristics, Z.⁶ Efficiency improvements to the building envelope (e.g., insulation improvements, window sealing, a furnace upgrade) reduce the energy required to deliver any given level of heating services. This implies that the slope of the relationship between heating services and energy consumption becomes less steep following an efficiency improvement, depicted as the pivot in the function that determines energy consumption from E(H; SQ, Z) to E(H; W, Z) in the figure. Section V.C details our approach to empirically estimating the relationship between energy consumption and heating demand, which in practice is measured as indoor temperature.

We use the empirical relationship between indoor temperatures and energy consumption to construct bounds on the utility gains from any efficiency-induced increase in the demand for indoor temperature using revealed-preference logic. Since the agent chooses to increase heating services from H₀ to H₁ following the efficiency improvement, it follows that a lower bound for the associated increased utility is the increase in heating costs incurred after weatherization. This value is represented by P_E*(E_{1, W} − E_{0, W}), where P_E is the exogenous price of energy. Note that the agent chose less heating than H₁ prior to the efficiency improvement. Thus, the cost of the change in heating services prior to weatherization, measured by P_E*(E_{1, SQ} − E_{0, SQ}), provides an upper bound on the welfare gain. Preferences revealed prior to the efficiency improvement suggest that the agent values the increase in heating services less than this incremental cost.

With this conceptual framework as a guide, the article estimates the causal effect of WAP participation on annual energy consumption and WTP for changes in heating services.

III. Background and Study Design

III.A. WAP

The WAP is the nation’s largest residential energy-efficiency program. WAP supports improvements in the energy efficiency of dwellings occupied by low-income families. Since its inception in 1976, over 7 million low-income households have received weatherization assistance through the program. The American Recovery and Reinvestment Act PL111-5 (ARRA) dramatically increased the scale and scope of WAP.⁷ Our analysis seeks to estimate the impacts of weatherization assistance over the ARRA-funded time period.

WAP funds are distributed to states based on a formula tied to a state’s climate, the number of low-income residents, and their typical energy bills. The states distribute WAP money to over 1,000 local subgrantees, which are typically community action agencies (CAAs) or similar nonprofit groups. These subgrantees are then tasked with identifying and serving eligible households. The average participating household in our data received an average of $4,130 of energy efficiency investments and over $1,000 worth of additional house improvements at zero out-of-pocket costs.⁸

Before implementing a weatherization retrofit, CAA program staff conduct an energy audit of the home. The purpose of the audit is to recommend specific efficiency improvements. During the visit, program auditors collect detailed information about the building structure, heating and cooling systems, appliances, insulation, ventilation, and so on. This information is combined with local climate conditions and retrofit measure costs, then fed into a computer-based audit tool: the National Energy Audit Tool (NEAT). This tool is composed of a set of easy-to-use but advanced energy audit computer programs that identify the cost-effective energy-efficiency retrofit measures for a home after taking into account local weather conditions, retrofit measure costs, fuel costs, and specific construction details of the home. NEAT was designed specifically to help states and local weatherization agencies implement WAP, although it is also a widely used tool for private-sector weatherization audits (EERE 2010).⁹

NEAT produces an estimate of the energy savings and costs associated with different combinations of efficiency measures. The present value of projected energy savings are calculated using a discount rate of 3% and an engineering estimate of the life span of the measures. The 3% discount rate is consistent with Office of Management and Budget guidance on how to evaluate benefits of federal spending but is substantially lower than the cost of borrowing for most households, especially low-income ones. The WAP program requires that all recommended measures return a minimum of $1.00 in incremental savings for every $1.00 spent in labor and material costs.

The process of applying for weatherization is highly onerous and time intensive, at least partially to prevent fraud. Applicants must submit extensive paperwork documenting their eligibility, including utility bills, earnings documentation, Social Security cards for all residents of the home, and deeds to the home. Local agencies often identify potential applicants from the pool of households that are receiving other social services, although walk-in clients are routinely admitted. Agencies screen potential applicants for eligibility. Eligible applicants are then prioritized following guidelines that recommend CAAs assign the household a high rank if it has an elderly resident, a person with disabilities, or children, or where the occupants typically face a high energy burden (energy as a share of income) or have high residential energy use (see 10 CFR 440.16(b) (1–5)).¹⁰ Given the nonrandom nature of the process by which households end up in the program, any comparisons of energy consumption across weatherized and unweatherized households risks confounding the effect of the program with preexisting differences in determinants of energy consumption.

III.B. Research Design

Our analysis focuses on a sample of low-income Michigan households. Michigan is one of the largest recipients of WAP program funding on account of its cold winters and large low-income population. Furthermore, we were able to secure collaborative agreements with a major Michigan utility and five CAAs working in this utility’s service territory. This allowed us access to detailed, household-level energy consumption and weatherization program data. A close collaboration with two of these agencies allowed us to implement a large-scale field experiment.

Michigan received over $200 million in ARRA funding for weatherization assistance. A series of bureaucratic delays—for example, ensuring that contractors were paid prevailing wages—delayed spending until early 2011 (Radnofsky 2010). Around March 2011, weatherization activities in Michigan increased markedly. All stimulus funds had to be spent by March 2012. After that point, the pace of weatherization activity dropped precipitously. We stopped collecting data in April 2014. A number of households that applied for WAP had not yet received services by the end of our study period.

The parameter of interest is β, which measures the mean difference in energy consumption subsequent to the completion of WAP energy efficiency investments, after adjustment for the fixed effects. It is a difference-in-differences estimator that compares the change in energy consumption after weatherization to before, relative to consumption among households that have either not yet weatherized through WAP or never did during our sample period. We take two approaches to developing unbiased estimates of β, described in the next two subsections.

1. Randomized Encouragement Design

The article’s primary empirical estimates are derived from an experimental research design. The basis of the experiment is a randomly assigned encouragement intervention that aims to increase the probability of treatment households’ participation in WAP through recruitment and significant application assistance. This randomized encouragement design is particularly useful in contexts such as this one where program participation cannot be directly denied—we cannot prohibit eligible households from participating in this federal program.

The recruitment and assistance was conducted by FieldWorks, a private company that specializes in running neighborhood canvassing operations and managing outreach campaigns. We chose them because they had substantial experience working with low-income populations, and their staff had generated millions of phone calls and knocked on millions of doors in previous engagements.

The experimental sample comprised 34,161 households that were both presumptively eligible for WAP and located within the counties served by the two CAAs that partnered with us for the field experiment. Approximately one quarter of the sample households were randomly assigned to our encouragement treatment. For the remaining households assigned to the control group, we simply observe energy consumption and program participation decisions.

The encouragement campaign got under way as ARRA funds began flowing to the implementing agencies. Encouragement activities ran from March to May 2011. Table I summarizes our encouragement and enrollment activities in detail. During the encouragement phase, field staff made almost 7,000 initial in-person house visits.¹² The ground operations were complemented with 23,500 targeted robo-calls to raise awareness of the weatherization program and our encouragement campaign.

After the encouragement phase, we transitioned to an enrollment phase, which lasted through February 2012. Our staff made over 9,000 personal phone calls to provide assistance with the onerous application process and to coordinate in-person meetings. Over the course of 2,720 home visits, field staff helped individuals assemble documentation and complete paperwork. In some cases, field staff provided transportation to and from the program agency offices.

The final row of Table I reports that we spent around $475,000 on the encouragement or a little more than $55 per household in the treatment group. It is noteworthy that we did not initially intend to devote such extensive efforts and resources to the encourage and enrollment phases. However, the early results suggested that we were failing to have a substantial impact on applications, and concerns about the ultimate precision of our estimated treatment effects motivated us to raise additional funds to expand the share of treated households.¹³

We substitute |${\mathbf {1}\lbrace \widehat{WAP}\rbrace _{imt}}$| from the estimation of equation (2) to fit equation (1) and obtain |$\hat{\beta} _{IV}$|⁠. In this instrumental variables (IV) framework, |$\hat{\beta} _{IV}$| is identified using the exogenous variation in program participation that is generated via the random assignment of encouragement.

2. Quasi-experimental Design

The quasi-experimental research design uses data collected from households that applied for WAP after March 2011 with one of the five CAAs that shared data with us to estimate a version of equation (1). In this sample of households, we compare patterns in energy consumption among weatherized households and households that applied for WAP but had not been weatherized by mid-2014, when our data ends. Forty percent of applicants in our data were weatherized through WAP during this period.

A critical issue for the validity of the quasi-experimental estimates is how households in this sample were chosen for weatherization. The road from application to energy efficiency investments is long and there are many potential off-ramps. Applicant households may fail to complete the necessary—and involved—paperwork or may be deemed ineligible based on the information they provide. Once paperwork is completed successfully, households are put on a list, and the waiting times can exceed one year. Once at the top of the list, homeowners must schedule an energy audit. Households may fail to receive weatherization if they miss an audit appointment, or if the auditors discover risks to WAP contractors (e.g., asbestos in the home). Because of significant delays in ramping up weatherization activities under ARRA, agencies were unable to complete the weatherizations they anticipated prior to the March 2012 ARRA deadline, which helps explain why fewer than half of the applicants in our sample were weatherized by mid-2014. Some of the explanations for variation in treatment status among applicants are orthogonal to household characteristics that determine energy consumption patterns, whereas others clearly are not.

The primary threat to consistent estimation of β using the quasi-experimental approach is the possibility that time-varying factors that affect household demand for energy, also influence WAP participation. For instance, households may push forward their WAP application more aggressively when they anticipate an increase in their demand for energy, as would be the case when the number of people in their household increases or they lose a job and expect to spend more time at home. Although the quasi-experimental approach does not have a direct solution to this threat to identification, we take steps to balance observable characteristics and trends across weatherized and unweatherized households. To evaluate the robustness of our findings, we reestimate equation (1) using alternative sets of controls and regression weights. Our preferred quasi-experimental specification reweights control observations to achieve covariate balance across weatherized and unweatherized controls, explained in more detail below.

Of course, it is possible that the effect of a WAP weatherization varies across households (i.e., β_i) based on observables or unobservables. Indeed, it is possible that households sort into WAP participation based on knowledge of their idiosyncratic returns; this has been labeled “essential heterogeneity” (Heckman, Urzua, and Vytlacil 2006). In the presence of heterogeneity in how weatherization retrofits affect residential energy consumption, the expectation of unbiased quasi-experimental and experimental estimates of β_i need not be equivalent. The quasi-experimental approach is designed to provide an estimate of the average treatment effect on all treated households (ATET). In contrast, with some additional assumptions (Angrist, Imbens, and Rubin 1996), the randomized encouragement design estimates the so-called local average treatment effect (LATE), or the average effect for the subset of the population who must be encouraged to participate in the program (i.e., the compliers). Therefore, significant differences in the quasi-experimental and experimental estimates could be due to bias in the former or differences in the LATE and ATET. Section 5.2 and Section 6 of the Online Appendix explore the sources of the differences in our quasi-experimental and experimental estimates of the effect of WAP weatherization.

IV. Data Sources and Summary Statistics

IV.A. Data Sources

The data collected to support this analysis correspond to two overlapping groups of households. The first group comprises the 34,161 households in our experimental sample drawn from the counties served by our two partner CAAs. The second group of households corresponds to our quasi-experimental research design. Because this design did not require the same degree of coordination with our agency partners, we were able to expand the scope of this sample by collecting detailed data from three more implementing agencies. The quasi-experimental sample includes the 7,304 households that applied for weatherization assistance at these five agencies. The quasi-experimental sample is smaller overall but has a larger number of applicants and weatherized households relative to the experimental sample. The two groups overlap as 1,773 applicant households are also part of our experimental sample.

1. Energy Consumption Data

We obtained monthly natural gas and electricity consumption data over the period June 2008 to May 2014. This period includes at least two years of preretrofit data for all weatherized households in our sample. The utility data track monthly kilowatt-hours (kWh) of electricity and thousand cubic feet (Mcf) of gas used at the dwelling. Some specifications convert both natural gas and electricity consumption to million British thermal units (MMBtu) using the conversion factor employed by the NEAT audit tool.

Energy consumption records obtained from the utility are merged with households in our experimental sample and the applicant data obtained from the five implementing agencies. Data are merged using detailed name and address information. Not all households find an exact match in consumption records. Match rates are 88% and 81% in our experimental and quasi-experimental samples, respectively. The higher match rate in our experimental sample is to be expected; when selecting this sample we focused exclusively on postal codes within the territory of our partner utility. Online Appendix 2 provides more detail on missing data and attrition. After dropping observations with no match, our experimental sample is 28,888.

2. Application Data

We obtained the data about households collected through the application process, including information used to determine program eligibility (e.g., income, household size, number of children) and information that can be used to prioritize successful applicants (e.g., elderly residents). Application materials also collect other demographic characteristics, including race and education. These data are used to assess balance across the quasi-experimental treatment and control groups.

3. Efficiency Audit Data

Detailed information about the dwelling is collected during the household-level energy efficiency audit, which is a critical part of the WAP implementation. An output of these audits is the list of energy efficiency measures for which projected energy savings exceed projected costs. These are the measures that comprise the weatherization retrofit. We acquired the job reports filed after the weatherization retrofit is completed. These allow us to confirm that the recommended measures were installed and to compare realized costs with projections.

4. Indoor Temperature Data

Two years after the encouragement effort was initiated, we randomly selected a subset of weatherized and unweatherized households for a field survey. These households were selected from the quasi-experimental sample. The primary purpose of the survey was to measure thermostat set points and indoor temperatures to test for a direct rebound effect.

Michigan field staff attempted to contact 6,400 households on cold days (projected maximum temperature below 45°F) in March and early April 2013. With the homeowner’s permission, surveyors entered the home, closed the door, moved to the center of the room, and recorded multiple indoor air temperature measurements using two different thermometers. Of our initially targeted sample, surveyors spoke with 1,658 homeowners. Of these, 899 allowed us to enter their homes and record their thermostat set point, and 688 allowed us to close the door and collect two or more indoor thermometer readings.

5. Demographic Data

Detailed data on occupant and dwelling characteristics are not available for the households in our study that did not apply for weatherization assistance. To construct a more complete picture, we map households in the study to census-block-level data on variables of interest (such as home age, poverty level of residents). We use these data to assess balance in observable characteristics across our experimental encouraged and control groups.

IV.B. Summary Statistics

Table II summarizes pretreatment information on the households in both the experimental and quasi-experimental samples. The top panel summarizes monthly energy consumption during the two years immediately preceding the treatment period. The first two columns summarize means for the randomized encouragement and experimental control groups, respectively. The third column reports p-values for the differences between the treatment and control groups.

There are seasonal patterns in energy consumption that differ across natural gas and electricity. Winter natural gas consumption (which is dominated by space heating) is significantly higher than summer gas use (composed primarily of hot water heating and cooking). Electricity usage is fairly consistent across seasons. Because households in the experimental sample were randomly assigned to the encouraged and control groups, it is unsurprising that differences in energy consumption across these two groups are all small and statistically indistinguishable from zero.

The table also provides an opportunity to judge the credibility of the comparisons that underlie the quasi-experimental estimates. The fourth column reports on all weatherized households in the territory covered by the five CAAs. The fifth column summarizes households in these territories that applied for weatherization but had not received weatherization assistance as of April 2014. In practice, the variation in the weatherization dates means that the identification of the quasi-experimental estimator is not just based on comparisons between the samples summarized in columns (4) and (5), but also relies on within-household comparisons. It is nevertheless informative to compare these sets of households. p-values for the mean differences in column (6) show that weatherized households have historically consumed significantly less natural gas than the unweatherized applicants during both winter and summer months.

To assess the balance in characteristics other than energy consumption across the experimental control and encouraged groups, we locate households in the experimental sample within census blocks and summarize dwelling and household information at the census-block level. Panel B summarizes these data. p-values in column (3) account for clustering within census blocks. None of these differences in group means are statistically distinguishable from zero.

Table II, Panel C summarizes the detailed demographic information and dwelling characteristics that are collected for most clients as part of the application process. Our summary of these variables focuses exclusively on program applicants because these data are not available for the majority of households in the experimental sample that did not apply to the program. Column (6) documents important differences between the weatherized and unweatherized applicant subsamples. Note that differences across Panels B and C should be interpreted carefully because variables are measured differently across the two data sources. For example, income reported in Panel B summarizes incomes at the census block level and includes all income received on a regular basis. Income reported for the purpose of determining weatherization program eligibility (measured at the household level and summarized in Panel C) includes cash receipts (e.g., wages, salaries, pensions) but excludes other potential income sources (capital gains, tax refunds, or housing allowances). Weatherized households have higher incomes compared with unsuccessful applicants. In addition, applicants who ultimately receive weatherization are further above the poverty line, which adjusts for household size, than are the unsuccessful applicants. Weatherized households also report having more children, are more likely to report an elderly or disabled resident, and are more likely to use natural gas as their primary heat source.

The significant differences between the weatherized and unweatherized applicants motivate us to reweight observations in the control group so that observable factors are distributed similarly in applicant groups. Although our preferred specifications include household-by-month-of-sample fixed effects that control for all time-invariant differences between households, we are concerned that observable fixed differences might be correlated with time-varying differences. We use an estimated propensity score to balance covariates that presumably play a role in determining program take-up. Online Appendix 3 provides a detailed discussion of the participation equation we estimate. Table II, column (7) reports p-values for the mean differences between weatherized households and matched controls. Reassuringly, the differences in average covariate values across weatherized households and propensity-score weighted controls are statistically indistinguishable.

In the Online Appendix, Table A.1 summarizes some of the detailed information collected during the household energy efficiency audits. A typical weatherization retrofit involved several measures such as furnace replacement (34% of retrofits), attic insulation (85%), wall insulation (44%), and infiltration reduction (76%). The NEAT model predicts that on average these investments will reduce natural gas consumption by 46% and electricity consumption by 15%. The average project involved over $5,150 in total expenditures. This includes materials, labor, and construction costs, but does not include any program overhead. Using a 3% discount rate, the projected net present value of energy bill savings average $10,611. The average projected savings:investment ratio (across measures) exceeds 2:1.

V. Results

V.A. Experimental Estimates of Energy Savings

1. First-Stage: Program Take-Up

It may seem straightforward to encourage households to participate in a program that provides free efficiency retrofits worth an average of approximately $5,000 that are designed to significantly reduce energy expenditures. In our experience, that was hardly the case. The impact of reducing barriers to participation (e.g., information and process costs) on program uptake is of independent interest both to policy makers and researchers. This section evaluates the impact of our intervention on a multipart participation process.

Table III summarizes program take-up at three separate junctures. In the first stage, our goal was to increase the share of households filing applications. The coefficient estimate in column (1) indicates that the encouragement intervention increased the rate of application to the program by 13 percentage points from the control group mean of 2%. Column (2) reveals that the fraction of households who received an energy audit was 5 percentage points higher in the encouraged group (off a base of about 1%). As we discussed already, several factors can explain why so many households that submit an application are not audited, including if the household fails to follow through on requests for more information or if the submitted information indicates that the household does not meet the program’s eligibility requirements.

Table III, Column (3), documents that the treatment increased the fraction of households that were successfully weatherized by about 5 percentage points, against a 1% rate in the control group.¹⁴ The encouragement treatment is a statistically significant predictor of weatherization, which we will use to instrument for program participation.

The low take-up rates in the encouraged group are quite striking. Program participants receive substantive home improvements, yet incur no out-of-pocket expenses. All households in the encouraged group received some information about the program via a phone call or door hanger. A majority of households (i.e., those who spoke with our canvassers in person or by phone) received further information about our offer of application assistance. Given that households had detailed, specific information about the program, it seems reasonable to surmise that some combination of high perceived costs of applying for the program, low expectation of an application leading to a weatherization, high unmeasured process costs, and low expected benefits of participating in the program are impediments to WAP participation. In the end, the average cost of encouragement per completed weatherization was about $1,050, which is more than 20% of the average costs of measures installed. See Fowlie, Greenstone, and Wolfram (2015) for further details.

2. IV Estimates of Energy Savings

|$\phantom{.}$|Figure II provides a graphical overview of the experimental estimates. The broken line shows the cumulative effect of the randomly assigned encouragement intervention on the monthly rate of weatherization, relative to the control group which received no encouragement. This effect increases over time as the treatment households submit applications and receive weatherization assistance. The figure also plots month-by-month estimates of intent to treat (ITT) effects on energy consumption. Conceptually, monthly estimates of the local average effect of weatherization on energy consumption can be constructed as a ratio of the monthly ITT estimates and the corresponding effect of encouragement on program participation.

Table IV, Panel A summarizes results that relate measures of energy consumption to the WAP participation indicator. In the first two columns, the dependent variable is the log of total energy consumption (MMBtu/month). In the third and fourth columns, the dependent variables are the log of natural gas and electricity consumption, respectively. Standard errors are clustered by household in all specifications.

For comparison, the first column reports the coefficient associated with the WAP indicator from an OLS regression with month of-sample and household-month fixed effects that is fit to the experimental sample. This specification indicates that WAP participation is associated with a 10% decline in energy consumption.

The IV estimates based on our randomized encouragement design are reported in columns (2)–(4). The estimate in column (2) indicates that WAP participation causes a reduction in monthly energy consumption of approximately 20% among households that were encouraged into the WAP program. This is the first indication that the realized savings from the WAP-induced energy efficiency investments are substantially smaller than the projections from the engineering model. Columns (3) and (4) report local average treatment effects for natural gas and electricity, respectively. Natural gas accounts for 94% of projected savings (measured in MMBtu), so it is not surprising that natural gas consumption is more significantly affected by weatherization.¹⁵

Our IV strategy is predicated on an exclusion restriction: we assume that our encouragement activities affected energy consumption only through its effect on participation in the WAP program. To informally test whether the treatment’s encouragement activities had a direct effect on energy consumption, we test for an effect of our encouragement activities on the households in the encouraged group that did not receive weatherization assistance. We fail to reject the null of no effect on energy consumption among these households.¹⁶

Table IV, Panel B computes the present value of the estimated energy savings under alternative assumptions about investment time horizons and discount rates. To express the estimates of monthly energy savings in dollar terms, we first impute the average consumption among unweatherized compliers.¹⁷ Estimates of monthly natural gas savings (in percentage terms, based on the coefficient in column (3)) are multiplied by the product of this imputed average monthly gas consumption among control group compliers and the residential retail price of natural gas in Michigan in 2013 (EIA 2015).¹⁸ Similarly, average 2013 electricity savings (in percentage terms, based on the coefficient in column (4)) are multiplied by the product of average monthly electricity consumption among control group compliers and the retail electricity price ($0.11/kWh).¹⁹ Taken together, our estimate of average energy savings is approximately $233 a year.

To compute the net present value of energy savings over the useful life of the improvements, we invoke some additional assumptions. First, we rely on the NEAT simulation program’s assumptions about measure-specific life spans. These projected life spans range from 3 years (for a furnace tune-up) to 20 years (for attic insulation). The energy savings-weighted average life span for installed measures in our data set is 16 years. In the table, we report discounted benefits with assumed life spans of 10, 16, and 20 years. We also assume that the effect of weatherization on energy consumption—and real energy prices—do not vary over the life of the measure. Discounted benefits are calculated at discount rates of 3%, 6%, and 10% (see columns).

The estimated energy savings are small relative to the projected savings and the upfront costs. Our estimates imply energy savings of 17 MMBtu a year, whereas the average projected savings among compliers is 56 MMBtu (aggregating electricity and gas measured in terms of site energy content). Our central estimate of the realized average savings per household is roughly $2,349. This is significantly lower than the average net present value of savings predicted by the ex ante engineering analysis which is $9,810 among compliers (and $10,611 averaged across all weatherized households in our data).²⁰ The nine estimates of the present value of the savings range from approximately $1,428 (high discount rate and short time horizon) to about $3,500 (low discount rate and long time horizon). These estimates are just 31% to 76% of the average upfront cost of the energy efficiency measures among compliers.²¹

V.B. Quasi-experimental Estimates of Energy Savings

Table V presents the quasi-experimental estimates based on the estimation of equation (1). The dependent variable in all regressions is the log of monthly energy consumption (i.e., the sum of electricity and natural gas, both measured in MMBtu). The first two columns use data from all weatherization applicants, with the second specification allowing time period effects to vary across counties. Columns (3) and (4) trim the sample to obtain estimates that are more directly comparable to the experimental estimates. Specifically, this sample is limited to the implementing agencies that participated in the experiment and to applicants that applied after the encouragement intervention was initiated. Columns (5) and (6) report estimates comparable to columns (1) and (2) reweighted by the propensity score.²²

The first row in Table V reports the estimated average treatment effect. Across the columns, the estimates suggest that WAP participation reduces energy consumption by roughly 8–10%.

Figure III provides another perspective on the results in Table V. It reports on the estimation of a version of the column (1) specification, except the weatherization indicator is interacted with indicator variables for each of the potential quarters before and after weatherization was completed (the time zero effect captures energy consumption in the month of weatherization). We also interact the quarter indicators with heating degree days (HDD) and HDD squared, to account for the fact that weatherization retrofits may lead to higher energy savings during colder quarters.²³ The figure then plots the coefficients on the quarter dummy variables plus the coefficients on the HDD and HDD squared variables multiplied by the average HDD and HDD squared for the corresponding month. We also plot 95% confidence intervals associated with these coefficient sums. The decline in energy consumption is apparent and seems roughly constant throughout the period of our study.

Table V, Panel B computes the present value of the estimated energy savings using essentially the same approach described in Section II. To estimate monetary savings, we need to disaggregate our energy savings estimates by fuel. We estimate equation (1) separately for gas and electricity, respectively (see Online Appendix 4). Again, the energy savings are small relative to the upfront costs. The nine estimates of the present value of the savings range from approximately $1,000 (high discount rate and short time horizon) to about $2,400 (low discount rate and long time horizon). These estimates are between 22% and 53% of the average upfront cost of the energy efficiency retrofits.

The experimentally estimated LATE of weatherization on energy consumption is approximately twice as large as the quasi-experimental estimate of the average effect among all households receiving weatherization assistance. Online Appendix 6 documents that there are significant observable differences between the group of complier households in the experimental sample and the weatherized households in the quasi-experimental sample, but they only explain about 30% of the difference between the experimental and quasi-experimental treatment effect estimates.

“Essential heterogeneity” (Heckman, Urzua, and Vytlacil 2006), where households sort into participation based on their gains, is a potential explanation, but the data fail to support standard sorting stories because the experimentally estimated LATE exceeds the estimated ATET from the quasi-experimental sample where households choose to participate in WAP without our encouragement. Measurement error is perhaps a more likely explanation; specifically, the results are consistent with the possibility that the quasi-experimental estimates are attenuated due to measurement error and that the experimentally derived IV removes this bias. Indeed, the finding that the OLS estimate in column (1) of Table IV is comparable to the quasi-experimental estimator is striking and consistent with measurement error playing an important role here.²⁴ Finally, we are using different (albeit overlapping) data sets to implement the experimental and quasi-experimental estimation, so the differences in the estimated parameters could be due to sampling variation.

V.C. Household Reoptimizing Behavior, Building Thermal Properties, and the Welfare Implications of Rebound

This section implements the utility-maximization framework outlined in Section II to bound the average willingness to pay for any efficiency-induced increase in energy services.

Does Weatherization Lead to Temperature “Take Back”? We first test for an effect of weatherization on household demand for space heating. Table VI summarizes the results from our survey of indoor air temperatures and thermostat set points collected during the winter of 2013 at a subset of our quasi-experimental households. All weatherized households surveyed had received efficiency improvements at least one year before the survey was administered, allowing plenty of time for residents to observe the extent to which the weatherization retrofit affected winter heating costs. A total of 899 households allowed our survey team to enter their home to record the thermostat set point, and 688 allowed our surveyors to linger long enough to record the actual indoor temperature. Approximately half of the households for which we have thermostat or temperature data had received weatherization assistance (453/899 and 349/688). The remaining households in the control group had applied for–but not received–weatherization assistance. Anticipating some measurement error, we used two different devices to measure indoor temperatures at each home.

Table VI reports results from regressing thermometer readings and thermostat set points on a binary variable indicating whether the household had been weatherized.²⁵ Columns (1) and (3) report results from our base specification. Survey respondents comprise a nonrandom subset of our sample as only a fraction of the targeted household were home and/or willing to open the door to receive our surveyors. Furthermore, only 41% of households who opened the door to our surveyors allowed us to come in and collect temperature measurements. We observe differences between survey respondents and the larger sample along potentially important dimensions (such as past energy use and demographics). We estimate specifications where survey household observations are weighted to improve the match in covariate means with the larger quasi-experimental sample.²⁶ These results are reported in columns (2) and (4). All specifications control for the outdoor temperature on the day of the survey, measured by heating degree days (HDDs).

Columns (1) and (2) show that after controlling for outdoor temperatures, indoor temperatures may be slightly warmer at weatherized households. The point estimate is imprecisely estimated; we fail to reject the null of 0. The coefficient associated with the HDD variable is statistically significantly negative in the temperature specifications, suggesting that indoor temperatures measure lower on colder days. Although we asked surveyors to wait several minutes before recording temperatures, this finding suggests that cold air brought with the surveyor could be affecting the measurements. Because the standard errors of our estimates do not allow us to reject a small increase in indoor temperatures, we estimate an upper bound on possible welfare gains using an estimated increase in temperature of 0.67°F based on the point estimate in column (2).²⁷

Columns (3) and (4) suggest that weatherized households set their thermostats lower, on average, by approximately 0.6°F. This is inconsistent with a rebound in demand for indoor heat. A decrease in thermostat settings may seem inconsistent with the suggestive evidence of an increase in indoor air temperatures. One possible explanation is that retrofits, by reducing cold air infiltration, allow households to maintain the same (or slightly higher) levels of indoor comfort at lower thermostat set points. It is also potentially relevant that the study period covers the aftermath of the Great Recession when the marginal utility of income was elevated.

2. Building Energy Performance

To estimate how a given increase in demand for indoor space heating translates to an increase in monthly gas consumption, we need to estimate the thermal properties of a representative building. We use the so-called degree day method to model the energy required to increase temperatures in an average home in our data (Thorpe 2013). The literature that analyzes energy use in buildings commonly assumes a linear relationship between energy consumption and the difference between outdoor temperatures and a temperature that people find comfortable indoors (e.g., Friedman 1987; Dyson et al. 2014). We find empirical support for this assumption in our setting.

All of the estimated coefficients in equation (3) are highly statistically significant and very precisely estimated. The R-squared value is 0.69. The estimated slope of the relationship is less steep among weatherized homes. That is, weatherizations effectively reduce the marginal cost of indoor space heating during the winter. Moreover, it is noteworthy that this relationship is approximately linear though we estimated quadratic terms. Online Appendix Figure A.2 summarizes the estimated relationship, which is analogous to the bottom panel of Figure I (flipped into the top quadrant).

We can use this estimated relationship between HDD and monthly energy consumption to estimate the effect of increased demand for indoor temperatures on energy consumption and associated expenditures. This approach assumes that a household’s choice of the indoor temperature is independent of outdoor temperatures, thus outdoor temperatures are a valid proxy for the desired level of heating services.²⁸ The product of the slope of the relationship between natural gas consumption and HDD (estimated from equation (3)) and the average natural gas price in the postencouragement period provides a measure of the marginal cost of gas heating. Among unweatherized households, this product is equal to approximately $0.072 per HDD (or $2.17 per heating degree month). The analogous calculation for weatherized households leads to an estimated marginal cost of $0.056 per HDD (or $1.67 per heating degree month). This implies that weatherization led to a reduction in the marginal cost of approximately 20%.²⁹

3. Bounding the Average Valuation of Increased Indoor Heat

Estimates of the marginal costs of heating among weatherized and unweatherized households can be combined with the estimated 0.67°F increase in indoor temperatures (see Table VI, column (2)) to bound the average welfare gain from weatherized households’ reoptimization.³⁰ The lower bound of households’ valuation of the higher temperatures is given by 0.67°F * $1.67/degree month or $1.12 per winter month. At this lower bound, the utility gains from increased warmth are exactly offset by the increase in the energy expenditures incurred to achieve the temperature increase, implying a zero gain in welfare.

To define the upper bound, we note that by revealed preference, unweatherized households chose not to pay 0.67°F * $2.17/degree month or $1.45 per winter month to achieve this incremental increase in temperature. It follows that average marginal benefits from this temperature increase cannot exceed $1.45 per winter month. Since increasing indoor temperatures by 0.67°F costs the average weatherized household $1.12 per winter month, the average net gain from the weatherization-induced increase in warmth does not exceed $0.33 per winter month. Assuming six full winter months in Michigan, this implies an upper bound on the annual welfare gain of roughly $1.98 from higher indoor temperature.

In sum, this bounding exercise suggests that the welfare gains from any efficiency-induced rebound in heating demand are very small, likely less than 1% of the energy expenditure savings. Put another way, the efficiency-induced rebound in demand for indoor heating appears to be inconsequential in this setting. Our methodology also allows us to calculate the potential welfare gains from much larger increases in indoor temperatures. Note that even a 10°F increase in indoor temperatures would lead to a welfare gain of less than $30 a year.

VI. Interpretation

VI.A. Returns on Residential Energy Efficiency Investments

Table VII, Panel A evaluates the internal rate of return (IRR) on energy efficiency investment from a private perspective. More precisely, we report the discount rate at which the discounted value of average avoided energy expenditures exactly equals the average upfront investment. Although these investments were free to participating households, it is nevertheless informative to estimate the private returns if households had been responsible for the upfront costs because most households that consider the exact same investments do not qualify for WAP. That is, in the broader population, the IRR is a critical factor in determining take-up of energy efficiency investments.

Column (1) computes the IRR using the average upfront investment costs³¹ and the average reductions in annual energy expenditures projected by the WAP program audit. These savings are valued at the average retail residential natural gas and electricity prices in 2013 as described already. Over a range of time horizons, the estimated IRR is quite high, as they should be given the investment rule that required projected savings exceed costs. The rate of return associated with the savings-weighted average life span (i.e., 16 years) is approximately 12%. By this measure, efficiency investments supported under WAP appear to be very attractive investments that greatly exceed typical returns available in equity, real estate, and bond markets.

The second column of Table VII replaces the projected savings with an estimate of the actual energy savings (in monetary terms), derived from the randomized encouragement design, plus a generous estimate of the welfare gain from higher indoor temperatures (i.e., the estimated upper bound of the monetized value of the net welfare gain).³² The estimate of realized savings is about $232, which is only 36% (= $232/$653) of the average projected annual monetary savings.³³ When the upper bound of the monetized value of the higher indoor temperatures is added to the energy savings, the annual benefits are approximately $234. Using this measure of annual benefits, the IRR is −10.6% for the 10-year horizon, −2.3% for 16 years, and 0.2% for 20 years. Negative returns suggests that, at least for residential home retrofits, there may not be much of an efficiency gap to explain. Investments with these returns are infrequently taken up in the broader economy.

Panel B conducts a similar exercise but adds the value of avoided emissions to the benefit side of the ledger. Avoided emissions of CO₂ are valued at $38 per ton (Greenstone, Kopits, and Wolverton 2013). Nitrogen oxide and sulfur dioxide emissions from residential gas consumption are valued at $250 per ton and $970 per ton, respectively (Muller and Mendelsohn 2009).³⁴ The IRRs for the 10-, 16-, and 20-year horizons are −7.6%, 0.1%, and 2.3%, respectively.

The societal perspective is an especially important for judging these investments, because a broad range of policies encourage residential energy efficiency investments. Panel C reports estimates of the social IRR that are calculated using estimates of the avoided marginal costs, rather than the retail prices, to monetize the energy savings. These avoided costs are then added to the monetized value of the avoided emissions (as in Panel B). The reason for this adjustment is that retail electricity and natural gas prices used in Panels A and B include a fixed cost component; energy providers recover a significant fraction of the fixed costs of supplying energy (e.g., transmission and distribution investments) in their volumetric rates (Davis and Muehlegger 2010). These fixed costs are not avoided when efficiency investments reduce residential natural gas and electricity consumption. Most electricity and natural gas distribution utilities are subject to cost-plus regulation, and in these cases, the fixed costs recovery will be shifted to other customers, meaning that households that reduce energy consumption receive a transfer from other households by reducing their contribution to covering fixed costs. We use the average of the 2013 spot prices ($3.73/MMBtu) set at the Henry Hub distribution point, a standard reference price for natural gas in North America, to reflect the true marginal cost of natural gas over the lifetime of the measures.³⁵ To value reductions in electricity consumption, we use the average wholesale price in the Midwest electricity market and assume transmission and distribution losses of 5%.

These adjustments are reported in Panel C and lead to a meaningful decrease in the IRR, relative to the IRRs in Panels A and B. For example, the social IRR for the 16-year time horizon is −7.8%; it is −17.8% and −4.6% for the 10- and 20-year horizons, respectively. Overall, these residential energy efficiency investments have a negative rate of return across all reasonable time horizons.³⁶

An alternative method to summarize the return on WAP energy efficiency investments is to estimate the cost per ton of CO₂ avoided. This is calculated as the ratio of the net cost of the investments (i.e., the annual rental cost of the upfront investment less the value of annual energy savings and avoided damages from regional pollutants) and the tons of CO₂ emissions reduced a year. Panel D uses a 3% discount rate to calculate the annual rental cost of capital, while Panel E uses 7%. Both panels report results based on assumptions of 10-, 16-, and 20-year life spans for the investments and use wholesale energy prices as above.

As in Panels A through C, the conclusions differ depending on whether one uses projected or realized energy savings. Using projected energy savings values generated by the NEAT program audits and the 16-year life span, the cost per ton of CO₂ avoided is $38 with a 3% discount rate. With these estimates, the energy efficiency investments would break even, from a social perspective, because the abatement cost per ton is approximately equal to the U.S. government’s official value of the social cost of carbon of $38.

However, the estimates that are based on actual energy savings again tell a different story. When the experimental estimates of actual natural gas and electricity savings are used in column (2), the analogous costs per ton of CO₂ avoided are $201 (3% discount rate) and $285 (7% discount rate). These costs exceed the U.S. government’s social cost of carbon by a significant margin. On the basis of the costs and benefits we account for here, these residential energy efficiency investments are not a cost-effective approach to mitigating climate change.

VI.B. What Explains the Low Rate of Return on These Efficiency Investments?

It is natural to ask why the returns to residential energy efficiency investments are so low. After all, WAP contractors only implement measures with projected savings to costs ratios greater than one. Section V.C found that the low returns cannot be explained by a rebound effect, and this subsection examines other potential explanations with the aim of shedding light on whether these findings are externally valid to other settings.

One possibility is that the costs of the investments are inflated in the government-funded WAP program, especially relative to costs in a private market, where most households make energy efficiency investments. Online Appendix Tables A.8 and A.9 compare the costs of four common measures in our data with costs incurred by homeowners participating in two residential audit-based energy efficiency programs administered by the Wisconsin Energy Conservation Corporation (WECC), analyzed by Allcott and Greenstone (2017). The WECC program is a standard residential audit-based energy efficiency program, so it seems reasonable to presume that most of its consumers have incomes that exceed the WAP threshold. The clear conclusion from these tables is that even after adjustment for observables, costs were statistically lower by an economically meaningful amount under WAP for three of the four measures and roughly equivalent for the fourth measure. One can speculate on the reasons, but the data fail to support the possibility that the costs used to calculate the rate of return associated with WAP measures are inflated. The analysis is described in greater detail in the Online Appendix.

Another potential explanation is that contractor quality is low in the WAP program and that this was particularly the case during the Great Recession when WAP greatly increased its scale, making the article’s results unrepresentative of returns under more normal conditions. We note that ARRA funding faced tremendous scrutiny for mismanagement, and during this period WAP agencies indicated an increased commitment to training, monitoring, and quality assurance. Furthermore, it is possible that the quality of contractors could either increase or decrease during this period because, while WAP was scaling up, the home construction industry was declining dramatically, potentially increasing the supply of skilled contractors.

This is ultimately an empirical question, so the Online Appendix investigates the possibility that realized savings vary with contractor experience. More precisely, we test for systematic variation in realization rates (i.e., the share of projected savings that manifest as actual savings). The Online Appendix describes this exercise in detail, including some important data limitations, but the key finding is that we fail to find any relationship between realization rates and our imperfect measure of contractor experience in the WAP program (see Online Appendix Figure A.1). Based on the available data, it appears that contractor heterogeneity cannot explain the low rates of return to energy efficiency investments through the WAP program.

Given that the data reject large rebound effects, and we find no evidence that costs incurred in this government program were abnormally high, we believe that the most likely explanation is that the NEAT efficiency audit tool systematically overstates the real returns to these investments by a significant margin.³⁷ It is noteworthy that we found in our data that the NEAT program predicts baseline natural gas consumption that exceeds actual consumption by more than 25% prior to weatherization, suggesting that the auditing tool could be underestimating homes’ efficiency properties prior to weatherization. This might help explain why the benefits of efficiency measures are overstated.³⁸ Furthermore, recent research (e.g., Davis, Fuchs, and Gertler 2014; Levinson 2016; Allcott and Greenstone 2017) has also found low realization rates for energy efficiency measures in other settings. Ultimately, we do not have a definitive explanation for why the engineering models overstate the savings, but we note that this is not an industry where there is a culture of objective ex post evaluation and that contracts are not written with penalties for a failure to deliver promised returns. Regardless of the cause, it seems clear that significant research into improving the functioning of the NEAT efficiency auditing tool (and similar tools) is warranted as it is used by state and local WAP subgrantees, utility companies, and home energy audit firms.³⁹

More broadly, it is possible that the conventional cost-benefit approach to these programs misses important nonenergy benefits and that such benefits help explain the popularity of energy efficiency programs and investments. The most oft-cited example is that weatherization leads to improvements in health, presumably due to decreasing thermal stress. The Department of Energy produced an evaluation of the weatherization program, and claim that the retrofits produce substantial health benefits (Oak Ridge National Laboratory 2015). Among other things, the analysis claims that the typical WAP retrofit leads to an astounding almost $5,000 of mortality reduction benefits. We investigated this claim and found that it is based on replies to the survey question, “In the past 12 months, has anyone in the household needed medical attention because your home was too cold (hot)?” A series of assumptions are invoked to map these responses to fatality rates. The estimates were also derived from a nonstandard statistical technique and included no effort to assess statistical significance.⁴⁰ Finally, the DOE’s own analysis of nonenergy benefits fails to account for any health costs associated with the documented increases in radon and formaldehyde levels (Pigg et al. 2014), both of which are known health risks. Based on the available evidence, particularly the small change in indoor temperature that we document (and confirmed by the DOE study), we find little reason to believe that the nonenergy benefits are important and some suggestion that they are negative.

It is possible that the weatherization retrofits improve the quality of the house in ways beyond energy savings and the nonenergy benefits discussed in the previous paragraph. These quality improvements could be reflected in the property markets through higher house valuations. A prime example is new windows, which are easily visible, can improve a house’s aesthetics, and are frequently undertaken as part of retrofits by non–low-income households. However, the program’s requirement that each investment pass a cost-benefit test rules out windows in most cases, because their energy savings generally fall short of their costs. At the same time, this is an immature area and more research is necessary to understand how housing markets capitalize energy efficiency investments.

VII. Conclusion

We conducted a large-scale randomized-encouragement design experiment on a sample of 30,000 households presumptively eligible for participation in WAP in the state of Michigan. Approximately one quarter of these households were randomly assigned to a treatment group that was encouraged to apply for the program and received significant application assistance. The control households were free to apply for WAP but were not contacted or assisted in any way by our team. We also analyze corroborating evidence from a quasi-experimental analysis covering over twice as many weatherizations as well as a survey of indoor conditions at weatherized and unweatherized homes.

We document three primary findings. First, the aggressive encouragement efforts were disappointing. This encouragement increased take-up rates from less than 1% in the control group to about 6% at a cost of over $1,000 per weatherized household. Second, we find that WAP participation reduced energy consumption by 10–20% among participating households. However, the upfront cost of the energy efficiency investments are about twice the cost of the realized energy savings. Furthermore, the projected savings are more than three times the actual savings. Third, while the modest energy savings might be attributed to the rebound effect, when demand for energy end uses increases as a result of greater efficiency, the article fails to find evidence of economically or statistically significant increases in indoor temperature at weatherized homes.

Overall, the energy efficiency investments we evaluate are poor performers on average across a variety of metrics. From a household’s perspective, the annual internal rate of return that would rationalize these efficiency investments is −2.3%. The household’s perspective differs from society’s because it fails to recognize the benefits of greenhouse gas and local pollutant emissions reductions and because the retail prices for natural gas and electricity exceed their marginal costs of delivery. Accounting for these two factors, the annual social internal rate of return that would justify these investments is −7.8%, which is even less favorable. Finally, we also calculate the average cost per ton of avoided CO₂ under a range of assumptions. The most plausible estimates exceed $200/ton, which is significantly larger than the U.S. government’s estimate of the monetized benefits of avoided emissions (i.e., the social cost of carbon) of roughly $38.

This study demonstrates that the returns to common residential energy efficiency investments are negative privately and socially among low-income households in Michigan. The results are striking because Michigan’s cold winters and the likelihood that the weatherized homes were not in perfect condition suggests that it may have been reasonable to expect high returns in this setting. Regardless of one’s priors, this article underscores that it is critical to develop a body of credible evidence on the true (rather than projected) returns to energy efficiency investments in the residential and other sectors. The findings also suggest that the past several decades may have seen too much investigation into the why of the energy efficiency gap and not enough into whether there really was one.

From a policy perspective, WAP does not appear to pass a conventional cost-benefit test, although its full set of goals may not be reflected in such tests. On the broader question of optimal climate change policy, this article’s findings indicate that residential energy efficiency retrofits are unlikely to provide the least expensive carbon reductions. Future research should examine whether the real-world returns to energy efficiency investments differ so starkly from engineering projections in other settings.

Supplementary Material

An Online Appendix for this article can be found at The Quarterly Journal of Economics online. Data and code replicating tables and figures in this article can be found in Fowlie, Greenstone, and Wolfram (2018), in the Harvard Dataverse, doi:10.7910/DVN/OJFYSR.

Footnotes

References