GBA Prime subscribers have access to many articles that aren’t accessible to non-subscribers, including Martin Holladay’s weekly blog series, “Musings of an Energy Nerd.” To whet the appetite of non-subscribers, we offer a “GBA Prime Sneak Peek.” This GBA Prime blog was originally published on July 3, 2015.

A recent paper on the cost-effectiveness of weatherization work has received much more attention in the popular press than have similar studies in the past. The researchers concluded that weatherization measures performed at five nonprofit community action agencies in Michigan weren’t cost-effective. Newspaper headline writers have had a field day, trumpeting generalizations that aren’t supported by the limited data collected by researchers. (A typical headline: “Study: Home efficiency upgrades fall short, don’t pay.”)

Energy experts familiar with weatherization programs have suggested that the headline writers should have dug a little deeper into this story than they apparently did, but these measured voices have gained less attention than the “Weatherization Doesn’t Pay” headlines.

The controversial paper, “Do Energy Efficiency Investments Deliver?,” was written by three economists: one from the University of Chicago, Michael Greenstone, and two from the University of California at Berkeley, Meredith Fowlie and Catherine Wolfram. (Some of the paper’s findings were summarized by Scott Gibson in a recent GBA news story, “Blows Against Two Carbon Reduction Strategies.”)

This decades-old program helps low-income Americans

The federal low-income weatherization program — officially known as the Weatherization Assistance Program, or WAP — is now 39 years old. Sponsored by the U.S. Department of Energy, the program pays for weatherization workers to perform air sealing work, to add insulation, to seal leaky duct seams, and to replace inefficient appliances in the homes of low-income Americans. The services are provided at no charge to recipients.

The weatherization program is administered by state employees; in most states, however, the actual weatherization work is performed by employees of local nonprofit groups known as community action agencies.

One of the guidelines promoted by WAP is that weatherization measures should be selected based on cost-effectiveness. Over the past decades, many researchers have looked at whether the program is successful at achieving this cost-effectiveness goal.

For example, in a 2010 paper, “Cost-Effectiveness of Weatherization in Low-Income Urban Housing Stock,” author Jonathan L. Bradshaw reported, “Most weatherization treatments examined are profitable. Although cost-effectiveness varies significantly between cities and treatment scenarios, almost all treatments in the cities examined were NPV-positive [net-present-value-positive] over either a 7 or 15 year period. These results indicate that we would expect the present value of energy savings earned through weatherization to exceed weatherization installation costs for programmable thermostats, attic insulation, and air sealing.”

Summing up several decades of program evaluations, Fred Sissine’s 2012 paper, “DOE Weatherization Program: A Review of Funding, Performance, and Cost-Effectiveness Studies,” reported, “Over the program’s history, DOE’s Oak Ridge National Laboratory (ORNL) and the Office of Management and Budget have used process and impact evaluation research methods to assess WAP operations and estimate cost-effectiveness. Virtually all the studies conducted through 2005 showed that the program was moderately cost effective.”

Does weatherization pass the cost-benefit test?

The three economists who authored “Do Energy Efficiency Investments Deliver?” came to a different conclusion from previous researchers. In their abstract, the authors wrote, “Conventional wisdom suggests that energy efficiency (EE) policies are beneficial because they induce investments that pay for themselves and lead to emissions reductions. … The findings suggest that the upfront investment costs are about twice the actual energy savings.”

In the section of the paper headed “Conclusion,” the authors wrote, “From a policy perspective, WAP does not appear to pass a conventional cost-benefit test, although its full set of goals may not be reflected in such tests.”

Basic facts about the study are hidden

The paper by Fowlie, Greenstone, and Wolfram is poorly written and opaque. It neglects to report information of obvious interest to those concerned with weatherization issues: for example, how many weatherized houses were included in the study, the average pre-weatherization utility bills for the studied houses, and the average post-weatherization bills of weatherized houses.

I contacted one of the researchers, Meredith Fowlie, to ask how many weatherized houses were studied. “If you want to know the number of households in our experimental sample that appear in all data sources needed for the estimation of causal impacts, the total number is 616,” she answered. “The 616 number conditions on a perfect match across all of our data sources. A small share of households in our initial experimental sample could not be perfectly matched with our billing data and thus were dropped from the sample.”

Although she also wrote, “You can find these numbers in Table 3 of the Appendix,” the numbers in the table don’t match her tally. The table lists a total of 436 “experimental encouraged” households, 1,473 “other weatherized” households, and 180 “experimental control” households.

Although the number of studied weatherized houses is not clearly reported in the researcher’s paper, I have no reason to doubt Fowlie’s statement that the researchers evaluated the cost-effectiveness of weatherization measures performed at 616 houses. Because it took me so much digging to determine how many houses were studied, it’s hard to fault the journalists who latched on to the prominently featured statement at the beginning of the researchers’ report: namely, the claim that the paper “reports on the results of an experimental evaluation of the nation’s largest residential EE [energy efficiency] program conducted on a sample of more than 30,000 households.”

To repeat: the number of weatherized houses that were evaluated by the researchers was 616, not 30,000.

Fuzzy numbers

Only by careful digging does one discover, in a confusingly labeled table, the type of weatherization measures that were implemented at the studied houses. The information can be found in the Appendix, Table 1. It appears that 85% of the houses received new attic insulation; 76% received a measure called “infiltration reduction” (more accurately called “air sealing”); 44% received new wall insulation; and 34% received new furnaces.

The authors wrote, “Recipient households in our study received approximately $5,150 worth of home improvements on average, at zero out-of-pocket costs. The most common measures included furnace replacement, attic and wall insulation, and infiltration reduction.” (Note that the authors didn’t list these measures in order of frequency.)

They also wrote, “We estimate that the WAP energy efficiency investments reduce monthly energy consumption by 10-20% on average.” It’s unclear why, if the researchers are reporting an “average,” they were unable to pin down energy savings with any better precision than the wide range cited here.

The authors continued, “Although this surely provides a substantial assist to participating low-income households in the form of reduced energy bills, the upfront investment costs are about twice the realized energy savings. Further, the model-projected savings are roughly 2.5 times the actual savings.”

Needless to say, most journalists reported that the cost of the weatherization measures was greater than the anticipated energy savings.

Almost all weatherization researchers report average energy savings in weatherized houses. The method of reporting chosen by the three economists, however, is surprisingly vague. “Our estimates imply energy savings of 17.2 MMBtu [of natural gas] per year,” the authors wrote. (That estimate of per-family annual savings would be worth about $155 in Michigan.)

It’s unclear why the authors report that their “estimates imply” this level of savings; why not just tell us the actual weather-normalized savings?

Putting the data into perspective

The three researchers made broad generalizations about residential energy efficiency improvements based on limited data from one state. It is worth noting:

• The researchers only looked at the work of 18% of the low-income weatherization agencies working in Michigan. The researchers didn’t evaluate weatherization programs in any other state.
• The researchers only analyzed heating fuel savings in homes that were heated with natural gas, excluding homes heated by propane, fuel oil, and electricity. Of course, weatherization work is much more likely to be cost-effective in homes heated by propane, fuel oil, or electricity, because these fuels are generally more expensive than natural gas.
• While natural gas is currently inexpensive, it may not be inexpensive forever. Rising energy prices would improve the cost-effectiveness of the weatherization work that was studied.
• The researchers only looked at WAP. They did not evaluate any energy-efficiency work performed at the homes of middle-class or upper-class Americans. Because low-income families often live in substandard housing, weatherization work in low-income homes is often less cost-effective than similar work in homes owned by middle-class Americans. Moreover, WAP has to serve every low-income family that qualifies for the program, even when the family’s home may not be an ideal candidate for cost-effective weatherization measures — a fact that tends to lower cost-effectiveness numbers.
• The researchers’ conclusions show that they didn’t appreciate the fact that low-income weatherization programs often perform work for reasons other than energy savings; and that while the non-energy measures reduce the cost-effectiveness of the work, the measures provide important benefits to the families who are served.
• The researchers sought and obtained funding that they used to encourage low-income families to enroll in the weatherization program. This “encouragement design” may have lowered the savings compared to regular WAP work. Low-income households apply for WAP voluntarily, presumably because they think they can benefit — for example, because their homes are drafty and poorly insulated. By pushing the program on people and increasing application rates by six times, the researchers’ marketing efforts may have resulted in a group of participants who were less likely to benefit from the program than the normal flow of applications. The “encouragement design” changed the treatment group to include more households that didn’t feel as though they needed the program. Moreover, the “encouragement group” was found to have smaller homes and more elderly occupants than other weatherized homes and apparently had much lower gas use (about 25% lower than the average home in Michigan).
• As Martin Kushler, a senior fellow at ACEEE, has pointed out, “This study looked at the WAP during the ‘stimulus package’ years. … WAP was extremely stressed at the time, with tremendous pressure to push money out to the field. Job creation was at least as big a goal as energy savings, and they were functioning with a lot of new and inexperienced employees in order to handle the huge increase in funding and the deadlines to get it spent. … The stimulus package demands on the system drove up the average cost per home in the program, in an effort to ‘get the money spent.’ This would naturally tend to diminish the apparent cost-effectiveness in terms of energy savings per dollar spent.”
• The researchers apparently made their cost-effectiveness calculations without performing any weather normalization. Weather normalization is the adjustment of fuel cost numbers to account for differences between colder-than-average winters, average winters, and warmer-than-average winters; without weather normalization, it’s impossible to compare one winter’s fuel bill with the next winter’s fuel bill. Instead of weather normalization, the researchers included month-specific effects which indirectly account for weather through how the control group’s usage shifted month by month. But these month-specific effects doesn’t actually capture the weather that each customer experienced and do not adjust the usage for a typical weather year.
• When trumpeting their findings in a press release, at least one of the three authors didn’t just report the data; he used the Michigan data to make sweeping generalizations about residential energy efficiency improvements. “Energy efficiency investments hold great potential as a means to fight climate change,” Michael Greenstone declared. “However, we found that, at least in the case of residential energy efficiency investments, the projected savings overestimate the reality on the ground.”

One of Greenstone’s co-authors appears to disagree with Greenstone’s generalizations. Asked about the study, Meredith Fowlie was quoted in the Washington Post as saying, “This is one study in one state looking at one subpopulation and one type of measure. I would not feel comfortable generalizing from our study in Michigan.”

Furnace replacements aren’t cost-effective

After the paper by Fowlie, Greenstone, and Wolfram was published, I called up a prominent weatherization expert known for his data-analysis skills and extensive experience evaluating energy-efficiency programs. Because of his current employment status, he asked that I withhold his name — a request to which I reluctantly acceded. I’ll have to call him “John Doe.”

“These researchers looked at the weatherization work of just five agencies in Michigan,” John Doe began. (There are a total of 28 community action agencies that provide weatherization services in Michigan.) “WAP is a national program in 50 states,” Doe continued. “Different agencies have different program designs and different levels of training. There are variations in energy savings between agencies that can vary from 2 or 3 to 1, and there are variations in energy savings between states that can vary from 2 or 3 to 1. So this is not a representative sample. You can’t draw any conclusions about the WAP program based on this sample.”

John Doe noted that replacing an old furnace with a new furnace usually has a long payback period. When weatherization jobs include furnace replacement — as opposed to just air sealing and insulation improvements — the furnace replacement jobs tend to drag down the cost-effectiveness numbers. Of the weatherization jobs studied by these researchers, 34% included furnace replacement.

“In some cases, weatherization agencies estimate the efficiency of the existing furnace at a really low number, low enough that the program rules will permit the furnace to be replaced, because they know that the furnace is on its last legs and isn’t particularly safe,” Doe told me. “It’s a stopgap approach, but these decisions are made for the best of reasons. Some of these programs have strict criteria for when certain measures can be performed, so they end up doing furnace replacements which are really health and safety measures, even though they are categorized as an energy-efficiency measure. These programs want to fix up a house as much as they can as well as reduce energy bills. We know that furnace replacement is almost never cost-effective, especially now that natural gas is cheap, but that doesn’t mean that you shouldn’t ever replace a furnace. … Remember, these agencies are doing an amazing job, juggling multiple funding streams, trying to help poor people have better energy-efficient homes.”

Doe continued, “How do you account for the costs and benefits? Maybe we shouldn’t be replacing these furnaces. Maybe $2,000 of the $5,000 that was spent on a weatherization job was invested in things that we never really thought would pay for themselves. But a cost-effectiveness analysis doesn’t account for the value of the improved quality of life and improved safety for the people who live there. There are a wide array of benefits when we weatherize people’s homes. When there is a leaky roof, maybe they patched the roof and fixed the attic. The work sometimes includes asbestos abatement or repairs for faulty wiring. It’s not fair to say this work isn’t cost-effective.”

Thinking like economists, not energy experts

The fact that authors of the paper are all economists affected their method of analysis.

“The authors did a terrible job of analyzing the data, so we don’t really know what the savings are,” Doe told me. “They didn’t use the standard analytical methods used in the energy-savings field. They ignored the fact that this is physics, not economics. They actually don’t model energy use, but instead model the logarithm of monthly energy use — which makes an additive process (heating use adds onto baseload use in the winter — it doesn’t multiply it by some percent) into something that makes no sense and can lead to unpredictable answers.”

An expensive way to reduce carbon emissions

The three economists — Fowlie, Greenstone, and Wolfram — noted that spending on WAP is not the best way to reduce carbon emissions. That’s true, but WAP was not established to lower carbon emissions. The program was established to help low-income families who are struggling with high energy bills.

One of the papers I cited at the beginning of this article — “Cost-Effectiveness of Weatherization in Low-Income Urban Housing Stock” by Jonathan L. Bradshaw — addresses the question of carbon emissions. “Weatherization strategies aimed at energy savings, carbon savings, and cost-effectiveness may not lead to the same conclusion,” Bradshaw wrote. “Because average energy consumption, carbon intensity of energy consumed, and energy prices all vary geographically and largely independently, energy savings, carbon savings, and cost-effectiveness are not necessarily aligned. Weatherization strategies that seek to minimize residential energy use may not be the same strategies that seek to minimize residential carbon emissions. Additionally, there are different ways to consider cost-effectiveness, including net present value or by abatement cost for energy or carbon. In evaluating existing and designing new weatherization programs, it will be important for policy-makers to recognize these differences and decide the priorities of weatherization programs.”

Taking a broader view

Is the low-income weatherization program cost-effective? It’s hard to say. The weatherization work performed by some community action agencies is almost certainly cost-effective, while the work of other agencies may not be. These differences depend on climate, fuel costs, housing stock types, worker skills, and a variety of assumptions, including assumptions about future energy cost increases.

“The WAP program is running out of high users,” John Doe told me. “Average natural gas usage has dropped. A few decades ago, the average house used 1300 therms of gas a year; now it is 900 therms. There used to be a lot of houses with inefficient refrigerators and naturally aspirated furnaces. The programs are running out of that type of house. Over time, the worst of the housing stock gets knocked down, or it has been insulated by some program or another. The housing stock is getting more efficient. Furnace efficiency standards, water heater efficiency standards, improved insulation — it has all added up to significant reductions in energy use.”

While it may be harder in 2015 to show that the work performed by WAP is cost-effective than it was in 1985 or 1995, that doesn’t mean that the money invested in the program is wasted. After a house is weatherized, the house becomes more comfortable to live in, and the low-income family living in the house sees lower fuel bills. All kinds of social benefits flow from this type of work.

“These programs are typically judged by including the associated ‘non-energy benefits’ in addition to the direct energy savings,” wrote Martin Kushler. “These multiple benefits include things like the effects on comfort, health, safety (e.g., WAP typically installs smoke detectors, CO detectors, fixes wiring problems, fixes gas leaks, etc.), increased value of the improved housing stock, reduced utility-bill payment arrearages and non-payment collection costs (which saves money for all ratepayers), improved ability to remain in the dwelling and not have to be relocated, etc. Studies exist in the industry that quantify these types of variables, and when taken in aggregate, the non-energy benefits’ value can nearly equal, or even exceed, the direct energy savings value… The study [by Fowlie, Greenstone, and Wolfram] … simply ignores those other multiple benefits, and does not quantify them in the analysis. This methodological flaw pre-ordains the conclusion.”

Setting up a straw man

Kushler worries that this paper might affect funding for programs aimed to help low-income families.

Kushler wrote, “The authors conducted a study of one particular low-income program (the federal Weatherization Assistance Program, or WAP), as implemented in portions of one state (Michigan), and somehow ended up with the sweeping headline ‘Study Finds Costs of Residential Energy Efficiency Investments are Double the Benefits.’ Some of the popular press is already picking up on this theme, and the concern is that a misunderstanding (or misuse) of this study will lead to low-income families having less access to important programs that drive down their utility bills. Or worse yet, as a broad-brush attack on all types of energy efficiency programs.”

He continued, “In short, this study cherry-picked the worst possible program for comparing total costs to just direct energy savings, then set up a straw man to knock down, then tried to suggest, from an extremely limited sample of one program type, that all ‘residential energy efficiency investments’ are suspect.”

Energy savings don’t amount to much when energy is cheap

In my opinion, WAP is an excellent program. I’m proud that some of my federal tax dollars are used to perform energy-efficiency improvements in the homes of low-income families. (Unfortunately, I can’t really say the same thing about some of the other ways that the federal government spends our tax money — but that’s another story.)

According to Merrian Borgeson, a senior scientist for energy for the Natural Resources Defense Council, the study by Fowlie, Greenstone, and Wolfram has not been peer-reviewed. The three researchers might have been able to repair some of the paper’s flaws if it had undergone the usual peer-review process. Moreover, Greenstone’s attempt to generalize from the study’s limited data is indefensible.

That said, the researchers make some valid points — some of which are frequently mentioned on GBA. Here are two:

• When weatherization workers enter data into existing energy modeling programs, the software usually overpredicts future energy savings, for two reasons: the inputs are often wrong, and the software programs have flaws.
• Energy retrofit work on existing houses is not the fastest or most cost-effective way to address our looming climate change crisis.

Weatherizing old houses isn’t easy work. The most cost-effective examples of this work occur when all of the stars are aligned: when the house being worked on is poorly insulated and leaky; when there is good access to the basement and attic; when fuel costs are high; when the work is not impeded by hazards like asbestos, faulty electrical wiring, or a wet basement; when the crew is well trained and well motivated; and when the implemented measures are limited to measures with a proven track-record for cost-effectiveness.

When it comes to weatherization work, things have gotten a little tougher recently. The average house is using less energy than it used to, so there are fewer opportunities for dramatic savings. At the same time, energy — especially natural gas — is cheap. Meanwhile, labor isn’t getting any cheaper. And many older homes still have the same grab-bag of problems that have always plagued weatherization workers: water entry into basements, small access hatches, asbestos duct insulation, lead paint, knob-and-tube wiring, and cramped attics filled with long-forgotten possessions.

Fourteen years ago, the Environmental Protection Agency launched Home Performance with Energy Star, a program that hoped to lure contractors into a lucrative new field: providing energy retrofits for middle-class and upper-class Americans. As Nate Adams (among many others) discovered, it’s hard to make money doing this sort of work. (Nate Adams wrote a series of blogs on this topic for GBA; the first blog in his series was titled How an Efficiency Program Killed My Business.) Customers for this type of work are hard to find, so client-acquisition costs are high. Homeowners require lots of hand-holding. And most old houses have a lot of problems that need to be fixed before energy retrofits can start.

While these facts about weatherization work need to be acknowledged, it’s important not to throw out the baby with the bathwater. In many areas of the country, there are lots of opportunities to perform cost-effective work: for example, by replacing incandescent bulbs with CFLs or LEDs, or by swapping old single-speed swimming pool pumps for new variable-speed pumps. It still pays to seal up the large holes in our nation’s attics, and the attic insulation in many homes is still appallingly thin. We need to continue to work at addressing these problems, while simultaneously investing in a variety of other measures designed to reduce our nation’s carbon emissions.

 

GBA Prime subscribers can read a great many posted comments at the page where this article was originally published: Is Weatherization Cost-Effective? If you are a GBA Prime subscriber, that page remains the best place to post comments. Non-subscribers are invited to post comments below.