Image Credit: Office of Energy Efficiency and Renewable Energy
More Musings of an Energy Nerd
A recent paper on the cost-effectiveness of weatherization work has received much more attention in the popular press than have similar studies in the past. The researchers concluded that weatherization measures performed at five nonprofit community action agencies in Michigan weren’t cost-effective. Newspaper headline writers have had a field day, trumpeting generalizations that aren’t supported by the limited data collected by researchers. (A typical headline: “Study: Home efficiency upgrades fall short, don’t pay.”)
Energy experts familiar with weatherization programs have suggested that the headline writers should have dug a little deeper into this story than they apparently did, but these measured voices have gained less attention than the “Weatherization Doesn’t Pay” headlines.
The controversial paper, “Do Energy Efficiency Investments Deliver?,” was written by three economists: one from the University of Chicago, Michael Greenstone, and two from the University of California at Berkeley, Meredith Fowlie and Catherine Wolfram. (Some of the paper’s findings were summarized by Scott Gibson in a recent GBA news story, “Blows Against Two Carbon Reduction Strategies.”)
This decades-old program helps low-income Americans
The federal low-income weatherization program — officially known as the Weatherization Assistance Program, or WAP — is now 39 years old. Sponsored by the U.S. Department of Energy, the program pays for weatherization workers to perform air sealing work, to add insulation, to seal leaky duct seams, and to replace inefficient appliances in the homes of low-income Americans. The services are provided at no charge to recipients.
The weatherization program is administered by state employees; in most states, however, the actual weatherization work is performed by employees of local nonprofit groups known as community action agencies.
One of the guidelines promoted by WAP is that weatherization measures should be selected based on cost-effectiveness. Over the past decades, many researchers have looked at…
Weekly Newsletter
Get building science and energy efficiency advice, plus special offers, in your inbox.
This article is only available to GBA Prime Members
Sign up for a free trial and get instant access to this article as well as GBA’s complete library of premium articles and construction details.
Start Free TrialAlready a member? Log in
18 Comments
Thanks for doing the work
I've seen this study cited in various places, including by other contractors. I am glad you did the hard work of slogging through it all to see that it doesn't hold up very well to scrutiny.
Improvements to Weatherization
I worked as a Weatherization Trainer for 3-1/2 years and got to know one state's program and personnel pretty well. In my experience, there was a wide range of quality between the different offices and crews in the state. I attribute that to lack of energy accountability. Utility bills were analyzed in order to pre-qualify clients, but there was never any follow-up after the work was completed. Without post Wx utility data there is no way to find out which measures, materials and techniques were most effective. There was also no basis on which to manage the program at its primary purpose; saving energy.
The state office and individual agencies often seemed most preoccupied with following increasingly burdensome DOE regulations. Health and Safety regulations, a growth field within WAP, often used up so much of project budgets, that Wx measures had to be sacrificed.
One recent case involved mandating ASHRAE 62.2 for ventilation without providing additional funding for the measure. This was an opportunity to sharply increase energy savings with more thorough air sealing measures, but a variety of institutional impediments got in the way. Clients had better air quality but also higher energy bills.
I came to the conclusion that these problems could only be solved from the top. The Weatherization program has been shrinking since the stimulus program ended. Protecting turf has evidently become more important than program improvement.
In spite of all the problems, there are many wonderful, committed workers out in the field and thousands of grateful clients.
Response to Jim Baerg
Jim,
Thanks for sharing some of the challenges for weatherization work these days.
The recent increased focus on health and safety measures, as you point out, often reduces the cost-effectiveness of the program. But I think we can all agree that focusing on health and safety is a good thing.
You wrote, "Utility bills were analyzed in order to pre-qualify clients, but there was never any follow-up after the work was completed. Without post-weatherization utility data, there is no way to find out which measures, materials and techniques were most effective."
While it's certainly possible that the agencies you worked with did not look at post-weatherization data during the time period when you worked as a trainer, lots of researchers have looked very closely at post-weatherization utility bills. This type of research has been performed for decades, for precisely the reasons you noted: to determine "which measures, materials and techniques were most effective."
It's not as if we don't know which measures, materials, and techniques are most effective; we do. However, as you point out, different agencies apply this knowledge unevenly. Some agencies are doing a terrific job; others aren't doing as well.
I'm all for homeowners
I'm all for homeowners upgrading their homes including taking steps to reduce their energy bills but the idea that tax dollars should be used to "swap out old single- speed swimming pool pumps for new variable-speed pumps" is troubling. If you can afford a pool you can afford the pump change on your own dime.
Over the winter I worked on a number of apartments where the tenants didn't pay for the heating costs. I walked past doors wide open, windows open, and air conditioners running in sub zero weather all because the tenants didn't have any skin in the game.
Maybe a better program would require the occupants to share in the expense of the improvements since they should see savings in energy bills.
Resonse to Dan Vandermolen
Dan,
I can assure you that the low-income weatherization program does not pay to swap out swimming pool pumps. So you can put that worry to rest.
Trumpeting the findings of his recent study, Michael Greenstone announced, "we found that, at least in the case of residential energy efficiency investments, the projected savings overestimate the reality on the ground.”
This sweeping generalization did not mention the low-income weatherization program, and the generalization got a lot of press. My article tries to establish that, contrary to Greenstone's false statement, many residential energy efficiency investments are cost-effective -- both investments made by the low-income weatheriztion program, and investments made by middle-class and upper-class Americans who swap out inefficient swimming pool pumps for variable-speed pumps.
Your comments raise an issue that I did not address: whether landlords or tenants should pay for heating fuel and electricity. I agree with you that in most cases, it makes more sense for tenants, not landlords, to pay for heating fuel and electricity. I used to work for a nonprofit developer of low-income housing, and I can assure you that in the vast majority of low-income housing projects, it is the tenants, not the landlord, who pays for heating fuel and electricity.
How did this train wreck happen?
The paper by Fowlie, Greenstone, and Wolfram is something of a train wreck. So how could three educated economists from two premier universities publish a paper without (apparently) conducting a thorough literature search; claim to be writing a ground-breaking study when they were in fact visiting a well-trod field; fail to educate themselves about methods of weatherization; fail to clearly disclose essential facts for any research paper on weatherization cost-effectiveness, including the number of weatherized houses that were analyzed, pre-weatherization blower door test results, post-weatherization blower door test results, average pre-weatherization fuel bills, and average post-weatherization fuel bills; use a logarithmic method of analysis when logarithms distort the results; and report their analysis in such an opaque, poorly written paper with almost indecipherable and poorly labeled tables of data?
Absent any evidence of a different explanation, one can only conclude that, like many economists, they were overconfident of their ability to master a new field, quickly got in over their heads, and were prevented from consulting with more knowledgeable experts in the field by hubris and a desire to publish a counterintuitive paper that might make a splash.
While this explanation is possible -- and is the most charitable explanation -- politically savvy observers note that it may be naive. These political analysts assume that this paper was driven by a political agenda, with the conclusions written before the data were collected. In essence, the paper came out of the Chicago school of economic theory; to quote Wikipedia, "The Chicago school's methodology has historically produced conclusions that favor free market policies and little government intervention ... The school has been blamed for growing income inequality in the United States. Economist Brad DeLong of the University of California, Berkeley says the Chicago School has experienced an ‘intellectual collapse,’ while Nobel laureate Paul Krugman of Princeton University, says that recent comments from Chicago school economists are ‘the product of a Dark Age of macroeconomics in which hard-won knowledge has been forgotten.’ Critics have also charged that the school's belief in human rationality contributed to bubbles such as the recent financial crisis, and that the school's trust in markets to self-regulate has offered no aid to the economy in the wake of the crisis."
According to a press release announcing the publication of this paper, the research report is "a part of The E2e Project." According to the E2e website, the E2e Project's advisory board consists of five people, including John Deutch (former Director of the CIA), George Shultz (who served in the Nixon and Reagan administrations) and Cass Sunstein (husband of Samantha Powers, and creator of the “conspiracy theory” method of opposition dismissal).
One of George Shultz's memorable pronouncements came during his testimony to Congress in 1983, when he said that the Sandinista government in Nicaragua was "a cancer in our own land mass" that must be "cut out."
According to an article on Salon.com, "In 2008, while at Harvard Law School, [Cass] Sunstein co-wrote a truly pernicious paper proposing that the U.S. Government employ teams of covert agents and pseudo-‘independent’ advocates to ‘cognitively infiltrate’ online groups and websites — as well as other activist groups — which advocate views that Sunstein deems ‘false conspiracy theories’ about the Government. This would be designed to increase citizens’ faith in government officials and undermine the credibility of conspiracists."
If a Hollywood studio were preparing a script for a movie about a right-wing conspiracy to undermine the low-income weatherization program, I don't think that the studio could come up with a more suspicious trio than Deutch, Shultz, and Sustein. But their presence on the advisory board of E2e might be a coincidence.
One of this paper's authors, Catherine Wolfram, has recieved grant funding from the Smith Richardson Foundation, a foundation known to support right-wing political causes. The Smith Richardson Foundation has also contributed $6 million to the American Enterprise Institute, a cornerstone think tank of neoconservatism.
For the time being, I'm sticking with the charitable explanation of the train wreck. But I'm willing to share any information I learn that undermines the charitable explanation.
A grid aware single-speed pump controller
A retrofit grid aware single-speed pump controller would be more valuable to the taxpayers/ratepayers than an ECM drive pump. Even though the clunky single speed uses more total energy, if it can be taken off line during peak grid load periods it reduces wear & tear on the grid infrastructure, and lowers the amount of peaking generator power required during peak grid loading events, lowering the spinning reserve requirements. (It's better to turn off a million pool pumps for a few hours that to keep a fast-ramping peaker up to temp & spinning to manage peak loads.)
I'd be happy to subsidize grid-aware equipment even for the well-off even for fairly low-efficiency equipment, since it lowers the grid infrastructure cost and increases the net grid efficiency.
Thinking only in terms of the raw energy use of grid-powered items is WAY too 19th century. Unlocking the value of grid-aware and grid-operator controlled equipment on the ratepayer's side of the meter may require changes in utility regulations in most places, but that value is real, and will become standard practice in the coming decade or so.
Grid aware & responsive water heaters are currently not in Energy Star program, but will be soon enough- see lines 185-196 (p.6) of this draft:
http://www.energystar.gov/sites/default/files/specs//ENERGY%20STAR%20Water%20Heaters%20Version%203%200%20Final%20Draft.pdf
http://www.esource.com/ES-WP-18/GIWHs
There are already commercial products, not just retrofits:
http://www.greentechmedia.com/articles/read/the-water-heater-to-smart-grid-connector-goes-commercial
For decades many utilities have been retrofitting water heater controls for these purposes, and giving the homeowners breaks on the power bill for participating, but FERC Order 745 would allow even broader participation and capacity payments to aggregators & participants, if the Supreme Court overturns the D.C. District Court's ruling on that. (Could happen this summer.) It's SO cheap to just turn non-critical stuff off/down than it is to maintain & operator a generator that runs at low capacity factors that owners of generating equipment objected to competition from nega-watt aggregators, which depresses the value of (and payments for) their standby capacity, which is why they sued (and succeeded, for now) to get FERC Order 745 overturned.
With a sufficiently smart grid they really SHOULD start decommissioning some of the older low efficiency junk generating equipment that's currently only staying in the market because they are fully amortized, and can make a living on capacity payments. The current regulatory environment is slowing full implementation of Grid 2.0, but it won't forever.
Again, I'd be happy to subsidize the equipment for putting low efficiency pool pumps into a demand response program under grid operator control along with grid-aware water heaters, (even for gazillion-aires!). The cost is low relative to the benefits to all ratepayers, and the planet.
Martin, great comment [#6].
Martin, great comment [#6]. I had no idea we were so similar in views. Now that you've let the cat out of bag it might be time to put the cat back in. All the better to pose as a somewhat right wing advisor to inflame left wing thinking here at GBA. Absolutely kidding (I think...)
This non-peer-reviewed paper
This non-peer-reviewed paper certainly seems to be a pile of garbage, thanks for sifting through it Martin.
I am confused by the statement you quoted from the paper that "The findings suggest that the upfront investment costs are about twice the actual energy savings." Setting aside that the methodology was fatally flawed, isn't this a nonsense statement? The energy savings accrue over a period of years, so to have any meaning the energy savings has to have a time-frame attached to it, ie the payback period. It would be perfectly logical and acceptable to make an argument that the payback period for certain types of upgrades, like furnaces, is too long, and perhaps even that the savings from a furnace replacement wouldn't be realized over the lifetime of that furnace.The same definitely goes for windows. But if you're talking about attic insulation, wall insulation, or air sealing, don't we need to speak in terms of the payback period??
Response to Peter Rogers
Peter,
I agree that the sentence you quote is poorly written. The reference to "actual energy savings" refers not to annual energy savings but to the net present value of the energy retrofit measures. For more information on net present value, see Payback Calculations for Energy-Efficiency Improvements.
run the numbers
The paper is really simple, though it looks complex. There is a simple comparison: the cost of the energy upgrades ($4143 shown on page 9) and the present value of all the savings ($2304 shown in Panel B of Table 3). The takeaway, they say, is that the present value of the savings is far less than the investment, so damn the whole WAP program. Well, they don’t say that, but they let the various journalists in all the papers who get the press release say that.
Let’s check their numbers, by reproducing Panel B of Table 3. Easy. In any spreadsheet, simply enter the formula they give us in the bottom paragraph of page 20. Present value (PV) is calculated from the Rate, the Number of Periods (nper), and the Payment (pmt). Page 21 tells us that the annual savings are $155 (pmt). They give discount rates of 3%, 6% and 10%, and nper of 10, 16 and 20 years. If you calculate PV using these inputs, bingo, you reproduce Panel B of Table 3.
Let’s check their assumptions. How long do the measures last? I consider 25 years to be a fair estimate. If we use a 25 year term, then the numbers look a little better.
Which brings us to discount rate. This is based on the question of whether investing now in a set of energy saving measures is a good idea compared to investing it somewhere with a positive rate of return.
Suppose you have your money in a hedge fund that makes 15% annual return, and you’re considering greening your building. You compare the savings you’ll generate against the return if you kept your money in the hedge fund. No-brainer. You’d only invest in energy savings to show how much you love the environment, and keep all the money you can in your hedge fund. You’re not worried about inflation.
Suppose you hold T-bills that pay 3% interest. Well, you’re ahead of inflation, often assumed since 2010 to be about 2%. There are formulas for inflation adjustment of returns, but you know you’re ahead of inflation, so your discount rate is positive, maybe 1%.
Suppose your investments match inflation. Your discount rate is 0%. However, fuel cost inflation may not match general inflation. It may be more. And the inflation that matters in this example is fuel cost inflation. Martin shows how to calculate NPV in the comment above, and in that post he has John Straube using 8% inflation in fuel costs. Perhaps high. Perhaps not.
Suppose you are a low-income weatherization client and you have no investments, or perhaps an interest-bearing checking account that pays 0.02% interest. Your discount rate is -2% thanks to inflation, or maybe -3% or -6% or -10% depending on future fuel costs.
Suppose you pay your utility bill with a credit card you don’t pay off, or with a payday loan. Your discount rate is downhill from -20%. One thing for sure is that energy savings are a real big deal for you.
So let’s run these numbers using non-hedge fund discount rates. I hope this formats.
yrs% 10% 6% 3% 0% -3% -6% -10%
10 $952 $1,141 $1,322 $1,550 $1,840 $2,213 $2,895
16 $1,213 $1,566 $1,947 $2,480 $3,245 $4,369 $6,815
20 $1,320 $1,778 $2,306 $3,100 $4,334 $6,321 $11,199
25 $1,407 $1,981 $2,699 $3,875 $5,897 $9,550 $20,041
At -3% discount rate (decent assumption), and 20 years, we’ve paid back the $4134 investment in five years less than the 25-year service life.
I think these authors just showed us that the WAP program pays for itself. They deserve our praise. If they didn’t write such godawful gobbledegook.
This analysis and the writeup took me an hour and a half this morning. I really wish that the journalists who picked up on the paper had taken just a little time to check what they were writing. Should GreenBuildingAdvisor send out a press release on this?
Response to Bill Rose
Bill,
Thanks very much for your excellent comments.
I suppose GBA could issue a press release (although I doubt if we have a good list of e-mail addresses for potential recipients). I might ask Scott Gibson to write up a summary of this blog, and relevant comments, as a news story.
One thing's for sure, though: When the Dept. of Economics of the University of Chicago issues a press release, in concert with a "project" whose 5-member advisory board includes the former head of the CIA and George Shultz, I think it gets more press attention than a press release from GBA. We're outgunned.
Money is a resource that should be conserved too
Just because this study is flawed it doesn't mean that the Weatherization program should not be examined as to how effective it is.
In Nicholas Kristof and Sherryl WuDunn's book A Path Appears they write about how important it is to study antipoverty programs to find the most effective way to spend limited resources to have the maximum impact. It isn't just good enough to have good intentions about things like poverty or energy conservation and then throw money at those issues. An example would be instead of drilling new wells in a third world country why not use that same money to fix existing wells impacting a far greater number of people. Another example would be to ship seeds to impoverished areas instead of food giving the individual the ability to grow a surplus of food that can then be traded for other goods.
Under the Stimulus Plan the Weatherization Program approximately doubled overnight in the county where I live. It wasn't because there was a well thought out new plan on how to use the money but the money was there so lets take it. It is similar to say a lot of local governments that are buying this used military equipment not because they have a use for it but because the Federal government is giving it away to them on the cheap. Would any of them pay full sticker value if they had to? Would have our county doubled their weatherization program if it came out of the local budget?
Just like gas, water, and wood should be conserved and used efficiently so should the limited resource called money. Especially tax dollars. Readers of GBA wouldn't want to allow water to pour down the drain from a leaky or inefficient toilet. Why is allowing the scarce resource money pour down the drain any different?
Every dollar has a whole lot of energy and resources embedded in it. In order to earn a dollar I need to drive to work. I use resources to repair or build things for others to earn that dollar. I wear through shoes and clothing to earn that dollar.
Money is a scarce resource and how our tax dollars are spent should be examined and studied.
Great article, Martin &
Great article, Martin & thanks Bill for explaining the math. Occurred to me while reading about the findings that the study actually proved the opposite & Bill nailed that. You really don't have to be a conspiracy theorist to believe in industry funded "studies"; they;ve been common for decades. This one was a home run, thanks to the writers who had to figure out how to misreport the actual findings, which they did in part by making it impossible to read. Our issue is how it fight this false perception - probably the best way is to keep doing what we are all doing and improving the housing stock and methods of building.
They aren't total slouches, even if there is another agenda.
They did normalize energy use to heating degree days. They also were careful in attempting to measure rebound effect, and found it statisitically insignificant. The crux of the problem may be the gross inaccuracy &/or mis-use of the NEAT tool as a qualifying measure for the retrofits.
They pointed out on p35 (pdf pagination) that the NEAT audit tool regularly overestimated natural gas consumption of the "before" picture in this program by about 25%, and suggest that it's likely those inaccuracies are a major factor leading to underwhelming actual performance gains of low double-digits percentages, whereas 40% savings was promised. While a 40% savings from a mythical use 125% greater than reality would still be a 25% savings from reality (which is more than what was measured), there are plenty of reasons to mistrust these types of tools (as Nate Adams, Mike MacFarland and many others have pointed out.) Overpromising and underdelivering on energy efficiency measures is not a problem unique to these programs.
It's quite likely that this particular program failed it's own economic test, narrowly defined, but it's just silly to generalize from that.
Many of these programs were intentionally designed as employment programs, a deeper analysis would also take into account the direct reduction of costs such as unemployment benefits.
They also assumed zero energy price inflation, which is arguably cherry-picking, given that natural gas during that period (as now) is trading near all-time historical lows. There is no good argument that gas will remain this cheap over the lifecycle of long term weatherization or equipments upgrades, and in fact there's plenty of reason to believe otherwise. The current (and then) natural gas glut is a byproduct of high oil prices. Without the liquids fractions garnering a high market price, the gas byproduct of fracked shale would never support the drilling rates required to sustain current production levels. Now that the oil price and drilling rates have fallen, we'll likely see the gas-glut begin to dry up, and prices are likely to rise. In fact, they have no way to go BUT up, since gas production at current levels does not break even on price in the shale-fields, it only lowers cost to the developers. (There are a few dry-gas shale fields that can make money, but they are the exceptions that prove the rule.) An assumption of zero gas price inflation is only credible to those drinking the frack-water.
Also, requiring ASHRAE 62.2 ventilation rates would undo a good chunk of the energy savings, even if executed with heat recovery ventilation (at an even higher up front cost and lousier NPV.)
"Do Energy Efficiency Investments Deliver?" is the first line of the title, but the more appropriate question might be "Do Energy Efficiency Programs Deliver?". As in the many analyses preceding this one, it's a mixed bag. The "right" efficiency investments clearly deliver, but it's also clear that efficiency PROGRAMS don't have a great track record of always finding and hitting the sweet spot.
Response to Dan Vandermolen (Comment #13)
Dan,
I agree that it makes sense to analyze whether existing weatherization programs are cost-effective. Fortunately, WAP has been doing that for decades. Politicians may differ concerning whether investing in a program to lower the energy bills of low-income families is a good way to spend tax money, but we have a pretty good idea of what the effects of this program are.
I also agree that the boom-and-bust cycle of weatherization funding is nuts. The program is much better served when federal funding is fairly consistent from year to year. I wrote about this problem in an earlier article titled, "Weatherization Funding Has Been Slashed." In that article, I wrote, "For the past 40 years, ever since the oil price shocks of the 1970s, our country has lacked a coherent energy policy. While the governments of Sweden, Denmark, and Germany have made consistent investments that bring them closer to achieving their countries' national energy goals, the U.S. has been lurching from one misguided incentive to the next, with a boom-and-bust cycle that creates and destroys industries with wanton abandon."
This article is now easier to share
In order to make it easier for GBA readers to share this story with friends or colleagues, GBA has published the article on the free side of the GBA paywall.
Here is the link to the free article: GBA Prime Sneak Peak: Is Weatherization Cost-Effective?
Use of logarithms
The use of logs in the report disturbed me. I looked back on the evaluation from the Illinois weatherization Program Year 2009 (not a great year, but not a bad year) to see the distribution of energy savings by house. The data are better represented as a normal distribution rather than as a lognormal distribution. I think the authors made a wrong choice. Readers who want to see more weatherization evaluation reports and data should pester DOE to issue the most recent national evaluation.
.
Log in or become a member to post a comment.
Sign up Log in