Steven Nadel is the executive director of the American Council for an Energy-Efficient Economy. This post originally appeared on the ACEEE blog. GBA posted a news story about the original study in January.

ACEEE is a strong supporter of analyzing energy efficiency programs in order to see what they have accomplished and to learn lessons so we can do even better. It was thus with interest that we reviewed How Much Energy Do Building Energy Codes Really Save? Evidence from California by Arik Levinson.

In this paper Levinson conducts several analyses and concludes that “there is no evidence that homes constructed since California instituted its building energy codes use less electricity today than homes built before the codes came into effect.”

On the surface his conclusions about the efficacy of building codes are very different from other recent analyses, so we took a deeper look.

Building energy codes in the United Sates primarily address energy used for space heating and air conditioning, with some impact on water heating energy use. In addition, commercial building codes also address lighting in a substantial way. Therefore, any analysis of the effect of codes needs to look at energy consumption for these end uses.

Energy use in California

Levinson is examining California, so we should start by looking at how California heats and cools its buildings and water. The California Energy Commission had a consultant prepare a report looking at residential appliance saturations in 2009. They found that 93 percent of California homes are heated with gas and only 5 percent are heated with electricity. Likewise they found that 87 percent of homes have gas water heating and only 7 percent use electricity for water heating. Central air conditioners are used in 49 percent of homes, with an additional 15 percent using room air conditioners.

California added a limited lighting provision to its 2008 residential code, but the savings are too recent and too small to show up in a long-term analysis.

Based on this information, an analysis of California residential building codes should concentrate on natural gas use and only secondarily on electricity. Unfortunately, this is the opposite of what Levinson does. Most of his analysis is on electricity use, and he generally excludes homes with electric space or water heat. He does so because the saturation of electric space heat has been changing over time, and according to Levinson, therefore doesn’t fit well into his time series analysis. Thus his analysis looks for the effect of building codes on air conditioning energy use.

California’s mild climate is a factor

The fact is, though, that California has a mild climate, and, according to a recent analysis by the Energy Information Administration (EIA), only 4 percent of California home energy use is for air conditioning. Using EIA figures we can go on to calculate that air conditioning represents about 10 percent of home electricity (as opposed to energy) use. If we make a ballpark estimate that codes reduce air conditioning electricity use by 30 percent, Levinson is looking for 3 percent savings in the data.

Furthermore, a majority of the savings in air conditioning energy use is probably due to air conditioner efficiency standards that also apply to replacement equipment in existing homes. Since Levinson is comparing new and existing homes, of the 3 percent savings mentioned above, perhaps 2 percent are also being achieved in his comparison group of existing homes.

Thus, he’s looking for a 1 percent savings effect in new homes. It’s very difficult to find such a small effect in a statistical analysis; with an effect that small, it would be more surprising to see such savings show up in a statistical analysis than if the savings did not show up.

In addition, it should be noted that the rising saturation of electronic gadgets in U.S. homes may be affecting new home energy use, a factor Levinson does not examine but that perhaps explains some of the increasing energy use he found in new homes.

Study should focus on natural gas

A good analysis of the impact of California’s building codes should focus on natural gas use. Levinson does do one analysis of natural gas use, finding that homes built since California’s building energy code began use less natural gas than earlier homes.

However he then dismisses this finding since the trend started before the building codes took effect. He presents no evidence that prior trends would have continued, and therefore his claim that building codes had no effect is speculation.

Finally, if the objective is to examine all building codes and not just some codes, then it’s also important to look at commercial buildings. According to an analysis by Pacific Northwest National Laboratory, nationwide about 69 percent of building code savings in 2012 were in the commercial sector and only 31 percent in the residential sector.

More than 80 percent of the commercial savings were in electricity.

Bottom line: Levinson is on the wrong path if he’s trying to see the impacts of building codes. If he wants to see the forest — the impacts of codes or their absence — he needs to understand the trees. He needs to look where the impacts are supposed to be, and in California this means residential natural gas and commercial building electricity use.