Stove manufacturer documents scores of errors in NESCAUM report

NESCAUM responds to Morrissey as he issues Part 2 of his review

Tom Morrissey, owner of Woodstock Soapstone, a wood stove manufacturer, has written a blistering critique of a report that claimed the EPA wood stove certification process is “dysfunctional.”  

Tom Morrissey at his factory 
in West Lebanon, NH

The review by this independent manufacturer further ratchets up a conflict that has engulfed a section of EPA’s Office of Air and Radiation.  The original report, “Assessment of EPA’s Residential Wood Heater Certification Program" by the Northeast States for Coordinated Air Use Management (NESCAUM), issued in March 2021, was based on a review of all test lab certification documents that had been sent to EPA to certify stoves to EPA’s regulatory standards.  The review was conducted jointly by the Alaska Department of Conservation (ADEC) and NESCAUM, with funding from the New York State Energy Research and Development Authority (NYSERDA). 

(June update: Tom Morrissey released part 2 of his review on June 15 and three days later, NESCAUM released their rebuttal to Morrissey's initial review.  NESCAUM appears to be standing behind its entire report and did not find any merit in Morrissey's critique.  On page 21, NESCAUM also addressed editorial comments in AGH's blog. Part 2 of Morrissey's critique mainly focuses on burn rate requirements adopted by ADEC as a deficiency, but not used by NYSERDA's IDC test method.  Morrissey also describes why the IDC will likely result in lower efficiencies because BTUs in the tail of the burn are not included and questions if the EPA should ever approve a test method where underlying calculations are not public.  AGH agrees with a substantial amount of the content in both the NESCAUM report and Morrissey's rejoinders and believe it's important for EPA officials, states and experts to consider both.)

NESCAUM and ADEC accused the EPA of running a dysfunctional certification program that “is easily manipulated by manufacturers and testing laboratories” and “EPA has done little to no oversight and enforcement.” Morrissey counters that NESCAUM and ADEC also have produced a dysfunctional assessment of the EPA program that is full of “error, bias and conflict of interest.”

We have excerpted key parts of Morrissey’s review below.  The full review, with scores of photos, can be downloaded here.  The Alliance for Green Heat believes it is important to consider both the NESCAUM report and rejoinders from Morrissey and another one from veteran lab technician Ben Myren together.  We believe there is substantial truth in all three of these documents.  Our initial response to the NESCAUM report was that it “overstated” its case, and AGH welcomes expert rejoinders that bring some balance to the conversation.  However, AGH still believes that EPA should take the NESCAUM report seriously because it is clear that the EPA certification and enforcement programs need significant improvement, and that these have been underfunded, understaffed and overlooked within the EPA.  

AGH also agrees that much of ADEC’s and NESCAUM’s review was hastily done without sufficient fact-checking.  The report carries an unfounded degree of authority that is now reverberating throughout the wood stove industry and its state and federal regulators, and the wider public is less likely to see these rejoinders.  NESCAUM and ADEC now have the opportunity to respond and explain some of the allegations in Morrissey's review or continue to stand behind that data that led to their conclusions. Numerous individuals have approached northeastern state government agencies who are members of NESCAUM, urging them to distance themselves from the report and not let their agency names be used on such reports in the future without more due diligence.

In early April, in response to the NESCAUM report, the EPA announced they were conducting an in-depth review of the certification process.  They placed most of the blame on the test labs and third party certifiers.  Morrissey’s review may cause the EPA to rethink its strategy and not put undue trust in the finding of the NESCAUM report.  Morrissey’s review is significant because he is not a member of the wood stove industry association and has often taken sides for and against industry interests – and the EPA’s. This is part 1 of his review and he expects to complete the second part in June.

AGH is concerned that the NESCAUM report, media coverage of it, and Morrissey and Myren’s rejoinder put so much focus on the inherent problems with wood stoves and the pollution they can cause, that the benefits of pellet stoves get overlooked.  Virtually nowhere is there recognition of the technology and reliability of pellet stoves and boilers as a far cleaner, renewable energy technology that can help households and businesses get off fossil heating fuel.  

Excerpts:

REVIEW (PART 1)

of

“ASSESSMENT OF EPA’s RESIDENTIAL WOOD HEATER CERTIFICATION PROGRAM” Written by NESCAUM, March 2021 

Page 1: In March 2021, the Northeast States for Coordinated Air Use Management (NESCAUM) published a document entitled “Assessment of EPA’s Residential Wood Heater Program (“Assessment”). The “Assessment” is the result of a review conducted by NESCAUM “in collaboration with the Alaska Department of Environmental Conservation” (ADEC). 

The “Assessment” is intended to influence “policymakers” by claiming 1) that the EPA Certification Program is dysfunctional and a systemic failure, 2) that there are a significant number of discrepancies and omissions in test reports submitted to EPA for approval, 3) that EPA has failed to conduct compliance audits, and 4) that the NESCAUM and ADEC could do a better job than EPA in, a) deter- mining which stoves are in fact the cleanest burning and, b) developing a test method for certifying wood burning appliances. NESCAUM has provided scant data to back up these major claims, and some of the data that it does present is riddled with discrepancies, omissions, bias errors, and conflict of interest, as detailed below. Bias is evident everywhere; in tone and use of language, in lack of transparency, in the selection of subjective criteria to attempt to discredit test methods and results, and in its attempt to advance NESCAUM’s own agenda4 with its “policy recommendations.” The bias is so pervasive that it undermines much of the “Assessment.” 

Page 3: In order to assess NESCAUM/ADEC’s data analysis and conclusions, we need to first look care- fully at its data collection methods and ask whether the underlying data is complete, credible, and unbiased. The main focus of this Part One of A Review of the “Assessment” is on how data was collected and tallied on “Summary Review Sheets” by ADEC. 

On the following pages I raise concerns about quality control, bias, and conflict of interest in NESCAUM/ADEC’s acquisition of data. It is clear that NESCAUM/ADEC reviewers lacked objectivity in assessing information, particularly with regards to the ASTM E-3053 method, and they reviewed individual test reports (knowingly or not) with the intent to discredit the ASTM E-3053 and advance their own interest in promoting the IDCTM method. 

Page 4: All ADEC data sheets that I have reviewed are undated and unsigned. Most have few, if any comments.  Many have unfilled spreadsheet boxes (data not collected). All of the ADEC reports of stoves made by Woodstock Soapstone Company have serious omissions, errors of fact, misreporting, and untrue statements. Of six Woodstock Soapstone Company models approved to the EPA 2020 Standards, two models were missing entirely, and one model was reviewed twice, on separate data sheets that were inconsistent and did not match (i.e., different reviewers looking at the same data, or the same reviewer on different dates looking at the same data). The fact that ADEC reviewed the same data twice, and the two completed spreadsheets are markedly different, speaks to the concern (also noted in footnote #2) about quality control. 

Page 5: On the pair of summary sheets where NESCAUM/ADEC inadvertently reviewed the same test report twice, there were 25 discrepancies between the two reports, including errors of transcription, op- posing claims that data was or was not reported, rounding errors, conflicting or inconsistent “flags” and numeric/arithmetic errors. This is not reassuring in terms of NESCAUM’s claimed consistency in generating the summary results, and raises the issue of whether NESCAUM’s own consistency and repeatability should be the subject of an audit. 

These two ADEC Summary Reports are reproduced on page 6, and an explanation of most of the errors on page 7. For simplicity sake, I refer to the report that is captioned Model 210a (but really Model 210) as Report A, and the Report that was (correctly) reviewing Model 210 as Report B. Both reports were posted and properly labeled on the Woodstock Soapstone website. But that’s not the point; these two reviews of the same report should produce similar, if not identical results, but they did not. 

These two Summary Sheets, which review the same test report,8 disclose obvious problems in the research and reporting methods employed by NESCAUM/ADEC, and the ability/willingness of NESCAUM/ADEC to impose meaningful quality controls on their inquiry. As noted early in this review (see footnote 2), there is little, if any, evidence of NESCAUM/ADEC cross-checking or vetting of the reviews or data in the “Assessment”. The task of auditing the “Assessment” and validating its so-called “data” and its various claims will now, probably, fall squarely on EPA. 

This is the central irony of this situation; NESCAUM’s own data and reporting is guilty of the same failures it attributes to EPA, namely failures in transparency, documentation, and auditing its own work product for consistency, impartiality, and accurateness. The EPA will now become responsible for cleaning up the NESCAUM mess. 

Page 8:  For the purpose of this initial review, I will focus mainly on stoves made by Woodstock Soap- stone Company. Next, I will examine the ADEC Summary Sheet for our Model 202/204. This is a “plain vanilla” Summary Sheet, compared to Model 210, on pages 6 and 7, above. 

ADEC encourages manufacturers to “review their certification test report summaries and submit corrections, and that any substantiated errors or corrections will be applied to the summary sheet.” So, I’ll just make the corrections here. On our Model 202/204, the initial ADEC summary sheet (see next page, LEFT COLUMN) makes the following errors (WHICH CUMULATIVELY TOTAL 13 FLAGS). I intend to address THREE ADDITIONAL RED FLAGS (related to Documentation of 1) run appropriateness, 2) run Validity, and 3) run anomalies on Part 2 of this Review. 

What is fascinating is that between early April 2021, when I downloaded the original Summary Review Sheet, and today (mid-May, 2021), ADEC performed an additional review and corrected some of its original errors, and made some new errors. Here are comments on the initial ADEC Summary Sheet. Comments on the revised Summary Sheet are on the next page. 

Page 10: According to the ADEC “PROCESS”, I am supposed to address all of these “issues” by

One of the many photos that
appear to provide evidence
contradicting ADEC "flags"

discussing them with ADEC, and maybe submitting modified or reformed test reports. Then ADEC will makes changes as it deems appropriate. Or not. 

Page 14: The “Assessment” claims that “90% of the stoves tested using ASTM E-3053 used debarked wood or failed to provide information about whether there was bark on the fuel.” (Assessment page 38) The “Assessment further claims that “61% of the stoves tested with ASTM E-3053 used squared wood for more than 50% of the pieces” (Assessment page 33), including the stove immediately above (Model 205) and the fuel for Model 202/204, pictured on pp 10-11. 

Based on my review, I cannot believe either of these claims. If the basic data underlying the “Assessment” is defective, then its claims of numerous deficiencies in testing and reporting, and its criticisms of ASTM E-3053 are suspect, because they are based on bad data. 

Page 19: I hope to have Part 2 in mid-June. There is a lot more to unpack in the “Assessment,” including: 

* The overall “Assessment” review strategy, and whether it is a credible basis for proceeding to the conclusions that the “Assessment” tries to come to. This strategy is basically to make a list of each and every requirement imposed by the NSPS, and then see if each and every item on the list can be identified in test reports, no matter how obscure or irrelevant the requirement might be. Otherwise, deficiencies are claimed by NESCAUM/ADEC without any apparent oversight or review, or any basis in fact. 

* The nexus of firebox size and calculation, loading direction, fuel length, and loading density. These are the second set of elements that the “Assessment” uses to criticize ASTM E-3053. 

* I’ll review this sentence, and how it has spawned innumerable “flags” (i.e., claims of violations of the NSPS) in the “Assessment’s flag-collection effort."

Alaska building list of properly certified wood and pellet stoves

Lab certification test are designed to
burn as cleanly as possible and all
emission results must now be publicly
disclosed as of 2015.

Alaska Finds widespread deficiencies in EPA wood stove certification process 

Officials in Alaska are in the process of compiling lists of wood stoves and boilers that have met all the requirements of EPA certification and emit fewer particulates during the first hour of the test burn. Alaska regulations requires new wood heating appliances installed in the Fairbanks nonattainment area to meet additional regulatory requirements beyond obtaining a federal U.S. EPA certification. The regulations went into effect January 8, 2020 and Alaska has been working to implement those regulations since then.  The new regulatory requirements involve reviewing certification test reports for deficiencies and collected data regarding additional PM levels and then approve specific models of wood stoves and pellet stoves by updating their approved device list.

The Alaska Department of Environmental Conservation (ADEC) began this process as part of their efforts to address excess wood smoke pollution that contributed to Fairbanks becoming a non-attainment area.  The State’s Implementation Plan (SIP) to improve air quality began including measures years ago to crack down on dirtier wood burning appliances, such as outdoor wood boilers.

The current effort involves reviewing every stove’s certification paperwork to ensure that it includes all the elements that EPA regulations require, such as average CO, the manufacturer’s written instructions to the lab, firebox dimensions, efficiency calculations, burn rate calculations, raw data sheets, documentation of run anomalies, etc. etc. 

 

If you are not familiar with this brewing controversial initiative, you are not alone.  The first time most people outside a small group of manufacturers and regulators heard about this was a month ago, Sept. 18,  when a group of states weighed in on the litigation between HPBA and the EPA.  Their brief mentioned that the Alaska Department of Environmental Conservation was conducting a systematic review of wood-burning devices that have been certified to be compliant with EPA standards. The Department found that 59% of the certifications had inaccurate certification data, and that the EPA must have a way to check on manufacturers through audits. 

The start-up phase of a stove in the lab
should be consistent with how it is
described in the owner's manual.

The lists being developed by Alaska officials only pertain to what can and can’t be sold in the Fairbanks Nonattainment Area, a very small market.  But regulators, manufacturers and test labs we spoke to all say that this is having major repercussions.  For the EPA, it’s a wakeup call that they have not been sufficiently reviewing test lab reports before certifying stoves.  Test labs are under more scrutiny for various practices and are already  being asked by manufacturers to help ensure their tests come in under 6 grams during the first hour.  And scores of manufacturers are scurrying to provide additional information to Alaska and showing them the details that were in their test reports that Alaska officials missed.  

 

One of the main reasons that everyone is paying attention is that most of the people AGH spoke to agree that other states and change out programs could adopt the Alaska lists instead of using the full EPA list of certified heaters.  If more change-out programs, or even states adopt these stricter requirements, the efforts of a small city in Alaska will have greater national ramifications.  Some managers of change out programs that AGH spoke to say they interested to explore ways to identify cleaner cord wood stoves and are uncertain whether the EPA’s reduction from a de facto 4.5 to 2 grams an hour actually resulted in cleaner cord wood stoves. Incentive and change out programs often have adopted stricter efficiency and/or emission requirements and this may represent the next vehicle for those programs to guide how taxpayer dollars should best be used.

 

The last time state regulators made changes that went on to have national implications may have been in 1995 when Washington State adopted a 4.5 gram an hour state standard, when the EPA allowed up to 7.5 grams an hour.  The 4.5 gram an hour limit soon became a de facto national standard and 20 years later, HPBA insisted it was still the lowest that the EPA should go for the 2015 NSPS.  

Non-confidential portions of emission
test reports made the Alaska investigation
possible for the first time.

Aside from developing new lists of stoves that met all the requirements of the NSPS and emitted less than 6 grams for the first hour of all their certification test runs, Alaska is providing a wealth of information to the EPA, NESCAUM and others who are already in the process of developing a new federal reference test method for certifying wood stoves.  Some regulators the Alliance for Green Heat (AGH) spoke to now acknowledge that they see the current system as “broken” and that no one knew it was so broken.  On November 16, Alaska will be making its data public, showing which requirements in the stove certification process are most commonly ignored or overlooked and which ones are complied with.  

 

The data to be released on Nov. 16 includes a two-page data sheet on individual stoves, showing any deficiencies in their test lab reports.   Each manufacturer will have had up to 2 months to review their own sheets and provide corrections to ADEC prior to their public release.  The initial sheets prepared by ADEC have numerous mistakes according to several manufactures AGH spoke to and include many of the data points that ADEC initially said were not in their test report.  Test labs have been helping manufacturers find relevant data in their test reports, and ADEC will continue making corrections before and after Nov. 16.

 

It’s not yet clear if any test lab reports may be so deficient that the EPA could revoke their certification or require that the stove be tested again.  EPA officials are just beginning to grapple with how widespread the problems may be and what they can do moving forward to help fix problems that should have been in plain sight for so long. The EPA has identified and tried to correct some testing deficiencies in the past.  An AGH blog in July 2019 covered an EPA memo asking labs to correct lapses in reports on stoves tested with the ASTM E3053 test method.  This incident may have contributed to greater scrutiny by states including Alaska’s far more in-depth investigation.

An ADEC official taking
air quality readings on top
of a school.

The EPA certifies stoves based on the weighted average of the entire burn but requires labs to also report the amount of PM in the first hour.  Like efficiency, its data that must be collected and reported by the lab to the EPA, but there is no regulatory limit.  The Alaska initiative is making everyone ask whether the first hour of emissions may be an equally important indicator of a stoves cleanliness than the entire burn cycle.  

 

Currently, the way EPA approved test labs test stoves is by using the standard Method 28 or a variation of it. Lab technicians load stoves with an amount of wood based on the size of the firebox and let the fire go until all the wood is burned, which usually takes anywhere from 4 – 9 hours but can be longer.  During the last several hours of the burn, known as the “tail”, there is virtually no particulate matter being released, but those hours are still averaged into the overall calculation, with much be less than 2 grams an hour with cribs, or 2.5 grams an hour with cordwood. 

 

AGH reviewed scores of test reports and found that single PM reading from the first hour could be as high as 20 grams an hour and the average of all the first hour burns could be as high as 10 grams an hour, but it would be less than 2 grams when the cleaner parts of the burn and especially the tail end of the burn was included.  For Alaska, if any single run went over 6 grams, it was rejected.  Since start-up is the dirtiest part of the burn, identifying stoves that have cleaner start-up may help airsheds improve air quality.

 

Test labs that AGH spoke to noted that this will likely have the result of disqualifying a greater proportion of larger fireboxes.  The average firebox size is 2.2 cubic feet and an initial small sample of stoves with average first hour emissions over 6 grams was 3 grams an hour.  One test lab also said that this could disadvantage catalytic stoves that have no secondary combustion during start up prior to engaging the catalyst, which often occurs 20 – 30 minutes after lighting the fire.  Hybrid stoves, however, that use both air tubes and a catalyst are likely to have cleaner start-up, according to test labs.

 

Some manufacturers are angry that a state is using a brand-new emission metric – first hour emissions – that they could have designed for, if they knew it would be used in some markets to regulate stoves.  Now that it has been flagged, manufacturers certifying stoves going forward can try to meet that – or at least urge the lab to build  the type of start-up fire that will come in under 6 grams.  One of the primary goals of a lab is to familiarize themselves with the stove being tested so they can “optimize stove operations during certification testing.” Some regulators fear that this will just become another factor that manufacturers and labs will use to “game the system.”

 

The Alaska initiative will have a far greater impact on wood stoves than pellet stoves.  Extremely few pellet stoves emit more than 6 grams in their first hour and the testing regimen is more straightforward.  Ultimately, only a dozen or fewer pellet stoves may be disqualified by their review, out of the 98 models that are currently certified. Of the 144 currently certified wood stoves, up to a quarter to a third could be impacted.   New outdoor wood boilers are not allowed in Fairbanks already.  They will be reviewing pellet boilers for compliance.

An inversion in Fairbanks that traps
wood smoke close to the ground.,

Next steps

 

As of Nov. 16, Alaska will publish its review on virtually every EPA certified stoves.  On December 1, only those stoves that ADEC found had complete test reports will remain on the approved device list and be allowed to be sold in Fairbanks.  If a manufacturer needs more time to address potential report deficiencies, they can contact ADEC and if they commit to working to correct the deficiencies, their device may remain on the approved device list.  The first hour emission of 6 grams on each test run requirement, went into effect on September 1, 2020, and those devices have already been removed from the approved device list. 

 

Other jurisdictions, incentive and change-out programs will likely begin assessing whether the Alaska list represents better stoves for public funding.  Consumers who really care about a cleaner stove could also check that list before buying a stove.  For now, there are more questions than answers but the bottom line for everyone in the industry and the wider renewable heating community is that this is a story to watch.

Related stories

Veteran lab technician challenges Alaska's wood stove criteria (March 2021)

Alaska releases deficiency details on wood and pellet stove test reports (Nov. 2020)