Stove manufacturer documents scores of errors in NESCAUM report

NESCAUM responds to Morrissey as he issues Part 2 of his review

Tom Morrissey, owner of Woodstock Soapstone, a wood stove manufacturer, has written a blistering critique of a report that claimed the EPA wood stove certification process is “dysfunctional.”  

Tom Morrissey at his factory 
in West Lebanon, NH

The review by this independent manufacturer further ratchets up a conflict that has engulfed a section of EPA’s Office of Air and Radiation.  The original report, “Assessment of EPA’s Residential Wood Heater Certification Program" by the Northeast States for Coordinated Air Use Management (NESCAUM), issued in March 2021, was based on a review of all test lab certification documents that had been sent to EPA to certify stoves to EPA’s regulatory standards.  The review was conducted jointly by the Alaska Department of Conservation (ADEC) and NESCAUM, with funding from the New York State Energy Research and Development Authority (NYSERDA). 

(June update: Tom Morrissey released part 2 of his review on June 15 and three days later, NESCAUM released their rebuttal to Morrissey's initial review.  NESCAUM appears to be standing behind its entire report and did not find any merit in Morrissey's critique.  On page 21, NESCAUM also addressed editorial comments in AGH's blog. Part 2 of Morrissey's critique mainly focuses on burn rate requirements adopted by ADEC as a deficiency, but not used by NYSERDA's IDC test method.  Morrissey also describes why the IDC will likely result in lower efficiencies because BTUs in the tail of the burn are not included and questions if the EPA should ever approve a test method where underlying calculations are not public.  AGH agrees with a substantial amount of the content in both the NESCAUM report and Morrissey's rejoinders and believe it's important for EPA officials, states and experts to consider both.)

NESCAUM and ADEC accused the EPA of running a dysfunctional certification program that “is easily manipulated by manufacturers and testing laboratories” and “EPA has done little to no oversight and enforcement.” Morrissey counters that NESCAUM and ADEC also have produced a dysfunctional assessment of the EPA program that is full of “error, bias and conflict of interest.”

We have excerpted key parts of Morrissey’s review below.  The full review, with scores of photos, can be downloaded here.  The Alliance for Green Heat believes it is important to consider both the NESCAUM report and rejoinders from Morrissey and another one from veteran lab technician Ben Myren together.  We believe there is substantial truth in all three of these documents.  Our initial response to the NESCAUM report was that it “overstated” its case, and AGH welcomes expert rejoinders that bring some balance to the conversation.  However, AGH still believes that EPA should take the NESCAUM report seriously because it is clear that the EPA certification and enforcement programs need significant improvement, and that these have been underfunded, understaffed and overlooked within the EPA.  

AGH also agrees that much of ADEC’s and NESCAUM’s review was hastily done without sufficient fact-checking.  The report carries an unfounded degree of authority that is now reverberating throughout the wood stove industry and its state and federal regulators, and the wider public is less likely to see these rejoinders.  NESCAUM and ADEC now have the opportunity to respond and explain some of the allegations in Morrissey's review or continue to stand behind that data that led to their conclusions. Numerous individuals have approached northeastern state government agencies who are members of NESCAUM, urging them to distance themselves from the report and not let their agency names be used on such reports in the future without more due diligence.

In early April, in response to the NESCAUM report, the EPA announced they were conducting an in-depth review of the certification process.  They placed most of the blame on the test labs and third party certifiers.  Morrissey’s review may cause the EPA to rethink its strategy and not put undue trust in the finding of the NESCAUM report.  Morrissey’s review is significant because he is not a member of the wood stove industry association and has often taken sides for and against industry interests – and the EPA’s. This is part 1 of his review and he expects to complete the second part in June.

AGH is concerned that the NESCAUM report, media coverage of it, and Morrissey and Myren’s rejoinder put so much focus on the inherent problems with wood stoves and the pollution they can cause, that the benefits of pellet stoves get overlooked.  Virtually nowhere is there recognition of the technology and reliability of pellet stoves and boilers as a far cleaner, renewable energy technology that can help households and businesses get off fossil heating fuel.  

Excerpts:

REVIEW (PART 1)

of

“ASSESSMENT OF EPA’s RESIDENTIAL WOOD HEATER CERTIFICATION PROGRAM” Written by NESCAUM, March 2021 

Page 1: In March 2021, the Northeast States for Coordinated Air Use Management (NESCAUM) published a document entitled “Assessment of EPA’s Residential Wood Heater Program (“Assessment”). The “Assessment” is the result of a review conducted by NESCAUM “in collaboration with the Alaska Department of Environmental Conservation” (ADEC). 

The “Assessment” is intended to influence “policymakers” by claiming 1) that the EPA Certification Program is dysfunctional and a systemic failure, 2) that there are a significant number of discrepancies and omissions in test reports submitted to EPA for approval, 3) that EPA has failed to conduct compliance audits, and 4) that the NESCAUM and ADEC could do a better job than EPA in, a) deter- mining which stoves are in fact the cleanest burning and, b) developing a test method for certifying wood burning appliances. NESCAUM has provided scant data to back up these major claims, and some of the data that it does present is riddled with discrepancies, omissions, bias errors, and conflict of interest, as detailed below. Bias is evident everywhere; in tone and use of language, in lack of transparency, in the selection of subjective criteria to attempt to discredit test methods and results, and in its attempt to advance NESCAUM’s own agenda4 with its “policy recommendations.” The bias is so pervasive that it undermines much of the “Assessment.” 

Page 3: In order to assess NESCAUM/ADEC’s data analysis and conclusions, we need to first look care- fully at its data collection methods and ask whether the underlying data is complete, credible, and unbiased. The main focus of this Part One of A Review of the “Assessment” is on how data was collected and tallied on “Summary Review Sheets” by ADEC. 

On the following pages I raise concerns about quality control, bias, and conflict of interest in NESCAUM/ADEC’s acquisition of data. It is clear that NESCAUM/ADEC reviewers lacked objectivity in assessing information, particularly with regards to the ASTM E-3053 method, and they reviewed individual test reports (knowingly or not) with the intent to discredit the ASTM E-3053 and advance their own interest in promoting the IDCTM method. 

Page 4: All ADEC data sheets that I have reviewed are undated and unsigned. Most have few, if any comments.  Many have unfilled spreadsheet boxes (data not collected). All of the ADEC reports of stoves made by Woodstock Soapstone Company have serious omissions, errors of fact, misreporting, and untrue statements. Of six Woodstock Soapstone Company models approved to the EPA 2020 Standards, two models were missing entirely, and one model was reviewed twice, on separate data sheets that were inconsistent and did not match (i.e., different reviewers looking at the same data, or the same reviewer on different dates looking at the same data). The fact that ADEC reviewed the same data twice, and the two completed spreadsheets are markedly different, speaks to the concern (also noted in footnote #2) about quality control. 

Page 5: On the pair of summary sheets where NESCAUM/ADEC inadvertently reviewed the same test report twice, there were 25 discrepancies between the two reports, including errors of transcription, op- posing claims that data was or was not reported, rounding errors, conflicting or inconsistent “flags” and numeric/arithmetic errors. This is not reassuring in terms of NESCAUM’s claimed consistency in generating the summary results, and raises the issue of whether NESCAUM’s own consistency and repeatability should be the subject of an audit. 

These two ADEC Summary Reports are reproduced on page 6, and an explanation of most of the errors on page 7. For simplicity sake, I refer to the report that is captioned Model 210a (but really Model 210) as Report A, and the Report that was (correctly) reviewing Model 210 as Report B. Both reports were posted and properly labeled on the Woodstock Soapstone website. But that’s not the point; these two reviews of the same report should produce similar, if not identical results, but they did not. 

These two Summary Sheets, which review the same test report,8 disclose obvious problems in the research and reporting methods employed by NESCAUM/ADEC, and the ability/willingness of NESCAUM/ADEC to impose meaningful quality controls on their inquiry. As noted early in this review (see footnote 2), there is little, if any, evidence of NESCAUM/ADEC cross-checking or vetting of the reviews or data in the “Assessment”. The task of auditing the “Assessment” and validating its so-called “data” and its various claims will now, probably, fall squarely on EPA. 

This is the central irony of this situation; NESCAUM’s own data and reporting is guilty of the same failures it attributes to EPA, namely failures in transparency, documentation, and auditing its own work product for consistency, impartiality, and accurateness. The EPA will now become responsible for cleaning up the NESCAUM mess. 

Page 8:  For the purpose of this initial review, I will focus mainly on stoves made by Woodstock Soap- stone Company. Next, I will examine the ADEC Summary Sheet for our Model 202/204. This is a “plain vanilla” Summary Sheet, compared to Model 210, on pages 6 and 7, above. 

ADEC encourages manufacturers to “review their certification test report summaries and submit corrections, and that any substantiated errors or corrections will be applied to the summary sheet.” So, I’ll just make the corrections here. On our Model 202/204, the initial ADEC summary sheet (see next page, LEFT COLUMN) makes the following errors (WHICH CUMULATIVELY TOTAL 13 FLAGS). I intend to address THREE ADDITIONAL RED FLAGS (related to Documentation of 1) run appropriateness, 2) run Validity, and 3) run anomalies on Part 2 of this Review. 

What is fascinating is that between early April 2021, when I downloaded the original Summary Review Sheet, and today (mid-May, 2021), ADEC performed an additional review and corrected some of its original errors, and made some new errors. Here are comments on the initial ADEC Summary Sheet. Comments on the revised Summary Sheet are on the next page. 

Page 10: According to the ADEC “PROCESS”, I am supposed to address all of these “issues” by

One of the many photos that
appear to provide evidence
contradicting ADEC "flags"

discussing them with ADEC, and maybe submitting modified or reformed test reports. Then ADEC will makes changes as it deems appropriate. Or not. 

Page 14: The “Assessment” claims that “90% of the stoves tested using ASTM E-3053 used debarked wood or failed to provide information about whether there was bark on the fuel.” (Assessment page 38) The “Assessment further claims that “61% of the stoves tested with ASTM E-3053 used squared wood for more than 50% of the pieces” (Assessment page 33), including the stove immediately above (Model 205) and the fuel for Model 202/204, pictured on pp 10-11. 

Based on my review, I cannot believe either of these claims. If the basic data underlying the “Assessment” is defective, then its claims of numerous deficiencies in testing and reporting, and its criticisms of ASTM E-3053 are suspect, because they are based on bad data. 

Page 19: I hope to have Part 2 in mid-June. There is a lot more to unpack in the “Assessment,” including: 

* The overall “Assessment” review strategy, and whether it is a credible basis for proceeding to the conclusions that the “Assessment” tries to come to. This strategy is basically to make a list of each and every requirement imposed by the NSPS, and then see if each and every item on the list can be identified in test reports, no matter how obscure or irrelevant the requirement might be. Otherwise, deficiencies are claimed by NESCAUM/ADEC without any apparent oversight or review, or any basis in fact. 

* The nexus of firebox size and calculation, loading direction, fuel length, and loading density. These are the second set of elements that the “Assessment” uses to criticize ASTM E-3053. 

* I’ll review this sentence, and how it has spawned innumerable “flags” (i.e., claims of violations of the NSPS) in the “Assessment’s flag-collection effort."