Air Quality in the Barnett Shale - Part 25: What a statistically valid report looks like

So here we are with a bunch of reports...looking at a bunch of numbers...and making a bunch of different observations and conclusions.  Should we trust the TCEQ's findings?  Should we discount what the FWLN's report claims in terms of possible exposure to carbon disulfide?  What about ERG's report and all the work Alisa Rich did for the Town of Dish Texas?

It seems daunting, but it is not if you follow a certain - well recognized - set of basic scientific principles when collecting samples and analyzing them. So lets compare and contrast these reports.

One of my main complaints centers on the identification and quantification of the carbon disulfide concentration Dr. Sattler bases the setback on as well as an indication of potential health concerns.  And one of my main issues with the work performed by Dr. Sattler is the "backed in" data she uses to calculate her emission rate for the air dispersion modeling.  I believe I have laid out a fairly well substantiated reasoning for why her model's predictions are invalid.

Dr. Sattler assumes that with one air sample taken at a distance from the source you can back in to the air dispersion model and calculate the emission rate of the contaminant detected from that source.  With this emission rate, you can then plug it into the air dispersion model along with historical wind and weather information and predict the upper most concentration and distance from the source that could exceed a health based limit.

In my last post I attempted to show how that assumption - based on one single sample - assumes that one single sample accurately represents the air in and around the source.  n = 1 makes the model's predictions invalid, let alone the assumption that you can back in data to obtain the emission rate.  How confident is Dr. Sattler and her team on the representativeness of that one sample used to calculate carbon disulfide in Plot 1 and, most importantly, what do they base that confidence on?

In the ERG interim Report an emission rate was also calculated.  So why should ERGs stand and Dr. Sattler's be discounted?  Lets look at the assumption and the methodology used by ERG:

At 66 sites, preliminary quantitative emission estimates have been generated. Estimated annual emission rates were calculated from the short-term test data, assuming the measured emissions on the day of testing are representative of the emissions that occur over the year.

This is an assumption - it states it as such.  Scientist make assumptions all the time.  Then we test, test again, test some more, and research other similar published work.  Then when we look at all the data collected and we make a reasonable conclusion that our assumption is sound.

Even though ERG's emission rates were calculated from one SUMMA canister's results, there were 66 similar tests performed at 66 different locations.  This gives a representative snapshot as to what is most likely - reasonably - taking place.

A researcher can never know the true value of the population being sampled.  This is why we use statistics to calculate a mean, standard deviation, and variance.  It is with those numbers that we look at our assumptions and make a judgement to its soundness.

With 66 different samples of the population (air within oil & gas sites) collected by ERG, we can look at the report's tables to get a pretty good idea of what is being emitted - provided some additional methodologies were also performed.

Because bias is always present, the sampling team and laboratory need to follow a strict protocol on how they will collect the sample, transport the sample, prepare the sample, test the sample, and report the results.  This is all part of what is referred to as QA/QC.  If you look at the ERG report you will see a section QA/QC and if you look at the TCEQ report you will see the same.  If you look at the FWLN report and the reports by Alisa Rich for the Town of Dish, Texas, you will not see a QA/QC section. (The Dish Final Report does describe the sampling procedures and the analysis method performed.)

So the assumption is this; If the researchers follow their QA/QC plan, the impact of any bias will be reduced and the results can be considered representative and valid.  Additionally, the laboratory follows a similar QA/QC plan and also may choose to follow a strict methodology and become NELAC accredited.  (To be valid in Texas the laboratory must be NELAC accredited.)

For the laboratory that performs the analysis, it must be able to show that they are able to "accurately determine a compound and that they can acquire the same concentrations from different instruments or samples while they are sampling the same gas stream, with an acceptable level of uncertainty."  This is known as measurement precision.  ERG calculated analytical precision by collecting two sets of duplicate samples at two different sites and analyzed them in replicate (page 64). (1)

As well as precision, the laboratory is required to show that it has the "ability to acquire the correct concentration data from an instrument or sample analysis with an acceptable level of uncertainty while measuring a reference gas stream of a known concentration."  This is known as accuracy. (1)

All of this data to show precision and accuracy is identified and report by both ERG and the TCEQ in their two separate reports on air quality in the Barnett Shale.  This data is needed to support their assumption that the results they have put forth are representative and valid.  Since we can never know the actual - or true - value, we need this information to support what we can reasonably ascertain.

This information is missing from Alisa Rich's reports as well as the FWNL report.  With one exception...sort of...GD Air Testing, the laboratory used by FWLN and Alisa Rich - is NELAC  Accredited.

However, when they perform analysis to identify carbon disulfide - a TIC - that analytical method is "not included in the Scope of NELAC Accreditation" (see note 'N' on the Lab report).  Without the FWLN report's laboratory performing the analysis as per their NELAC Accreditation, the results for carbon disulfide is invalid.  That's not the fault of GD Air Testing, its just the way we have to look at data that has not been generated under a strict protocol and methodology designed to minimize the impact of bias and substantiate validity.

To meet this, the samples had to be prepared and analyzed using a method that is designed specifically for the contaminant of concern.

So how did ERG and the TCEQ analyze for carbon disulfide?  They used the correct method, TO-15.

ERG Report

TCEQ Report

And what did Alisa Rich (2) and Dr. Sattler (3) use for their reports?

Dish, Texas Report

FWLN Report

So the wrong method - TO-14 - was used by Alisa Rich and Dr. Sattler to identify the carbon disulfide.  This creates uncertainty on the validity of all the assumptions that were based off the carbon disulfide quantification.

On the other hand, the use of the correct method - TO-15 - reduces the uncertainty and adds to the validity of the results and assumptions made by the TCEQ and ERG in their reports.

Because we cannot know the true value, assumptions are made.  It then becomes necessary for the report writer to show how sound those assumptions are.  Without representative sampling, without QA/QC, and without using the correct analytical method whereby precision and accuracy could be shown, the assumption presented by the FWLN are invalid.

Our assumptions are only as good as our data is sound.

Air Quality in the Barnett Shale - Part 26: That's a lot of n's!


.

Air Quality in the Barnett Shale - Part 24: How confident in that 10.8 ppbv are you?

It has taken me a bit of time to get through the ERG "Interim Ambient Air Monitoring" Report as well as to re-read the Ft. Worth League of Neighborhoods (FWLN) "Recommendations for policy Changes for Gas Drilling Near Schools" report.

There is a lot of data to take in as well as looking at the methodology and conclusions.  I also try not to take things at face value and check the sources of the information (which is why I cite and link my sources).

So here is the issue;  Holy lots-o-dots Batman!  There's a bat-boat load of oil & gas related activity in the Ft. Worth area! (1)

So it is only natural that the public in and around these sites would question if their health and safety is being impacted.  And, when it comes to their children, as in "ensuring the safety of the 80,000 children who attend FWISD schools (2)" the need to be assured becomes elevated.  So what to do...what to do.

I am critical of bad science because it can cause people to do things that are more detrimental to their health then the thing they they were trying to avoid.  The issue in all of this should not be carbon disulfide.  Carbon disulfide is a non-issue and it is dominating the discussion and taking away from a real dialog which should be; What can we do to make oil & gas activity in and around homes and schools safer and reduce its impact on the environment?

The discussion should be; You want to extract here, then you need to adopt an "Environmentally Systems Drilling Friendly Program" I spoke about in a previous post.  Instead we have a "team of scientists and experts – Dr. Ramon Alvarez, Dr. Melanie Sattler, Dr. DavidSterling, and Carl Weimer – who donated their expertise and time to the League to produce this report (2)" describing a situation of concern that does not exist.

Whether a setback is a prudent idea is not what I am being critical of.  The FWLN report is basing this "one mile setback" on an air dispersion model's finding that carbon disulfide could be found at concentrations over three times the OSHA permissible exposure level (PEL) one mile from an O&G facility. (241 mg/m3 = 78 ppm)

Now reading this, and knowing that a "team of scientists and experts" - two of them Texas University professors - signed off on the findings, one might reasonably conclude; Holy exposure Batman!  The children!

But a closer look at how that number was derived would lead you to a different conclusion; Holy poor science Batman!  That data for carbon disulfide is invalid!

I have tried to present as objective and scientifically based argument as to why this is so in over 24 posts on this topic.  If I have not made my case then either you don't believe me, don't want to believe me, don't want to consider it, or have some other reason to ignore my conclusion as to why the methodology and findings in the Town of Dish Texas and the FWLN reports are wrong.

These blogs are open to anyone who wants to show where my premise, conclusion, or understanding of the science is incorrect, faulty, or wrong.  If I can dish it out (no pun intended) then I sure as better be able to take it.

So it comes down to this.  The FWLN report includes laboratory analysis for carbon disulfide for one single sample:

With the carbon disulfide concentration as:

So the air dispersion model generating Plot 1:

...was based on the analytical report shown above:

And if for no other reason than that - the results of the model and the conclusion for Plot 1 is invalid.  One sample cannot be used to test the null hypothesis that there is a relationship between the ambient air concentration at one location (SUMMA canister for 300 McNaughton Ln.)  and the expected ambient air concentration at another point (one mile out in Plot 1).

One sample is not representative of the air at that location.  Here is what EPA has to say about samples:

Representative: “a sample of a universe or whole (e.g., waste pile, lagoon, ground water) which can be expected to exhibit the average properties of the universe or whole."

Inferences about the population are made from samples selected from the population. For example, the sample mean (or average) is a consistent estimator of the population mean. In general, estimates made from samples tend to more closely approximate the true population parameter as the number of samples increases. The precision of these inferences depends on the theoretical sampling distribution of the statistic that would occur if the sampling process were repeated over and over using the same sampling design and number of samples.

This then leads to:

[a]fter a sample of a certain size, shape, and orientation is obtained in the field (as the primary sample), it is handled, transported, and prepared for analysis. At each stage, changes can occur in the sample (such as the gain or loss of constituents, changes in the particle size distribution, etc.). These changes accumulate as errors throughout the sampling process such that measurements made on relatively small analytical samples (often less than 1 gram) may no longer “represent” the population of interest.  Because sampling and analysis results may be relied upon to make decisions about a waste or media, it is important to understand the sources of the errors introduced at each stage of sampling samples and take steps to minimize or control those errors. In doing so, samples will be sufficiently “representative” of the population from which they are obtained.

When scientists make statements regrading their observations, the concept of precision and bias come into play:

• Precision is a measurement of the closeness of agreement between repeated measurements. 
• Bias is the systematic or consistent over- or underestimation of the true value

Precision is the ability to get the same - or very close - result each and every time you collect or analyze the sample.  Bias, on the other hand, results from a number of problems inherent in sampling and analysis. 

Sampling Bias: 

• Bias can be introduced in the field and the laboratory through the improper selection and use of devices for sampling and subsampling. Bias related to sampling tools can be minimized by ensuring all of the material of interest for the study is accessible by the sampling tool.

• Bias can be introduced through improper design of the sampling plan. Improper sampling design can cause parts of the population of interest to be over- or under-sampled, thereby causing the estimated values to be systematically shifted away from the true values. Bias related to sampling design can be minimized by ensuring the sampling protocol is impartial so there is an equal chance for each part of the waste to be included in the sample over both the spatial and temporal boundaries defined for the study. 

• Bias can be introduced in sampling due to the loss or addition of contaminants during sampling and sample handling. This bias can be controlled using sampling devices made of materials that do not sorb or leach constituents of concern, and by use of careful decontamination and sample handling procedures. For example, agitation or homogenization of samples can cause a loss of volatile constituents, thereby indicating a concentration of volatiles lower than the true value. Proper decontamination of sampling equipment between sample locations or the use of disposable devices, and the use of appropriate sample containers and 

Analytical Bias:

• Analytical (or measurement) bias is a systematic error caused by instrument contamination, calibration drift, or by numerous other causes, such as extraction inefficiency by the solvent, matrix effect, and losses during shipping and handling.

Statistical Bias:

• When the assumptions made about the sampling distribution are not consistent with the underlying population distribution, or
• When the statistical estimator itself is biased.

Because bias is always in play, the number of samples collected (replicates) and the dates/areas (representative) is increased to minimize the impact of these issues.

So how confident in the number 10.8 ppbv is the report's "team of scientists and experts?" How sure are they that 10.8 ppbv represents the true value of the air at that sample point?  I mean. look at all the potential errors that could have impacted it.  Shouldn't at least a duplicate sample have also been collected and analyzed?

It was that one single value that was used to determine the concentration of 78 ppm determined by the model to be in the plume:

Plume extends 1 mile from the source in this graphic. Full extent of plume was in excess of 2 miles. Plot 1 multiples were up to 1000 times the short term health benchmark for carbon disulfide.

How confident would (could) any reputable scientist be if one - and only one - sample - was used in their published research?

For that reason alone, Plot 1 is not valid.

Now, couple that with this:

 Notice that "N" in the very last column?  That's a note from the laboratory:

What the lab is saying is that all the statistical stuff they do - precision & accuracy - was not performed on this sample.  How confident are they in the number 10.8 ppbv?

So there you have it.  Plot 1 was developed using one sample and analyzed on an instrument that was not calibrated for carbon disulfide.

Holy bias Batman!  Plot 1 values are not statistically valid!


Yes Robin, that's what I have been trying to say all along.  The whole enchilada is an example of bad methodology, sampling, and analysis - and the results and conclusion are invalid.


Next post: Air Quality in the Barnett Shale - Part 25: What a statistically valid report looks like.


.