Validation - Statistics

[JA]

Fellow Chromatographers,

A question popped up during a recent audit on which I am hoping to gain some of your insight. During a linearity experiment (GC-headspace for acetone) we were questioned over our determination and validity of the "back calculations" stated. In bringing the data, presented below, to the forum I have renamed "back calculation" as "% Recovery". The analyst had calculated the % Recovery at each concentration level versus the theoretical response predicted by the straight line equation, and as the data demonstrates, this recovery falls off at the low end. However, when one calculates the % Recovery versus a predetermined (or "nominal") concentration the data fits nicely throughout the whole experiment.

Internally, we are not unanimous in our choice of the most appropriate method. When dealing with HPLC methodology, I feel that the identification of a "nominal" sample concentration, and thus a reference point for the calculation (cf. the column entitled "From B") makes sense as our work (let's say, an assay) is based around a single-point standard. In the case of GC-headspace, however, everything is quantified by means of a calibration curve. In an equivalent case for headspace I guess this could have been calculated from the "Stock" solution (although it doesn't get injected as it's considered too strong), thus I've used B for the example. As you can see, there is some resulting confusion

As we do not have an "expert" on site, have a QA dept. who know less about it than me and a management barrier-to-development who will no doubt draw out my request for a relevant training course for months, I would sincerely appreciate your advice!



The criterion for our validation is that the % Recovery should be within a certain tolerance, with that value being widened at the low levels. As it stands, it passes by one method and would fail by the other.

Additional criteria are the intercept expressed as a % of the peak area observed at "nominal" concentration. This is another idea brought over from HPLC - it is a confusing concept in GC-headspace due to the aforementioned ambiguity over what constitutes "nominal". Lastly, the % Residual is noted in the final column although I am not sure of it's importance - can it be used to help qualify the assignment of LOQ? At the moment this is done only by demonstrating sufficient precision at low level.

This message has been posted in the LC forum simply due to the greater number of views and the general mathematical topic.

[AdrianF]

My first observation is that the intercept is clearly incorrect - it would be reasonable to force it through zero.

[JI2002]

Your problem is not uncommon for linear regression, especially when the calibration range is wide, in your case, 1000 folds. I put your data in a spreadsheet and find some solutions.

Solution 1: You can try to narrow down your calibration range to from 2.1128 to 211.28. The data fit very well if backcalculating the calibration points.

Solution 2: You can calculate response factors (response/concentration) for all 9 levels. Take the average and use it as the slope of the curve. The curve will be: y = 5.7152 * x

As an alternative, you can try a linear regression by forcing the curve through origin suggested by AldrianF. The curve will be : y= 5.53 * x

Solution 3: Use weighting factor inverse of concentration (1/x) or inverse of concentration square (1/x^2) for the linear regression. Either should work for you. Based on my calculation, 1/x^2 works better. The curve will be: y = 5.70 * x + 0.271

The idea is that linear regression curve must have an intercept much smaller than the response of lowest standartd, in this case 12.23612.

[tom jupille]

I'll make one addition to the suggestions from JI2002: use log-log calibration.

Your problem is actually not unusual for wide-range data. It comes from the fact that linear regression is based on the implicit assumption that absolute errors are approximately constant across the entire range ("homoscedastic"), whereas chromatography data generally has percentage errors approximately constant ("heteroscedastic"). There is a more detailed write-up on this available on our web site (an excerpt from our Bioanalytical LC-MS course). Go to http://www.lcresources.com/resources/resources.html , click on the "Exchange" link, and then download "weighting.pdf".

[JA]

Thanks for the replies so far. Not a popular topic though, I think

I need convincing on the idea of weighting the data, particularly when we only weight one variable - on the abscissa. The log-log example works nicely but why is it used. Can one say it is useable, or preferred, when the data spans a range of a couple of orders of magnitude? Should it be used exclusively in chromatography?

Here is a set of further processed data. I didn't find any real difference between log(x) and log(1/x^2) - In the latter case I presume plotting it this way is equivalent to plotting the log-log and trying to fit a 1/x^2 line to it? Do we need a more focussed data processing package to produce line fits like those in the weighting.pdf example?



When comparing the areas now predicted by the log-log graph with the measured data we see a differing recovery depending on whether you compared log with log, or convert to base 10 and compare there. Can anyone comment on this?

Following JI2002's comment we note that the intercept area (in base 10) is lower than the lowest calibration point. Is it low enough?

If anyone wants to play with the data, you can download it by clicking on the screenshot above

[Bruce Hamilton]

Oh no - it's probably a very popular topic, but most of us aren't capable of improving on the original replies.

My only question would be, what sort of calibration curve should a GC headspace technique produce at the concentrations used? and then that would result in "are the mathematical manipulations valid for that technique?".

Please do continue to develop this, it's interesting, but probably not yet enthralling

Please keep having fun,

Bruce Hamilton

[tom jupille]

When dealing with heteroscedastic data, two approaches are suggested:
http://www.itl.nist.gov/div898/handbook ... pmd452.htm

- Transform the data in such a way as to meet the assumptions of least squares, or
- Weight the values appropriately.

In the case of chromatographic data, a transform to log-log renders the errors homoscedastic ("proof is left to the student as an exercise"); I find this a lot more straightforward than weighted least squares. If somebody questions the appropriateness, point them to the NIST handbook referenced above.

[LocoLobo]

There are some good replies here. All I can add is that where I work QA doesn't let us quantitate based on so many calibration levels. This is because no detector is perfectly linear over such a range. See Tom Jupile's comments about homoscedastic vs heteroscedastic. We definitely see this effect with our instruments. Especially when trying to quantitate from higher levels to the detectors detection limits.

It is possible to have a linear calibration curve over the whole range but the lower end be non-linear when you examine it. In other cases there is a subtle deflection (inflection?) point in the curve and the slope is different at the lower end than the higher end. If you are interested in the lower concentrations you want to make sure the calibration is good at that level.

Calculations based on a single point are OK provided that your standard and sample responses are close enough.

[AdrianF]

I would like to ask some basic questions.

1.Why do you want to calibrate over such a huge range?

2. I think calibration using doubling dilutions is inappropriate - calibrations
should be evenly spaced.

3. Given that the response is linear why not use single point calibration repeated at suitable intervals? Most of the run must be creating a calibration!

[Peter Apps]

How sure are you that the deviations from the straight lines of the various raw and transformed data sets are due to calibration per se, rather than bias and/or repeatability rsd of the HS analysis and the standard dilutions ?. Even a perfectly linear calibration will have points off the line due to analytical variation.

You can (in principal) reduce the repeatability rsd by running multiple replicates and making up replicate dilutions, and then plotting the means, but this does nothing for bias. What happens to the linear fit if you use a different dilution scheme ?

I agree 100% with Adrian F that calibration points should be evenly spaced, it is a common questionable practice by instrument manufacturers to put a single point high and right on a calibration graph to improve their R-squareds.

Peter

[JI2002]

When log-log is used for linear regression, it's assumed that b is 0 in
y= ax + b, otherwise relationship between log y and log x won't be linear methmatically.

Linear regression of log y to log x and log y to log x^2 is basically same curve, they have same intercept, and slope is half in the later case. When calculating %R, you should use the recovery after converting the area to base 10. It's not surprising to see difference between
y/y (expected) and log y/ log y(expected). When compared to lowest standard response, intercept is not a concern in log-log equation because it's assumed b=0.

[JA]

Thanks for the contributions from those who replied. I've dragged the thread up from unintentional neglect to try and clear the points up so I've got a proper understanding.

1.Why do you want to calibrate over such a huge range?

The work is performed over a reasonably large range as our intentions are to be able to quantify the solvent level from above the ICH limit to the LOQ. If I recall correctly, data plotted in the acetone example above represents a concentration level of 20,000 - 20 ppm (2 - 0.002%) in samples. I agree that the upper end could be trimmed back, perhaps by one data point to twice the ICH limit (sample C). Would you please elaborate further if this does not answer the question.

I think calibration using doubling dilutions is inappropriate - calibrations should be evenly spaced.

I agree that inappropriate spacing can lead to the leverage effect for any outliers but, as the log graph shows, perhaps it is not doubling dilutions which lead to this problem rather the way we are graphically presenting it?

I am now fairly well convinced that ordinary least squares linear regression is inappropriate for chromatographic data. My hang-up was due to the observation that a good fit (four 9's in the current example) can still be found using this technique. However, the low end is unquestionably in error as demonstrated by the % recovery or residual values.

All I can add is that where I work QA doesn't let us quantitate based on so many calibration levels. This is because no detector is perfectly linear over such a range.

If I take your comment as written, you are inferring your QA have defined the limits of a calibration with respect to having too many data points, or that it spans too wide of a concentration range, without even seeing any data from a study. Pragmatically, I would of thought that the linear range of a detector isn't of immediate concern given that a satisfactory linear regression can be produced from the real world results after appropriate transformation.

When log-log is used for linear regression, it's assumed that b is 0 in y= ax + b, otherwise relationship between log y and log x won't be linear methmatically.

I'm not sure I understand this. I'm wondering that if the result of "y" before logging was due to a nonlinear process (I can only think of the HPLC-UV example right now, where area equals absorbance x time, and absorbance is nonlinear w.r.t light intensity measured in the detector) and thus logging it generates additional variables which manifest themselves as an intercept (I've confused myself ).

I feel that a good review of the analysis and use of data at my workplace is needed. In one sense I post here hoping to hear back from experienced industry employees who point out how things are done there and/or which practices stand up to regulatory scrutiny. I appreciate that people might not want to stand out and explain things fully - that may be somebody's paying profession.

I am left wondering why our laboratory has never before heard back from customers with enquiries from regulatory bodies about something as fundamental as a calibration curve, especially if we are analysing the data inappropriately.

HPLC
All of the above discussion centred around a GC-HS-FID analysis for residual solvents. For our HPLC assay methods for API drug substance we produce a linearity graph (and currently analyse it in a similar fashion to that described for the GC-HS) from around 150% to 0.1% of the nominal concentration. The LOQ is tested by both repeatability and recovery vs. the nominal response. Clearly we are stressing the data analysis by trying to ensure the response is sufficiently linear 'from top to bottom' without any transformations, going against the definition of HPLC data as heteroscedastic. In addition, are data points being wasted throughout the middle of the concentration range which would be better served in the typical 80-120% range?

[tom jupille]

I feel that a good review of the analysis and use of data at my workplace is needed. In one sense I post here hoping to hear back from experienced industry employees who point out how things are done there and/or which practices stand up to regulatory scrutiny. I appreciate that people might not want to stand out and explain things fully - that may be somebody's paying profession.

If you want someone to vet your statistical approaches, the best bet might be to find a statistician as a consultant. Perhaps someone from a university math department?

[bartjoosen]

The work is performed over a reasonably large range as our intentions are to be able to quantify the solvent level from above the ICH limit to the LOQ. If I recall correctly, data plotted in the acetone example above represents a concentration level of 20,000 - 20 ppm (2 - 0.002%) in samples. I agree that the upper end could be trimmed back, perhaps by one data point to twice the ICH limit (sample C). Would you please elaborate further if this does not answer the question.

If I remember correctly, this is a limit test, and you should only prove accuracy and no linearity.

So you can take the ICH limit ±20% or so, and validate this range.

If you want to use the entire range you specified, and you're dealing with heteroscedastic (as this is the problem, not the doubling of the concentration), you have to use weighing, and/or you can prove that the intercept is through zero for your linearity, and as this is true (which is the case for your data), you can use a single point calibration for your accuracy (because the intercept is zero).


Good luck

Bart
PS: This is my point of view, not necesary that of my company.


Bart

[JA]

We are aware that the determination of residual solvents is usually based upon the definition of a limit, although it is my understanding that we perform a full linearity at my place of work such that each solvent can be accurately quantified. This (hopefully) allows us a better understanding of the material's mass balance, or to quantify the potency of a reference standard, for example.

Can you please explain your statement of my acetone linearity data having an intercept of zero? Both on cartesian and log-log scale there is a significant non-zero intercept.

Thanks.