Validation of calculations

[cody84]

Hi, I work in a cGMP/FDA/Health Canada regulated company and I am wondering if you know how normal companies go about validating the custom calculations. They have an extremely tedious and time-consuming method that they use here, including checking averages, standard deviations and relative standard deviations. Meanwhile they assume all calculators work fine...but empower software which is $$$$ cannot possibly be reliable.

I plan on just manually calculating all results from a method validation and comparing to the empower reports. Shouldn't that be enough???

If you have any advice or documentation that I could use to help convince QA here that would be very helpful.

Right now they check all calculations that empower does on three different batches of product in triplicate. There is even a protocol and report just for this calculation verification...I think it would be much more logical to check the calcs during validation of the method and really, if the validation passes all parameters, doesn't that mean the calculations would have to be correct?



Thanks

[unmgvar]

in my view you need to brake it into 3 categories when doing calculations using a report made inside a chromatography software.
1. check that your report works for 1-3 works in comparison to a manual check(or excel if you already checked it to a manual check, do not trust excel either )
2. see that you implement a good set of working instruction that will help your users to correctly use the report. implement a set of fields in the report to show that they are no mistakes in the results, like, date of process, ID or last update date, printing date, the number of page total, the user name and things like that
3. implement the user management in such a way that users possibility to make changes or errors in the reports are minimized and hard to perform

as part of the validation show that the fails safes work and that if an error message or comment from the software appears to say that a given procedure is forbidden, document it in the validation

[lynzjm]

When I worked in a GMP lab we put in place excel spreadsheets for common calculations. We would then generate a "pass" and a "fail" sheet and these would then be checked by QA/compliance. They would contain locked calculation cells and unlocked input cells. These once approved did not need to be checked as the spreadsheets were validated. Only QA and the system administrator had the locked cells password and to make any changes you had to submit a changes form. If you use a lot of the same calculations a lot of the time this might be something to look into, certainly saves QC checking!

lynz x

[cody84]

Well the whole idea for me is to use Empower to its full extent, by having it do all the calculations in the software, it would be the same thing as using an excel sheet, but one less step.

[unmgvar]

try to make your reports to be as less paper consuming as possible as well.
i remember i hate the duplicate reporting in empower because it takes so much paper when printed

[cody84]

I have, using overlays for all duplicates and working towards 1 report for entire runs to make it easier for the analysts.

[Johnny Rod]

This does seem a bit like overkill, like when the 21CFR11 regs came out and peopel were validating coffe machines. Speak to Waters, they can supply you with info regarding DQ etc. of the software, and maybe also conduct a vendor audit if necessary. This will reduce the risk of the inbuilt calculations so that you don't have to prove them. I mean, are you also cutting out peaks and weighing them to prove you have correct relative areas? No, of course not, there is also the built-in IQ/OQ tool that runs dummy data through the client software (on the PC/LACE/processing server, however you have it set up) to show that it's doing the same processing as expected. You'll have to check the custom calcs you do yourself of course, and maybe you could check the odd built in calc (e.g. RSD, the ones you can't modify).

[unmgvar]

johnny you have to remember that you are responsible to correctly implement and use all the GLP and CFR 21 part 11 features of the software you use
and the reason is to create a working procedure that will allow you to answer the 5 "w"
who, when, why, how, what.
if you can track that you are fine for the regulation

now when you use empower to calculate automatically for you, how do you know all the necessary fields have the correct figures in them? and the result the report is giving you is right?
for that you need to check it, for example many fields come from the alter sample, and it is best to have them on the report as well, or you should have also a print out of the alter sample
for audit trail issues, you best have the ID number of the process method used and show which process method was used. for other softwares, it is the last changed date field.
you need to see that all the standard used for the calibration are the right ones, the first common error with empower is to forget to clean the last used curve, and the other mistake is to simply take the wrong standard in the calculation because empower dumps all the channels together and you need to sort it out correclty. so you need to have a good standard report
it is not overdoing, itis simply based on the mistakes that users do make while making use of empower and therefore you are required to be capable of answering the 5 "w"

[Johnny Rod]

Odds are you'll acquire and process data as part of the validation scheme anyway, if only to show it is possible, so these issues will be covered there without extra work. To simply show that the software can process and do basic calcs that can't be customised, the built-in OQ tool is sufficient plus you cna use this whevener you apply a software patch or install a new client, as a benchmark, which is why Waters came up with it in the first place. There's also an IQ module that verifies the install and prints a report for you.

[unmgvar]

Johnny waters and the other vendors have created products that are capable of being GLP and CFR 21 part 11 compliant. it does not mean that you have to work with them that way.
therefore it is your responsibility to show that you are correctly setting the product to work has you intend it to be used.
Waters is never responsible to that. their documents show that if you want to the tools work as intended when used.
and also you need to make a difference between IQ, OQ, PV/PQ
you are responsible to have them in a GLP environment and you can use for IQ OQ the supplier settings and procedures, but especially PV is for your applications and environment and so you must validate something of them to show that the product is working well for you.
with softwares you have several ways to report the same things (well at least this is how we have it for chromeleon), you need to see that the way you choose will allow you to meet regulations and SOPs, hence you have to validate.
let's compare to a matter that is also very important.
back ups. softwares can do that, and in the OQ it should show that it can do it. but if you do not implement it then it will not work, and if you use it for sure you will check that it is running like you want it.
same for the reports,
this is PV- seeing that you are using the tools sa intended in your environment and your procedures

[Johnny Rod]

All true but to validate to this degree is, to my mind, more work than is necessary. If Empower produces a table, and gives averages and RSDs for the columns, then these calculations aren't something the user can alter. If you did the same thing in a spreadsheet then you can't alter tha calculation but you would validate it as you could have wrong cell references. So how do you know Empower is using the right numbers? Either validate it (which only needs doing once), or take the vendor's qualification material to show that you have already bought a product which has been written and checked and is suitable for use. This is the bit where they sell it as capable of being 21CFR11 compliant - they've done as much as they can i.e. the background work, so you can put it in and make it compliant. This means your risk assessment of many parts of the software can be reduced so you don't have to test it to death. Doing so may seem to be of little consequence, but it engenders a lot of wasted time, and a culture where things are so hard to validate that nothing ever gets changed (i.e. improved).

Cody, I'd say if you haven't already spoken to Waters then you should. They have a load of supporting info on their website on how to do this, and for them, GMP compliance is a big deal because GSK and many others use Empower. In terms of custom calcs, if you mean ones you define yourself, yes you'll have to test them, but only once (as computers always do the same thing), and it is worth retaining a set of data as a test set - this is bascially what the built in OQ tool does - so your revalidation will be easier in future.

[cody84]

I figured I will just do one calculation manually of each custom field for each test in each method validation. So, I would just do one example calculation of the result and any other custom calc once each for accuracy + range, precision, linearity, etc. I'm just hoping there's some data out there that will confirm this is enough. It makes sense that as part of a method validation, the calculations should be checked at least once. For normal practise here in our labbooks we must show one full example calculation and then we can just write the result for the rest.
We also don't validate our hand calculators here and assume they are correct when we use them for manual calculations, which seems rediculous to me to assume that Empower is less reliable than a hand calculator.
As for average, SD and RSD, I think that if these need to be proven at all you shouldn't be working with anything that involves math...

[unmgvar]

johnny you are missing a basic view of the matter.
we are not checking the math. no we know this works,
we are checking the input procedure, workflow, and the capacity to review the report in order to find errors if they happen. that is why you do the PV. to show that when the procedure is done correctly the results fit (BTW in general +- a 1 difference in the last decimal, be aware of that, it depends on the accuracy you set yourself). and on the other hand to be certain that if a critical field of the calculation that requires a user input or manipulation is done incorrectly then it can be spotted.
the same as for a spreadsheet. there you simply need to check everything all the time
in softwares, you mainly need to see that the weights and dilutions and purities are correct and that the integration was done all with the same process of the same ID with the correct runs (because in empower you are free to use any standards and samples you want if you do not know what you are doing and each one can be processed differently).
once this is done then you can take an impurity or dissolution profile calculation that was taking hours to check and do it in 15-20 minutes for example
all this is because no software come GLP and cfr 21 compliant ready to use. it is your responsibility to make it so.

and Cody, it is not enough to show that it does the math correctly. you need to show you can spot when a mistake is made that brings a wrong reporting, this is a lot more crucial then the math, which we know is done right.

[Johnny Rod]

That would be writing an analysis procedure and checking that the right data has been entered, so it's not software validation, it's data checking. The software won't know if the numbers entered are right or wrong, but it will use them for calcs anyway. It's the transition from input data to results that is in question. The data system has to be supported by some documentation to complete the package as there are things beyonds its ability to control. As you seem determined to misunderstand me and find a way to say I'm wrong, I'm going to end the discussion there.

Cody you should be fine, I presume the main validation of the system is done as you're already using it, so we don't need to worry about that, you just need evidence of the data in and out and that it does what you expected it to do.