Significant figures in a calibration table

Discussions about chromatography data systems, LIMS, controllers, computer issues and related topics.

5 posts Page 1 of 1
If I had a standard material mass of 0.2055g and a Certificate of Analysis of the material of 99%, should I only have 2 significant figures in the calibration table, i.e. 0.20g?

I have seen several labs use the full assay corrected value of 0.203445g in the calibration table. Should I be using all that are certain 0.20 plus one digit in doubt 0.203 in the calibration table?

I should add that my reported results are two places to the right of the decimal.
An often cited rule is to carry all digits until the end and then round. Your rules (or SOP) may vary.
An often cited rule is to carry all digits until the end and then round. Your rules (or SOP) may vary.

Do I understand you to mean carry all digits through a calculation until I get to the report results, and then only use the 2 digits to the right of the decimal as in my initial example for the least number of significant figures - unless my SOP rules expect all significant figures plus one which is in doubt?
From statistics point of view if you round your intermediate results you increase the uncertainty of the final result. Rounding adds additional error since you're basically adding a random value with some variance to your measurement. And the variance of sum (of the measurement and the random value) is the sum of variances (your measurement variance plus the variance of the random value).

So when doing any type of measurement or calculations the common practice is to never round the intermediate results.
Software Engineer at elsci.io (my contact: stanislav.bashkyrtsev@elsci.io)
by sbashkyrtsev » Sat Jan 29, 2022 12:56 pm
From statistics point of view if you round your intermediate results you increase the uncertainty of the final result. Rounding adds additional error since you're basically adding a random value with some variance to your measurement. And the variance of sum (of the measurement and the random value) is the sum of variances (your measurement variance plus the variance of the random value).

So when doing any type of measurement or calculations the common practice is to never round the intermediate results.


Thank you for your answer! I do appreciate it.
5 posts Page 1 of 1

Who is online

In total there is 1 user online :: 0 registered, 0 hidden and 1 guest (based on users active over the past 5 minutes)
Most users ever online was 1117 on Mon Jan 31, 2022 2:50 pm

Users browsing this forum: No registered users and 1 guest

Latest Blog Posts from Separation Science

Separation Science offers free learning from the experts covering methods, applications, webinars, eSeminars, videos, tutorials for users of liquid chromatography, gas chromatography, mass spectrometry, sample preparation and related analytical techniques.

Subscribe to our eNewsletter with daily, weekly or monthly updates: Food & Beverage, Environmental, (Bio)Pharmaceutical, Bioclinical, Liquid Chromatography, Gas Chromatography and Mass Spectrometry.

Liquid Chromatography

Gas Chromatography

Mass Spectrometry