Page 1 of 1
How does Empower round this number?
Posted: Mon Sep 25, 2017 7:18 pm
by EmpowersBane
I am working on a custom field (Boolean) that calculates a result of Pass or Fail but my spec is NMT 2.0%. What is the limit that you can put in the CConst field of the proc method that will return a Pass? If I have a result of 1.5 does Empower round it to 2.0 or does it depend on reported precision eg is 1.499999 still less than 2.0?
I'm having trouble trying to decide what to put when I'm trying to decide what would exceed a spec of for example NLT 98%, NMT 1.0%. Any tips?
Re: How does Empower round this number?
Posted: Tue Sep 26, 2017 6:26 am
by ydna1977
Hi,
Not sure , Im guessing but depending on dp shown empower rounds up
ie 1.49 shown = 1.5 (to 1dp)
Maybe worth having custom field with ROUND or truncation formula , though may need to check /follow internal SOP's ie
combination of both (probably not needed here but good example):
ROUND(Amount+0.005,-2)-0.01
Re: How does Empower round this number?
Posted: Fri Oct 20, 2017 8:14 pm
by bundy5686
When comparing two numbers, ALWAYS use ROUND function to round your result to the same precision as your specification limit before making the comparison. Never compare two number with different precision. Never assume the program has the same rounding rule as what you are expecting.
Re: How does Empower round this number?
Posted: Fri Oct 20, 2017 8:26 pm
by DR
1 good example above...
This has been covered at great length in Waters Inform conferences over the years - Google that for custom field information.