|
code
newsgroups
|
|||||||||||||||||||||||
|
|||||||||||||||||||||||
VB6 - Round() Error after compileinconsistent in some circumstances. Below is some code to demonstrate it, but does anyone have any idea why the behaviour changes post compile?: Dim c1 As Currency Dim c2 As Currency Dim c3 As Currency c1 = 41.5 c2 = 5 c3 = c1 / 100 * c2 MsgBox ("the following calculations should all return the same value based on bankers rounding" & _ ", but they differ between IDE debugging and running it compiled.") 'If you put these in as numbers directly the result is wrong consistently. MsgBox ("c1 is " & c1 & " and c2 is " & c2 & vbCrLf & _ "Round(c1 / 100 * c2, 2) returns: " & Round(c1 / 100 * c2, 2)) MsgBox ("c3 is " & c3 & vbCrLf & _ "Round(c3, 2) returns: " & Round(c3, 2)) 'If you put them in as variables then they're correct when you run 'in code through the IDE. 'However, when you run this as a compiled executable then you 'get the incorrect result again. MsgBox ("Round(2.075, 2) reutrns :" & Round(2.075, 2))
Show quote
Hide quote
> There seems to be a bug with the rounding function that means it's If you put the numbers into Currency variables first, then the calculations > inconsistent in some circumstances. Below is some code to demonstrate it, > but > does anyone have any idea why the behaviour changes post compile?: > > Dim c1 As Currency > Dim c2 As Currency > Dim c3 As Currency > > c1 = 41.5 > c2 = 5 > c3 = c1 / 100 * c2 > > MsgBox ("the following calculations should all return the same value based > on bankers rounding" & _ > ", but they differ between IDE debugging and running it compiled.") > 'If you put these in as numbers directly the result is wrong consistently. > MsgBox ("c1 is " & c1 & " and c2 is " & c2 & vbCrLf & _ > "Round(c1 / 100 * c2, 2) returns: " & Round(c1 / 100 * c2, 2)) > > MsgBox ("c3 is " & c3 & vbCrLf & _ > "Round(c3, 2) returns: " & Round(c3, 2)) > > 'If you put them in as variables then they're correct when you run > 'in code through the IDE. > 'However, when you run this as a compiled executable then you > 'get the incorrect result again. > > MsgBox ("Round(2.075, 2) reutrns :" & Round(2.075, 2)) are carried out as integer calculations (Currency values are scaled integers which is why they have a fixed maximum number of decimal places; basically, the decimal point is eliminated, the integer calculation takes place and then the decimal point is reinserted)... integer calculation have no roundings when being carried out. However, when you use the numbers directly, the calculations take place as a floating point operation and is subject to all of the problems associated with them. These two links should give you a better insight into the floating point calculation problem... INFO: Visual Basic and Arithmetic Precision http://support.microsoft.com/default.aspx?scid=http://support.microsoft.com:80/support/kb/articles/Q279/7/55.ASP&NoWebContent=1 (Complete) Tutorial to Understand IEEE Floating-Point Errors http://support.microsoft.com/default.aspx?scid=http://support.microsoft.com:80/support/kb/articles/Q42/9/80.ASP&NoWebContent=1 Rick Thanks Rick - that explains the rounding error.
Any idea why it seems to work (at least partially) during Debug in the IDE, but then fails when run as a compiled application? Cheers, Tom. Show quoteHide quote "Rick Rothstein (MVP - VB)" wrote: > > There seems to be a bug with the rounding function that means it's > > inconsistent in some circumstances. Below is some code to demonstrate it, > > but > > does anyone have any idea why the behaviour changes post compile?: > > > > Dim c1 As Currency > > Dim c2 As Currency > > Dim c3 As Currency > > > > c1 = 41.5 > > c2 = 5 > > c3 = c1 / 100 * c2 > > > > MsgBox ("the following calculations should all return the same value based > > on bankers rounding" & _ > > ", but they differ between IDE debugging and running it compiled.") > > 'If you put these in as numbers directly the result is wrong consistently. > > MsgBox ("c1 is " & c1 & " and c2 is " & c2 & vbCrLf & _ > > "Round(c1 / 100 * c2, 2) returns: " & Round(c1 / 100 * c2, 2)) > > > > MsgBox ("c3 is " & c3 & vbCrLf & _ > > "Round(c3, 2) returns: " & Round(c3, 2)) > > > > 'If you put them in as variables then they're correct when you run > > 'in code through the IDE. > > 'However, when you run this as a compiled executable then you > > 'get the incorrect result again. > > > > MsgBox ("Round(2.075, 2) reutrns :" & Round(2.075, 2)) > > If you put the numbers into Currency variables first, then the calculations > are carried out as integer calculations (Currency values are scaled integers > which is why they have a fixed maximum number of decimal places; basically, > the decimal point is eliminated, the integer calculation takes place and > then the decimal point is reinserted)... integer calculation have no > roundings when being carried out. However, when you use the numbers > directly, the calculations take place as a floating point operation and is > subject to all of the problems associated with them. These two links should > give you a better insight into the floating point calculation problem... > > INFO: Visual Basic and Arithmetic Precision > http://support.microsoft.com/default.aspx?scid=http://support.microsoft.com:80/support/kb/articles/Q279/7/55.ASP&NoWebContent=1 > > (Complete) Tutorial to Understand IEEE Floating-Point Errors > http://support.microsoft.com/default.aspx?scid=http://support.microsoft.com:80/support/kb/articles/Q42/9/80.ASP&NoWebContent=1 > > Rick > > >
Function that returns an Array
Reason for 'Set' keyword SendKeys command fails on Vista Error in reading large csv file Sending email Should I or shouldn't I include excel.exe in my app distribution package? Getting Info from a .TTF File Run-time Error 430, WITH non-broken binary compatibility? TaskBar won't show when hidden. My DataReport Problem |
|||||||||||||||||||||||