Home All Groups Group Topic Archive Search About

VB6 - Round() Error after compile

Author
28 Feb 2007 10:44 AM
Tom3030
There seems to be a bug with the rounding function that means it's
inconsistent in some circumstances. Below is some code to demonstrate it, but
does anyone have any idea why the behaviour changes post compile?:

Dim c1 As Currency
Dim c2 As Currency
Dim c3 As Currency

c1 = 41.5
c2 = 5
c3 = c1 / 100 * c2

MsgBox ("the following calculations should all return the same value based
on bankers rounding" & _
    ", but they differ between IDE debugging and running it compiled.")
'If you put these in as numbers directly the result is wrong consistently.
MsgBox ("c1 is " & c1 & " and c2 is " & c2 & vbCrLf & _
    "Round(c1 / 100 * c2, 2) returns: " & Round(c1 / 100 * c2, 2))

MsgBox ("c3 is " & c3 & vbCrLf & _
    "Round(c3, 2) returns: " & Round(c3, 2))

'If you put them in as variables then they're correct when you run
'in code through the IDE.
'However, when you run this as a compiled executable then you
'get the incorrect result again.

MsgBox ("Round(2.075, 2) reutrns :" & Round(2.075, 2))

Author
28 Feb 2007 3:05 PM
Rick Rothstein (MVP - VB)
Show quote Hide quote
> There seems to be a bug with the rounding function that means it's
> inconsistent in some circumstances. Below is some code to demonstrate it,
> but
> does anyone have any idea why the behaviour changes post compile?:
>
> Dim c1 As Currency
> Dim c2 As Currency
> Dim c3 As Currency
>
> c1 = 41.5
> c2 = 5
> c3 = c1 / 100 * c2
>
> MsgBox ("the following calculations should all return the same value based
> on bankers rounding" & _
>    ", but they differ between IDE debugging and running it compiled.")
> 'If you put these in as numbers directly the result is wrong consistently.
> MsgBox ("c1 is " & c1 & " and c2 is " & c2 & vbCrLf & _
>    "Round(c1 / 100 * c2, 2) returns: " & Round(c1 / 100 * c2, 2))
>
> MsgBox ("c3 is " & c3 & vbCrLf & _
>    "Round(c3, 2) returns: " & Round(c3, 2))
>
> 'If you put them in as variables then they're correct when you run
> 'in code through the IDE.
> 'However, when you run this as a compiled executable then you
> 'get the incorrect result again.
>
> MsgBox ("Round(2.075, 2) reutrns :" & Round(2.075, 2))

If you put the numbers into Currency variables first, then the calculations
are carried out as integer calculations (Currency values are scaled integers
which is why they have a fixed maximum number of decimal places; basically,
the decimal point is eliminated, the integer calculation takes place and
then the decimal point is reinserted)... integer calculation have no
roundings when being carried out. However, when you use the numbers
directly, the calculations take place as a floating point operation and is
subject to all of the problems associated with them. These two links should
give you a better insight into the floating point calculation problem...

INFO: Visual Basic and Arithmetic Precision
http://support.microsoft.com/default.aspx?scid=http://support.microsoft.com:80/support/kb/articles/Q279/7/55.ASP&NoWebContent=1

(Complete) Tutorial to Understand IEEE Floating-Point Errors
http://support.microsoft.com/default.aspx?scid=http://support.microsoft.com:80/support/kb/articles/Q42/9/80.ASP&NoWebContent=1

Rick
Author
28 Feb 2007 5:03 PM
Tom3030
Thanks Rick - that explains the rounding error.

Any idea why it seems to work (at least partially) during Debug in the IDE,
but then fails when run as a compiled application?

Cheers,
Tom.

Show quoteHide quote
"Rick Rothstein (MVP - VB)" wrote:

> > There seems to be a bug with the rounding function that means it's
> > inconsistent in some circumstances. Below is some code to demonstrate it,
> > but
> > does anyone have any idea why the behaviour changes post compile?:
> >
> > Dim c1 As Currency
> > Dim c2 As Currency
> > Dim c3 As Currency
> >
> > c1 = 41.5
> > c2 = 5
> > c3 = c1 / 100 * c2
> >
> > MsgBox ("the following calculations should all return the same value based
> > on bankers rounding" & _
> >    ", but they differ between IDE debugging and running it compiled.")
> > 'If you put these in as numbers directly the result is wrong consistently.
> > MsgBox ("c1 is " & c1 & " and c2 is " & c2 & vbCrLf & _
> >    "Round(c1 / 100 * c2, 2) returns: " & Round(c1 / 100 * c2, 2))
> >
> > MsgBox ("c3 is " & c3 & vbCrLf & _
> >    "Round(c3, 2) returns: " & Round(c3, 2))
> >
> > 'If you put them in as variables then they're correct when you run
> > 'in code through the IDE.
> > 'However, when you run this as a compiled executable then you
> > 'get the incorrect result again.
> >
> > MsgBox ("Round(2.075, 2) reutrns :" & Round(2.075, 2))
>
> If you put the numbers into Currency variables first, then the calculations
> are carried out as integer calculations (Currency values are scaled integers
> which is why they have a fixed maximum number of decimal places; basically,
> the decimal point is eliminated, the integer calculation takes place and
> then the decimal point is reinserted)... integer calculation have no
> roundings when being carried out. However, when you use the numbers
> directly, the calculations take place as a floating point operation and is
> subject to all of the problems associated with them. These two links should
> give you a better insight into the floating point calculation problem...
>
> INFO: Visual Basic and Arithmetic Precision
> http://support.microsoft.com/default.aspx?scid=http://support.microsoft.com:80/support/kb/articles/Q279/7/55.ASP&NoWebContent=1
>
> (Complete) Tutorial to Understand IEEE Floating-Point Errors
> http://support.microsoft.com/default.aspx?scid=http://support.microsoft.com:80/support/kb/articles/Q42/9/80.ASP&NoWebContent=1
>
> Rick
>
>
>