|
code
newsgroups
|
|||||||||||||||||||||||
|
|||||||||||||||||||||||
Windows or VB problem - Please helpI noticed the following problem in all of the VB, VBA or VBScript.
E.g. When I do 1/3 it always equal 0.333331383035028 instead of 0.33333333333333 The problem originally appeared in one of my Excel VBA program, and when later I found the problem consistent among all the visual basic related applications or scripts. Some one suggested to re-install Office, which I did and of no help. I have also done a complete re-install of Windows 98 SE, and still no help. Could someone advise how I can go about fixing this problem. fiazid***@gmail.com wrote:
Show quoteHide quote > I noticed the following problem in all of the VB, VBA or VBScript. you need to find a general primer on floating point math.> > E.g. When I do 1/3 it always equal 0.333331383035028 instead of > 0.33333333333333 > > The problem originally appeared in one of my Excel VBA program, and > when later I found the problem consistent among all the visual basic > related applications or scripts. > > Some one suggested to re-install Office, which I did and of no help. > I have also done a complete re-install of Windows 98 SE, and still no > help. > > Could someone advise how I can go about fixing this problem. > there is nothing unique to VB about it. The reason this problem is unique is.
I have tried 5 computers with similar settings and tried out the same and all the computers return 1/3 = 0.3333333333333333 and only the computer that has this problem returns 1/3 = 0.333331383035028 You may try this on your own computer on the immediate window like ? 1/3 and I am certain it will return 0.33333333333 I am just stating the problem in the simplest terms. Infact, the problem is even worse like when I type in a VBA module 1.23 and enter it automatically changes to 1.2300293334234 for example. When I open the same module in a different computer it shows as 1.23. Strange isn't it. Bob O`Bob wrote: Show quoteHide quote > fiazid***@gmail.com wrote: > > I noticed the following problem in all of the VB, VBA or VBScript. > > > > E.g. When I do 1/3 it always equal 0.333331383035028 instead of > > 0.33333333333333 > > > > The problem originally appeared in one of my Excel VBA program, and > > when later I found the problem consistent among all the visual basic > > related applications or scripts. > > > > Some one suggested to re-install Office, which I did and of no help. > > I have also done a complete re-install of Windows 98 SE, and still no > > help. > > > > Could someone advise how I can go about fixing this problem. > > > > > you need to find a general primer on floating point math. > > there is nothing unique to VB about it. <fiazid***@gmail.com> wrote in message
news:1142924841.052966.60220@j33g2000cwa.googlegroups.com... Did you call the dilbert help line? ;-) Some people's answer to everything >I noticed the following problem in all of the VB, VBA or VBScript. > > E.g. When I do 1/3 it always equal 0.333331383035028 instead of > 0.33333333333333 > > The problem originally appeared in one of my Excel VBA program, and > when later I found the problem consistent among all the visual basic > related applications or scripts. > > Some one suggested to re-install Office, which I did and of no help. > I have also done a complete re-install of Windows 98 SE, and still no > help. is to reinstall the entire world. :-) > Could someone advise how I can go about fixing this problem. This isn't a problem or a bug, it's standard behaviour for floating point data types. They only have so much accuracy because they are stored as binary values in a certain number of bits. You could try changing the data type to "double" or "currency" to see if either of these help (most likely it is using "single" by default which is half the accuracy of double. Michael <fiazid***@gmail.com> wrote in message
Show quoteHide quote news:1142924841.052966.60220@j33g2000cwa.googlegroups.com... To amplify on the others responses...> I noticed the following problem in all of the VB, VBA or VBScript. > > E.g. When I do 1/3 it always equal 0.333331383035028 instead of > 0.33333333333333 > > The problem originally appeared in one of my Excel VBA program, and > when later I found the problem consistent among all the visual basic > related applications or scripts. > > Some one suggested to re-install Office, which I did and of no help. > I have also done a complete re-install of Windows 98 SE, and still no > help. > > Could someone advise how I can go about fixing this problem. > The "Classic" paper: "What Every Computer Scientist Should Know About Floating-Point Arithmetic" http://docs.sun.com/source/806-3568/ncg_goldberg.html "FP Representation" http://www.cprogramming.com/tutorial/floating_point/understanding_floating_point_representation.html "(Complete) Tutorial to Understand IEEE Floating-Point Errors:" http://support.microsoft.com/?scid=kb;[LN];42980 "INFO: Visual Basic and Arithmetic Precision" http://support.microsoft.com/default.aspx?scid=kb;en-us;279755 "INFO: Visual Basic and Arithmetic Precision" http://support.microsoft.com/default.aspx?scid=http://support.microsoft.com:80/support/kb/articles/Q279/7/55.ASP&NoWebContent=1 "Microsoft Basic Logical Expression Evaluation:" http://vb.mvps.org/tips/truth.asp hth -ralph fiazid***@gmail.com wrote:
Show quoteHide quote > I noticed the following problem in all of the VB, VBA or VBScript. What others have said about FP math is true, but not necessarily germane.> > E.g. When I do 1/3 it always equal 0.333331383035028 instead of > 0.33333333333333 > > The problem originally appeared in one of my Excel VBA program, and > when later I found the problem consistent among all the visual basic > related applications or scripts. > > Some one suggested to re-install Office, which I did and of no help. > I have also done a complete re-install of Windows 98 SE, and still no > help. > > Could someone advise how I can go about fixing this problem. There is no ggod reason why dividing two doubles (1# / 3#) to get a double result should give you such a discrepancy. What might account for it is if you are dividing two singles (1! / 3!) and assigning the result to a double. The extra bits of mantissa are necessarily assigned zeros. You can't gain precision by assignment. Try explicitly causing all constants to be doubles, and be very wary of slipping down in precision as you calculate. One single in a set of values or expressions can poison the entire result. Try the following in the immediate window: z! = 1! / 3! : ? z! : X# = z! : ? X# "Jim Mack" <jmack@mdxi.nospam.com> wrote And then try> > E.g. When I do 1/3 it always equal 0.333331383035028 instead of > > 0.33333333333333 > Try the following in the immediate window: > z! = 1! / 3! : ? z! : X# = z! : ? X# ? 1# / 3# LFS Larry Serflaten wrote:
> "Jim Mack" <jmack@mdxi.nospam.com> wrote When I do that I get just what I'd expect: 1.333333333... > >>> E.g. When I do 1/3 it always equal 0.333331383035028 instead of >>> 0.33333333333333 > >> Try the following in the immediate window: > >> z! = 1! / 3! : ? z! : X# = z! : ? X# > > And then try > > ? 1# / 3# What do you get, and why is it surprising to you? :-) -- Jim There is something stange with what your VB is doing and your take on maths
1# / 3# = 1.333333333 no, I don't think so how about 0.333333333333333 "Jim Mack" <jmack@mdxi.nospam.com> wrote in message Larry Serflaten wrote:news:u$qh8wPTGHA.1728@TK2MSFTNGP11.phx.gbl... > "Jim Mack" <jmack@mdxi.nospam.com> wrote When I do that I get just what I'd expect: 1.333333333...> >>> E.g. When I do 1/3 it always equal 0.333331383035028 instead of >>> 0.33333333333333 > >> Try the following in the immediate window: > >> z! = 1! / 3! : ? z! : X# = z! : ? X# > > And then try > > ? 1# / 3# What do you get, and why is it surprising to you? :-) -- Jim Dave wrote:
> There is something stange with what your VB is doing and your take on Right -- I should cut & paste. But the point is, I see 3's all the way out, just as you would expect. I wonder what Larry sees that he finds unexpected.> maths 1# / 3# = 1.333333333 > > no, I don't think so > > how about > 0.333333333333333 -- Show quoteHide quoteJim > > > "Jim Mack" <jmack@mdxi.nospam.com> wrote in message > news:u$qh8wPTGHA.1728@TK2MSFTNGP11.phx.gbl... > Larry Serflaten wrote: >> "Jim Mack" <jmack@mdxi.nospam.com> wrote >> >>>> E.g. When I do 1/3 it always equal 0.333331383035028 instead of >>>> 0.33333333333333 >> >>> Try the following in the immediate window: >> >>> z! = 1! / 3! : ? z! : X# = z! : ? X# >> >> And then try >> >> ? 1# / 3# > > When I do that I get just what I'd expect: 1.333333333... > > What do you get, and why is it surprising to you? :-) "Jim Mack" <jmack@mdxi.nospam.com> wrote I get the same answer as you, the answer the OP was looking for.> What do you get, and why is it surprising to you? :-) It was just the case in support of using doubles to produce the OP's expected results. You posted examples that showed rounding errors, so I posted an example that did as the OP wanted, which could be seen in the Immediate window. You indiciated as much, but did not provide an example: "There is no ggod reason why dividing two doubles (1# / 3#) to get a double result should give you such a discrepancy." It just seemed natural to me, to include an example of doing it the right way.... LFS  
Other interesting topics
|
|||||||||||||||||||||||
