Hi all,

In my app, I use an array (one dimension) to store a lot of variables
during processing. the number of variables may rise to 100000.

Now during that process, I have to pass variables from one range to
another within the same array. FE I have to pass the variables
10000-20000 to lets say 80000-90000.
So i have to copy 10000 variables.

For now,  I use the 'brute force' method to copy the variables:
I loop through the variables 10000-20000 to copy them to 80000-90000
Something like

   For i = 10000 to 20000
     array(i+70000)= array(i)
   Next


Problem is that this is time consuming.
Question: is there a better (faster) method to accomplish the job

Thanks for your help

Chris

Here is a vb project I was experimenting on before to speed test the
different methods of array coping
http::\\www.randem.com/cgi-bin/randem/countdown.cgi?ArrayTest.zip You could
use this to help make your decision.

Show quoteHide quote

The link got mussed. This will do it.

http://www.randem.com/cgi-bin/randem/countdown.cgi?ArrayTest.zip

Show quoteHide quote

Why do you say it is time consuming?  Have you profiled the
compiled code and found it to be unacceptable?

If you compile to native code (which is very likely considering
your situation), you may find it really isn't all that bad....

Your other option is to use CopyMemory which is (surprise)
compiled (micro-)code that does the brute force method!

LFS

Larry Serflaten wrote:
CopyMemory is *considerably* faster than native array assignments.

Several orders of magnitude improvement ought to be easily achievable.

You skipped the first part:

Try his example, it runs nearly instantaniously.  Kinda hard to
get magnitudes of increase from instantanious....

I was just suggesting that he look for the real trouble before
trying to optimize out any percieved trouble....

LFS

Larry Serflaten wrote:
Show quoteHide quoteI didn't try his code, no.  But, I was forced into a loop like this today:

      For i = 1 To Zones
         For j = 1 To Zones
            ' Input array is 0-based...
            mtx.Cell(i, j) = mat(i - 1, j - 1)
         Next 'j
      Next 'i

Where Zones was 2013, and both the object and Variant array were filled with
Doubles.  Okay, so that's 4-million-plus 8-byte elements, and it was running in VBS
<g>, but it did provide a quite noticable delay.  Enough so that I'm going to, So
Help Me Great Turtle, find a way to sling the bits all at once!

Larry Serflaten wrote:
Show quoteHide quoteHi Larry,

In fact, my problem is more complex than I explained:
I have a function to which I pass the 'move' range and the 'to' range.
Now, the 'move' range can be something like
"3000,3250,3260,20000-30000,32000-33000,40000". The to range will than
be fe "6000,6250,6260,70000-90000,62000-63000,10000". As you can see,
the interval between groups can differ.
What I do is putting ALL the elemnts of the parsed string in another
array for both the 'move' and the 'to' range. Then I check if the ranges
have the same number of elements and the loop through the 'move'
elements and copy them to the 'to' elements.
This takes some time and I would like to reduce that time.

Here is the code

Sub move_variabelen(move_reeks, to_reeks)
   'bepaal de reeks van de "move" variabelen
   If move_reeks <> "" And to_reeks <> "" Then
     Dim move_tabel(), to_tabel()


     'bepaal eerst het aantal door te geven variabelen
     xxx = 0: yyy = 0: mv_teller& = 0

     Do
       yyy = InStr(xxx + 1, move_reeks, ",")
       If yyy = 0 And xxx <= Len(move_reeks) + 1 Then yyy =
Len(move_reeks) + 1
       zzz$ = Mid$(move_reeks, xxx + 1, yyy - (xxx + 1))
       xxx = yyy
       Call splits(zzz$, "-", ev$, lv$, "", "", "", "", "")
       If lv$ = "" Then lv$ = ev$

       mv_teller& = mv_teller& + (varnum&(lv$) - varnum&(ev$) + 1)
     Loop Until yyy > Len(move_reeks)

     'bepaal eerst het aantal "to" variabelen
     xxx = 0: yyy = 0: tv_teller& = 0

     Do
       yyy = InStr(xxx + 1, to_reeks, ",")
       If yyy = 0 And xxx <= Len(to_reeks) + 1 Then yyy = Len(to_reeks) + 1
       zzz$ = Mid$(to_reeks, xxx + 1, yyy - (xxx + 1))
       xxx = yyy
       Call splits(zzz$, "-", ev$, lv$, "", "", "", "", "")
       If lv$ = "" Then lv$ = ev$

       tv_teller& = tv_teller& + (varnum&(lv$) - varnum&(ev$) + 1)
     Loop Until yyy > Len(to_reeks)

     'dimensioneer de move_tabel array
     ReDim move_tabel(mv_teller&)

      'dimensioneer de move_tabel array
     ReDim to_tabel(tv_teller&)



     xxx = 0: yyy = 0: mv_teller& = 0

     Do
       yyy = InStr(xxx + 1, move_reeks, ",")
       If yyy = 0 And xxx <= Len(move_reeks) + 1 Then yyy =
Len(move_reeks) + 1
       zzz$ = Mid$(move_reeks, xxx + 1, yyy - (xxx + 1))
       xxx = yyy
       Call splits(zzz$, "-", ev$, lv$, "", "", "", "", "")
       If lv$ = "" Then lv$ = ev$
       For vrbl& = varnum&(ev$) To varnum&(lv$)
          'opsplitsen in variabelen tabel
         incr mv_teller&, 1
  '       ReDim Preserve move_tabel(mv_teller&)
         move_tabel(mv_teller&) = vrbl&
       Next vrbl&
     Loop Until yyy > Len(move_reeks)


     'bepaal de reeks van de "to" variabelen
     xxx = 0: yyy = 0: tv_teller& = 0
     Do
       yyy = InStr(xxx + 1, to_reeks, ",")
       If yyy = 0 And xxx <= Len(to_reeks) + 1 Then yyy = Len(to_reeks) + 1
       zzz$ = Mid$(to_reeks, xxx + 1, yyy - (xxx + 1))
       xxx = yyy
       Call splits(zzz$, "-", ev$, lv$, "", "", "", "", "")
       If lv$ = "" Then lv$ = ev$
       For vrbl& = varnum&(ev$) To varnum&(lv$)
          'opsplitsen in variabelen tabel
         incr tv_teller&, 1
'        ReDim Preserve to_tabel(tv_teller&)
         to_tabel(tv_teller&) = vrbl&
       Next vrbl&

     Loop Until yyy > Len(to_reeks)




     If mv_teller& = tv_teller& Then

       If mv_teller& > 0 And tv_teller& > 0 Then
         For loopteller = 1 To UBound(move_tabel)
           mv = move_tabel(loopteller)
           tv = to_tabel(loopteller)


           maatpar$(tv) = maatpar$(mv)
         Next loopteller
       End If



     End If



     Erase move_tabel, to_tabel
   End If



End Sub

I suppose this can be done faster

Show quote Hide quote <snipped for brievity>


I'd have to wonder how your built those 'move" and 'to' strings.

As you may know, computers work better with numbers than strings.
For that reason, you might look into doing the copy work  when you
know what to copy, instead of storing the values to a string.

In any case, programming is all about breaking large tasks into smaller,
more managable tasks.

In your case I might have created a MoveData routine that does
the copying (something like):

Sub MoveData(ByVal Source&, ByVal Destination&, ByVal Length&)
Dim idx As Long
  If Destination > Source Then
    For idx = Length - 1 to 0 step -1
      maatpar(Destination + idx) = maatpar(Source + idx)
    Next
  ElseIf Destination < Source Then
    For idx = 0 to Length - 1
      maatpar(Destination + idx) = maatpar(Source + idx)
    Next
  End If
End Sub


Then, as soon as you know what needs to move (Ex  3000 to 6000 from
your example) you could call the move routine instead of storing those
values to a string.  In that way you bypass (or eliminate) all of the string
parsing process which you are currently using.

Even if you need to delay moving anything until you've completed some
other process, you could still store your values as numbers instead of strings:

Set Moves = New Collection
....  <processing>
Moves.Add Array(3000, 6000, 1)
....  <processing>
Moves.Add Array(20000, 70000, 10001)
.... etc...

The routine you posted would then become something more like:

For Each itm in Moves
  MoveData itm(0), itm(1), itm(2)
Next

As you can see, removing the string parsing greatly reduces the
code needed to do the task....

Does that help?

LFS

Thank you Larry,

Indeed, your way of doing is very interesting to me and I will dig into
that.

For information  what the process is (see dpb):

I am calculating values in a macro that is interpreted by my code.
In the macro's, I use numbered variables which contain the calculated
values. Thes values are numbers, strings etc. That 's why my array
maatpar() is declardes as String.
So when I call a macro to calculte values, I give it a range of
variables in which the results should be copied after calculting.
So, I can call the same macro's several times with another range of
variables as argument. The process stores the results in different
ranges whitout loosing the other ranges. Maybe this approach is a little
clumsy, but it is hard to change this mechanism now.

That's why I am optimising it as much as possible

Chris


Larry Serflaten wrote:
Show quoteHide quote

Chris D wrote:
....
I still fail to see the need to copy the inputs but it's your choice
whether to continue to do so or not.

Chris D wrote:
....
They always are... :)

....

I'm in full agreement w/ Larry's comment to keep the to/from indices as
numeric; the type of list is quite a common i/o "trick" and not
translating to/from/to numeric/character/numeric will help in code logic
even if not speed up the process too much since it's an overhead as
compared to the moving.

You've still not provided sufficient information on what the processing
is, but I'd still be looking at eliminating the need to make the move at
all as my first choice.

Karl's suggestion re: SafeArray is good inside VB; being a Fortran'er
first w/ VB as the front end mostly, I still like the idea of the DLL
personally.  Subarray syntax in modern Fortran could reduce the actual
code to a one liner eliminating the loops almost entirely and the
compiler optimizer could/would likely accomplish a memory move
automagically behind the scenes for you as well.

But, I reiterate--does the end result _have_ to be the subarray entities
in a contiguous array?

dpb wrote:
Heckuva good point.  Why not pass the array, a starting index, and a count?

dpb wrote:
I intended to note that if you were to create the list as ...3260 20000
-30000 32000 -33000 ... stored in an array or collection instead the
elements are already parsed individually and all the logic needs is to
note that a negative value is inclusive from the previous.  Much simpler
as Larry notes.

/CodgerAlert _WAY_ back in the dark ages when mainframes and punch cards
were all that existed, the comma-delimited list w/ the same
interpretation was quite a common input technique for large
computational programs in order to squeeze as much input as possible on
single card.  The input decks for these (nuclear design codes
specifically) programs could be as much as 4-5 full boxes of punch cards
anyway (1500 cards/box, roughly).  When first drum storage became
available and we modified the codes to read changes from a stored input
deck that seemed like pure magic and a blessed relief to the production
users/engineers. CodgerAlert/

Show quote Hide quote You haven't said what kind of variables your array contains, but in general
you can do it very much faster than looping. The CopyMemory API function is
just about as fast as you can get. It doesn't matter if the target area
overlaps the source area, or at which end it overlaps, because as far as I
recall CopyMemory is fairly intelligent in that respect and it will
automatically copy downwards or upwards as appropriate to avoid data overlap
problems. I see that Randem has just posted a link to some of his code, and
I'm sure that one of the functions it compares will be CopyMemory, but if
not then post again. Just make sure that you are copying the correct number
of bytes (the number of elements you need to shift multiplied by the byte
size of each element).

Mike

FWIW, an interesting read...

http://vb.mvps.org/hardcore/html/bringyourhatchet.htm

LFS

Show quote Hide quote I think you'll also find Williams is wrong about his claim overlapping
doesn't matter.  He is probably only testing DWord aligned structures or
values.

I think you'll find I'm correct, McCarthy, and that you are wrong.

Don't jump to conclusions, McCarthy, especially erroneous conclusions as you
are in the habit of doing. And if you want to talk to me then be a man for a
change and talk directly to me instead talking about me in the third person.
Here is an example of shifting odd numbers of bytes into various positions
in an array. There are two separate tests, which overlap in different ways,
the results of which should show you that I am correct and that you are
wrong. And the fact that it is a Byte array is irrelevant. As far as
rtlMoveMemory is concerned it is just a block of memory.

Option Explicit
Private Declare Sub CopyMemory Lib "kernel32" _
   Alias "RtlMoveMemory" (pDest As Any, _
    pSource As Any, ByVal dwLength As Long)

Private Sub Form_Load()
Me.Height = 8000
End Sub

Private Sub Command1_Click()
Dim b(1 To 255) As Byte, n As Long
For n = 1 To 255
  b(n) = n
Next n
Cls
Print "Initial contents"
For n = 1 To 15
  Print n, b(n)
Next n
CopyMemory b(3), b(5), 7
Print
Print "New Contents"
For n = 1 To 15
  Print n, b(n)
Next n
End Sub

Private Sub Command2_Click()
Dim b(1 To 255) As Byte, n As Long
For n = 1 To 255
  b(n) = n
Next n
Cls
Print "Initial contents"
For n = 1 To 15
  Print n, b(n)
Next n
CopyMemory b(6), b(3), 7
Print
Print "New contents"
For n = 1 To 15
  Print n, b(n)
Next n
End Sub

I think you'll find I'm correct, McCarthy, and that you are wrong.

Don't jump to conclusions, McCarthy, especially erroneous conclusions as you
are in the habit of doing. And if you want to talk to me then be a man for a
change and talk directly to me instead talking about me in the third person.
Here is an example of shifting odd numbers of bytes into various positions
in an array. There are two separate tests, which overlap in different ways,
the results of which should show you that I am correct and that you are
wrong. And the fact that it is a Byte array is irrelevant. As far as
rtlMoveMemory is concerned it is just a block of memory.

Option Explicit
Private Declare Sub CopyMemory Lib "kernel32" _
   Alias "RtlMoveMemory" (pDest As Any, _
    pSource As Any, ByVal dwLength As Long)

Private Sub Form_Load()
Me.Height = 8000
End Sub

Private Sub Command1_Click()
Dim b(1 To 255) As Byte, n As Long
For n = 1 To 255
  b(n) = n
Next n
Cls
Print "Initial contents"
For n = 1 To 15
  Print n, b(n)
Next n
CopyMemory b(3), b(5), 7
Print
Print "New Contents"
For n = 1 To 15
  Print n, b(n)
Next n
End Sub

Private Sub Command2_Click()
Dim b(1 To 255) As Byte, n As Long
For n = 1 To 255
  b(n) = n
Next n
Cls
Print "Initial contents"
For n = 1 To 15
  Print n, b(n)
Next n
CopyMemory b(6), b(3), 7
Print
Print "New contents"
For n = 1 To 15
  Print n, b(n)
Next n
End Sub

Oh how cute .. changing your posting name trying to get attention huh ??

Wanna bet ???


Ah huh, other than deal with FACTS you open with personal attacks.  that's
why I have you on blocked sender.


see above.

Ah huh... that's "rtlMoveMemory".  You said "CopyMemory".  CopyMemory API,
which is what you said is defined as :
http://msdn.microsoft.com/en-us/library/aa366535(VS.85).aspx

And *clearly* says :
If the source and destination blocks overlap, the results are undefined.

You know very well what I meant, McCarthy. I meant the function that is
called CopyMemory in the VB6 API Viewer and which is indexed as such and
which uses rtlMoveMemory, exactly as in the example code I posted. This is a
VB6 group McCarthy and everyone here will know exactly what I was talking
about. You knew very well what I was saying, but you chose to deliberately
interpret it wrongly in your pathetic attempt to cause trouble on this
group. If the OP uses his VB6 API Viewer or the API Viewer downloadable from
http://www.activevb.de and if he selects CopyMemory he will get exactly the
routine I was suggesting that he should use, and it will copy his data
precisely with no problems whatsoever regarding overlap. Larry Serflaten
knew exactly what I was talking about, as indicated by his own response, and
I believe that so did others here. It's only you who deliberately
interpreted it wrongly. You're a dick brain, McCarthy.

Why won't they let you back on the VB.Net group by the way? You've been
absent from it for many months, unless of course you're masquerading as
someone else there. Have you deserted VB.Net and have you come back to VB6?
Or are you just here with the intention of causing trouble? Dick brain.

You know very well what I meant, McCarthy. I meant the function that is
called CopyMemory in the VB6 API Viewer and which is indexed as such and
which uses rtlMoveMemory, exactly as in the example code I posted. This is a
VB6 group McCarthy and everyone here will know exactly what I was talking
about. You knew very well what I was saying, but you chose to deliberately
interpret it wrongly in your pathetic attempt to cause trouble on this
group. If the OP uses his VB6 API Viewer or the API Viewer downloadable from
http://www.activevb.de and if he selects CopyMemory he will get exactly the
routine I was suggesting that he should use, and it will copy his data
precisely with no problems whatsoever regarding overlap. Larry Serflaten
knew exactly what I was talking about, as indicated by his own response, and
I believe that so did others here. It's only you who deliberately
interpreted it wrongly. You're a dick brain, McCarthy.

Why won't they let you back on the VB.Net group by the way? You've been
absent from it for many months, unless of course you're masquerading as
someone else there. Have you deserted VB.Net and have you come back to VB6?
Or are you just here with the intention of causing trouble? Dick brain.

I know very well you were wrong as I pointed out.  You then claimed you mean
RtlMoveMemory, but that is not what you said.  As I pointed out to you the
documentation on CopyMemory is clear.  Sadly, your childish rude name
calling is all we can expect from you.  If you didn't resort to those
tactics this would have been a simple point of clarification, but no, you
want to wage some silly war.  Well good luck with that.  No doubt you'll
change your posting alias yet again like a little child starving for
attention, but one can hope you will grow up (one day)

Bill McCarthy wrote:
Show quoteHide quoteWe can only hope that Mike Williams soon retires from the VB "community"
and takes up self-flagellation as a hobby.

Jesus Christ! Another dick brain from Melbourne! That's three dick brains,
Jason Keats, Michael C and Bill McCarthy, all living within not more than a
few miles from each other in Melbourne in the great arid wasteland of
Australia. Must be something in the water!

Mike

No, he is correct.  In fact, VB's own Mid function uses MoveMemory
also.  You can copy a string to itself in overlapping fashion and the
low level loop iterator will either decrement toward the beginning,
or increment toward the end to make the correct move.

See for yourself:

Const tst = "123ABC"
Dim s As String

  s = tst
  Mid(s, 3, 3) = Mid(s, 1, 3)
  ' Moves: 3>B 2>A 1>3  (DEC)
  Debug.Print s  ' 12123C

  s = tst
  Mid(s, 3, 3) = Mid(s, 4, 3)
  ' Moves: 4>3 5>4 6>5  (INC)
  Debug.Print s  '12ABCC


As you see, the usage determines which way it will copy.

Then of course there is that handy Replicate routine that puts that
knowlege to use:

Public Function Replicate(ByVal Number&, Pattern$) As String
' LFS:  Replicate(3, "abc")   returns  "abcabcabc"

  If Number > 0 Then
    If Len(Pattern) = 1 Then
      Replicate = String$(Number, Pattern)
    Else
      If Number = 1 Then
        Replicate = Pattern
      Else
        Replicate = Pattern & Space$((Number - 1) * Len(Pattern))
        Mid$(Replicate, Len(Pattern) + 1) = Replicate
      End If
    End If
  End If

End Function



LFS

Show quote Hide quote See the link I posted before.  MoveMemory is very different from CopyMemory.
For CopyMemory it clearly states "If the source and destination blocks
overlap, the results are undefined. For overlapped blocks, use the
MoveMemory function."

That is hardly fair since 'the industry'  and you YOURSELF have declared
CopyMemory as an alias for RtlMoveMemory:

Public Declare Sub CopyMemory Lib "kernel32" Alias "RtlMoveMemory" _
       (Destination As Any, Source As Any, ByVal Length As Long)


http://groups.google.com/group/microsoft.public.vb.winapi.networks/browse_thread/thread/d4a6ea887cf1e3b6/236f62ac03d776f0?hl=en&lnk=gst&q=copymemory+alias#236f62ac03d776f0



LFS

*alias* being the keyword here.  No *alias* was defined, instead he said
"CopyMemory is fairly intelligent in that respect and it will
automatically copy downwards or upwards as appropriate to avoid data
overlap problems."

Most folks wouldn't notice the slightly lesser overhead of using the
CopyMemory API (memcpy) versus MoveMemory (memmove). The significant
difference is in the way they work around overlapped sections. So when one
says that CopyMemory works with overlapped sections they need to be clear
they are talking about an *alias* for MoveMemory, and don't mean the
CopyMemory function.   CopyMoery does not work for overlapped sections,
rather in that case it is undefined. The API documentation is clear on this.

http://msdn.microsoft.com/en-us/library/aa366535(VS.85).aspx

Show quote Hide quote                                         CopyMemory
(must have been overlapped there <g>)



Show quoteHide quote

Hi Chris,

you can use CopyMemory for structures or numeric values pretty simply by
just copying, eg:
Copymemory   array(1000), array(8000), 1000 * 4  'assuming values are long

For object references and strings you can actually use the same approach but
you also need to zero fill the original are so that the reference counting
remains correct.




Show quoteHide quote

Chris D wrote:
....

Another possible approach depending on what you're really doing would be
to write subroutine(s) in (say) Fortran and pass the starting array
position and do the work in the subroutine(s).

Since (altho not Standard-speak, for the purposes here it's good enough)
Fortran passes by value you could potentially eliminate the copy
entirely unless for some reason the original array really must be
rearranged.

For example...

Dim x(100000) as single
idx = 80000
call ftnsubx(x(idx))
....

subroutine ftnsubx(x)
   real x(1)
   integer i

   do i = 1, 20000
     z = operations on x(i)
   enddo
end

Inside ftnsubx the address of x(1) would be that of x(80000) in the VB
program.  The x(1) in the declaration is an idiom in older FORTRAN that
simply declares that x is an array and bounds checking is up to the user.

If such an option were to fit the problem, there are other more
sophisticated ways to use current features of Fortran that don't give up
bounds checking, etc., that could be used.  If interested repost and
could cobble up actual example.

_IF_ (the proverbial "big if") such rereferencing the array subsections
were to fit the problem well, it could as noted eliminate copy
operations entirely.

The unfortunate thing in VB is that afaik there's no way to pass array
references other than the whole array so can't do equivalent inside
"pure" VB.

Similar addressing tricks could, of course, be done in C as well as
Fortran but the addressing and similarity of Fortran syntax to VB makes
it a natural for writing DLLs to use w/ VB.

Anyway, a different approach.

dpb wrote:
That's the best option *if* the answer it offers applies, no doubt.

And it's pretty easy to do with SafeArray pointers.  You just create a new
SafeArray, and point to the element within the other that you want to be first in
line for the new one.  All there is to it.  :-)

That CSafeArray class Bill wrote is just the ticket, in this case.  It's available,
with a demo of using it to memory-map an entire file as a byte array, at
http://vb.mvps.org/samples/MapFile

PaintPicture
My concept for a very basic stripboard layouter in VB6
Put Date/Time on Right Side of Form.Caption
Option buttons become black - no text shown
Procedure limit for Form?
Where to store database login information in VB6
ActiveX Control Crashes - VB6
Convert statements from VB 2008 to 2003?
config file
Is one color brighter than the other one?