|
code
newsgroups
|
|||||||||||||||||||||||
|
|||||||||||||||||||||||
Copy & Move filesI have an app that basically polls a folder looking for new files which can be updated at any time.When it finds relevant files, it copies them to another folder for further processing and then moves them to a back up folder. The three folders (poll, work & backup) can be on different computers (servers running Win 2K Advanced or Win 2003). When the process polls it gets a snapshot of the files currently there then processes each file, copying it to the work folder and then moving it t the backup folder. Files not having a predefined extension are ignored. This process has been working fine, except that a few day ago I got a report that the process of just doing the copy&move for 600 files (avarage size 2K) took 2 hours. And this was a batch of 6,000 files total. So now I am going to do some research and testing to see if I can speed up the process. I would like to know your opinions as to which would be a good strategy to accomplish this in the least amount of time possible. Thanks for your time and input. Saga How are you doing it now? Through VB command or the SHFileOp API?
-- Show quoteHide quote2025 If you do not believe in time travel, your beliefs are about to be tempered. http://www.facebook.com/group.php?gid=43606237254 "Saga" <antiSpam@nowhere.com> wrote in message news:uuslXoc9JHA.200@TK2MSFTNGP05.phx.gbl... | Hi all, I would like your opinion on the folowing issue. | | I have an app that basically polls a folder looking for new files which can | be updated at any time.When it finds relevant files, it copies them to | another folder for further processing and then moves them to a back up | folder. The three folders (poll, work & backup) can be on different | computers (servers running Win 2K Advanced or Win 2003). | | When the process polls it gets a snapshot of the files currently there then | processes each file, copying it to the work folder and then moving it t the | backup folder. Files not having a predefined extension are ignored. | | This process has been working fine, except that a few day ago I got a report | that the process of just doing the copy&move for 600 files (avarage size 2K) | took 2 hours. And this was a batch of 6,000 files total. | | So now I am going to do some research and testing to see if I can speed up | the process. I would like to know your opinions as to which would be a good | strategy to accomplish this in the least amount of time possible. Thanks for | your time and input. Saga | | | | The copy to the work folder is being done using FileCopy
The move to the backup folder is being done using FileCopy/Kill Initially I was using the Name x As y command, but it had some issue that I can't recall, so I rebuilt the process to use the FileCopy/Kill combonation. I also was using dir(sPathSpec) to get the initial file and dir() subsequently to get the next file in the poll folder, but this had some issues, so I changed the process to fill an array with the files found in the poll folder using FindFirstFile/FindNextFile/FindClose API calls. Thanks, Saga Show quoteHide quote "Kevin Provance" <Bill.McCarthy.Is.Stalking.TPASoft.com***@nowhere.edu> wrote in message news:OEr2Btc9JHA.5044@TK2MSFTNGP05.phx.gbl... > How are you doing it now? Through VB command or the SHFileOp API? > > -- > 2025 > If you do not believe in time travel, > your beliefs are about to be tempered. > > http://www.facebook.com/group.php?gid=43606237254 > "Saga" <antiSpam@nowhere.com> wrote in message > news:uuslXoc9JHA.200@TK2MSFTNGP05.phx.gbl... > | Hi all, I would like your opinion on the folowing issue. > | > | I have an app that basically polls a folder looking for new files which > can > | be updated at any time.When it finds relevant files, it copies them to > | another folder for further processing and then moves them to a back up > | folder. The three folders (poll, work & backup) can be on different > | computers (servers running Win 2K Advanced or Win 2003). > | > | When the process polls it gets a snapshot of the files currently there > then > | processes each file, copying it to the work folder and then moving it t > the > | backup folder. Files not having a predefined extension are ignored. > | > | This process has been working fine, except that a few day ago I got a > report > | that the process of just doing the copy&move for 600 files (avarage size > 2K) > | took 2 hours. And this was a batch of 6,000 files total. > | > | So now I am going to do some research and testing to see if I can speed > up > | the process. I would like to know your opinions as to which would be a > good > | strategy to accomplish this in the least amount of time possible. Thanks > for > | your time and input. Saga > | > | > | > | > > Hi Saga,
Show quoteHide quote "Saga" <antiSpam@nowhere.com> wrote in message Try using ShFileOp to copy them all at once.news:eu3NqOd9JHA.5040@TK2MSFTNGP04.phx.gbl... > The copy to the work folder is being done using FileCopy > The move to the backup folder is being done using FileCopy/Kill > > Initially I was using the Name x As y command, but it had > some issue that I can't recall, so I rebuilt the process to use the > FileCopy/Kill combonation. > > I also was using dir(sPathSpec) to get the initial file and dir() > subsequently to get the next file in the poll folder, but this had > some issues, so I changed the process to fill an array with the files > found in the poll folder using FindFirstFile/FindNextFile/FindClose > API calls. Thanks, Saga > http://vbnet.mvps.org/index.html?code/shell/shdirectorycopy.htm
http://vbnet.mvps.org/code/shell/shdirectorycopy.htm
That's actually the URL. I don't know if this is true for everyone, but when I use one of those server-side parameter URLs at Randy's site it just dumps me at the front page.
Show quote
Hide quote
"mayayana" <mayaXXy***@rcXXn.com> wrote in message It's true for everyone who has JavaScript disabled. The script is necessary news:Ofr7ZHh9JHA.5704@TK2MSFTNGP03.phx.gbl... > >> >> http://vbnet.mvps.org/index.html?code/shell/shdirectorycopy.htm >> > > http://vbnet.mvps.org/code/shell/shdirectorycopy.htm > > That's actually the URL. I don't know if this > is true for everyone, but when I use one > of those server-side parameter URLs at > Randy's site it just dumps me at the front > page. to insure that the left frame is visible when someone visits the page, if that user clicked on a search result for example. Thanks! I will add this suggestion to my tests. Saga
Show quoteHide quote "Bill McCarthy" <TPASoft.com Are Identity Thieves> wrote in message news:OP5zUJg9JHA.4168@TK2MSFTNGP05.phx.gbl... > Hi Saga, > > "Saga" <antiSpam@nowhere.com> wrote in message > news:eu3NqOd9JHA.5040@TK2MSFTNGP04.phx.gbl... >> The copy to the work folder is being done using FileCopy >> The move to the backup folder is being done using FileCopy/Kill >> >> Initially I was using the Name x As y command, but it had >> some issue that I can't recall, so I rebuilt the process to use the >> FileCopy/Kill combonation. >> >> I also was using dir(sPathSpec) to get the initial file and dir() >> subsequently to get the next file in the poll folder, but this had >> some issues, so I changed the process to fill an array with the files >> found in the poll folder using FindFirstFile/FindNextFile/FindClose >> API calls. Thanks, Saga >> > > Try using ShFileOp to copy them all at once. > > http://vbnet.mvps.org/index.html?code/shell/shdirectorycopy.htm > >
Show quote
Hide quote
"Saga" <antiSpam@nowhere.com> wrote in message Would shelling to xcopy or robocopy with the correct switches set work fornews:uuslXoc9JHA.200@TK2MSFTNGP05.phx.gbl... > Hi all, I would like your opinion on the folowing issue. > > I have an app that basically polls a folder looking for new files which > can be updated at any time.When it finds relevant files, it copies them to > another folder for further processing and then moves them to a back up > folder. The three folders (poll, work & backup) can be on different > computers (servers running Win 2K Advanced or Win 2003). > > When the process polls it gets a snapshot of the files currently there > then processes each file, copying it to the work folder and then moving it > t the backup folder. Files not having a predefined extension are ignored. > > This process has been working fine, except that a few day ago I got a > report that the process of just doing the copy&move for 600 files (avarage > size 2K) took 2 hours. And this was a batch of 6,000 files total. > > So now I am going to do some research and testing to see if I can speed up > the process. I would like to know your opinions as to which would be a > good strategy to accomplish this in the least amount of time possible. > Thanks for your time and input. Saga > > > > your application? It might be faster. John Thanks John. I had not really considered shelling out. I will
include it in my tests to determine its feasibility for this project. Again, thanks, Saga Show quoteHide quote "John Simpson" <jasimpNON@SPAMMEearthlink.net> wrote in message news:%23vliyJd9JHA.1376@TK2MSFTNGP02.phx.gbl... > > "Saga" <antiSpam@nowhere.com> wrote in message > news:uuslXoc9JHA.200@TK2MSFTNGP05.phx.gbl... >> Hi all, I would like your opinion on the folowing issue. >> >> I have an app that basically polls a folder looking for new files which >> can be updated at any time.When it finds relevant files, it copies them >> to another folder for further processing and then moves them to a back up >> folder. The three folders (poll, work & backup) can be on different >> computers (servers running Win 2K Advanced or Win 2003). >> >> When the process polls it gets a snapshot of the files currently there >> then processes each file, copying it to the work folder and then moving >> it t the backup folder. Files not having a predefined extension are >> ignored. >> >> This process has been working fine, except that a few day ago I got a >> report that the process of just doing the copy&move for 600 files >> (avarage size 2K) took 2 hours. And this was a batch of 6,000 files >> total. >> >> So now I am going to do some research and testing to see if I can speed >> up the process. I would like to know your opinions as to which would be a >> good strategy to accomplish this in the least amount of time possible. >> Thanks for your time and input. Saga >> >> >> >> > > Would shelling to xcopy or robocopy with the correct switches set work for > your application? It might be faster. > > John > "Saga" <antiSpam@nowhere.com> a écrit dans le message de news: uuslXoc9JHA.***@TK2MSFTNGP05.phx.gbl...Show quoteHide quote > Hi all, I would like your opinion on the folowing issue. In addition to the suggestions already posted I would like to add...> > I have an app that basically polls a folder looking for new files which > can be updated at any time.When it finds relevant files, it copies them to > another folder for further processing and then moves them to a back up > folder. The three folders (poll, work & backup) can be on different > computers (servers running Win 2K Advanced or Win 2003). > > When the process polls it gets a snapshot of the files currently there > then processes each file, copying it to the work folder and then moving it > t the backup folder. Files not having a predefined extension are ignored. > > This process has been working fine, except that a few day ago I got a > report that the process of just doing the copy&move for 600 files (avarage > size 2K) took 2 hours. And this was a batch of 6,000 files total. > > So now I am going to do some research and testing to see if I can speed up > the process. I would like to know your opinions as to which would be a > good strategy to accomplish this in the least amount of time possible. > Thanks for your time and input. Saga Win2K Server can be very slow with File Moves, it is often far faster (and safer with appropriate code) to do a "manual" move by copying then deleting the original. If the file extension is specific to your app, you might want to exclude it from any real-time anti-virus checking done on the source and destination computers. You may wish also to check the speed of doing the same operations with Windows Explorer - there could be a network problem somewhere that is slowing things down. Good luck Thanks for the info. The W2K FileMove issue is interesting. I'll
be taking into account all suggestions posted so far. Saga Show quoteHide quote "Clive Lumb" <clumb2@gratuit_en_anglais.fr.invalid> wrote in message news:u9%23ayqi9JHA.4176@TK2MSFTNGP02.phx.gbl... > > "Saga" <antiSpam@nowhere.com> a écrit dans le message de news: > uuslXoc9JHA.***@TK2MSFTNGP05.phx.gbl... >> Hi all, I would like your opinion on the folowing issue. >> >> I have an app that basically polls a folder looking for new files which >> can be updated at any time.When it finds relevant files, it copies them >> to another folder for further processing and then moves them to a back up >> folder. The three folders (poll, work & backup) can be on different >> computers (servers running Win 2K Advanced or Win 2003). >> >> When the process polls it gets a snapshot of the files currently there >> then processes each file, copying it to the work folder and then moving >> it t the backup folder. Files not having a predefined extension are >> ignored. >> >> This process has been working fine, except that a few day ago I got a >> report that the process of just doing the copy&move for 600 files >> (avarage size 2K) took 2 hours. And this was a batch of 6,000 files >> total. >> >> So now I am going to do some research and testing to see if I can speed >> up the process. I would like to know your opinions as to which would be a >> good strategy to accomplish this in the least amount of time possible. >> Thanks for your time and input. Saga > > In addition to the suggestions already posted I would like to add... > > Win2K Server can be very slow with File Moves, it is often far faster (and > safer with appropriate code) to do a "manual" move by copying then > deleting the original. > > If the file extension is specific to your app, you might want to exclude > it from any real-time anti-virus checking done on the source and > destination computers. > > You may wish also to check the speed of doing the same operations with > Windows Explorer - there could be a network problem somewhere that is > slowing things down. > > Good luck > > > Bad hard disk?
Show quoteHide quote "Saga" wrote: > Hi all, I would like your opinion on the folowing issue. > > I have an app that basically polls a folder looking for new files which can > be updated at any time.When it finds relevant files, it copies them to > another folder for further processing and then moves them to a back up > folder. The three folders (poll, work & backup) can be on different > computers (servers running Win 2K Advanced or Win 2003). > > When the process polls it gets a snapshot of the files currently there then > processes each file, copying it to the work folder and then moving it t the > backup folder. Files not having a predefined extension are ignored. > > This process has been working fine, except that a few day ago I got a report > that the process of just doing the copy&move for 600 files (avarage size 2K) > took 2 hours. And this was a batch of 6,000 files total. > > So now I am going to do some research and testing to see if I can speed up > the process. I would like to know your opinions as to which would be a good > strategy to accomplish this in the least amount of time possible. Thanks for > your time and input. Saga > > > > > Close :-)
First of all thanks to everyone who gave their input. I hope to follow through with all suggestions for future needs. The problem was found. When I "move" the file to the backup folder I first check to see whether a file with the same name already exists in the destination folder. We found that that folder already has 135,000 files in it, so it is taking about a minute to check for each file. The backup folder is cleaned out once per month (will be adjusted now<g>) and there are thousands of files coming through every day, so the checking routine was taking way too long. Thanks again, Saga Show quoteHide quote "Bee" <B**@discussions.microsoft.com> wrote in message news:E742F91D-EBC4-4BEB-9292-88FAE52D8343@microsoft.com... > > Bad hard disk? > > "Saga" wrote: > >> Hi all, I would like your opinion on the folowing issue. >> >> I have an app that basically polls a folder looking for new files which >> can >> be updated at any time.When it finds relevant files, it copies them to >> another folder for further processing and then moves them to a back up >> folder. The three folders (poll, work & backup) can be on different >> computers (servers running Win 2K Advanced or Win 2003). >> >> When the process polls it gets a snapshot of the files currently there >> then >> processes each file, copying it to the work folder and then moving it t >> the >> backup folder. Files not having a predefined extension are ignored. >> >> This process has been working fine, except that a few day ago I got a >> report >> that the process of just doing the copy&move for 600 files (avarage size >> 2K) >> took 2 hours. And this was a batch of 6,000 files total. >> >> So now I am going to do some research and testing to see if I can speed >> up >> the process. I would like to know your opinions as to which would be a >> good >> strategy to accomplish this in the least amount of time possible. Thanks >> for >> your time and input. Saga >> >> >> >> >> "Saga" <antiSpam@nowhere.com> wrote Might that be why IE has an index.dat file for its internet cache?> The problem was found. (...) We found that that folder already has 135,000 > files in it, so it is taking about a minute to check for each file. LFS Hi Larry, I am somewhat dense, since I did not understand how
IE having an index file for its internet cache is related to the issue that I documented. Or was that just a general observation? :-) Saga Show quoteHide quote "Larry Serflaten" <serflatenNON@SPAMMEusinternet.com> wrote in message news:OtbAV8R%23JHA.1380@TK2MSFTNGP02.phx.gbl... > > "Saga" <antiSpam@nowhere.com> wrote > >> The problem was found. (...) We found that that folder already has >> 135,000 >> files in it, so it is taking about a minute to check for each file. > > Might that be why IE has an index.dat file for its internet cache? > > LFS > > "Saga" <antiSpam@nowhere.com> wrote I was just thinking out loud. It might be quicker to read and parse> Hi Larry, I am somewhat dense, since I did not understand how > IE having an index file for its internet cache is related to the issue > that I documented. Or was that just a general observation? :-) a 4 MB file than to hit the directory each time you want to check for a file. That may be 'part' of the reason IE uses a file... Of course that is dependant on the code used. I would suppose the system would cache a directory making the API pretty quick it you went that route... LFS LFS Understood :-)
Yeah, agreed. For now the solution was to simply clear the backup folder more often. Saga Show quoteHide quote "Larry Serflaten" <serfla***@usinternet.com> wrote in message news:OzITAol%23JHA.5092@TK2MSFTNGP03.phx.gbl... > > "Saga" <antiSpam@nowhere.com> wrote >> Hi Larry, I am somewhat dense, since I did not understand how >> IE having an index file for its internet cache is related to the issue >> that I documented. Or was that just a general observation? :-) > > I was just thinking out loud. It might be quicker to read and parse > a 4 MB file than to hit the directory each time you want to check > for a file. That may be 'part' of the reason IE uses a file... > > Of course that is dependant on the code used. I would suppose > the system would cache a directory making the API pretty quick > it you went that route... > > LFS > > LFS > > |
|||||||||||||||||||||||
Other interesting topics