|
code
newsgroups
|
|||||||||||||||||||||||
|
|||||||||||||||||||||||
Download multiple files concurrently using WinHttpRequestI did ask already in the german NG but did not find any one so far who knows the WinHttpRequest object (WHR in short for now). The WHR wraps the APIs in the Winhttp.dll. In my app I use it to download up to about 60 files asynchronously and concurrently. I create as much WHR object instances as needed, start for each instance the download via the Send method and wait until the data comes in, caught in the OnResponseFinished event of the objects. Then I destroy the objects. Works well and the GUI of the app stays responsive during download. However viewing the TCP site of my app, using SysInternals TCPView, following happens: - for each downloading object used in parallel a connection to the server is created. - after the downloads finished and the objects are destroyed, the connections are still there and stay there until the app is closed. - further downloads by creating in parallel and using for downloading other, newly created objects seem to reuse the existing connections, as no new connections are created, as long as the total number of newly created objects running in parallel does not exceed the total number of parallel running objects used in an earlier run. If eg. in a first run 6 objects are created, run in parallel and destroyed, 6 connections are created and stay there, even when the objects are destroyed. If then afterwards a run with 3 objects is started, no new connections are created, and there are still 6 connections. If then another run with eg. 8 objects is started, 2 new connections are created, and the now total 8 connections stay there after the run. If then all other runs following never would use more than 8 objects, the number of 8 connections stays at 8. So the total number of open connections never exceeds the greatest number of all objects used during the "biggest" run. But, as already told, as soon as connections have been opened, they never are closed until the app closes down. I wonder wether this (having open connections that live until the app dies) is ok. Could those open connections in any way hinder - the server by forcing him to hold connections alive? - other apps on the PC the app is running on? - the OS itself? With hindering I mean eating up to many ressources that then might be missing for other apps or the servers work, or slowing down delivering of data of the server or in general any negative influence that would be noticeable. If there are any objections against having such much connections open for a considerable long time (the app may run for hours, but not necessarily download data all the time), what should I do? Am I missing something in serving the objects that could stop this behavior?
Show quote
Hide quote
"Ulrich Korndoerfer" <ulrich_wants_nospam@prosource.de> wrote in message Are you calling WinHttpCloseHandle() when done?news:uXzGQ9h7JHA.356@TK2MSFTNGP03.phx.gbl... > Hi, > > I did ask already in the german NG but did not find any one so far who > knows the WinHttpRequest object (WHR in short for now). > > The WHR wraps the APIs in the Winhttp.dll. In my app I use it to download > up to about 60 files asynchronously and concurrently. I create as much WHR > object instances as needed, start for each instance the download via the > Send method and wait until the data comes in, caught in the > OnResponseFinished event of the objects. Then I destroy the objects. > > Works well and the GUI of the app stays responsive during download. > However viewing the TCP site of my app, using SysInternals TCPView, > following happens: > > - for each downloading object used in parallel a connection to the server > is created. > > - after the downloads finished and the objects are destroyed, the > connections are still there and stay there until the app is closed. > > - further downloads by creating in parallel and using for downloading > other, newly created objects seem to reuse the existing connections, as no > new connections are created, as long as the total number of newly created > objects running in parallel does not exceed the total number of parallel > running objects used in an earlier run. > > If eg. in a first run 6 objects are created, run in parallel and > destroyed, 6 connections are created and stay there, even when the objects > are destroyed. If then afterwards a run with 3 objects is started, no new > connections are created, and there are still 6 connections. If then > another run with eg. 8 objects is started, 2 new connections are created, > and the now total 8 connections stay there after the run. If then all > other runs following never would use more than 8 objects, the number of 8 > connections stays at 8. > > So the total number of open connections never exceeds the greatest number > of all objects used during the "biggest" run. But, as already told, as > soon as connections have been opened, they never are closed until the app > closes down. > > I wonder wether this (having open connections that live until the app > dies) is ok. Could those open connections in any way hinder > > - the server by forcing him to hold connections alive? > - other apps on the PC the app is running on? > - the OS itself? > > With hindering I mean eating up to many ressources that then might be > missing for other apps or the servers work, or slowing down delivering of > data of the server or in general any negative influence that would be > noticeable. > > If there are any objections against having such much connections open for > a considerable long time (the app may run for hours, but not necessarily > download data all the time), what should I do? Am I missing something in > serving the objects that could stop this behavior? Servers operators do not like keeping connections open without using them. They use some resources. They most likely ban the IP that these connections are coming from. Hi,
Nobody schrieb: >> If there are any objections against having such much connections open for Good to know. However I use the WinHttpRequest COM object, which does >> a considerable long time (the app may run for hours, but not necessarily >> download data all the time), what should I do? Am I missing something in >> serving the objects that could stop this behavior? > > Are you calling WinHttpCloseHandle() when done? > > Servers operators do not like keeping connections open without using them. > They use some resources. They most likely ban the IP that these connections > are coming from. not offer a CloseHandle method. WinHttpCloseHandle is an API method. Normally, when tearing down an object, it should release its ressources too, including those adressed by handles. This is the reason why I create and destroy for every download a WinHttpRequest object. But seemingly the WinHttpRequest object does not do this. I think I even can not get this handle from the object, so I can not resort to free it before destroying the object. And if I could get this handle from the object, it probably would not be good to mix object driven access and API calls. Besides Larry's suggestion, check these articles for WinInet based
approaches: SAMPLE: Vbhttp.exe Demonstrates How to Use HTTP WinInet APIs in Visual Basic http://support.microsoft.com/kb/259100/en-us SAMPLE: VBFTP.EXE: Implementing FTP Using WinInet API from VB http://support.microsoft.com/kb/175179/en-us SAMPLE: Using FTP WinInet APIs in Visual Basic with SimpleFtp http://support.microsoft.com/kb/195653/en-us Hi,
Nobody schrieb: > Besides Larry's suggestion, check these articles for WinInet based Thanks for the links. IIRC I've had already a glance on this sample > approaches: > > SAMPLE: Vbhttp.exe Demonstrates How to Use HTTP WinInet APIs in Visual Basic > http://support.microsoft.com/kb/259100/en-us > > SAMPLE: VBFTP.EXE: Implementing FTP Using WinInet API from VB > http://support.microsoft.com/kb/175179/en-us > > SAMPLE: Using FTP WinInet APIs in Visual Basic with SimpleFtp > http://support.microsoft.com/kb/195653/en-us > > projects, especially the first one. The problem was that it was not possible or either to complicated to get many downloads in parallel in a non blocking way. WHR makes it really easy to have asynchronous, concurrent downloads, that execute outside the apps main thread and leave this thread during downloading (except for the short moments when data arrives) alone, so allowing the apps GUI to stay responsive during downloads. Also the amount of control the WHR allows is sufficient for me, especially now when I learned about the mysteries of Http protocol headers and how to use them with the WHR :-) "Ulrich Korndoerfer" <ulrich_wants_nospam@prosource.de> wrote I didn't know the WHR either, but the web is a fatastic place to learn new things....> > I did ask already in the german NG but did not find any one so far who > knows the WinHttpRequest object (WHR in short for now). <snipped for brievity> > - for each downloading object used in parallel a connection to the Apparently HTTP 1.1 specification has a way to indicate you want the> server is created. > > - after the downloads finished and the objects are destroyed, the > connections are still there and stay there until the app is closed. ...l. > If there are any objections against having such much connections open > for a considerable long time (the app may run for hours, but not > necessarily download data all the time), what should I do? Am I missing > something in serving the objects that could stop this behavior? connection closed after use. See: http://www.ietf.org/rfc/rfc2616.txt (Section 14-10) eg MSG HEADER; Connection : Close Apply that to the WHR SetRequestHeader method prior to your call to Send: WHR.Open (...) WHR.SetRequestHeader "Connection", "Close" WHR.Send I haven't tried it, so I won't know if that is what you need. Another route might be to wait until you get your data, then set the header and call Send again. See if that helps and do read the linked section. It says once its closed, don't use it, but with the WHR I wonder if you have full control... ??? Good luck! LFS Hi Larry,
Larry Serflaten schrieb: Show quoteHide quote > "Ulrich Korndoerfer" <ulrich_wants_nospam@prosource.de> wrote Apparently this seems to do the trick! I just inserted your suggested >> I did ask already in the german NG but did not find any one so far who >> knows the WinHttpRequest object (WHR in short for now). > > I didn't know the WHR either, but the web is a fatastic place to learn new things.... > > > <snipped for brievity> > >> - for each downloading object used in parallel a connection to the >> server is created. >> >> - after the downloads finished and the objects are destroyed, the >> connections are still there and stay there until the app is closed. > ...l. >> If there are any objections against having such much connections open >> for a considerable long time (the app may run for hours, but not >> necessarily download data all the time), what should I do? Am I missing >> something in serving the objects that could stop this behavior? > > > Apparently HTTP 1.1 specification has a way to indicate you want the > connection closed after use. > > See: http://www.ietf.org/rfc/rfc2616.txt (Section 14-10) > eg MSG HEADER; Connection : Close > > Apply that to the WHR SetRequestHeader method prior to your call to Send: > > WHR.Open (...) > WHR.SetRequestHeader "Connection", "Close" > WHR.Send > > I haven't tried it, so I won't know if that is what you need. Another route > might be to wait until you get your data, then set the header and call Send again. > > See if that helps and do read the linked section. It says once its closed, don't use > it, but with the WHR I wonder if you have full control... ??? line as shown, and files got downloaded as fast as before, but TCPView shows connections just flashing up and vanishing then. Many thanks, I will read a little bit more and then probably will stay at this solution. If any problems arouse, I will come back later :-) Ulrich Korndoerfer schrieb:
>> See if that helps and do read the linked section. It says once its The rfc states:>> closed, don't use >> it, but with the WHR I wonder if you have full control... ??? HTTP/1.1 defines the "close" connection option for the sender to signal that the connection will be closed after completion of the response. So its main purpose is to tell the *receiver* that the connection will not be used any more after the request finished. To close the connection on the sender's side the sender has to do this. Apparently the WHR interprets the request headers: - WHR sees a "connection:close" part in the request headers - WHR transmits the request - because it was told to tell the receiver that this connection will be closed after completion of the request, it also closes the connection after the completion of the request. So by setting this request header one is sending information to the receiver (this is what the headers are for) *and* instructs the WHR to close the connection afterwards, such controlling both the receiver and the sender! Funny. Btw, I just had a look how Firefox behaves. Firefox also creates many persistent connections. But after some time, when the connections were not used, he kills them one by one. May be I should go this route too: - create persistent connections - after some idle time kill them I know now how to create and kill afterwards immediately. The question is: how to create and kill later? Will have to do some digging, perhaps I find the answer in the docs for the API calls. "Ulrich Korndoerfer" <ulrich_wants_nospam@prosource.de> wrote You described the 'persistant' behaviour in your first post. That you create> I know now how to create and kill afterwards immediately. The question > is: how to create and kill later? Will have to do some digging, perhaps > I find the answer in the docs for the API calls. and destroy WHR objects, but they re-use previous connections if present. It would seem to me, if you keep track of how many simultanious connections you create. You could create and destroy your WHR's as before and when you want to destroy a connection you create a new WHR open for HEAD (Not Get, Post, or Put) with the connection:close header info. When that is sent you destroy that object and decrement the connection count you've been accumulating. (Something to try) FWIW that RFC also indicated that clients should typically not create more than 2 connections to a server to avoid congestion. You might limit your own use to some low number of simultanious connections. LFS "Ulrich Korndoerfer" <ulrich_wants_nospam@prosource.de> wrote If you read that somewhere that may be, but this may also be a result of> So its main purpose is to tell the *receiver* that the connection will > not be used any more after the request finished. To close the connection > on the sender's side the sender has to do this. Apparently the WHR > interprets the request headers: the client monitoring the connection and when it sees the server close the connection, the client also closes. See section 8.1.4 : "Clients and servers SHOULD both constantly watch for the other side of the transport close, and respond to it as appropriate." LFS 
Other interesting topics
When the compiler freezes.... what is the rescue?
How to Change $(AppPath) in PDWizard? Karl, I need to understand your Timer! RAM Drive Run-Time Error 5 - 'Invalid procedure call or argument' error msg Overcoming a MsgBox's shyness File not found How to obtain the new GUIID key on a new build of an ActiveX DLL or an ActiveX OCX Autosize DataGrid Columns How to create a VB6 app that opens a file when the file is double |
|||||||||||||||||||||||
