What is the fastest way to deliver a "pass-through" document?

Discussion in 'ASP .Net' started by Benjamin Joldersma, Jan 20, 2004.

  1. Hello all,

    My company is installing a Bluecoat caching machine to help serve some
    of our dynamic content. This is fine for free documents, but we have
    to make a minor architectural tweak for the documents that require

    Our solution is to run the requests through an HttpModule that
    determines if the request is cacheable, and the user is authorized to
    view it (if the document is not cacheable, it is just processed
    regularly and bypasses the bluecoat machine.) Assuming the user is
    authorized, the HttpModule generates a System.Net.WebClient instance,
    and requests the document from the cache machine, which manages the
    private origin servers. We then take the stream from the OpenRead
    method, and ReadToEnd directly into the Response buffer.

    Preliminary tests seem reasonable, but I am in my gut a bit concerned
    about scalablity.

    Is this the fastest way we can retrieve a document? Would it be
    faster/ more performant to open a socket ourselves and avoid the
    overhead of the WebClient class?

    Is there a better way to transfer whole documents from a private
    server to a public server, a la Response.WriteFile, but via a URI
    instead of a physical file path?

    Thanks in advance for any feedback.

    Benjamin Joldersma,
    Sr. Software Engineer,
    Citadel Media, Inc.
    Benjamin Joldersma, Jan 20, 2004
    1. Advertisements

  2. Hi Benjamin,

    Thank you for using MSDN Newsgroup! My name is Steven, and I'll be
    assisting you on this issue.
    From your description, you're developing an ASP.NET web application in
    which you need to transfer a document from a private server to another
    public server. And currently you are using the webclient class to perform
    the transfering function, so you'd like to look for some suggestions on
    whether there're some other means to implement this or the strong points
    and weak points of each?
    If there is anything I misunderstood, please feel free to let me know.

    Based on my experience, since the webclient class itself just encapsulate
    the using of the socket and http layer communication, so the efficiency or
    performance is hard to conclude simply. The result may be different under
    all kinds of conditions. But as for the convenience and higher abstracted,
    I recommend that you use the webclient class rather than manually implement
    yourself using socket programming. That'll save alot of time, do you think
    so? Also, I think you may have a detailed test on them if you think the
    runtime condition is not very hard to estimate in a certain range.

    In addition, since you use the webclient class to transfer a document
    between two different server. Are they are in the same intranet, is there
    any net bandwidth problem with the network between them? If the bandwidth
    is ok ,I think you may consider the ASP.NET XML Webservice, it is also a
    very wonderful feature on distribute operations. And the webservice provide
    a light weight component for implement may newwork computing functionality.
    Here is tech references on the XML webservice for DOTNET in MSDN:

    #Getting Started with XML Web Services in Visual Basic.NET and Visual C#

    #Remote Scripting in a .NET World

    #How ASP.NET Web Services Work

    #Large Data Strategies

    You may have a look to see whether they'll be helpful to you. In the mean
    time, if you need any further assistance, please feel free to let me know.


    Steven Cheng
    Microsoft Online Support

    Get Secure! www.microsoft.com/security
    (This posting is provided "AS IS", with no warranties, and confers no
    Steven Cheng[MSFT], Jan 21, 2004
    1. Advertisements

  3. Steven

    Thanks so much for your response. Your suggestion about the relationship between the WebClient class and the socket layer was reassuring

    We've since put out the code and tested it against live traffic. Mixed results. The good news is that it uses very little CPU (which makes sense), about 1% or so. The retrieval time is very quick, just a few milliseconds, sometimes it doesn't even register

    The problem we are having though, is that there seems to be a bottleneck. We can only get to around 200 - 250 connections on the machine, before it the queue starts oscillating pretty wildly, but the CPU still stays low

    We've tried to change the ServicePoint ConnectionLimit for the Cache Host IP to 100, and close / dispose the WebClient objects as soon as possible. But I'm not sure how to check to ensure that the new connection limit is taking - the ServicePoint classes are a bit confusing, and I can't find any examples that relate them to the WebClient (or WebRequest/Response) class(es.

    Do you just have to call ServicePointManager.FindServicePoint( "Cache-Ip", null ); - and then set the ConnectionLimit once, and that is it? Could we set it to a very large number, like 2,000 or 3,000? During our big days we run about 3,000 - 3,500 connections per box, so 200 - 250 is still too low

    thanks again for the help

    =?Utf-8?B?QmVuamFtaW4gSm9sZGVzbWE=?=, Jan 29, 2004
  4. Okay, I think we are looking good, I found this article (very helpful) http://msdn.microsoft.com/webservic...ary/en-us/dnbda/html/bdadotnetarch14.asp?_r=1

    In the article, they mention the System.Net configuration setting --
    <system.net><connectionManagement><add address="*"

    We put this in our machine.config, and now we are rocking full on - we are at 800 connections, and hoping to go much higher.

    thanks for the help,

    =?Utf-8?B?QmVuamFtaW4gSm9kZXJzbWE=?=, Jan 29, 2004
  5. Hi Ben,

    Thanks for your followup. I'm very glad that my suggestions has helped you.
    Also, I've viewed the
    <connectionManagement><add address="*" maxconnection="100"

    setting, it looks so cool :). In addtion, as for the potential bottleneck
    of the high connections requirement, would you consider implement a pooling
    mechanism on the connections? Just like what is widely used for many other
    objects, for example, database connection pool , thread pool, .... In the
    meantime, if you've got any good ideas , please feel free to post here. I
    believe that'll be very helpful to many other community customers. Thanks.


    Steven Cheng
    Microsoft Online Support

    Get Secure! www.microsoft.com/security
    (This posting is provided "AS IS", with no warranties, and confers no
    Steven Cheng[MSFT], Jan 30, 2004
    1. Advertisements

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. Miguel Dias Moura
    Cor Ligthert
    Apr 12, 2004
  2. Ken Dopierala Jr.
    Ken Dopierala Jr.
    Jun 16, 2005
  3. Replies:
  4. Matt Clara
    Mark Parnell
    Dec 9, 2004
  5. =?Utf-8?B?UGV0ZXI=?=

    Deliver video files - best way of practice?

    =?Utf-8?B?UGV0ZXI=?=, Sep 26, 2007, in forum: ASP .Net
    Sep 26, 2007