Poor performance with large datasets

Discussion in 'ASP .Net Web Services' started by Luther Miller, Oct 1, 2004.

  1. I have noticed that performance is very poor when a web service
    returns a large dataset.

    Further testing has shown that the XML version of the dataset is
    approximately 1.3GB in size - about 10x the binary representation of
    the data. It is the transfer of this data over the network that is
    causing the performance problem.

    We are pulling a large result set into an Excel application for pivot
    table processing.

    Connecting directly to the database using ADO takes only about 30
    seconds to finish retrieving all the data - presumably because the
    data is coming over efficiently in binary format.

    We also looked into using a 3rd party product to compress the XML
    dataset before returning it from the web service, but that too is
    prohibitively slow.

    How is Microsoft recommending that applications deal with large
    datasets? How are other people dealing with this issue (if at all)? I
    realize that .NET 2.0 is supposed to support binary data over web
    services, but this needs to be a 1.1 solution.
    Luther Miller, Oct 1, 2004
    #1
    1. Advertising

  2. Luther Miller

    Lee_Nover Guest

    > We also looked into using a 3rd party product to compress the XML
    > dataset before returning it from the web service, but that too is
    > prohibitively slow.


    check www.remobjects.com .. .NET SDK 3 is still in beta but you can get
    beta access if you ask nicely :)
    their binary format supports compression + encryption and is really fast
    I've used RO SDK for over a year (with delphi) and have no complaints .. :)
    current RO SDK 2 is only a client version and only supports their binary
    format
    contact them and I'm sure you can arrange something :)
    Lee_Nover, Oct 1, 2004
    #2
    1. Advertising

  3. Luther Miller

    Guest

    Sorry, this is not an answer.. but

    I too am facing the same problem.. Having tried various techniques, asp.net can't handle huge Datasets..It simply recycles... Now how Can I make my application work? I can't tell my people to buy .Net 2.0...

    My DataSets will generally contain 100,000 records with relations... Its killing ...

    I tried storing data in Session, Cache, other "Compact" ing techniques.. No use!!

    Can any expert help pleaseeeee......



    **********************************************************************
    Sent via Fuzzy Software @ http://www.fuzzysoftware.com/
    Comprehensive, categorised, searchable collection of links to ASP & ASP.NET resources...
    , Oct 21, 2004
    #3
  4. Luther Miller

    Dan Rogers Guest

    Hi Luther,

    I think the crux of the issue comes down to why are you deciding to
    implement this large-data-set requirement on a Web Service stack? When I
    think thru all of the distributed computing (WS is distributed computing)
    wisdom that I've encountered over the years, one of the repeating themes is
    "the farther away you are, the smaller your data transmission requirements
    should be". In a web service world, requests are likely to be more
    serviceable if you keep the payloads of the requests and responses small.

    If you find yourself needing to send large binary payloads, perhaps the
    answer is to send large binary payloads. You can use remoting, save to
    file and use things like FTP, or open direct streams for these kind of
    "batch transfers". With web services, if you are using the WS interface to
    transfer what amounts to reference copies of data, some might suggest that
    a different approach to managing the publication of large reference data
    sets is called for.

    Compression to binary will certainly reduce the number of packets required,
    and thus effect your transfer speed by a factor of the compression ratio.
    But compression adds several complicating factors. An easy way to get mass
    compression is to use some of the hardware solutions available at both ends
    of predictable high latency connections (I say high latency because of the
    end to end response time). In a controlled corporate network setting, or
    high volume commercial web infrastructures, this is economically feasible
    and eliminates the processing intensive and coding intensive overhead of
    both sides having to know some new compression protocol.

    Products from Netscalar, Cisco and F5 all can do what you want without
    adding any coding overhead to your network. Another option is faster
    network connections as well (gigabit ethernet, etc). I know these all may
    seem daunting to the project manager/budget minded, but they are a real
    part of the overall solution you should consider before you add in a
    complicated software fix. One real advantage is to the budget - these are
    typically capital budget items and not R&D expenses, so that added cost can
    be amoritized more rapidly.

    Hardware solutions to this issue also has the added benefit of degrading
    gracefully. Requests from clients that are outside of your corporate
    network (and thus who cannot make the efficient direct binary calls) can
    still come in thru an HTTP security gateway with a normal web service
    request. Yes, they will see large files, and the requisite effect of
    aggregate bandwidth and latency, but it'll work without having to code a
    special version of the application.

    I hope this adds some aspects to your trade-off matrix,

    Dan Rogers
    Microsoft Corporation
    --------------------
    >From: (Luther Miller)
    >Newsgroups: microsoft.public.dotnet.framework.aspnet.webservices
    >Subject: Poor performance with large datasets
    >Date: 1 Oct 2004 09:50:01 -0700
    >Organization: http://groups.google.com
    >Lines: 23
    >Message-ID: <>
    >NNTP-Posting-Host: 66.238.191.14
    >Content-Type: text/plain; charset=ISO-8859-1
    >Content-Transfer-Encoding: 8bit
    >X-Trace: posting.google.com 1096649401 13991 127.0.0.1 (1 Oct 2004

    16:50:01 GMT)
    >X-Complaints-To:
    >NNTP-Posting-Date: Fri, 1 Oct 2004 16:50:01 +0000 (UTC)
    >Path:

    cpmsftngxa06.phx.gbl!TK2MSFTNGXA03.phx.gbl!TK2MSFTNGP08.phx.gbl!newsfeed00.s
    ul.t-online.de!t-online.de!border2.nntp.dca.giganews.com!border1.nntp.dca.gi
    ganews.com!nntp.giganews.com!news.glorb.com!postnews1.google.com!not-for-mai
    l
    >Xref: cpmsftngxa06.phx.gbl

    microsoft.public.dotnet.framework.aspnet.webservices:25711
    >X-Tomcat-NG: microsoft.public.dotnet.framework.aspnet.webservices
    >
    >I have noticed that performance is very poor when a web service
    >returns a large dataset.
    >
    >Further testing has shown that the XML version of the dataset is
    >approximately 1.3GB in size - about 10x the binary representation of
    >the data. It is the transfer of this data over the network that is
    >causing the performance problem.
    >
    >We are pulling a large result set into an Excel application for pivot
    >table processing.
    >
    >Connecting directly to the database using ADO takes only about 30
    >seconds to finish retrieving all the data - presumably because the
    >data is coming over efficiently in binary format.
    >
    >We also looked into using a 3rd party product to compress the XML
    >dataset before returning it from the web service, but that too is
    >prohibitively slow.
    >
    >How is Microsoft recommending that applications deal with large
    >datasets? How are other people dealing with this issue (if at all)? I
    >realize that .NET 2.0 is supposed to support binary data over web
    >services, but this needs to be a 1.1 solution.
    >
    Dan Rogers, Nov 17, 2004
    #4
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. =?Utf-8?B?QmlsbCBUaG9ybmU=?=

    ASP.NET Poor performance with long-running COM object calls

    =?Utf-8?B?QmlsbCBUaG9ybmU=?=, Oct 12, 2004, in forum: ASP .Net
    Replies:
    4
    Views:
    657
    =?Utf-8?B?QmlsbCBUaG9ybmU=?=
    Nov 15, 2004
  2. news.microsoft.com
    Replies:
    0
    Views:
    435
    news.microsoft.com
    Apr 12, 2006
  3. Francisco Garcia
    Replies:
    2
    Views:
    467
    Fran Garcia
    Apr 13, 2006
  4. news.microsoft.com
    Replies:
    0
    Views:
    187
    news.microsoft.com
    Apr 12, 2006
  5. Francisco Garcia
    Replies:
    3
    Views:
    250
    vincent
    Apr 13, 2006
Loading...

Share This Page