Poor performance with large datasets

[Luther Miller]

I have noticed that performance is very poor when a web service
returns a large dataset.

Further testing has shown that the XML version of the dataset is
approximately 1.3GB in size - about 10x the binary representation of
the data. It is the transfer of this data over the network that is
causing the performance problem.

We are pulling a large result set into an Excel application for pivot
table processing.

Connecting directly to the database using ADO takes only about 30
seconds to finish retrieving all the data - presumably because the
data is coming over efficiently in binary format.

We also looked into using a 3rd party product to compress the XML
dataset before returning it from the web service, but that too is
prohibitively slow.

How is Microsoft recommending that applications deal with large
datasets? How are other people dealing with this issue (if at all)? I
realize that .NET 2.0 is supposed to support binary data over web
services, but this needs to be a 1.1 solution.

 

[Lee_Nover]

We also looked into using a 3rd party product to compress the XML

dataset before returning it from the web service, but that too is
prohibitively slow.

check www.remobjects.com .. .NET SDK 3 is still in beta but you can get
beta access if you ask nicely
their binary format supports compression + encryption and is really fast
I've used RO SDK for over a year (with delphi) and have no complaints ..
current RO SDK 2 is only a client version and only supports their binary
format
contact them and I'm sure you can arrange something

 

[mohan]

Sorry, this is not an answer.. but

I too am facing the same problem.. Having tried various techniques, asp.net can't handle huge Datasets..It simply recycles... Now how Can I make my application work? I can't tell my people to buy .Net 2.0...

My DataSets will generally contain 100,000 records with relations... Its killing ...

I tried storing data in Session, Cache, other "Compact" ing techniques.. No use!!

Can any expert help pleaseeeee......



**********************************************************************
Sent via Fuzzy Software @ http://www.fuzzysoftware.com/
Comprehensive, categorised, searchable collection of links to ASP & ASP.NET resources...

 

[Dan Rogers]

Hi Luther,

I think the crux of the issue comes down to why are you deciding to
implement this large-data-set requirement on a Web Service stack? When I
think thru all of the distributed computing (WS is distributed computing)
wisdom that I've encountered over the years, one of the repeating themes is
"the farther away you are, the smaller your data transmission requirements
should be". In a web service world, requests are likely to be more
serviceable if you keep the payloads of the requests and responses small.

If you find yourself needing to send large binary payloads, perhaps the
answer is to send large binary payloads. You can use remoting, save to
file and use things like FTP, or open direct streams for these kind of
"batch transfers". With web services, if you are using the WS interface to
transfer what amounts to reference copies of data, some might suggest that
a different approach to managing the publication of large reference data
sets is called for.

Compression to binary will certainly reduce the number of packets required,
and thus effect your transfer speed by a factor of the compression ratio.
But compression adds several complicating factors. An easy way to get mass
compression is to use some of the hardware solutions available at both ends
of predictable high latency connections (I say high latency because of the
end to end response time). In a controlled corporate network setting, or
high volume commercial web infrastructures, this is economically feasible
and eliminates the processing intensive and coding intensive overhead of
both sides having to know some new compression protocol.

Products from Netscalar, Cisco and F5 all can do what you want without
adding any coding overhead to your network. Another option is faster
network connections as well (gigabit ethernet, etc). I know these all may
seem daunting to the project manager/budget minded, but they are a real
part of the overall solution you should consider before you add in a
complicated software fix. One real advantage is to the budget - these are
typically capital budget items and not R&D expenses, so that added cost can
be amoritized more rapidly.

Hardware solutions to this issue also has the added benefit of degrading
gracefully. Requests from clients that are outside of your corporate
network (and thus who cannot make the efficient direct binary calls) can
still come in thru an HTTP security gateway with a normal web service
request. Yes, they will see large files, and the requisite effect of
aggregate bandwidth and latency, but it'll work without having to code a
special version of the application.

I hope this adds some aspects to your trade-off matrix,

Dan Rogers
Microsoft Corporation
--------------------