Google Adwords-like app with ASP.NET / Index Server

[thinkfr33ly]

We're implementing a "Google Adwords"-like service for an affiliate
program for one of our ASP.NET sites. (Google Adwords is the
advertising program where affiliates place iFrames on their site and
pass keywords to Google via the SRC attribute of the iFrame. Google
then uses those keywords to generate applicable results for the
contents of the iFrame.)

Our catalog will contain between 7,000 and 15,000 files that each
corrispond to a piece of content we offer. With the results we get back
from Index Server we can piece together the URLs/content for the
iFrame.

I'm currently using an HTTP Handler and the OleDB provider for Index
Server, along with a simple ExecuteReader command.

From my preliminary performance testing, it appears that we can

probably do around 750 RPS on each of our dual Zeon (3.2 Ghz) / 1.5 GB
of RAM Win2k3 web servers. This, however, pegs the CPUs. So,
realistically, we're looking at more like 350 to 450 RPS.

This isn't horrible, but I was hoping for more. The first thought I had
was to implement an output caching solution, but because we don't
control the input of keywords we would be creating a scenario where we
might be caching far too many combinations of keywords and essentially
opening ourselves up to a DoS attack. (Somebody could easily craft a
script that submitted millions of random keywords to our application,
thereby quickly filling up the memory cache.)

Does anybody have any suggestions for improving performance of a Index
Server based searching scenario like ours?

Thanks,
RMD

 

[clintonG]

You might consider evaluating Lucene ported to C# as open-source from Java.
I think its correct to say that Index Server has been a big failure and
Microsoft has chosen to continue to consider it as a bastard step child. I'd
surmise they do so because they have an entirely new stratgey that will be
available in future platform releases.


<%= Clinton Gallagher
METROmilwaukee (sm) "A Regional Information Service"
NET csgallagher AT metromilwaukee.com
URL http://metromilwaukee.com/
URL http://clintongallagher.metromilwaukee.com/

[1] http://www.dotlucene.net/

 

[thinkfr33ly]

Why do you say that Index Server is a big failure?

What limitations are you aware of?

By the way, I've switched from using the OleDB provider to the IXSSO
COM interop method, and this method is about 15% faster.

 

[clintonG]

AFIK while never testing it much myself it appears to be considered a
failure simply because there is little information to be found about using
it, developing solutions that require its use and it is nowhere to be
provided and supported by thje typical Windows hosting provider noting there
are likely exceptions but generally speaking the deafening silence speaks
loud enough for me.

<%= Clinton Gallagher

 

[thinkfr33ly]

I agree there are all that many articles on using Index Server, but it
appears to be a fairly good product.

I also like DotLucene, but it doesn't do things like word stemming,
which is extremely cool and helpful in my application.