Generating "most popular article" lists without crushing the server? ASP.NET caching considerations?

[Guest]

The New York Times and many other online publications automatically generate
"most popular article" lists that cover, say, the last 24 hours.

I am looking for guidance and/or code on the best way to do this in ASP.NET
2 in a way that

a) won't crush the server
and
b) won't be surpressed by any of ASP.NET's caching options: you want the
"hits" to still be registered even if someone is clicking through to a
cached instance of the page.

What are the common solutions for dealing with this? Is it common to use a
stats package and operate on the data store that's generated by the package,
or is this overkill for this intended purpose?

SQL Server is available, and I have full control of the server.

-KF

 

[John Timney \(MVP\)]

This is the kind of data you want about every hour on a busy site, or every
half day on a less busy site. Ideally, you should be getting the data from
the webserver logs not from the database and analysing it that way. If I
remember correctly, even cached data generates a head request in the IIS log
files so you should be fine querying the that file, which certainly wont
crush your server - but you'll need to check.

Regards

John Timney (MVP)
VISIT MY WEBSITE:
http://www.johntimney.com
http://www.johntimney.com/blog

 

[Guest]

Thanks for the pointer, John. Interesting, I had never even thought of this
approach, nor have I seen it written about.

So you'd run a simple string analysis and count how many times files have
been hit?

Or maybe has someone written a class for interrogating log files. Does
anyone know of such a beast?

-KF

 

[John Timney \(MVP\)]

There is one, cant remember the name though but heres a couple of links to
get you started to prove the idea. Shouldn't take you too long.

http://awstats.sourceforge.net/
http://www.codeproject.com/aspnet/aspnetweblog.asp

Regards

John Timney (MVP)
VISIT MY WEBSITE:
http://www.johntimney.com
http://www.johntimney.com/blog

 

[Guest]

As an alternative to John's recommendation, you could try hooking the
Application_PreRequestHandlerExecute event in either global.asax or in an
HttpModule that handles the event. You can thus interrogate the request just
before it gets passed to the page processor, and log all the data you want
into your Sql Server.
Peter

 

[Guest]

Thank you. Here is an article that discusses the tactic you describe:
http://www.superdotnet.com/Article.aspx?ArticleID=122

They're hooking Application_BeginRequest in global.asax rather than
Application_PreRequestHandlerExecute as you suggested; I wonder if there's a
material difference.

I'm a little surprised nobody has written an all-knowing, all-seeing class
and stored procedure set for addressing site statistic issues. For my needs,
I'm finding after-the-fact log analysis less appealing than the idea of a
SQL database I could interrogate.

Some other options: "Log Parser" allows SQL-type queries against any most
any windows log.
http://www.asp.net/sandbox/app_logparser.aspx
I'm a little disappointed the package doesn't have much in the way of
instructions or example for how to get up and running working with IIS. In
theory, this could be a great solution for my needs.

I'm also surprised that many of the stats packages don't seem to expose
their report data in any sort of structured form, say, in XML. You're stuck
either "rolling your own" solution and coding everything, or working with
packages that seem more directed at Desktop users. Not happy with any of the
prebuilt solutions that I can afford.

-KF

 

[John Timney \(MVP\)]

Not happy with any of the prebuilt solutions that I can afford.

....and therein lies the problem. Its a complex subject, and experience
around these areas tends to be expensive as its more niche than mainstream
forms development for example, which is why your struggling to find an all
seeing class. Unfortunately there many ways to crack the same nut and the
pre-built nutcrackers that seem to work are not cheap for all of those
reasons. Whatever solution you turn to - make sure the solution itself
doesn't degrade your web servers performance.

--
--
Regards

John Timney (MVP)
VISIT MY WEBSITE:
http://www.johntimney.com
http://www.johntimney.com/blog

 

[Guest]

Thanks, John. I've identified one another option: as of version 8, SurfStats
(a traffic analysis program) will export reports in Excel format, which I
could presumably interrogate with my ASP.NET app. We bought a license for
SurfStats, so this may be a good choice for me.

Microsoft should sell a good analysis program, or bundle it with IIS. It
seems like a basic need for people who may run sites.

-KF

 

[Juan T. Llibre]

re:

Microsoft should sell a good analysis program, or bundle it with IIS. It seems like a basic need
for people who may run sites.

Microsoft recently (a few months ago) acquired Livestats from Deepmetrix:
http://www.deepmetrix.com/.
I doubt it will be bundled for free with IIS, but check them out.

I've been using SmarterStats free version:
http://www.smartertools.com/Products/SmarterStats/Free.aspx

I understand their paid-for version is quite complete.