Hyper-efficient Text Importing

[Awah Teh]

I am working on a project that involves importing IIS Log files into a SQL
Server database (and these logfiles are big --> Some up to 2GB in size).

Up until now I thought that DTS or the BULK INSERT command was the fastest
method out there, but still proves to be long (taking an average of 45mns to
an hour to process each log file).

Because I have to import the log files from three web heads in my cluster
(therefore 2GB log files per server times three servers), this really throws
off the concept of "real time" or near "real time" reporting.

So that begs the question. WHAT IS THE MOST EFFICIENT, AND QUICKEST
TEXT-FILE TO SQL SERVER IMPORTING TOOL, METHODOLOGY, SAMPLE CODE, WHATEVER,
OUT THERE???

Any help/lead will be greatly appreciated.

ps: I have seen some applications (such as Clicktracks 4.0 - which is a
logfile analyzer) gulp up to 2GB log files in less than 10 seconds so I know
there is faster and better than DTS or BULK IMPORT out there.

Thanks for your help in advance

--
Awah Teh
DigicentriQ Technologies, LLC
(e-mail address removed)
www.digicentriq.com
877 675 4742
805 732 9421

 

[Don Verhagen]

In Awah Teh <[email protected]> typed:
: I am working on a project that involves importing IIS Log files into
: a SQL Server database (and these logfiles are big --> Some up to 2GB

It seems the only groups you haven't cross-post to were the relevant ones,
try:

microsoft.public.sqlserver.*

This has nothing to do with ASP(.net). They will probably have the best
knowledge on how to bulk insert data from external sources (ie text/log
files).

Don

 

[William Tasso]

...
So that begs the question. WHAT IS THE MOST EFFICIENT, AND QUICKEST
TEXT-FILE TO SQL SERVER IMPORTING TOOL, METHODOLOGY, SAMPLE CODE,
WHATEVER, OUT THERE???

syncsort (sp?)