Flat files and ASP.NET

Discussion in 'ASP .Net' started by =?Utf-8?B?TmF2YW5lZXRoLksuTg==?=, Aug 30, 2007.

  1. Hello

    I am developing a system for user tracking. In this I am tracking all the
    visitors that came to my website. i developed this using a HTTPModule and
    storing the data in a SQL Server database. Everything worked fine till my
    website gets good traffic.

    When website is getting good traffic, this method is slowing down the entire
    application. So I am planning to use flat files instead of SQL Server
    database. My doubts are

    1 - What is a flat file and how can I connect to it from my ASP.NET
    application
    2 - If multiple users are browsing website at same time, how can I save
    data's to file, because that may be in use.
    3 - Is XML files are normal flat files ?

    Please help me on this issue.

    Sincerly
    Navaneeth
     
    =?Utf-8?B?TmF2YW5lZXRoLksuTg==?=, Aug 30, 2007
    #1
    1. Advertising

  2. =?Utf-8?B?TmF2YW5lZXRoLksuTg==?=

    marss Guest

    On 30 , 10:06, Navaneeth.K.N
    <> wrote:
    > Hello
    >
    > I am developing a system for user tracking. In this I am tracking all the
    > visitors that came to my website. i developed this using a HTTPModule and
    > storing the data in a SQL Server database. Everything worked fine till my
    > website gets good traffic.
    >
    > When website is getting good traffic, this method is slowing down the entire
    > application. So I am planning to use flat files instead of SQL Server
    > database. My doubts are
    >
    > 1 - What is a flat file and how can I connect to it from my ASP.NET
    > application
    > 2 - If multiple users are browsing website at same time, how can I save
    > data's to file, because that may be in use.
    > 3 - Is XML files are normal flat files ?
    >
    > Please help me on this issue.
    >
    > Sincerly
    > Navaneeth


    I have a doubt whether saving data to a file on the hard drive is
    faster then saving data in the database. Not to mention the speed of
    processing these data.

    Regards,
    Mykola
    http://marss.co.ua
     
    marss, Aug 30, 2007
    #2
    1. Advertising

  3. "Navaneeth.K.N" <> wrote in message
    news:...

    > So I am planning to use flat files instead of SQL Server database.


    ??? Surely not...

    > 2 - If multiple users are browsing website at same time, how can I save
    > data's to file, because that may be in use.


    You can't...

    > 3 - Is XML files are normal flat files ?


    XML files are text files, specifically designed as a lingua franca of data
    transmission between different systems - they should not be used as a
    substitute for a database...

    You need to look at optimising your database and how you are connecting to
    it - that's where the bottleneck is...


    --
    Mark Rae
    ASP.NET MVP
    http://www.markrae.net
     
    Mark Rae [MVP], Aug 30, 2007
    #3
  4. I suggest you stick with a database. Flat files will almost certainly not
    give you additional speed or scalability.
    Perhaps you could use some sort of a queue to save up log records and only
    write to the database every so often.

    --
    I hope this helps,
    Steve C. Orr,
    MCSD, MVP, CSM, ASPInsider
    http://SteveOrr.net



    "Navaneeth.K.N" <> wrote in message
    news:...
    > Hello
    >
    > I am developing a system for user tracking. In this I am tracking all the
    > visitors that came to my website. i developed this using a HTTPModule and
    > storing the data in a SQL Server database. Everything worked fine till my
    > website gets good traffic.
    >
    > When website is getting good traffic, this method is slowing down the
    > entire
    > application. So I am planning to use flat files instead of SQL Server
    > database. My doubts are
    >
    > 1 - What is a flat file and how can I connect to it from my ASP.NET
    > application
    > 2 - If multiple users are browsing website at same time, how can I save
    > data's to file, because that may be in use.
    > 3 - Is XML files are normal flat files ?
    >
    > Please help me on this issue.
    >
    > Sincerly
    > Navaneeth
     
    Steve C. Orr [MCSD, MVP, CSM, ASP Insider], Aug 30, 2007
    #4
  5. =?Utf-8?B?TmF2YW5lZXRoLksuTg==?=

    Aidy Guest

    Yes, the OP could look into MSMQ, however what levels of traffic are we
    talking here? If it isn't extremely heavy I can't understand why SQL would
    be causing a bottleneck. I'd look into what is causing the delay before
    looking for an alternative solution.

    "Steve C. Orr [MCSD, MVP, CSM, ASP Insider]" <> wrote in
    message news:...
    >I suggest you stick with a database. Flat files will almost certainly not
    >give you additional speed or scalability.
    > Perhaps you could use some sort of a queue to save up log records and only
    > write to the database every so often.
    >
    > --
    > I hope this helps,
    > Steve C. Orr,
    > MCSD, MVP, CSM, ASPInsider
    > http://SteveOrr.net
    >
    >
    >
    > "Navaneeth.K.N" <> wrote in message
    > news:...
    >> Hello
    >>
    >> I am developing a system for user tracking. In this I am tracking all the
    >> visitors that came to my website. i developed this using a HTTPModule and
    >> storing the data in a SQL Server database. Everything worked fine till my
    >> website gets good traffic.
    >>
    >> When website is getting good traffic, this method is slowing down the
    >> entire
    >> application. So I am planning to use flat files instead of SQL Server
    >> database. My doubts are
    >>
    >> 1 - What is a flat file and how can I connect to it from my ASP.NET
    >> application
    >> 2 - If multiple users are browsing website at same time, how can I save
    >> data's to file, because that may be in use.
    >> 3 - Is XML files are normal flat files ?
    >>
    >> Please help me on this issue.
    >>
    >> Sincerly
    >> Navaneeth

    >
     
    Aidy, Aug 30, 2007
    #5
  6. "Aidy" <> wrote in message
    news:...

    > If it isn't extremely heavy I can't understand why SQL would be causing a
    > bottleneck.


    Could be any number of reasons: poor design, no primary key / index,
    inefficient use of ADO.NET, not closing connections etc...


    --
    Mark Rae
    ASP.NET MVP
    http://www.markrae.net
     
    Mark Rae [MVP], Aug 30, 2007
    #6
  7. =?Utf-8?B?TmF2YW5lZXRoLksuTg==?=

    Aidy Guest

    > Could be any number of reasons: poor design, no primary key / index,
    > inefficient use of ADO.NET, not closing connections etc...


    That's what I meant, I don't see that it can be SQL Server's fault but
    something to do with the design/coding.

    BTW, why would not having an index *harm* insert performance? :p
     
    Aidy, Aug 30, 2007
    #7
  8. A flat file likely won't help you - ultimately your database does nothing
    more than write data to a file. XML is not he right approach either.

    You need to either change your approach to tracking data, using perhaps a
    hidden iframe in you pages that impacts a second web server instead of your
    primary - or look to scale out for load by adding web servers to create a
    farm.

    Ultimately, a fix is a design decision as your problem could be in the
    module rather than he fact that your talking to a database, you could try to
    spawn your write activities off to asynchronous threads and see if it
    reduces your problems or simply load your data into the application object
    and every 1000 requests or so write it to the DB.

    Regards

    John Timney (MVP)
    http://www.johntimney.com
    http://www.johntimney.com/blog


    "Navaneeth.K.N" <> wrote in message
    news:...
    > Hello
    >
    > I am developing a system for user tracking. In this I am tracking all the
    > visitors that came to my website. i developed this using a HTTPModule and
    > storing the data in a SQL Server database. Everything worked fine till my
    > website gets good traffic.
    >
    > When website is getting good traffic, this method is slowing down the
    > entire
    > application. So I am planning to use flat files instead of SQL Server
    > database. My doubts are
    >
    > 1 - What is a flat file and how can I connect to it from my ASP.NET
    > application
    > 2 - If multiple users are browsing website at same time, how can I save
    > data's to file, because that may be in use.
    > 3 - Is XML files are normal flat files ?
    >
    > Please help me on this issue.
    >
    > Sincerly
    > Navaneeth
     
    John Timney \(MVP\), Aug 30, 2007
    #8
  9. "Aidy" <> wrote in message
    news:...

    > BTW, why would not having an index *harm* insert performance? :p


    :)


    --
    Mark Rae
    ASP.NET MVP
    http://www.markrae.net
     
    Mark Rae [MVP], Aug 30, 2007
    #9
  10. =?Utf-8?B?TmF2YW5lZXRoLksuTg==?=

    Niels Ull Guest

    > Yes, the OP could look into MSMQ, however what levels of traffic are
    > we talking here? If it isn't extremely heavy I can't understand why
    > SQL would be causing a bottleneck. I'd look into what is causing the
    > delay before looking for an alternative solution.


    I definitely concur with analysing before fixing.

    However, it is very likely that a suitably buffered flat file writer will
    be faster than committing individual log entries to a database.

    Each thread could have its own buffered logger which only flushed to a common
    synchronized disk file logger when the buffer is full.
    If log entries are timestamped, the ordering of log messages are not important.

    Of course, this does run the risk of data loss in case of system failure.

    If it is undesirable to have even small waits when flushing to the synchronized
    logger, the logger could also be buffered with a separate thread to flush
    it to disk.

    MSMQ does seem like a bit overkill here, unlesss you really need transactional
    support, distribution etc.

    XML is not a very suitable format for log files, since they require all content
    to be wrapped in a single element. It would, however,
    be possible to have a log file containing lots of xml fragments.

    Good luck!

    Niels
     
    Niels Ull, Aug 30, 2007
    #10
  11. Hello,

    Using Application variables looks like a good solution to me. But what will
    be the performance issues where Application variable grows ? And I think I
    will loose unsaved data if any server crash happened
     
    =?Utf-8?B?TmF2YW5lZXRoLksuTg==?=, Aug 31, 2007
    #11
  12. Performance clearly depends on how much data you stick in the app object
    before you archive it off. You'll need to load test whatever your doing.

    You always run the risk of losing data in any application crash before its
    written to disk/database. What you need to work out is how many records are
    you prepared to lose and make sure your write events minimise the impact.
    So if you can only lose 10 records then dont wait until 100 to write to the
    database.

    Regards

    John Timney (MVP)
    http://www.johntimney.com
    http://www.johntimney.com/blog


    "Navaneeth.K.N" <> wrote in message
    news:...
    > Hello,
    >
    > Using Application variables looks like a good solution to me. But what
    > will
    > be the performance issues where Application variable grows ? And I think I
    > will loose unsaved data if any server crash happened
     
    John Timney \(MVP\), Aug 31, 2007
    #12
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. Homer
    Replies:
    230
    Views:
    10,882
    Roedy Green
    May 1, 2006
  2. MyClue
    Replies:
    5
    Views:
    3,003
    Oliver Wong
    Jul 5, 2006
  3. swordedotcom
    Replies:
    0
    Views:
    389
    swordedotcom
    Apr 23, 2006
  4. swordedotcom
    Replies:
    0
    Views:
    341
    swordedotcom
    Apr 23, 2006
  5. Stylus Studio
    Replies:
    0
    Views:
    558
    Stylus Studio
    Oct 26, 2007
Loading...

Share This Page