inputing, paging, sorting, a large text file

Discussion in 'ASP .Net' started by JJ, Jun 7, 2007.

  1. JJ

    JJ Guest

    I have a need to input a large tab delimited text file, which I will parse
    to check it has the expected columns, before allowing the user to submit it
    to the database. The user may paste the file into a textbox, or upload it
    (haven't decided yet).

    The problem I have is that the text file consists of around 3000 lines, and
    I want to display the formatted columns to the user before submitting it to
    the database (perhaps allowing them to edit some fields if they want, before
    clicking submit).
    Clearly 3000 lines is too much to display on one gridview page - so the
    question is::

    How can I page the data in the grid, without having to parse the text each
    time - i.e. where can I 'store' the parsed text so that I use that during my
    paging operations (prior to inserting it into the database)?

    Thanks,
    JJ
    JJ, Jun 7, 2007
    #1
    1. Advertising

  2. JJ

    Guest

    On Jun 7, 11:56 am, "JJ" <> wrote:
    > I have a need to input a large tab delimited text file, which I will parse
    > to check it has the expected columns, before allowing the user to submit it
    > to the database. The user may paste the file into a textbox, or upload it
    > (haven't decided yet).
    >
    > The problem I have is that the text file consists of around 3000 lines, and
    > I want to display the formatted columns to the user before submitting it to
    > the database (perhaps allowing them to edit some fields if they want, before
    > clicking submit).
    > Clearly 3000 lines is too much to display on one gridview page - so the
    > question is::
    >
    > How can I page the data in the grid, without having to parse the text each
    > time - i.e. where can I 'store' the parsed text so that I use that during my
    > paging operations (prior to inserting it into the database)?
    >
    > Thanks,
    > JJ


    JJ,
    What I'd consider doing is to read and parse the file into a
    DataTable, and use this to bind to a pageable grid.
    Peter
    , Jun 7, 2007
    #2
    1. Advertising

  3. JJ

    JJ Guest

    Hi Peter,

    Thats what I've done (used a dataTable that is). But I notice, unless I'm
    mistaken, that each time you change the page index, you need to rebind the
    grid to the datatable. I lose the contents of that datatable on each
    refresh. To recreate it I need to parse the text again, which will be a
    lengthy process if the text file is 3000 lines.

    I'm therefore at a loss as to how to keep hold of the large, formatted
    datatable throughout postbacks.
    Could I write the gridview as a user control and use viewstate for such a
    large datatable perhaps?

    JJ

    <> wrote in message
    news:...
    > On Jun 7, 11:56 am, "JJ" <> wrote:
    >> I have a need to input a large tab delimited text file, which I will
    >> parse
    >> to check it has the expected columns, before allowing the user to submit
    >> it
    >> to the database. The user may paste the file into a textbox, or upload it
    >> (haven't decided yet).
    >>
    >> The problem I have is that the text file consists of around 3000 lines,
    >> and
    >> I want to display the formatted columns to the user before submitting it
    >> to
    >> the database (perhaps allowing them to edit some fields if they want,
    >> before
    >> clicking submit).
    >> Clearly 3000 lines is too much to display on one gridview page - so the
    >> question is::
    >>
    >> How can I page the data in the grid, without having to parse the text
    >> each
    >> time - i.e. where can I 'store' the parsed text so that I use that during
    >> my
    >> paging operations (prior to inserting it into the database)?
    >>
    >> Thanks,
    >> JJ

    >
    > JJ,
    > What I'd consider doing is to read and parse the file into a
    > DataTable, and use this to bind to a pageable grid.
    > Peter
    >
    JJ, Jun 7, 2007
    #3
  4. "JJ" <> wrote in message
    news:...
    > Hi Peter,
    >
    > Thats what I've done (used a dataTable that is). But I notice, unless I'm
    > mistaken, that each time you change the page index, you need to rebind the
    > grid to the datatable. I lose the contents of that datatable on each
    > refresh. To recreate it I need to parse the text again, which will be a
    > lengthy process if the text file is 3000 lines.
    >
    > I'm therefore at a loss as to how to keep hold of the large, formatted
    > datatable throughout postbacks.
    > Could I write the gridview as a user control and use viewstate for such a
    > large datatable perhaps?
    >
    > JJ


    Why don't you save the data in the temporary table in the database? Once
    user confirmed the changes you will copy all data to the real table.
    Alexey Smirnov, Jun 7, 2007
    #4
  5. Store the DataTable in Session after you've first constructed it.
    Get it back out of Session after changing the PageIndex and rebind:

    grid.CurrrentPageIndex =e.newPageIndex;
    DataTable myTable = (DataTable)Session["myDataTable"];
    grid.DataSource=myTable;
    grid.DataBind();

    Peter
    --
    Site: http://www.eggheadcafe.com
    UnBlog: http://petesbloggerama.blogspot.com
    Short urls & more: http://ittyurl.net




    "JJ" wrote:

    > Hi Peter,
    >
    > Thats what I've done (used a dataTable that is). But I notice, unless I'm
    > mistaken, that each time you change the page index, you need to rebind the
    > grid to the datatable. I lose the contents of that datatable on each
    > refresh. To recreate it I need to parse the text again, which will be a
    > lengthy process if the text file is 3000 lines.
    >
    > I'm therefore at a loss as to how to keep hold of the large, formatted
    > datatable throughout postbacks.
    > Could I write the gridview as a user control and use viewstate for such a
    > large datatable perhaps?
    >
    > JJ
    >
    > <> wrote in message
    > news:...
    > > On Jun 7, 11:56 am, "JJ" <> wrote:
    > >> I have a need to input a large tab delimited text file, which I will
    > >> parse
    > >> to check it has the expected columns, before allowing the user to submit
    > >> it
    > >> to the database. The user may paste the file into a textbox, or upload it
    > >> (haven't decided yet).
    > >>
    > >> The problem I have is that the text file consists of around 3000 lines,
    > >> and
    > >> I want to display the formatted columns to the user before submitting it
    > >> to
    > >> the database (perhaps allowing them to edit some fields if they want,
    > >> before
    > >> clicking submit).
    > >> Clearly 3000 lines is too much to display on one gridview page - so the
    > >> question is::
    > >>
    > >> How can I page the data in the grid, without having to parse the text
    > >> each
    > >> time - i.e. where can I 'store' the parsed text so that I use that during
    > >> my
    > >> paging operations (prior to inserting it into the database)?
    > >>
    > >> Thanks,
    > >> JJ

    > >
    > > JJ,
    > > What I'd consider doing is to read and parse the file into a
    > > DataTable, and use this to bind to a pageable grid.
    > > Peter
    > >

    >
    >
    >
    =?Utf-8?B?UGV0ZXIgQnJvbWJlcmcgW0MjIE1WUF0=?=, Jun 7, 2007
    #5
  6. JJ

    JJ Guest

    Is there a limit to how large the datatable can be within a Session
    variable? We're talking a 3000 line tab delimited text file here.

    JJ

    "Peter Bromberg [C# MVP]" <> wrote in
    message news:...
    > Store the DataTable in Session after you've first constructed it.
    > Get it back out of Session after changing the PageIndex and rebind:
    >
    > grid.CurrrentPageIndex =e.newPageIndex;
    > DataTable myTable = (DataTable)Session["myDataTable"];
    > grid.DataSource=myTable;
    > grid.DataBind();
    >
    > Peter
    > --
    > Site: http://www.eggheadcafe.com
    > UnBlog: http://petesbloggerama.blogspot.com
    > Short urls & more: http://ittyurl.net
    >
    >
    >
    >
    > "JJ" wrote:
    >
    >> Hi Peter,
    >>
    >> Thats what I've done (used a dataTable that is). But I notice, unless I'm
    >> mistaken, that each time you change the page index, you need to rebind
    >> the
    >> grid to the datatable. I lose the contents of that datatable on each
    >> refresh. To recreate it I need to parse the text again, which will be a
    >> lengthy process if the text file is 3000 lines.
    >>
    >> I'm therefore at a loss as to how to keep hold of the large, formatted
    >> datatable throughout postbacks.
    >> Could I write the gridview as a user control and use viewstate for such a
    >> large datatable perhaps?
    >>
    >> JJ
    >>
    >> <> wrote in message
    >> news:...
    >> > On Jun 7, 11:56 am, "JJ" <> wrote:
    >> >> I have a need to input a large tab delimited text file, which I will
    >> >> parse
    >> >> to check it has the expected columns, before allowing the user to
    >> >> submit
    >> >> it
    >> >> to the database. The user may paste the file into a textbox, or upload
    >> >> it
    >> >> (haven't decided yet).
    >> >>
    >> >> The problem I have is that the text file consists of around 3000
    >> >> lines,
    >> >> and
    >> >> I want to display the formatted columns to the user before submitting
    >> >> it
    >> >> to
    >> >> the database (perhaps allowing them to edit some fields if they want,
    >> >> before
    >> >> clicking submit).
    >> >> Clearly 3000 lines is too much to display on one gridview page - so
    >> >> the
    >> >> question is::
    >> >>
    >> >> How can I page the data in the grid, without having to parse the text
    >> >> each
    >> >> time - i.e. where can I 'store' the parsed text so that I use that
    >> >> during
    >> >> my
    >> >> paging operations (prior to inserting it into the database)?
    >> >>
    >> >> Thanks,
    >> >> JJ
    >> >
    >> > JJ,
    >> > What I'd consider doing is to read and parse the file into a
    >> > DataTable, and use this to bind to a pageable grid.
    >> > Peter
    >> >

    >>
    >>
    >>
    JJ, Jun 7, 2007
    #6
  7. On Jun 7, 9:51 pm, "JJ" <> wrote:
    > Is there a limit to how large the datatable can be within a Session
    > variable? We're talking a 3000 line tab delimited text file here.
    >


    It depends on the web server's memory. Storing large objects in a
    user's session is maybe not always a good idea because of performance
    but storing and reloading data from memory will be faster than any
    other method.
    Alexey Smirnov, Jun 7, 2007
    #7
  8. As Alexey said. A 3000 line text file (or the DataTable you get from
    converting it) isn't going to bring down your server. But, if it is
    user-specific and you have to use Session, and you have a lot of users, well
    - you can do the math.
    Peter
    --
    Site: http://www.eggheadcafe.com
    UnBlog: http://petesbloggerama.blogspot.com
    Short urls & more: http://ittyurl.net




    "JJ" wrote:

    > Is there a limit to how large the datatable can be within a Session
    > variable? We're talking a 3000 line tab delimited text file here.
    >
    > JJ
    >
    > "Peter Bromberg [C# MVP]" <> wrote in
    > message news:...
    > > Store the DataTable in Session after you've first constructed it.
    > > Get it back out of Session after changing the PageIndex and rebind:
    > >
    > > grid.CurrrentPageIndex =e.newPageIndex;
    > > DataTable myTable = (DataTable)Session["myDataTable"];
    > > grid.DataSource=myTable;
    > > grid.DataBind();
    > >
    > > Peter
    > > --
    > > Site: http://www.eggheadcafe.com
    > > UnBlog: http://petesbloggerama.blogspot.com
    > > Short urls & more: http://ittyurl.net
    > >
    > >
    > >
    > >
    > > "JJ" wrote:
    > >
    > >> Hi Peter,
    > >>
    > >> Thats what I've done (used a dataTable that is). But I notice, unless I'm
    > >> mistaken, that each time you change the page index, you need to rebind
    > >> the
    > >> grid to the datatable. I lose the contents of that datatable on each
    > >> refresh. To recreate it I need to parse the text again, which will be a
    > >> lengthy process if the text file is 3000 lines.
    > >>
    > >> I'm therefore at a loss as to how to keep hold of the large, formatted
    > >> datatable throughout postbacks.
    > >> Could I write the gridview as a user control and use viewstate for such a
    > >> large datatable perhaps?
    > >>
    > >> JJ
    > >>
    > >> <> wrote in message
    > >> news:...
    > >> > On Jun 7, 11:56 am, "JJ" <> wrote:
    > >> >> I have a need to input a large tab delimited text file, which I will
    > >> >> parse
    > >> >> to check it has the expected columns, before allowing the user to
    > >> >> submit
    > >> >> it
    > >> >> to the database. The user may paste the file into a textbox, or upload
    > >> >> it
    > >> >> (haven't decided yet).
    > >> >>
    > >> >> The problem I have is that the text file consists of around 3000
    > >> >> lines,
    > >> >> and
    > >> >> I want to display the formatted columns to the user before submitting
    > >> >> it
    > >> >> to
    > >> >> the database (perhaps allowing them to edit some fields if they want,
    > >> >> before
    > >> >> clicking submit).
    > >> >> Clearly 3000 lines is too much to display on one gridview page - so
    > >> >> the
    > >> >> question is::
    > >> >>
    > >> >> How can I page the data in the grid, without having to parse the text
    > >> >> each
    > >> >> time - i.e. where can I 'store' the parsed text so that I use that
    > >> >> during
    > >> >> my
    > >> >> paging operations (prior to inserting it into the database)?
    > >> >>
    > >> >> Thanks,
    > >> >> JJ
    > >> >
    > >> > JJ,
    > >> > What I'd consider doing is to read and parse the file into a
    > >> > DataTable, and use this to bind to a pageable grid.
    > >> > Peter
    > >> >
    > >>
    > >>
    > >>

    >
    >
    >
    =?Utf-8?B?UGV0ZXIgQnJvbWJlcmcgW0MjIE1WUF0=?=, Jun 7, 2007
    #8
  9. JJ

    JJ Guest

    Looks like storing it in a temp table in the database may be the way to go
    then,
    Thanks,
    JJ

    "Peter Bromberg [C# MVP]" <> wrote in
    message news:...
    > As Alexey said. A 3000 line text file (or the DataTable you get from
    > converting it) isn't going to bring down your server. But, if it is
    > user-specific and you have to use Session, and you have a lot of users,
    > well
    > - you can do the math.
    > Peter
    > --
    > Site: http://www.eggheadcafe.com
    > UnBlog: http://petesbloggerama.blogspot.com
    > Short urls & more: http://ittyurl.net
    >
    >
    >
    >
    > "JJ" wrote:
    >
    >> Is there a limit to how large the datatable can be within a Session
    >> variable? We're talking a 3000 line tab delimited text file here.
    >>
    >> JJ
    >>
    >> "Peter Bromberg [C# MVP]" <> wrote in
    >> message news:...
    >> > Store the DataTable in Session after you've first constructed it.
    >> > Get it back out of Session after changing the PageIndex and rebind:
    >> >
    >> > grid.CurrrentPageIndex =e.newPageIndex;
    >> > DataTable myTable = (DataTable)Session["myDataTable"];
    >> > grid.DataSource=myTable;
    >> > grid.DataBind();
    >> >
    >> > Peter
    >> > --
    >> > Site: http://www.eggheadcafe.com
    >> > UnBlog: http://petesbloggerama.blogspot.com
    >> > Short urls & more: http://ittyurl.net
    >> >
    >> >
    >> >
    >> >
    >> > "JJ" wrote:
    >> >
    >> >> Hi Peter,
    >> >>
    >> >> Thats what I've done (used a dataTable that is). But I notice, unless
    >> >> I'm
    >> >> mistaken, that each time you change the page index, you need to rebind
    >> >> the
    >> >> grid to the datatable. I lose the contents of that datatable on each
    >> >> refresh. To recreate it I need to parse the text again, which will be
    >> >> a
    >> >> lengthy process if the text file is 3000 lines.
    >> >>
    >> >> I'm therefore at a loss as to how to keep hold of the large, formatted
    >> >> datatable throughout postbacks.
    >> >> Could I write the gridview as a user control and use viewstate for
    >> >> such a
    >> >> large datatable perhaps?
    >> >>
    >> >> JJ
    >> >>
    >> >> <> wrote in message
    >> >> news:...
    >> >> > On Jun 7, 11:56 am, "JJ" <> wrote:
    >> >> >> I have a need to input a large tab delimited text file, which I
    >> >> >> will
    >> >> >> parse
    >> >> >> to check it has the expected columns, before allowing the user to
    >> >> >> submit
    >> >> >> it
    >> >> >> to the database. The user may paste the file into a textbox, or
    >> >> >> upload
    >> >> >> it
    >> >> >> (haven't decided yet).
    >> >> >>
    >> >> >> The problem I have is that the text file consists of around 3000
    >> >> >> lines,
    >> >> >> and
    >> >> >> I want to display the formatted columns to the user before
    >> >> >> submitting
    >> >> >> it
    >> >> >> to
    >> >> >> the database (perhaps allowing them to edit some fields if they
    >> >> >> want,
    >> >> >> before
    >> >> >> clicking submit).
    >> >> >> Clearly 3000 lines is too much to display on one gridview page - so
    >> >> >> the
    >> >> >> question is::
    >> >> >>
    >> >> >> How can I page the data in the grid, without having to parse the
    >> >> >> text
    >> >> >> each
    >> >> >> time - i.e. where can I 'store' the parsed text so that I use that
    >> >> >> during
    >> >> >> my
    >> >> >> paging operations (prior to inserting it into the database)?
    >> >> >>
    >> >> >> Thanks,
    >> >> >> JJ
    >> >> >
    >> >> > JJ,
    >> >> > What I'd consider doing is to read and parse the file into a
    >> >> > DataTable, and use this to bind to a pageable grid.
    >> >> > Peter
    >> >> >
    >> >>
    >> >>
    >> >>

    >>
    >>
    >>
    JJ, Jun 7, 2007
    #9
  10. On Jun 7, 11:03 pm, "JJ" <> wrote:
    > Looks like storing it in a temp table in the database may be the way to go
    > then,
    > Thanks,
    > JJ
    >



    If you go this way, create a table with columns you need to get from
    Excel and an additional column for SessionId (varchar 25). And use
    that SessionId to identify data of the current user.

    To clean table you can create a sql job and schedule to run... (in
    this case you might also need to have a datetime column to know when
    data were added)
    Alexey Smirnov, Jun 7, 2007
    #10
  11. JJ

    JJ Guest

    Good point - I didn't think of that.
    Thanks,
    JJ
    "Alexey Smirnov" <> wrote in message
    news:...
    > On Jun 7, 11:03 pm, "JJ" <> wrote:
    >> Looks like storing it in a temp table in the database may be the way to
    >> go
    >> then,
    >> Thanks,
    >> JJ
    >>

    >
    >
    > If you go this way, create a table with columns you need to get from
    > Excel and an additional column for SessionId (varchar 25). And use
    > that SessionId to identify data of the current user.
    >
    > To clean table you can create a sql job and schedule to run... (in
    > this case you might also need to have a datetime column to know when
    > data were added)
    >
    JJ, Jun 7, 2007
    #11
  12. JJ

    JJ Guest

    It doesn't look like I'm able to access the sql agent to create a scheduled
    job. Perhaps I could check for records which are 24 hours old when I do
    another insert/delete etc and remove them then?

    Thanks,
    JJ
    "Alexey Smirnov" <> wrote in message
    news:...
    > On Jun 7, 11:03 pm, "JJ" <> wrote:
    >> Looks like storing it in a temp table in the database may be the way to
    >> go
    >> then,
    >> Thanks,
    >> JJ
    >>

    >
    >
    > If you go this way, create a table with columns you need to get from
    > Excel and an additional column for SessionId (varchar 25). And use
    > that SessionId to identify data of the current user.
    >
    > To clean table you can create a sql job and schedule to run... (in
    > this case you might also need to have a datetime column to know when
    > data were added)
    >
    JJ, Jun 8, 2007
    #12
  13. On Jun 8, 11:46 am, "JJ" <> wrote:
    > It doesn't look like I'm able to access the sql agent to create a scheduled
    > job. Perhaps I could check for records which are 24 hours old when I do
    > another insert/delete etc and remove them then?


    Yes, it could be a good idea. In general, you need to delete temporary
    data when the user has been confirmed the changes and the data have
    been transferred to a working table. But I think it could be a
    situation where an user did imported his data to that temporary table,
    and didn't confirmed... So, in this case you will need this kind of
    cleaning function
    Alexey Smirnov, Jun 8, 2007
    #13
  14. JJ

    JJ Guest

    Yes - exactly my thoughts. I intend to delete each record with the current
    sessionID as it is inserted into the final database table, but, as this is
    quiet a lot of data, and the user may close the browser just after adding to
    the temp table and before committing to the final table, I think that before
    I insert any data into the temp table, I should run some sql:

    DELETE
    Temp_Table
    WHERE
    DATEDIFF (hour , AddedDate , GetDate(); ) > 2

    to delete any records older than 2 hours.

    Thanks,
    JJ

    "Alexey Smirnov" <> wrote in message
    news:...
    > On Jun 8, 11:46 am, "JJ" <> wrote:
    >> It doesn't look like I'm able to access the sql agent to create a
    >> scheduled
    >> job. Perhaps I could check for records which are 24 hours old when I do
    >> another insert/delete etc and remove them then?

    >
    > Yes, it could be a good idea. In general, you need to delete temporary
    > data when the user has been confirmed the changes and the data have
    > been transferred to a working table. But I think it could be a
    > situation where an user did imported his data to that temporary table,
    > and didn't confirmed... So, in this case you will need this kind of
    > cleaning function
    >
    JJ, Jun 8, 2007
    #14
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. Pmb
    Replies:
    2
    Views:
    498
    Mike Wahler
    May 29, 2004
  2. Kelly
    Replies:
    10
    Views:
    455
    Victor Bazarov
    Dec 23, 2004
  3. Keith Hughitt
    Replies:
    4
    Views:
    341
    Sion Arrowsmith
    Jul 10, 2008
  4. Patrick.O.Ige
    Replies:
    10
    Views:
    276
    Patrick.O.Ige
    Feb 28, 2007
  5. Replies:
    1
    Views:
    134
Loading...

Share This Page