inputing, paging, sorting, a large text file

[JJ]

I have a need to input a large tab delimited text file, which I will parse
to check it has the expected columns, before allowing the user to submit it
to the database. The user may paste the file into a textbox, or upload it
(haven't decided yet).

The problem I have is that the text file consists of around 3000 lines, and
I want to display the formatted columns to the user before submitting it to
the database (perhaps allowing them to edit some fields if they want, before
clicking submit).
Clearly 3000 lines is too much to display on one gridview page - so the
question is::

How can I page the data in the grid, without having to parse the text each
time - i.e. where can I 'store' the parsed text so that I use that during my
paging operations (prior to inserting it into the database)?

Thanks,
JJ

 

[pbromberg]

I have a need to input a large tab delimited text file, which I will parse
to check it has the expected columns, before allowing the user to submit it
to the database. The user may paste the file into a textbox, or upload it
(haven't decided yet).

The problem I have is that the text file consists of around 3000 lines, and
I want to display the formatted columns to the user before submitting it to
the database (perhaps allowing them to edit some fields if they want, before
clicking submit).
Clearly 3000 lines is too much to display on one gridview page - so the
question is::

How can I page the data in the grid, without having to parse the text each
time - i.e. where can I 'store' the parsed text so that I use that during my
paging operations (prior to inserting it into the database)?

Thanks,
JJ

JJ,
What I'd consider doing is to read and parse the file into a
DataTable, and use this to bind to a pageable grid.
Peter

 

[JJ]

Hi Peter,

Thats what I've done (used a dataTable that is). But I notice, unless I'm
mistaken, that each time you change the page index, you need to rebind the
grid to the datatable. I lose the contents of that datatable on each
refresh. To recreate it I need to parse the text again, which will be a
lengthy process if the text file is 3000 lines.

I'm therefore at a loss as to how to keep hold of the large, formatted
datatable throughout postbacks.
Could I write the gridview as a user control and use viewstate for such a
large datatable perhaps?

JJ

 

[Guest]

Hi Peter,

Thats what I've done (used a dataTable that is). But I notice, unless I'm
mistaken, that each time you change the page index, you need to rebind the
grid to the datatable. I lose the contents of that datatable on each
refresh. To recreate it I need to parse the text again, which will be a
lengthy process if the text file is 3000 lines.

I'm therefore at a loss as to how to keep hold of the large, formatted
datatable throughout postbacks.
Could I write the gridview as a user control and use viewstate for such a
large datatable perhaps?

JJ

Why don't you save the data in the temporary table in the database? Once
user confirmed the changes you will copy all data to the real table.

 

[Guest]

Store the DataTable in Session after you've first constructed it.
Get it back out of Session after changing the PageIndex and rebind:

grid.CurrrentPageIndex =e.newPageIndex;
DataTable myTable = (DataTable)Session["myDataTable"];
grid.DataSource=myTable;
grid.DataBind();

Peter
--
Site: http://www.eggheadcafe.com
UnBlog: http://petesbloggerama.blogspot.com
Short urls & more: http://ittyurl.net

 

[JJ]

Is there a limit to how large the datatable can be within a Session
variable? We're talking a 3000 line tab delimited text file here.

JJ

Store the DataTable in Session after you've first constructed it.
Get it back out of Session after changing the PageIndex and rebind:

grid.CurrrentPageIndex =e.newPageIndex;
DataTable myTable = (DataTable)Session["myDataTable"];
grid.DataSource=myTable;
grid.DataBind();

Peter
--
Site: http://www.eggheadcafe.com
UnBlog: http://petesbloggerama.blogspot.com
Short urls & more: http://ittyurl.net

Hi Peter,

Thats what I've done (used a dataTable that is). But I notice, unless I'm
mistaken, that each time you change the page index, you need to rebind
the
grid to the datatable. I lose the contents of that datatable on each
refresh. To recreate it I need to parse the text again, which will be a
lengthy process if the text file is 3000 lines.

I'm therefore at a loss as to how to keep hold of the large, formatted
datatable throughout postbacks.
Could I write the gridview as a user control and use viewstate for such a
large datatable perhaps?

JJ

 

[Guest]

Is there a limit to how large the datatable can be within a Session
variable? We're talking a 3000 line tab delimited text file here.

It depends on the web server's memory. Storing large objects in a
user's session is maybe not always a good idea because of performance
but storing and reloading data from memory will be faster than any
other method.

 

[Guest]

As Alexey said. A 3000 line text file (or the DataTable you get from
converting it) isn't going to bring down your server. But, if it is
user-specific and you have to use Session, and you have a lot of users, well
- you can do the math.
Peter
--
Site: http://www.eggheadcafe.com
UnBlog: http://petesbloggerama.blogspot.com
Short urls & more: http://ittyurl.net

Is there a limit to how large the datatable can be within a Session
variable? We're talking a 3000 line tab delimited text file here.

JJ

Store the DataTable in Session after you've first constructed it.
Get it back out of Session after changing the PageIndex and rebind:

grid.CurrrentPageIndex =e.newPageIndex;
DataTable myTable = (DataTable)Session["myDataTable"];
grid.DataSource=myTable;
grid.DataBind();

Peter
--
Site: http://www.eggheadcafe.com
UnBlog: http://petesbloggerama.blogspot.com
Short urls & more: http://ittyurl.net

Hi Peter,

Thats what I've done (used a dataTable that is). But I notice, unless I'm
mistaken, that each time you change the page index, you need to rebind
the
grid to the datatable. I lose the contents of that datatable on each
refresh. To recreate it I need to parse the text again, which will be a
lengthy process if the text file is 3000 lines.

I'm therefore at a loss as to how to keep hold of the large, formatted
datatable throughout postbacks.
Could I write the gridview as a user control and use viewstate for such a
large datatable perhaps?

JJ

I have a need to input a large tab delimited text file, which I will
parse
to check it has the expected columns, before allowing the user to
submit
it
to the database. The user may paste the file into a textbox, or upload
it
(haven't decided yet).

The problem I have is that the text file consists of around 3000
lines,
and
I want to display the formatted columns to the user before submitting
it
to
the database (perhaps allowing them to edit some fields if they want,
before
clicking submit).
Clearly 3000 lines is too much to display on one gridview page - so
the
question is::

How can I page the data in the grid, without having to parse the text
each
time - i.e. where can I 'store' the parsed text so that I use that
during
my
paging operations (prior to inserting it into the database)?

Thanks,
JJ

JJ,
What I'd consider doing is to read and parse the file into a
DataTable, and use this to bind to a pageable grid.
Peter

 

[JJ]

Looks like storing it in a temp table in the database may be the way to go
then,
Thanks,
JJ

As Alexey said. A 3000 line text file (or the DataTable you get from
converting it) isn't going to bring down your server. But, if it is
user-specific and you have to use Session, and you have a lot of users,
well
- you can do the math.
Peter
--
Site: http://www.eggheadcafe.com
UnBlog: http://petesbloggerama.blogspot.com
Short urls & more: http://ittyurl.net

Is there a limit to how large the datatable can be within a Session
variable? We're talking a 3000 line tab delimited text file here.

JJ

Store the DataTable in Session after you've first constructed it.
Get it back out of Session after changing the PageIndex and rebind:

grid.CurrrentPageIndex =e.newPageIndex;
DataTable myTable = (DataTable)Session["myDataTable"];
grid.DataSource=myTable;
grid.DataBind();

Peter
--
Site: http://www.eggheadcafe.com
UnBlog: http://petesbloggerama.blogspot.com
Short urls & more: http://ittyurl.net




:

Hi Peter,

Thats what I've done (used a dataTable that is). But I notice, unless
I'm
mistaken, that each time you change the page index, you need to rebind
the
grid to the datatable. I lose the contents of that datatable on each
refresh. To recreate it I need to parse the text again, which will be
a
lengthy process if the text file is 3000 lines.

I'm therefore at a loss as to how to keep hold of the large, formatted
datatable throughout postbacks.
Could I write the gridview as a user control and use viewstate for
such a
large datatable perhaps?

JJ

I have a need to input a large tab delimited text file, which I
will
parse
to check it has the expected columns, before allowing the user to
submit
it
to the database. The user may paste the file into a textbox, or
upload
it
(haven't decided yet).

The problem I have is that the text file consists of around 3000
lines,
and
I want to display the formatted columns to the user before
submitting
it
to
the database (perhaps allowing them to edit some fields if they
want,
before
clicking submit).
Clearly 3000 lines is too much to display on one gridview page - so
the
question is::

How can I page the data in the grid, without having to parse the
text
each
time - i.e. where can I 'store' the parsed text so that I use that
during
my
paging operations (prior to inserting it into the database)?

Thanks,
JJ

JJ,
What I'd consider doing is to read and parse the file into a
DataTable, and use this to bind to a pageable grid.
Peter

 

[Guest]

Looks like storing it in a temp table in the database may be the way to go
then,
Thanks,
JJ

If you go this way, create a table with columns you need to get from
Excel and an additional column for SessionId (varchar 25). And use
that SessionId to identify data of the current user.

To clean table you can create a sql job and schedule to run... (in
this case you might also need to have a datetime column to know when
data were added)

 

[JJ]

Good point - I didn't think of that.
Thanks,
JJ

If you go this way, create a table with columns you need to get from
Excel and an additional column for SessionId (varchar 25). And use
that SessionId to identify data of the current user.

To clean table you can create a sql job and schedule to run... (in
this case you might also need to have a datetime column to know when
data were added)

 

[JJ]

It doesn't look like I'm able to access the sql agent to create a scheduled
job. Perhaps I could check for records which are 24 hours old when I do
another insert/delete etc and remove them then?

Thanks,
JJ

 

[Guest]

It doesn't look like I'm able to access the sql agent to create a scheduled
job. Perhaps I could check for records which are 24 hours old when I do
another insert/delete etc and remove them then?

Yes, it could be a good idea. In general, you need to delete temporary
data when the user has been confirmed the changes and the data have
been transferred to a working table. But I think it could be a
situation where an user did imported his data to that temporary table,
and didn't confirmed... So, in this case you will need this kind of
cleaning function

 

[JJ]

Yes - exactly my thoughts. I intend to delete each record with the current
sessionID as it is inserted into the final database table, but, as this is
quiet a lot of data, and the user may close the browser just after adding to
the temp table and before committing to the final table, I think that before
I insert any data into the temp table, I should run some sql:

DELETE
Temp_Table
WHERE
DATEDIFF (hour , AddedDate , GetDate(); ) > 2

to delete any records older than 2 hours.

Thanks,
JJ