Array contains too much data

R

Robert TV

Hi. I have a small perl script that opens a file, collects and assigns each
line of the file to an array, and then displays the data to the screen. This
script is working just fine, but is starting to get difficult to manage ...
you see, the data file now contains about 1,000 lines of data and it's
taking very long for perl to parse the data, assemble it into html and then
show in my browser (obviously it was very quick when the data file only had
10 enties total) Here is the jist of the script:

###### Start Script Example ######

#!/usr/bin/perl

open (FH, "<$user/datafile.db") or die "Can't open: $!";
@entires=<FH>;
close(FH);

print "Content-type: text/html \n\n";
print <<HTML;
<html>
<head>
<title>Database Results</title>
</head>
<body>
<table>
HTML

# DISPLAY DATABASE TABLES
chomp @entires;
foreach $entry(@entires) {
print <<HTML;
<tr>
<td width="42" height="19">Entry</td>
<td width="279" height="19">$entry</td>
</tr>
PRINTHTML
}
print <<HTML;
</table>
</body>
</html>
HTML
exit;

###### End Script Example ######

The database now at 1000 entires it takes very long to essemble and print to
screen, and it's bogging down my browser because the HTML is too long. Now,
I start to think ... I need a way to only show 20 or 30 results at a time,
and have "next results" or something similair ... kinda like search engines
do. So in an ideal world, I run the script and it only shows the first 30
lines of the array and has links to more results ... like this:

Showing 0-30 ---back 1 2 3 4 5 6 next #something like this.

I'm posting this inquiry because I have no idea how I would approach or
create such a feature, and I'm hoping that maybe a few of you out there
might be able to point me in the right direction, maybe there are some
tutorials, code snippets etc etc. I appriciate you taking time to read my
post! TIA

Robert TV
 
M

Matthew Braid

Robert said:
Hi. I have a small perl script that opens a file, collects and assigns each
line of the file to an array, and then displays the data to the screen. This
script is working just fine, but is starting to get difficult to manage ...
you see, the data file now contains about 1,000 lines of data and it's
taking very long for perl to parse the data, assemble it into html and then
show in my browser (obviously it was very quick when the data file only had
10 enties total) Here is the jist of the script:

###### Start Script Example ######

#!/usr/bin/perl

open (FH, "<$user/datafile.db") or die "Can't open: $!";
@entires=<FH>;
close(FH);

print "Content-type: text/html \n\n";
print <<HTML;
<html>
<head>
<title>Database Results</title>
</head>
<body>
<table>
HTML

# DISPLAY DATABASE TABLES
chomp @entires;
foreach $entry(@entires) {
print <<HTML;
<tr>
<td width="42" height="19">Entry</td>
<td width="279" height="19">$entry</td>
</tr>
PRINTHTML
}
print <<HTML;
</table>
</body>
</html>
HTML
exit;

###### End Script Example ######

The database now at 1000 entires it takes very long to essemble and print to
screen, and it's bogging down my browser because the HTML is too long. Now,
I start to think ... I need a way to only show 20 or 30 results at a time,
and have "next results" or something similair ... kinda like search engines
do. So in an ideal world, I run the script and it only shows the first 30
lines of the array and has links to more results ... like this:

Showing 0-30 ---back 1 2 3 4 5 6 next #something like this.

I'm posting this inquiry because I have no idea how I would approach or
create such a feature, and I'm hoping that maybe a few of you out there
might be able to point me in the right direction, maybe there are some
tutorials, code snippets etc etc. I appriciate you taking time to read my
post! TIA

Robert TV
Click to expand...

First off - I hope the line above consisting entirely of PRINTHTML was a
typo - it should have been HTML.

Secondly, you're going to have to accept parameters of what the current
display length is (say 30 for 30 items per page) and what offset is
required (30 would mean page 2 for a 30-items-per-page display length,
while 25 would mean someones playing with your URL :) ). You also need
to know how many entries are in your 'database' (although this should
_NOT_ be a parameter - figure it out yourself). This means using CGI
stuff. Read up on the CGI module (perldoc CGI). Your URL (using the GET
method at least) will end up looking something like:

http://www.example.com/show_entries.pl?offset=60&page_size=30

The back and next links should point to offsets that match the previous
and next page (if the current request is for offset 90 with a page_size
of 30, back should be a link to offset 60 with a page_size of 30 and
next should be a link to offset 120 with a page_size of 30 - remember
that there may not be a next or prev page though!). Since CGI is
stateless you can't tell what the _current_ page is unless you add
another parameter, and its easy enough to do without that.

There are a lot of traps to CGI programming since you have no control
over the user possibly typing in their own parameters to try and break
your code.

Of course, there is still one little problem - your 'database' is just a
flat text file. Its always going to be slow for any decent sized file.
Unless the entries are fixed-length you can't just skip the appropriate
part of the file to get entries at a given offset - you have to read
every line in the file up to that offset and count the newlines. You may
want to consider a proper database.

MB
 
G

Gunnar Hjalmarsson

Robert said:
###### Start Script Example ######

#!/usr/bin/perl
Click to expand...

You should enable strictures and warnings:

use strict;
use warnings;
I need a way to only show 20 or 30 results at a time, and have
"next results" or something similair ... kinda like search engines
do. So in an ideal world, I run the script and it only shows the
first 30 lines of the array
Click to expand...

To give you an idea what can be done, you can add these lines after
the file has been slurped into @entires:

my ($offset) = $ENV{QUERY_STRING} =~ /offset=(\d+)/;
$offset ||= 0;
@entires = splice @entires, $offset, 30;

Now, if you run the script as usual, only the first 30 results are
displayed. To display e.g. the next 30 results, you need to add a
query string to the URL, like this:

http://.../myscript.pl?offset=30
and has links to more results
Click to expand...

You can have the script generate such links by help of the $offset
variable.
 
R

Robert TV

Gunnar Hjalmarsson" wrote
use strict;
use warnings;
Click to expand...
To give you an idea what can be done, you can add these lines after
the file has been slurped into @entires:

my ($offset) = $ENV{QUERY_STRING} =~ /offset=(\d+)/;
$offset ||= 0;
@entires = splice @entires, $offset, 30;

Now, if you run the script as usual, only the first 30 results are
displayed. To display e.g. the next 30 results, you need to add a
query string to the URL, like this:

http://.../myscript.pl?offset=30
Click to expand...
You can have the script generate such links by help of the $offset
variable.
Click to expand...

Thanx for the reply Gunnar, I will research $offset.

Robert TV
 
T

Tad McClellan

Robert TV said:
Gunnar Hjalmarsson" wrote

Thanx for the reply Gunnar, I will research $offset.
Click to expand...


There is nothing special about the variable named $offset.

Don't go researching the wrong thing, that would be a big
waste of time.

Research the || and ||= operators in perlop.pod.
 
M

Malte Ubl

Gunnar said:
my ($offset) = $ENV{QUERY_STRING} =~ /offset=(\d+)/;
Click to expand...

Why would you ever want to do that rather than,

my $q = CGI->new;
<some code>
my $offset = $q->param('offset') || 0;
die "illegal offset" if $offset =~ /\D/;

???

malte
 
G

Gunnar Hjalmarsson

Malte said:
Why would you ever want to do that rather than,

my $q = CGI->new;
<some code>
my $offset = $q->param('offset') || 0;
Click to expand...

Why not? After all, matching that string isn't very complicated.

Personally I sometimes refrain from using modules, when I still can
achieve what I want with a line or two. One reason may be that the
program I have spent most time writing is a program where efficiency
matters, even if it is a CGI scipt. (If you would benchmark the two
alternatives above, using the heavy-weight CGI.pm would appear
absurdly inefficient.)

Anyway, thanks for posting the module alternative. As long as OP does
not fully understand how the first alternative works, and under which
conditions, I agree that he should better use CGI.pm. On the other
hand, it's not advisable to write CGI scripts without learning the
basics about how CGI works, is it?
 
M

Malte Ubl

Gunnar said:
Personally I sometimes refrain from using modules, when I still can
achieve what I want with a line or two. One reason may be that the
program I have spent most time writing is a program where efficiency
matters, even if it is a CGI scipt. (If you would benchmark the two
alternatives above, using the heavy-weight CGI.pm would appear
absurdly inefficient.)

Anyway, thanks for posting the module alternative. As long as OP does
not fully understand how the first alternative works, and under which
conditions, I agree that he should better use CGI.pm. On the other
hand, it's not advisable to write CGI scripts without learning the
basics about how CGI works, is it?
Click to expand...

If you know how CGI works, you will write code that doesn't run under
mod_perl or any other method of execution for that matter.

I for one appreciate every little bit I don't have to safe in my head.

BTW: The OP didn't even suggest he was using CGI :)

malte
 

Ask a Question

Want to reply to this thread or ask your own question?

You'll need to choose a username for the site, which only take a couple of moments. After that, you can post your question and our members will help you out.

Ask a Question

Similar Threads

Can someone tell me if this a real tracker? Or is it one designed to show you a different message at certain times, ie. acting like one? 0
Javascript DOM 1
Script to send email not working 1
Collect Excel Data from Website 5
Javascript fill function data from multidimensonal array 0
How to add dropdown selected data to table using jquery 2
How to create a JSON array with values from DOM(HTML TABLE) when I click a button using JQuery/Javascript? 0
Checking dynamically populated data using ajax with user entered value 5

Members online

Total: 33 (members: 1, guests: 32)
Robots: 346

Forum statistics

Threads
473,769
Messages
2,569,582
Members
45,061
Latest member
KetonaraKeto

Latest Threads

Top