How to read last line in a file specified by URL?

E

Eric Capps

Hi,

My subject basically says it all: I'm writing a Java applet that, among
other things, needs to read the last line of a rather large file.
RandomAccessFile has been suggested to me, however, I can't get it to
work with files over the web (for example:

r = new RandomAccessFile("http://mysite.com/whatever.dat");

fails.

My questions are:
1) SHOULD RandomAccessFile be able to work for a file on the internet?
My guess is no.
2) I am currently trying (unsuccesfully) to use the BufferedReader class
and the skip(long n) method to implement this. Does this method operate
in constant or linear time? I can't afford to have it read through the
entire file when it skips characters.
3) Can anyone offer any general advice as to how to approach this
problem? I can't download the entire fire or move through it linearly; I
CAN write my own random access I/O class if necessary and possible.

-Eric
 
C

Chris Smith

Eric Capps said:
My subject basically says it all: I'm writing a Java applet that, among
other things, needs to read the last line of a rather large file.
RandomAccessFile has been suggested to me, however, I can't get it to
work with files over the web (for example:

r = new RandomAccessFile("http://mysite.com/whatever.dat");

fails.

RandomAccessFile is not the right answer. HTTP is generally not
designed for this sort of things, but you could try issuing a request
with a Range header of something like "bytes=-500", assuming that 500
bytes would be enough to contain a line.
 
E

Eric Capps

Chris said:
RandomAccessFile is not the right answer. HTTP is generally not
designed for this sort of things, but you could try issuing a request
with a Range header of something like "bytes=-500", assuming that 500
bytes would be enough to contain a line.

Thanks for your reply. This would work for reading the last line for
sure, but I'm also trying to find a more general way to skip to
out-of-sequence parts of the file. But if you say HTTP is not good for
this sort of thing, then I think I may have to do this outside of Java.
Perhaps I could write PHP methods to generate small portions of the file
based on a requested line number, then just call the page within the
java applet and read it entirely.

Thanks again,
Eric
 
C

Chris Smith

Eric Capps said:
Thanks for your reply. This would work for reading the last line for
sure, but I'm also trying to find a more general way to skip to
out-of-sequence parts of the file.

More general than an HTTP Range header?
But if you say HTTP is not good for this sort of thing

With HTTP, you can't do various seek-and-read tactics in the same way
you would with a random access file. You just specify the range, and
you get what you get. Since each access to the resource is a different
HTTP request, you want to do as few reads as possible.
then I think I may have to do this outside of Java.

Certainly, if you can process the file on the server instead, such as
with PHP, and then only return the relevant bits, that would be better.
Nothing else would really work any better on the client, though. The
problem here is that you're accessing a file over a network, so certain
access patterns don't work so well. What language is being used on the
client is immaterial to this concern.
 
M

Mark Space

Chris said:
Certainly, if you can process the file on the server instead, such as
with PHP, and then only return the relevant bits, that would be better.
Nothing else would really work any better on the client, though. The
problem here is that you're accessing a file over a network, so certain
access patterns don't work so well. What language is being used on the
client is immaterial to this concern.


Ditto, but why even use HTTP at all? Make a Java program on the server.
Daemonize it, and have it listen for strings on a pre-arranged port
number. When it gets the string, have it look up the file, grab the
last line, then toss it back to the sender. You don't even need TCP/IP
for this, datagrams would work fine.

It's a little annoying seeing every service imaginable crammed onto port
80. There *are* other protocols out there guys. Some of them even work.
 
E

eruhk

Mark said:
Ditto, but why even use HTTP at all? Make a Java program on the server.
Daemonize it, and have it listen for strings on a pre-arranged port
number. When it gets the string, have it look up the file, grab the
last line, then toss it back to the sender. You don't even need TCP/IP
for this, datagrams would work fine.

It's a little annoying seeing every service imaginable crammed onto port
80. There *are* other protocols out there guys. Some of them even work.

Interesting, I hadn't considered an approach like this. Would there be
any advantages to this approach over using PHP? Or are these roughly
similar implementations of the same concept?
 
C

Chris Smith

eruhk said:
Interesting, I hadn't considered an approach like this. Would there be
any advantages to this approach over using PHP? Or are these roughly
similar implementations of the same concept?

They are roughly similar. Mark's approach is probably the more "pure",
but I'd actually recommend avoiding it. There are some good reasons
that everything gets sent over HTTP via port 80. They include, for
example, that other ports are sometimes blocked by firewalls but HTTP
rarely is, and that it's often easier to get HTTP servers set up on
servers that you don't control than to set up arbitrary other programs.
If you're looking for something hosted through another company, you'll
generally pay many times as much to run arbitrary code as you would to
get an account with PHP or Perl-CGI or a Java Servlet web container.
 
A

Andy Dingley

Eric said:
My subject basically says it all: I'm writing a Java applet that, among
other things, needs to read the last line of a rather large file.

Then write some server-side code that only serves you the tail of the
file.

If you "ought" to be sniffing around big files on web servers, then I
can only assume you ought to be able to get sufficient access to the
server to install such a tool -- even if you have to write it in CGI
Perl!


As far as a client-side fix in Java goes, then you're barking up
entirely the wrong tree. This thing is on the web, so it's a HTTP
document and not a file. Use the web- and HTTP-related classes to
access it, not the file-related classes.
 
M

Mark Space

Chris said:
They are roughly similar. Mark's approach is probably the more "pure",
but I'd actually recommend avoiding it. There are some good reasons
that everything gets sent over HTTP via port 80. They include, for
example, that other ports are sometimes blocked by firewalls but HTTP
rarely is, and that it's often easier to get HTTP servers set up on
servers that you don't control than to set up arbitrary other programs.
If you're looking for something hosted through another company, you'll
generally pay many times as much to run arbitrary code as you would to
get an account with PHP or Perl-CGI or a Java Servlet web container.

This is a good point. I hadn't even considered that he may be running
on some one else's highly restricted server. Still, once you have the
basic approach down, it's almost as easy to do the IO over stdin and
stdout, which can be run over a standard CGI interface. Perl is easy to
daemonize (I don't know about PHP). A Perl program that daemonized or
not according to command line parameters, and communicated via sockets
or stdin/stdout also according to command line parameters, would be
pretty flexible and also interesting and informative to write.

I would have said that the advantage to the daemon approach is that it
is far, far lighter weight than having to install (and maintain) a full
web server. "Any port" includes port 80, so problems with client
firewalls are a non-issue. If however you will only be using this on
leased servers that also already have a web server and various scripting
languages, then the web app idea makes more sense.

If you're doing this for yourself, tho, check out the daemon solution,
it'll be fun.
 
E

Eric Capps

Mark said:
This is a good point. I hadn't even considered that he may be running
on some one else's highly restricted server. Still, once you have the
basic approach down, it's almost as easy to do the IO over stdin and
stdout, which can be run over a standard CGI interface. Perl is easy to
daemonize (I don't know about PHP). A Perl program that daemonized or
not according to command line parameters, and communicated via sockets
or stdin/stdout also according to command line parameters, would be
pretty flexible and also interesting and informative to write.

I would have said that the advantage to the daemon approach is that it
is far, far lighter weight than having to install (and maintain) a full
web server. "Any port" includes port 80, so problems with client
firewalls are a non-issue. If however you will only be using this on
leased servers that also already have a web server and various scripting
languages, then the web app idea makes more sense.

If you're doing this for yourself, tho, check out the daemon solution,
it'll be fun.

Daemonizing is definitely something I can do, and I might want to look
into it later (in the meantime I've already finished the PHP
implementation). Writing a little random accesser in Perl would be easy
enough, but how do I daemonize something, and how do I access it in Java
once I have?

For that matter is there a gener
 
E

Eric Capps

Eric said:
Daemonizing is definitely something I can do, and I might want to look
into it later (in the meantime I've already finished the PHP
implementation). Writing a little random accesser in Perl would be easy
enough, but how do I daemonize something, and how do I access it in Java
once I have?

For that matter is there a gener

Disregard last line, obviously. I was going to ask it there is a general
way to call Perl from w/in Java but there's plenty about that on the web.
 
M

Mark Space

Ask a Question

Want to reply to this thread or ask your own question?

You'll need to choose a username for the site, which only take a couple of moments. After that, you can post your question and our members will help you out.

Ask a Question

Members online

No members online now.

Forum statistics

Threads
473,744
Messages
2,569,484
Members
44,903
Latest member
orderPeak8CBDGummies

Latest Threads

Top