finding subdirectories from starting URL

[Alan]

I want to find subdirectories from a starting URL. For example,
if I start with http://www.someplace.net, I want to be able to find
the subdirectories there, e.g.:

http://www.someplace.net/documentation/
http://www.someplace.net/about/
http://www.someplace.net/images/

Are there Java methods that facilitate this?

Thanks, Alan

 

[Daniel Pitts]

I want to find subdirectories from a starting URL. For example,
if I start with http://www.someplace.net, I want to be able to find
the subdirectories there, e.g.:

http://www.someplace.net/documentation/
http://www.someplace.net/about/
http://www.someplace.net/images/

Are there Java methods that facilitate this?

Thanks, Alan

There is no easy way to do that, unless someplace.net gives you a
listing page. Generally, in order to do that, you have to have either
direct access to the disk, access to an FTP account on that machine, or
you have to crawl the web site and parse out the urls.

Note, this is not a limitation of Java, but simply a result of the way
http works.

There are plenty of web-crawling libraries/programs out there, I suggest
you Google for them.

Good luck,
Daniel

 

[Andrew Thompson]

I want to find subdirectories from a starting URL. For example,
if I start with http://www.someplace.net, I want to be able to find
the subdirectories there, ..

Why? What business is that of yours?
What does this ability offer to the end user?

--
Andrew Thompson
http://www.athompson.info/andrew/

Message posted via JavaKB.com
http://www.javakb.com/Uwe/Forums.aspx/java-general/200711/1

 

[Roedy Green]

http://www.someplace.net/documentation/
http://www.someplace.net/about/
http://www.someplace.net/images/

Are there Java methods that facilitate this?

In general no. It is considered confidential information. Sometimes a
server will give you a directory listing in HTML if you give it an URL
of a directory without in index.html file in it.

 

[Sherman Pendley]

There is no easy way to do that, unless someplace.net gives you a
listing page. Generally, in order to do that, you have to have either
direct access to the disk, access to an FTP account on that machine,
or you have to crawl the web site and parse out the urls.

Note, this is not a limitation of Java, but simply a result of the way
http works.

In particular, note that documentation, about, and images may not even be
directories at all. A content-management system could use them as keys
into a database of managed documents, for example, or as category keywords
that are used to dynamically assemble a list of documents in the specified
category.

sherm--

 

[Lew]

In general no. It is considered confidential information. Sometimes a
server will give you a directory listing in HTML if you give it an URL
of a directory without in index.html file in it.

In addition, many directories on the server hard drive, while they are
subdirectories of the web document directory, will not be accessible to public
clients. A classic example is the WEB-INF/ directory tree in Java EE apps,
but Apache .htaccess can also restrict directories.

 

[Andrew Thompson]

In general no. It is considered confidential information. Sometimes a
server will give you a directory listing in HTML if you give it an URL
of a directory without in index.html file in it.

An ironic side note is that I'd like my server to *allow*
automatic 'directory listing' for dirs with no index.html,
but cannot figure how to achieve it using the arcane
..contol panel ..thingy the host offers.

If directory indexing *is* turned on, it is a perhaps tedious
but mundane task to parse the resulting HTML, looking for
links to sub-dirs and resources.

--
Andrew Thompson
http://www.athompson.info/andrew/

Message posted via JavaKB.com
http://www.javakb.com/Uwe/Forums.aspx/java-general/200711/1

 

[Chris ( Val )]

An ironic side note is that I'd like my server to *allow*
automatic 'directory listing' for dirs with no index.html,
but cannot figure how to achieve it using the arcane
.contol panel ..thingy the host offers.

[snip]

If it is an IIS web server, you will usually find a checkbox
in the IIS server properties, or for memory, you can even get
to it by right clicking on the virtual directory and editing
the properties.

Tomcat for example has the following setting in its web.xml file:

<init-param>
<param-name>listings</param-name>
<param-value>true</param-value>
</init-param>

But I think it's only good whilst developing.

 

[Alan]

Thanks for the information. I think I shall just follow href
links instead of finding directories.

Thanks, Alan

 

[Andrew Thompson]

Thanks for the information. I think I shall just follow href
links instead of finding directories.

In that case, as Daniel mentioned, search "Web crawler"/
"web crawling". There have been some interesting discussions
about crawlers in these groups, across the ages. As I vaguely
recall there was a source posted for one by Mr Omar Khan
..ahh yes.
<http://groups.google.com/group/comp.lang.java.programmer/msg/df4a6f43d57e3e6a
But please (please, please) respect the directions of the
site's robots.txt (if it has one).

--
Andrew Thompson
http://www.athompson.info/andrew/

Message posted via JavaKB.com
http://www.javakb.com/Uwe/Forums.aspx/java-general/200711/1