looking for wget-like module for getching software

[Robert P. J. Day]

just getting started with python, and i'm designing a program for
fetching software from the net, given the package name, version
number and/or date stamp, download method (tarball, CVS, etc.) and so
on. i've already got a shell script doing this, but python would
certainly clean up the code a lot.

first question -- does such a utility/module (or something like it)
already exist? that is, with the ability to say, "go get me gcc,
version 3.4.3, as a tarball"? or "go get me glibc, CVS date stamp
xxxx, from that CVS repo"?

as for the second question, the driving force behind a fair bit of
the current program is "wget", which has the ability to search
recursively and bring back a file of a given name (which "curl"
doesn't appear to have).

that is, i can just say, "go get file gcc-3.4.2.tar.bz2", and start
searching at "ftp://pub.gnu.org/pub/gcc". i may not know how far down
in the directory structure that file is, but wget will happily search
recursively until it finds it.

so, any pointers to python utils/modules that do any or all of the
above? thanks.

rday

 

[Stefan Behnel]

that is, i can just say, "go get file gcc-3.4.2.tar.bz2", and start
searching at "ftp://pub.gnu.org/pub/gcc". i may not know how far down
in the directory structure that file is, but wget will happily search
recursively until it finds it.

That sounds pretty inefficient and should produce some load on the server
side. Imagine everyone doing that. Many FTP-servers provide files like
"ls-R.gz" somewhere, meaning, a list of files and directories. And then,
there's Google. Why do you have to use FTP? Ask Google for
"yourfile.tar.bz2 site:the.domain.name". It will very likely return a
suitable URL for you that you can use for download. Just tried it with
"site:ftp.gnu.org gcc-3.4.2.tar.bz2" - works. You can even use that link
and try to replace "http://" with "ftp://" for download. Should work for
many source repositories.

You can use urllib2 for both querying and download.

Stefan

 

[Robert P. J. Day]

That sounds pretty inefficient and should produce some load on the
server side. Imagine everyone doing that. Many FTP-servers provide
files like "ls-R.gz" somewhere, meaning, a list of files and
directories.

i realize that and, as much as possible, i try to start the process
with what i *know* is the actual directory location. but
occasionally, if i can't guarantee an exact directory, i just want the
option to start a bit higher up the directory structure. i wasn't
going to make a habit of it, it's more an absolute last resort, if it
comes down to it.

And then, there's Google. Why do you have to use FTP? Ask Google for
"yourfile.tar.bz2 site:the.domain.name". It will very likely return
a suitable URL for you that you can use for download. Just tried it
with "site:ftp.gnu.org gcc-3.4.2.tar.bz2" - works. You can even use
that link and try to replace "http://" with "ftp://" for download.
Should work for many source repositories.

You can use urllib2 for both querying and download.

and that pretty much answers the question i had. much thanks. and
now that i have my answer, i might have to unsubscribe. you guys sure
are a chatty bunch.

rday

p.s. just kidding.