Need to download 20000 pdf files

Discussion in 'Perl Misc' started by Hemant, Mar 2, 2005.

  1. Hemant

    Hemant Guest

    I am working on a project that requires me to have access to more than
    20000 pdf files. Any suggestions on how to go about searching over the
    internet and be able to download the files?
    Hemant, Mar 2, 2005
    #1
    1. Advertising

  2. Hemant <> wrote:

    > be able to download the files?



    use LWP::Simple;
    my $pdf = get 'http://some.domain/directory/file.pdf';


    --
    Tad McClellan SGML consulting
    Perl programming
    Fort Worth, Texas
    Tad McClellan, Mar 2, 2005
    #2
    1. Advertising

  3. Hemant

    Peter Wyzl Guest

    "Hemant" <> wrote in message
    news:...
    :I am working on a project that requires me to have access to more than
    : 20000 pdf files. Any suggestions on how to go about searching over the
    : internet and be able to download the files?


    Here is something I ran to download a bunch of .swf files into the swf
    directory on my c: drive
    chdir 'c:/swf';
    for ('001' .. '340'){
    next if (-e "c:/swf/a_0${_}.swf");
    system "lwp-download http://www.swfsite.com/a_0${_}.swf";
    }
    exit;



    lwp-download comes with Activeperl

    The files I need were all named a_0***.swf where *** represents the numbers
    from 001 to 340

    Modify it to suit your needs (assuming you have Windows and lwp-download. I
    don't know if Activestate's perl for Unix includes lwp-download.



    P

    --
    print "Just another Perl Hacker";
    Peter Wyzl, Mar 2, 2005
    #3
  4. Hemant wrote:

    > I am working on a project that requires me to have access to more than
    > 20000 pdf files. Any suggestions on how to go about searching over the
    > internet and be able to download the files?


    Assuming one directory tree:
    http://www.gnu.org/software/wget/wget.html

    gtoomey
    Gregory Toomey, Mar 3, 2005
    #4
  5. On Wed, 02 Mar 2005 12:22:56 -0800, Hemant wrote:

    > I am working on a project that requires me to have access to more than
    > 20000 pdf files. Any suggestions on how to go about searching over the
    > internet and be able to download the files?


    er.. from all the other posts it seems an easy task
    but I dont get it.. are you asking how to just download the pdf
    documents.. or search them out over the net.. or search out random pdf
    documents for downloading?
    --

    Hardware, n.: The parts of a computer system that can be kicked
    Shane (aka froggy), Mar 3, 2005
    #5
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. Replies:
    1
    Views:
    447
    =?Utf-8?B?bGF0aGEgdmFsbGluYXlhZ2Ft?=
    May 5, 2005
  2. Peter

    20000 List Items

    Peter, Jan 27, 2006, in forum: ASP .Net
    Replies:
    3
    Views:
    354
  3. JuHui
    Replies:
    1
    Views:
    270
    Bruno Desthuilliers
    Mar 17, 2006
  4. Ricardo Pog
    Replies:
    1
    Views:
    376
    Austin Ziegler
    Mar 26, 2008
  5. David Mark

    SproutCore--over 20000 lines of new code!

    David Mark, Dec 31, 2009, in forum: Javascript
    Replies:
    49
    Views:
    334
    David Mark
    Jan 14, 2010
Loading...

Share This Page