How to download file from web site

Discussion in 'Ruby' started by Timothy Hunter, Mar 3, 2007.

  1. Okay, it's easy to download a file from an FTP server with Net::FTP.
    Given a URL like this:
    "http://rubyforge.org/frs/download.php/17670/RMagick-1.15.3.tar.gz" is
    there a way using pure Ruby to download this file?
    Timothy Hunter, Mar 3, 2007
    #1
    1. Advertising

  2. Timothy Hunter

    Guest

    On Sat, 3 Mar 2007, Timothy Hunter wrote:

    > Okay, it's easy to download a file from an FTP server with Net::FTP. Given a
    > URL like this:
    > "http://rubyforge.org/frs/download.php/17670/RMagick-1.15.3.tar.gz" is there
    > a way using pure Ruby to download this file?
    >


    harp:~ > cat a.rb
    require 'open-uri'
    uri = 'http://rubyforge.org/frs/download.php/17670/RMagick-1.15.3.tar.gz'

    open(uri) do |fin|
    open(File.basename(uri), 'w') do |fout|
    while(buf = fin.read(8192))
    fout.write buf
    end
    end
    end

    harp:~ > ruby a.rb

    harp:~ > new RMagick-1.15.3.tar.gz
    -rw-rw-r-- 1 ahoward ahoward 1048764 Mar 3 08:00 RMagick-1.15.3.tar.gz


    -a
    --
    be kind whenever possible... it is always possible.
    - the dalai lama
    , Mar 3, 2007
    #2
    1. Advertising

  3. wrote:
    > On Sat, 3 Mar 2007, Timothy Hunter wrote:
    >
    >> Okay, it's easy to download a file from an FTP server with Net::FTP.
    >> Given a URL like this:
    >> "http://rubyforge.org/frs/download.php/17670/RMagick-1.15.3.tar.gz"
    >> is there a way using pure Ruby to download this file?
    >>

    >
    > harp:~ > cat a.rb
    > require 'open-uri'
    > uri = 'http://rubyforge.org/frs/download.php/17670/RMagick-1.15.3.tar.gz'
    >
    > open(uri) do |fin|
    > open(File.basename(uri), 'w') do |fout|
    > while(buf = fin.read(8192))
    > fout.write buf
    > end
    > end
    > end
    >
    > harp:~ > ruby a.rb
    >
    > harp:~ > new RMagick-1.15.3.tar.gz
    > -rw-rw-r-- 1 ahoward ahoward 1048764 Mar 3 08:00
    > RMagick-1.15.3.tar.gz
    >
    >
    > -a

    Thanks again, Ara!
    Timothy Hunter, Mar 3, 2007
    #3
  4. Robert Klemme, Mar 3, 2007
    #4
  5. Timothy Hunter

    Guest

    On Sun, 4 Mar 2007, Timothy Hunter wrote:

    >>
    >> -a

    > Thanks again, Ara!
    >


    no way - thank you! i'm preparing a class right now, here's one of many
    examples i've got put together:


    #
    # gem install RMagick @ http://rmagick.rubyforge.org/
    #
    require "rubygems"
    require "RMagick"
    #
    # converts all images named on the command-line into thumbnails
    #
    ARGV.each do |path|
    path = File.expand_path path
    dirname, basename = File.split path
    base, ext = basename.split %r/[.]/, 2

    img = Magick::ImageList.new path

    thumb = img.thumbnail 96, 96
    thumb_path = File.join dirname, "#{ base }.thumb.#{ ext }"
    thumb.write thumb_path

    puts thumb_path
    end



    thanks for rmagick.

    -a
    --
    be kind whenever possible... it is always possible.
    - the dalai lama
    , Mar 3, 2007
    #5
  6. wrote:
    > On Sun, 4 Mar 2007, Timothy Hunter wrote:
    >
    >>>
    >>> -a

    >> Thanks again, Ara!
    >>

    >
    > no way - thank you! i'm preparing a class right now, here's one of many
    > examples i've got put together:
    >
    >
    > #
    > # gem install RMagick @ http://rmagick.rubyforge.org/
    > #
    > require "rubygems"
    > require "RMagick"
    > #
    > # converts all images named on the command-line into thumbnails
    > #
    > ARGV.each do |path|
    > path = File.expand_path path
    > dirname, basename = File.split path
    > base, ext = basename.split %r/[.]/, 2
    >
    > img = Magick::ImageList.new path
    >
    > thumb = img.thumbnail 96, 96
    > thumb_path = File.join dirname, "#{ base }.thumb.#{ ext }"
    > thumb.write thumb_path
    >
    > puts thumb_path
    > end
    >
    >
    >
    > thanks for rmagick.
    >
    > -a

    You're welcome!

    The reason I'm asking these questions is because I'm working on a Ruby
    program to download and install ImageMagick, all its delegate libraries,
    and RMagick, on OS X, starting from scratch and not using MacPorts.

    BTW, I saw your comment about the rmagick gem. You're right, it's
    screwy. I'm hoping to mothball it in RMagick 2.0.0, if I can figure out
    a way to replace my autoconf-generated configure script with pure Ruby.
    Timothy Hunter, Mar 3, 2007
    #6
  7. On Mar 3, 2007, at 9:39 AM, Timothy Hunter wrote:

    > The reason I'm asking these questions is because I'm working on a
    > Ruby program to download and install ImageMagick, all its delegate
    > libraries, and RMagick, on OS X, starting from scratch and not
    > using MacPorts.


    I just spent about a day fixing my RMagick install (on Mac OS X),
    that had become horribly broken, so releasing something like this
    would make you a hero in my eyes.

    James Edward Gray II
    James Edward Gray II, Mar 3, 2007
    #7
  8. James Edward Gray II wrote:
    > On Mar 3, 2007, at 9:39 AM, Timothy Hunter wrote:
    >
    >> The reason I'm asking these questions is because I'm working on a
    >> Ruby program to download and install ImageMagick, all its delegate
    >> libraries, and RMagick, on OS X, starting from scratch and not using
    >> MacPorts.

    >
    > I just spent about a day fixing my RMagick install (on Mac OS X), that
    > had become horribly broken, so releasing something like this would
    > make you a hero in my eyes.
    >
    > James Edward Gray II
    >

    Ouch! Sorry you had trouble. Email me when you have a problem, James.
    I'll be glad to do what I can to help.

    BTW, just last week I updated my "Installing RMagick on OS X HOWTO"
    here: http://rmagick.rubyforge.org/install-osx.html. I guess it's too
    late to help you but maybe it'll help somebody else.
    Timothy Hunter, Mar 3, 2007
    #8
  9. On Mar 3, 2007, at 11:53 AM, Timothy Hunter wrote:

    > James Edward Gray II wrote:
    >> On Mar 3, 2007, at 9:39 AM, Timothy Hunter wrote:
    >>
    >>> The reason I'm asking these questions is because I'm working on a
    >>> Ruby program to download and install ImageMagick, all its
    >>> delegate libraries, and RMagick, on OS X, starting from scratch
    >>> and not using MacPorts.

    >>
    >> I just spent about a day fixing my RMagick install (on Mac OS X),
    >> that had become horribly broken, so releasing something like this
    >> would make you a hero in my eyes.
    >>
    >> James Edward Gray II
    >>

    > Ouch! Sorry you had trouble. Email me when you have a problem,
    > James. I'll be glad to do what I can to help.


    No worries. It wasn't your fault. I broke it.

    James Edward Gray II
    James Edward Gray II, Mar 3, 2007
    #9
  10. On Mar 3, 2007, at 9:53 AM, Timothy Hunter wrote:

    > James Edward Gray II wrote:
    >> On Mar 3, 2007, at 9:39 AM, Timothy Hunter wrote:
    >>
    >>> The reason I'm asking these questions is because I'm working on a
    >>> Ruby program to download and install ImageMagick, all its
    >>> delegate libraries, and RMagick, on OS X, starting from scratch
    >>> and not using MacPorts.

    >>
    >> I just spent about a day fixing my RMagick install (on Mac OS X),
    >> that had become horribly broken, so releasing something like this
    >> would make you a hero in my eyes.
    >>
    >> James Edward Gray II
    >>

    > Ouch! Sorry you had trouble. Email me when you have a problem,
    > James. I'll be glad to do what I can to help.
    >
    > BTW, just last week I updated my "Installing RMagick on OS X HOWTO"
    > here: http://rmagick.rubyforge.org/install-osx.html. I guess it's
    > too late to help you but maybe it'll help somebody else.
    >


    Here is a shell script I wrote a while ago to build ruby/imagemagick/
    rmagick on OSX without using macports. Feel free to use it if you want.

    #!/bin/sh

    # Install Ruby & IMageMagick & RMagick on Macintosh OS X 10.4.7 (Intel)
    # Stock Apple Developer Tools (2.3)

    # Readline
    READLINE_VERSION="5.1"
    wget ftp://ftp.gnu.org/gnu/readline/readline-${READLINE_VERSION}.tar.gz
    tar xzvf readline-${READLINE_VERSION}.tar.gz
    pushd readline-${READLINE_VERSION}
    /configure --prefix=/usr/local
    make
    sudo make install
    popd


    # Ruby

    RUBY_VERSION="1.8.5"

    wget ftp://ftp.ruby-lang.org/pub/ruby/ruby-${RUBY_VERSION}.tar.gz
    tar xfz ruby-${RUBY_VERSION}.tar.gz

    pushd ruby-${RUBY_VERSION}
    /configure
    make
    sudo make install
    popd

    # Ruby Gems

    GEM_VERSION="0.9.2"

    wget http://rubyforge.org/frs/download.php/11289/rubygems-$
    {GEM_VERSION}.tgz
    tar xfz rubygems-${GEM_VERSION}.tgz

    pushd rubygems-${GEM_VERSION}
    sudo ruby setup.rb
    popd

    # ImageMagick

    LPNG_VERSION="1.2.12"

    wget ftp://ftp.simplesystems.org/pub/libpng/png/src/libpng-$
    {LPNG_VERSION}.tar.gz
    tar xfz libpng-${LPNG_VERSION}.tar.gz

    pushd libpng-${LPNG_VERSION}
    /configure
    make
    sudo make install
    popd

    JPEG_VERSION="6b"

    wget http://www.ijg.org/files/jpegsrc.v${JPEG_VERSION}.tar.gz
    tar xfz jpegsrc.v${JPEG_VERSION}.tar.gz

    pushd jpeg-${JPEG_VERSION}
    /configure
    make
    sudo make install
    sudo make install-lib
    popd

    TIFF_VERSION="3.8.2"

    wget ftp://ftp.remotesensing.org/pub/libtiff/tiff-${TIFF_VERSION}.tar.gz
    tar xfz tiff-${TIFF_VERSION}.tar.gz

    pushd tiff-${TIFF_VERSION}
    /configure
    make
    sudo make install
    popd

    IM_VERSION_MAJOR="6.2.9"
    IM_VERSION_MINOR="4"

    IM_VERSION="${IM_VERSION_MAJOR}-${IM_VERSION_MINOR}"

    wget ftp://ftp.imagemagick.net/pub/ImageMagick/ImageMagick-$
    {IM_VERSION}.tar.gz
    tar xfz ImageMagick-${IM_VERSION}.tar.gz

    pushd ImageMagick-${IM_VERSION_MAJOR}
    /configure
    make
    sudo make install
    popd


    sudo gem install rmagick


    Cheers-

    -- Ezra Zygmuntowicz
    -- Lead Rails Evangelist
    --
    -- Engine Yard, Serious Rails Hosting
    -- (866) 518-YARD (9273)
    Ezra Zygmuntowicz, Mar 4, 2007
    #10
  11. Ezra Zygmuntowicz wrote:
    >
    > Here is a shell script I wrote a while ago to build
    > ruby/imagemagick/rmagick on OSX without using macports. Feel free to
    > use it if you want.
    >

    Thanks, Ezra! That's a very nice gift. I'll incorporate all the best
    parts into my script.

    There are a couple of these kinds of bash scripts for installing RMagick
    floating around. None of them implement my exact set of superstitions
    about installing ImageMagick[1], so I thought it would be useful to do
    my own. I'm using Ruby instead of bash, though, since I get very little
    opportunity to write Ruby code.

    [1] "Speed up your RMagick apps in 1 easy step"
    http://rubyforge.org/forum/forum.php?thread_id=10975&forum_id=1618
    Timothy Hunter, Mar 4, 2007
    #11
  12. Timothy Hunter

    Bil Kleb Guest

    wrote:
    >
    > require 'open-uri'
    > uri = 'http://rubyforge.org/frs/download.php/17670/RMagick-1.15.3.tar.gz'
    >
    > open(uri) do |fin|
    > open(File.basename(uri), 'w') do |fout|
    > while(buf = fin.read(8192))
    > fout.write buf
    > end
    > end
    > end


    He asks naively...

    Why is this so complicated, e.g., the 8192 buffered write, etc.?

    Later,
    --
    Bil Kleb
    http://fun3d.larc.nasa.gov
    Bil Kleb, Mar 4, 2007
    #12
  13. On Mar 4, 2007, at 10:10 AM, Bil Kleb wrote:

    > wrote:
    >> require 'open-uri'
    >> uri = 'http://rubyforge.org/frs/download.php/17670/
    >> RMagick-1.15.3.tar.gz'
    >> open(uri) do |fin|
    >> open(File.basename(uri), 'w') do |fout|
    >> while(buf = fin.read(8192))
    >> fout.write buf
    >> end
    >> end
    >> end

    >
    > He asks naively...
    >
    > Why is this so complicated, e.g., the 8192 buffered write, etc.?


    So you don't try to slurp a very large file into Ruby's memory I'm
    guessing.

    James Edward Gray II
    James Edward Gray II, Mar 4, 2007
    #13
  14. Timothy Hunter

    Guest

    On Mon, 5 Mar 2007, Bil Kleb wrote:

    > wrote:
    >>
    >> require 'open-uri'
    >> uri = 'http://rubyforge.org/frs/download.php/17670/RMagick-1.15.3.tar.gz'
    >>
    >> open(uri) do |fin|
    >> open(File.basename(uri), 'w') do |fout|
    >> while(buf = fin.read(8192))
    >> fout.write buf
    >> end
    >> end
    >> end

    >
    > He asks naively...
    >
    > Why is this so complicated, e.g., the 8192 buffered write, etc.?


    well, this will do it too

    def wget src, dst
    open(dst,'w'){|b| open(src){|a| b.write a.read}}
    end

    but if you

    wget 'http://host/huge_giant_output.cgi', 'big.out'

    and then start 1000 of them in threads you might eat a lot of memory

    the 8192 thing is just being robust in the face of potential giant docs. not
    an issue for html probably but the OP was talking about downloading tar.gz
    files, so those could be big. we've got plenty of them that are > 10GB for
    download on our servers....

    cheers.


    -a
    --
    be kind whenever possible... it is always possible.
    - the dalai lama
    , Mar 4, 2007
    #14
  15. Bil Kleb wrote:
    > wrote:
    >>
    >> require 'open-uri'
    >> uri = 'http://rubyforge.org/frs/download.php/17670/RMagick-1.15.3.tar.gz'
    >>
    >> open(uri) do |fin|
    >> open(File.basename(uri), 'w') do |fout|
    >> while(buf = fin.read(8192))
    >> fout.write buf
    >> end
    >> end
    >> end

    >
    > He asks naively...
    >
    > Why is this so complicated, e.g., the 8192 buffered write, etc.?


    It does seem too low-level, doesn't it? Especially in the same snippet
    with open(uri), which abstracts so much.

    Didn't the rio library[1] have a way to abstract this operation?

    This is from the docs:

    > Copy a web-page to a file
    > rio('http://rubydoc.org/') > rio('afile')


    I suppose it's using the same buffering technique underneath that
    abstraction.

    [1] http://rio.rubyforge.org/ (never used it myself)

    --
    vjoel : Joel VanderWerf : path berkeley edu : 510 665 3407
    Joel VanderWerf, Mar 4, 2007
    #15
  16. Timothy Hunter

    Guest

    On Mon, 5 Mar 2007, Joel VanderWerf wrote:

    > Bil Kleb wrote:
    >> wrote:
    >>>
    >>> require 'open-uri'
    >>> uri = 'http://rubyforge.org/frs/download.php/17670/RMagick-1.15.3.tar.gz'
    >>>
    >>> open(uri) do |fin|
    >>> open(File.basename(uri), 'w') do |fout|
    >>> while(buf = fin.read(8192))
    >>> fout.write buf
    >>> end
    >>> end
    >>> end

    >>
    >> He asks naively...
    >>
    >> Why is this so complicated, e.g., the 8192 buffered write, etc.?

    >
    > It does seem too low-level, doesn't it? Especially in the same snippet with
    > open(uri), which abstracts so much.
    >
    > Didn't the rio library[1] have a way to abstract this operation?
    >
    > This is from the docs:
    >
    >> Copy a web-page to a file rio('http://rubydoc.org/') > rio('afile')

    >
    > I suppose it's using the same buffering technique underneath that
    > abstraction.
    >
    > [1] http://rio.rubyforge.org/ (never used it myself)


    reminds me of an RCR i've had in mind:

    io.relay another_io, bufsize

    seems like a nice one huh?

    -a
    --
    be kind whenever possible... it is always possible.
    - the dalai lama
    , Mar 4, 2007
    #16
  17. On 04.03.2007 20:50, wrote:
    > On Mon, 5 Mar 2007, Joel VanderWerf wrote:
    >
    >> Bil Kleb wrote:
    >>> wrote:
    >>>>
    >>>> require 'open-uri'
    >>>> uri =
    >>>> 'http://rubyforge.org/frs/download.php/17670/RMagick-1.15.3.tar.gz'
    >>>>
    >>>> open(uri) do |fin|
    >>>> open(File.basename(uri), 'w') do |fout|
    >>>> while(buf = fin.read(8192))
    >>>> fout.write buf
    >>>> end
    >>>> end
    >>>> end
    >>>
    >>> He asks naively...
    >>>
    >>> Why is this so complicated, e.g., the 8192 buffered write, etc.?

    >>
    >> It does seem too low-level, doesn't it? Especially in the same snippet
    >> with open(uri), which abstracts so much.
    >>
    >> Didn't the rio library[1] have a way to abstract this operation?
    >>
    >> This is from the docs:
    >>
    >>> Copy a web-page to a file rio('http://rubydoc.org/') > rio('afile')

    >>
    >> I suppose it's using the same buffering technique underneath that
    >> abstraction.
    >>
    >> [1] http://rio.rubyforge.org/ (never used it myself)

    >
    > reminds me of an RCR i've had in mind:
    >
    > io.relay another_io, bufsize
    >
    > seems like a nice one huh?


    Yes, although I'm not sure whether "relay" is immediately clear to
    everyone - especially non native speakers. I briefly thought of
    "copy_to" but that is not correct in every case, i.e. not always it is
    actually a copy. What about "forward_to"? Hm...

    Kind regards

    robert
    Robert Klemme, Mar 4, 2007
    #17
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. Ryan Taylor
    Replies:
    2
    Views:
    942
    Ryan Taylor
    Nov 10, 2004
  2. Steve C. Orr [MVP, MCSD]
    Replies:
    0
    Views:
    1,603
    Steve C. Orr [MVP, MCSD]
    Mar 7, 2005
  3. Sam --
    Replies:
    2
    Views:
    604
    Sam --
    Mar 17, 2005
  4. Brett  Kelly
    Replies:
    1
    Views:
    676
    Steve C. Orr [MVP, MCSD]
    Jun 16, 2006
  5. Replies:
    8
    Views:
    404
Loading...

Share This Page