remote file glob

Discussion in 'Ruby' started by greg, Nov 2, 2006.

  1. greg

    greg Guest

    What is the fastest way to do a recursive file glob on a remote server?
    Keep in mind that I want to do some filtering based on the directory
    name, file properties, and file extensions.
    (I want to copy these files, but I assume once I have the list, that
    using Net::FTP will be trivial)

    I am concerned that just using ftp.chdir and ftp.list will be slow.
    Perhaps there is a faster way using Net:SSH. I had an idea to try to
    run a ruby program on the server (ruby is installed on the remote
    server)
    cmd = "ruby -e ' #{ File.read( ruby_glob_program ) } ' "

    Net::SSH.start(SERVER, :username => u, :password => p) do |session|
    input, output, error = session.process.popen3( cmd )
    greg, Nov 2, 2006
    #1
    1. Advertising

  2. On 02.11.2006 16:00, greg wrote:
    > What is the fastest way to do a recursive file glob on a remote server?
    > Keep in mind that I want to do some filtering based on the directory
    > name, file properties, and file extensions.
    > (I want to copy these files, but I assume once I have the list, that
    > using Net::FTP will be trivial)
    >
    > I am concerned that just using ftp.chdir and ftp.list will be slow.
    > Perhaps there is a faster way using Net:SSH. I had an idea to try to
    > run a ruby program on the server (ruby is installed on the remote
    > server)
    > cmd = "ruby -e ' #{ File.read( ruby_glob_program ) } ' "


    That would rather be something like

    ruby -e 'puts Dir["base/**/*.txt"]'
    ruby -r find -e 'Find.find("base") {|f| puts f if /\.txt$/ =~ f }'

    > Net::SSH.start(SERVER, :username => u, :password => p) do |session|
    > input, output, error = session.process.popen3( cmd )


    However, if you just want to find files fast and then copy them, a
    combination of ssh, find, xargs, cp, tar,... might be more efficient.

    Kind regards

    robert
    Robert Klemme, Nov 2, 2006
    #2
    1. Advertising

  3. greg wrote:
    > What is the fastest way to do a recursive file glob on a remote server?
    > Keep in mind that I want to do some filtering based on the directory
    > name, file properties, and file extensions.
    > (I want to copy these files, but I assume once I have the list, that
    > using Net::FTP will be trivial)
    >
    > I am concerned that just using ftp.chdir and ftp.list will be slow.
    > Perhaps there is a faster way using Net:SSH. I had an idea to try to
    > run a ruby program on the server (ruby is installed on the remote
    > server)
    > cmd = "ruby -e ' #{ File.read( ruby_glob_program ) } ' "
    >
    > Net::SSH.start(SERVER, :username => u, :password => p) do |session|
    > input, output, error = session.process.popen3( cmd )
    >
    >
    >

    If you're talking about a Linux or other Unix-like server, indexing a
    filesystem is not something you want to do often. It *is* slow, even
    using native utilities like "find", or its cousin "slocate -u". What
    most sites do is run a cron job in the off hours to build the "slocate
    -u" database, and then do searches using the database.

    If you need more current indexing, you probably need to look at the
    application and have it do its own index maintenance.
    M. Edward (Ed) Borasky, Nov 2, 2006
    #3
  4. greg

    greg Guest

    I think I will just use the find command.

    I did notice that if I do the following:
    session.process.popen3( cmd ) do |input, output, error|
    puts output.read
    end

    For my find command, it will think it is finished reading before find
    is done!
    I solved this by using:

    shell = session.shell.sync
    puts shell.exec( find_cmd ).stdout

    Thanks guys!
    greg, Nov 2, 2006
    #4
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. Georgy Pruss
    Replies:
    15
    Views:
    706
    Tim Roberts
    Dec 1, 2003
  2. Tim Peters
    Replies:
    1
    Views:
    343
    Duncan Booth
    Dec 1, 2003
  3. Sean Berry

    Question about glob.glob <--newbie

    Sean Berry, May 4, 2004, in forum: Python
    Replies:
    3
    Views:
    333
    David M. Cooke
    May 4, 2004
  4. Elbert Lev

    glob.glob unicode bug or feature

    Elbert Lev, Jul 31, 2004, in forum: Python
    Replies:
    5
    Views:
    374
    Neil Hodgson
    Aug 2, 2004
  5. qanda
    Replies:
    1
    Views:
    103
    Tassilo v. Parseval
    Sep 15, 2003
Loading...

Share This Page