remote file glob

G

greg

What is the fastest way to do a recursive file glob on a remote server?
Keep in mind that I want to do some filtering based on the directory
name, file properties, and file extensions.
(I want to copy these files, but I assume once I have the list, that
using Net::FTP will be trivial)

I am concerned that just using ftp.chdir and ftp.list will be slow.
Perhaps there is a faster way using Net:SSH. I had an idea to try to
run a ruby program on the server (ruby is installed on the remote
server)
cmd = "ruby -e ' #{ File.read( ruby_glob_program ) } ' "

Net::SSH.start(SERVER, :username => u, :password => p) do |session|
input, output, error = session.process.popen3( cmd )
 
R

Robert Klemme

What is the fastest way to do a recursive file glob on a remote server?
Keep in mind that I want to do some filtering based on the directory
name, file properties, and file extensions.
(I want to copy these files, but I assume once I have the list, that
using Net::FTP will be trivial)

I am concerned that just using ftp.chdir and ftp.list will be slow.
Perhaps there is a faster way using Net:SSH. I had an idea to try to
run a ruby program on the server (ruby is installed on the remote
server)
cmd = "ruby -e ' #{ File.read( ruby_glob_program ) } ' "

That would rather be something like

ruby -e 'puts Dir["base/**/*.txt"]'
ruby -r find -e 'Find.find("base") {|f| puts f if /\.txt$/ =~ f }'
Net::SSH.start(SERVER, :username => u, :password => p) do |session|
input, output, error = session.process.popen3( cmd )

However, if you just want to find files fast and then copy them, a
combination of ssh, find, xargs, cp, tar,... might be more efficient.

Kind regards

robert
 
M

M. Edward (Ed) Borasky

greg said:
What is the fastest way to do a recursive file glob on a remote server?
Keep in mind that I want to do some filtering based on the directory
name, file properties, and file extensions.
(I want to copy these files, but I assume once I have the list, that
using Net::FTP will be trivial)

I am concerned that just using ftp.chdir and ftp.list will be slow.
Perhaps there is a faster way using Net:SSH. I had an idea to try to
run a ruby program on the server (ruby is installed on the remote
server)
cmd = "ruby -e ' #{ File.read( ruby_glob_program ) } ' "

Net::SSH.start(SERVER, :username => u, :password => p) do |session|
input, output, error = session.process.popen3( cmd )
If you're talking about a Linux or other Unix-like server, indexing a
filesystem is not something you want to do often. It *is* slow, even
using native utilities like "find", or its cousin "slocate -u". What
most sites do is run a cron job in the off hours to build the "slocate
-u" database, and then do searches using the database.

If you need more current indexing, you probably need to look at the
application and have it do its own index maintenance.
 
G

greg

I think I will just use the find command.

I did notice that if I do the following:
session.process.popen3( cmd ) do |input, output, error|
puts output.read
end

For my find command, it will think it is finished reading before find
is done!
I solved this by using:

shell = session.shell.sync
puts shell.exec( find_cmd ).stdout

Thanks guys!
 

Ask a Question

Want to reply to this thread or ask your own question?

You'll need to choose a username for the site, which only take a couple of moments. After that, you can post your question and our members will help you out.

Ask a Question

Members online

Forum statistics

Threads
473,744
Messages
2,569,483
Members
44,902
Latest member
Elena68X5

Latest Threads

Top