rsync functionality in Ruby?

P

Paul van Delst

Hello,

I wrote a ruby script to allow me to automatically update a repository working copy from
an unversioned hierarchy. I use rsync to synchronise the directories prior to svn
add/delete and commit. Worked great on my test system (linux). However, because of
security issues, on the system where this script will be used there is no rsync utility
available.

So, I was wondering, is there an existing rsync-like ruby solution? I checked this
newsgroup, rubyforge, raa, etc but couldn't find anything that replicated rsync
functionality (lots of utilities that use/manage rsync though.)

Note that I don't really need the "r" portion of rsync (the sync occurs on the same
filesystem), just the ability to synchronise two hierarchies with "--exclude <files>"
capability.

I thought I'd ask before embarking on a wheel-reinvention project.

cheers,

paulv

p.s. I'm pushing my luck here, but guidance on an algorithm or recipe for sync
functionality would also be appreciated. :eek:)
 
D

dusty

Hello,

I wrote a ruby script to allow me to automatically update a repository working copy from
an unversioned hierarchy. I use rsync to synchronise the directories prior to svn
add/delete and commit. Worked great on my test system (linux). However, because of
security issues, on the system where this script will be used there is no rsync utility
available.

So, I was wondering, is there an existing rsync-like ruby solution? I checked this
newsgroup, rubyforge, raa, etc but couldn't find anything that replicated rsync
functionality (lots of utilities that use/manage rsync though.)

Note that I don't really need the "r" portion of rsync (the sync occurs on the same
filesystem), just the ability to synchronise two hierarchies with "--exclude <files>"
capability.

I thought I'd ask before embarking on a wheel-reinvention project.

cheers,

paulv

p.s. I'm pushing my luck here, but guidance on an algorithm or recipe for sync
functionality would also be appreciated. :eek:)

I haven't found any either, but I do have a nagios plugin using rsync
to check the timestamps on remote files over rsync. New to posting
here, whats the best way to post code? Include it inline? I'd be
happy to share if you want it. I basically use open4 to call rsync
and then parse the results. It might be helpful in your wheel
project. :)

Dusty Doris
 
P

Paul van Delst

I haven't found any either, but I do have a nagios plugin using rsync
to check the timestamps on remote files over rsync. New to posting
here, whats the best way to post code? Include it inline? I'd be
happy to share if you want it. I basically use open4 to call rsync
and then parse the results. It might be helpful in your wheel
project. :)

Hi,

Thanks for the reply. But, if I understand your reply, I can't use anything that uses unix
rsync under the covers, because there is no rsync available on the system in question.

cheers,

paulv
 
C

Corey Jewett

You should look at FileUtils::cp_r with the :preserve option. It's
not the same as rsync by any stretch, but it should be fairly similar
assuming you didn't need --delete or --include/--exclude patterns. If
you do need those, then the source of FileUtils is probably a pretty
good place to start rolling your own since it properly does symlinks,
permissions, times, and other metadata.

Corey
 
R

Robert Klemme

You should look at FileUtils::cp_r with the :preserve option. It's not
the same as rsync by any stretch, but it should be fairly similar
assuming you didn't need --delete or --include/--exclude patterns. If
you do need those, then the source of FileUtils is probably a pretty
good place to start rolling your own since it properly does symlinks,
permissions, times, and other metadata.

Other options for unidirectional update on system with GNU utils are:

- use cp -au
- maybe you can also facilitate tar

You can build synchronization out of these by invoking then twice, i.e.
once for each direction.

Other than that it should not be too difficult to implement something on
your own using methods in class File.

Kind regards

robert
 
B

Bob Hutchison

I haven't found any either, but I do have a nagios plugin using rsync
to check the timestamps on remote files over rsync. New to posting
here, whats the best way to post code? Include it inline? I'd be
happy to share if you want it. I basically use open4 to call rsync
and then parse the results. It might be helpful in your wheel
project. :)

Is this rsync+ssh? If so, what did you handle passwords when making
the connection (I wound up going though disgusting conniptions using
expect)? I know that you can avoid login by setting up .ssh/
known_hosts but that is frowned upon in some circles.

Cheers,
Bob
Dusty Doris

----
Bob Hutchison -- tumblelog at http://
www.recursive.ca/so/
Recursive Design Inc. -- weblog at http://www.recursive.ca/
hutch
http://www.recursive.ca/ -- works on http://www.raconteur.info/
cms-for-static-content/home/
 
D

Dick Davies

Is this rsync+ssh? If so, what did you handle passwords when making
the connection (I wound up going though disgusting conniptions using
expect)? I know that you can avoid login by setting up .ssh/
known_hosts but that is frowned upon in some circles.

If you're dicking around with expect you have the
password in plain text somewhere along the line, so you'd be lucky to
avoid frowns even then :)

;Use RSA authentication - a blank passphrase on the key (for cronjobs), or
use ssh-agent (for everything else - unit testing etc).
 
B

Bil Kleb

Paul said:
Hello,
Hi.

I wrote a ruby script to allow me to automatically update a repository
working copy from an unversioned hierarchy.

Is there anyway to use SVN's import command to get you 80% of
the way there? See, for example,

http://subversion.tigris.org/faq.html#in-place-import

Or, some combination of `cp` and `svn diff`? E.g.,

svn co working_copy
cp -r unversioned working_copy
svn diff > patch
[process the patch results]

or perhaps some sort of inverse, e.g.,

U_DIRS=`find unversioned -type d`
U_FILES=`find unversioned -type f`
svn co working_copy
cd working_copy
S_DIRS=`find . \( -name .svn -prune \) -o -type d -print`
S_FILES=`find . \( -name .svn -prune \) -o -type f -print`
svn rm --force [ S_DIRS - U_DIRS ]
svn rm --force [ S_FILES - U_FILES ]
cp -r unversioned/* .
svn ci -m'To sync with unversioned.'

Regards,
 
B

Bob Hutchison

If you're dicking around with expect you have the
password in plain text somewhere along the line, so you'd be lucky to
avoid frowns even then :)

I know, funny isn't it :)
;Use RSA authentication - a blank passphrase on the key (for
cronjobs), or
use ssh-agent (for everything else - unit testing etc).

I think I'm going to push a bit on the RSA authentication, see where
I get.

Cheers,
Bob

----
Bob Hutchison -- tumblelog at http://
www.recursive.ca/so/
Recursive Design Inc. -- weblog at http://www.recursive.ca/
hutch
http://www.recursive.ca/ -- works on http://www.raconteur.info/
cms-for-static-content/home/
 
D

dusty

Is this rsync+ssh? If so, what did you handle passwords when making
the connection (I wound up going though disgusting conniptions using
expect)? I know that you can avoid login by setting up .ssh/
known_hosts but that is frowned upon in some circles.

Cheers,
Bob




----
Bob Hutchison -- tumblelog at http://www.recursive.ca/so/
Recursive Design Inc. -- weblog athttp://www.recursive.ca/
hutchhttp://www.recursive.ca/ -- works onhttp://www.raconteur.info/
cms-for-static-content/home/

Sorry I took so long to reply. Forgot about this thread. I am using
straight rsync. Running an rsync daemon on the servers I want. Then
I setup modules for the access. The reason I do this is to avoid
having ssh keys with blank passwords on my servers. Also, this is
only available on the internal network, so I'll also use things like
firewall rules and access-list in rsync to make sure only the machines
I want to get rsync access can.

Here's an example of a class that calls rsync to get a file listing in
a module.

----

require 'rubygems'
require 'open4'

class RsyncUtils

attr_reader :rsync

def initialize(path='')
paths = ['/bin',
'/usr/bin',
'/usr/local/bin',
'/opt/local/bin'
]
paths << path unless path.empty?
paths.each() do |path|
if(test(?x, "#{path}/rsync"))
@rsync = "#{path}/rsync"
end
end
raise "Cannot find rsync, please specify the path" unless @rsync
end

def list(hostname,path,recurs=false)
recursive = "--recursive" if recurs
r = run_command("#{@rsync} #{recursive} #{hostname}::#{path}")
if r.exitstatus == 0
r.results = RsyncFile.parse_list(r.stdout)
end
r
end

def list_recursive(hostname,path)
list(hostname,path,true)
end

private
def run_command(command)
pid, stdin, stdout, stderr = Open4.popen4(command)
ignored, status = Process::waitpid2 pid
RsyncResult.new(stdout.readlines,
stderr.readlines,
status.exitstatus)
end

end


Here is a class that hands back my results
----------

class RsyncResult

attr_reader :stdout, :stderr, :exitstatus
attr_accessor :results

def initialize(stdout,stderr,exitstatus)
@stdout = stdout
@stderr = stderr
@exitstatus = exitstatus
end

end

Here is the class that parses the filelisting to tell me about the
files

------

class RsyncFile

attr_reader :name, :type, :time, :size

FILE_TYPE = 0
DIRECTORY_TYPE = 1
UNKNOWN_TYPE = 2

def initialize(name,type,time,size)
@name = name
@time = time
@size = size
@type = type
end

def is_file?
@type == FILE_TYPE
end

def is_dir?
@type == DIRECTORY_TYPE
end

def self.parse_list(files)
results = []
files.each do |file|
parts = file.split
case parts[0]
when /^[rwx-]{10}$/
type = FILE_TYPE
when /^d[rwx-]{9}$/
type = DIRECTORY_TYPE
else
type = UNKNOWN_TYPE
end
time = Time.local(*ParseDate::parsedate("#{parts[2]}
#{parts[3]}"))
size = parts[1].to_i
name = parts[4]
results << RsyncFile.new(name,type,time,size)
end
results
end
end

Here is how I'd use it in a script.

rsync = RsyncUtils.new
check = rsync.list(@options["--hostname"],@options["--
path"],@options["--recursive"])
if check.exitstatus > 0
raise "#{check.stderr}"
end

#This would list all the files - the rsync output
if @options["--verbose"]
puts check.stdout
exit(0)
end

Now assume I have another class that is do a comparison of the results
for me. That is where I'm calling crit.compare. That's not as
important, as you can see I'm simply iterating through the results and
checking the Time on the current server vs. the time on the remove
file.

check.results.each do |r|
if r.type == RsyncFile::FILE_TYPE
diff = ((Time.now - r.time) - @options["--drift"].to_i).to_i
critical_files << r.name unless crit.compare(diff)
warning_files << r.name unless warn.compare(diff)
end
end


So, basically I would call the script

../rsync_file_age.rb --hostname somehost --path somedir/anotherdir/ --
warning 0:10 --critical 10:20

That would compile a command to RsyncUtils such as

rsync somehost::somedir/anotherdir/

It would then use open4 to run that command and would parse the
results.

Hope that is helpful.

BTW an rsyncd.conf file might look like this and would provide the
somedir module.


pid file = /var/run/rsyncd.pid
hosts allow = 10.0.0.10

uid = nobody
gid = nogroup
use chroot = no
max connections = 4
syslog facility = local5

[somedir]
path = /var/somedir
read only = true
 

Ask a Question

Want to reply to this thread or ask your own question?

You'll need to choose a username for the site, which only take a couple of moments. After that, you can post your question and our members will help you out.

Ask a Question

Members online

No members online now.

Forum statistics

Threads
473,763
Messages
2,569,562
Members
45,038
Latest member
OrderProperKetocapsules

Latest Threads

Top