Do arrays use up alot of memory space?

[Clement Ow]

I just started ruby not too long ago and I'm really new to the language.
But I came out with something; to copy files from path names to path
names using each respecive array elements.

--------------------------------------------------------------------------------
$options=
["2008*", "2008*", "700*", "2008*", "2008*"]

$source=
%w[C:/movtest/testing
C:/movtest/testing/new
U:/movtest/source
U:/movtest/new
U:/movtest/new1]

$dest=
%w[U:/test_1/
U:/dest1/
U:/dest2/
U:/dest3/
U:/dest4/]

while i<=j && i1<=j1 && i2<=j2

Dir.chdir($source)
print "\nSource: " + Dir.getwd + "\t\n"
print "Dest: " + $dest[i1] + "\n"
print "Options: " + $options[i2] +"\n"
FileUtils.cp_r Dir.glob($options[i2]), $dest[i1]
print "File Mov Test:Success"
i+=1
i1+=1
i2+=1
end

--------------------------------------------------------------------------------
Apparently, when moving large files (i.e file size 50mb) it takes
relatively long as when i use ROBOCOPY (a robust copying software by
MSServer), but I cant use robocopy because it has it's limitations,hence
i used ruby. As I will be running the ruby program on a server, I cant
afford to use too much memory while transferring files from one folder
to another as the server is used for more impt tasks like running impt
applications in the office.

So is there anything that i could do to quicken the copying process
without compromising on the ability to have different path names and
options for copying files? (i understand that arrays do take up more
memory)
Thanks in advance. =)

 

[7stud --]

--------------------------------------------------------------------------------
$options=
["2008*", "2008*", "700*", "2008*", "2008*"]

$source=
%w[C:/movtest/testing
C:/movtest/testing/new
U:/movtest/source
U:/movtest/new
U:/movtest/new1]

$dest=
%w[U:/test_1/
U:/dest1/
U:/dest2/
U:/dest3/
U:/dest4/]

You have 3 arrays with 5 short strings in them. A character in a string
takes up 1 byte of memory. I don't know exactly what the overhead for
an array is, but it is comparatively tiny--although you never know with
ruby. I would guess all your arrays would take up less than 1,000
bytes or 1 kB in memory. That's tiny.

while i<=j && i1<=j1 && i2<=j2

Dir.chdir($source)
print "\nSource: " + Dir.getwd + "\t\n"
print "Dest: " + $dest[i1] + "\n"
print "Options: " + $options[i2] +"\n"
FileUtils.cp_r Dir.glob($options[i2]), $dest[i1]
print "File Mov Test:Success"
i+=1
i1+=1
i2+=1
end

--------------------------------------------------------------------------------
Apparently, when moving large files (i.e file size 50mb) it takes
relatively long as when i use ROBOCOPY (a robust copying software by
MSServer), but I cant use robocopy because it has it's limitations,hence
i used ruby. As I will be running the ruby program on a server, I cant
afford to use too much memory while transferring files from one folder
to another as the server is used for more impt tasks like running impt
applications in the office.

So is there anything that i could do to quicken the copying process
without compromising on the ability to have different path names and
options for copying files? (i understand that arrays do take up more
memory)
Thanks in advance. =)

 

[Robert Dober]

--------------------------------------------------------------------------------
$options=
["2008*", "2008*", "700*", "2008*", "2008*"]

$source=
%w[C:/movtest/testing
C:/movtest/testing/new
U:/movtest/source
U:/movtest/new
U:/movtest/new1]

$dest=
%w[U:/test_1/
U:/dest1/
U:/dest2/
U:/dest3/
U:/dest4/]

You have 3 arrays with 5 short strings in them. A character in a string
takes up 1 byte of memory. I don't know exactly what the overhead for
an array is, but it is comparatively tiny--although you never know with
ruby.

A well judged statement again....

For Ruby 1.9 the overhead of an array is one long and 4 VALUES, I am
not a VM Guru but it would surprise me a lot if a VALUE used more than
4 bytes given their semantic meanings shared flags and klass, the
fourth being a pointer to the content.
I am sure that someone will correct me if I am wrong but I estimate
the overhead of less than 21 bytes.

I would guess all your arrays would take up less than 1,000

yes that seems indeed a save guess....

HTH
Robert

 

[Clement Ow]

Thanks Botp and Philip for the help! anyway, ans philip, the network
i'm running is quite optimal though not really sure about the speeds, as
I'm running this program in a company with alot of servers dealing with
loads of jobs a day

From: Clement Ow [mailto:[email protected]]
# $options=
you may remove the "$" sign

i use the "$" sign because I have a config file that holds all my
values, but I didnt include it for the purpose of this example

# U:/movtest/source
# U:/movtest/new
# U:/movtest/new1]
#
# $dest=
# %w[U:/test_1/
# U:/dest1/
# U:/dest2/
# U:/dest3/
# U:/dest4/]
#
# while i<=j && i1<=j1 && i2<=j2

hmmm, you just created 3 parallel arrays. Either you can create one
multi-dimensional or a hash.


# Dir.chdir($source)

you may not need this

# print "\nSource: " + Dir.getwd + "\t\n"
# print "Dest: " + $dest[i1] + "\n"
# print "Options: " + $options[i2] +"\n"
# FileUtils.cp_r Dir.glob($options[i2]), $dest[i1]
# print "File Mov Test:Success"
# i+=1
# i1+=1
# i2+=1

if you modify your code, you may not need those increments

# end
# ------------------
# Apparently, when moving large files (i.e file size 50mb) it takes
# relatively long as when i use ROBOCOPY (a robust copying software by
# MSServer),

robocopy seems to do stat on the filetransfer speed to get the optimum
chunk to transfer. You can do something like that in ruby, but you'll
have to work harder (since your code/logic will get a bit longer)

# but I cant use robocopy because it has it's limitations

it's not perfect, and w ruby, you can beat it

yea, limitations being that I cant have exceptions while selecting a
serious of folders. EG. I wanna select files and folders that start with
2008 but I want to keep the last day of the month untouched.(still yet
to figure out this part though)

So, if i were to tackle this, i'd first do, (note, this is ruby
pseudocode, ergo untested since i do not have time and space to test it
now...)


source=%w[C:/movtest/testing
C:/movtest/testing/new
U:/movtest/source
U:/movtest/new
U:/movtest/new1]

dest=%w[U:/test_1/
U:/dest1/
U:/dest2/
U:/dest3/
U:/dest4/]

selections=["2008*", "2008*", "700*", "2008*", "2008*"]

# here i combine those 3 arrays into 1 multi-dim array
sd_a=source.zip(dest,options)

hmmm, would combining the arrays be feasible if i were to add on more
values to the either of the arrays?

sd_a.each do |sd|
source, destination, selections = sd
src = File.join source,selections
puts "Source: #{src}"
puts "Dest: #{destination}"

# i comment the 2 lines below for you to choose bw recurse or not
# again, if you recurse, check for directory overlaps

# FileUtils.cp src, destination
# FileUtils.cp_r src, destination

This command runs an error because the value of src is
"C:/movtest/test/2008*" and it doesnt recognise this wildcard but
instead looks for the file which has the name 2008*.
so i tried to chang it alil:

Dir.chdir(source)
FileUtils.cp_r Dir.glob('2008*'), source, destination

But still runs an error here showing :
Source: C:/movtest/testing/2008*
Dest: U:/test_1/
C:/movtest/testing
c:/ruby/lib/ruby/1.8/fileutils.rb:1438:in `delete': can't convert Symbol
into St
ring (TypeError)
from c:/ruby/lib/ruby/1.8/fileutils.rb:1438:in
`fu_check_options'
from c:/ruby/lib/ruby/1.8/fileutils.rb:1437:in `each'
from c:/ruby/lib/ruby/1.8/fileutils.rb:1437:in
`fu_check_options'
from c:/ruby/lib/ruby/1.8/fileutils.rb:418:in `cp_r'
from testing.rb:44
from testing.rb:32:in `each'
from testing.rb:32

I have tried ways and means to try correct it but to no avail. Is there
anything majorly wrong with my code?

 

[Clement Ow]

From: (e-mail address removed)
# > # here i combine those 3 arrays into 1 multi-dim array
# > sd_a=source.zip(dest,options)
#
# hmmm, would combining the arrays be feasible if i were to add on more
# values to the either of the arrays?

i zipped it because,
1. i was too lazy to retype it again
2. i knew it would work since your arrays are one-to-one on each other

# > sd_a.each do |sd|
# > source, destination, selections = sd
# > src = File.join source,selections
# > puts "Source: #{src}"
# > puts "Dest: #{destination}"
# >
# > # i comment the 2 lines below for you to choose bw recurse or not
# > # again, if you recurse, check for directory overlaps
# >
# > # FileUtils.cp src, destination
# > # FileUtils.cp_r src, destination
#
# This command runs an error because the value of src is
# "C:/movtest/test/2008*" and it doesnt recognise this wildcard but
# instead looks for the file which has the name 2008*.
# so i tried to chang it alil:

no problem, just do

FileUtils.cp Dir.glob(src), destination

that should work because i just tested it here

kind regards -botp

it works
But one issue, when i changed to FileUtils.mv Dir.glob(src), destination
because i needed to move the files, it shows a big error msg:
c:/ruby/lib/ruby/1.8/fileutils.rb:505:in `rename': Permission denied -
C:/movtes
t/testing/20080321 or U:/test_1/20080321 (Errno::EACCES)
from c:/ruby/lib/ruby/1.8/fileutils.rb:505:in `move'
from c:/ruby/lib/ruby/1.8/fileutils.rb:1395:in
`fu_each_src_dest'
from c:/ruby/lib/ruby/1.8/fileutils.rb:1404:in
`fu_each_src_dest0'
from c:/ruby/lib/ruby/1.8/fileutils.rb:1402:in `each'
from c:/ruby/lib/ruby/1.8/fileutils.rb:1402:in
`fu_each_src_dest0'
from c:/ruby/lib/ruby/1.8/fileutils.rb:1393:in
`fu_each_src_dest'
from c:/ruby/lib/ruby/1.8/fileutils.rb:494:in `move'
from testing.rb:42
from testing.rb:32:in `each'
from testing.rb:32

It seems like some problem with the access rights.. Any advice?
I'm still quite new to ruby so ya, but I'm really thankful for the help
so far though.

 

[Phillip Gawlowski]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Clement Ow wrote:

|
| It seems like some problem with the access rights.. Any advice?
| I'm still quite new to ruby so ya, but I'm really thankful for the help
| so far though.

Yeah, contact your sysadmin, if you have write permission.

If you are the admin (and executing the script with admin rights), make
sure no other programs are using the directory/file (file locks).

That means, you can't have the script in the directory that you move.

(Sounds obvious, but it happens to the best...)

- --
Phillip Gawlowski
Twitter: twitter.com/cynicalryan

You were ever good at sudden commendations.
~ -- William Shakespeare (1564-1616), King Henry VIII
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.8 (MingW32)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iEYEARECAAYFAkf8NYIACgkQbtAgaoJTgL+/8wCfbBnRMgzG/oV0pkaAa67EmTLX
eHcAn2db9lJ0bNi6Iti/NUQU71lIn3w6
=SjNH
-----END PGP SIGNATURE-----

 

[Clement Ow]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Clement Ow wrote:

|
| It seems like some problem with the access rights.. Any advice?
| I'm still quite new to ruby so ya, but I'm really thankful for the help
| so far though.

Yeah, contact your sysadmin, if you have write permission.

If you are the admin (and executing the script with admin rights), make
sure no other programs are using the directory/file (file locks).

Im running the script with admin rights.

That means, you can't have the script in the directory that you move.

The script is residing in C:\rubytest which is not in any of the path
names above.
And I'm quite sure it's not due to the options i.e Verbose, force or
noop.
Tried it with the force option (returns no error) but no file move was
done.
hmmm, which makes it more puzzling..

 

[Phillip Gawlowski]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Clement Ow wrote:

| The script is residing in C:\rubytest which is not in any of the path
| names above.
| And I'm quite sure it's not due to the options i.e Verbose, force or
| noop.
| Tried it with the force option (returns no error) but no file move was
| done.
| hmmm, which makes it more puzzling..

Are you sure there are no open file handles? FileMon should reveal this:
http://technet.microsoft.com/en-us/sysinternals/bb896642.aspx

If there are no file handles shown, check if you can move the
directories/files via explorer.

- --
Phillip Gawlowski
Twitter: twitter.com/cynicalryan

[Abstract art is] a product of the untalented, sold by the unprincipled
~ to the utterly bewildered.
~ -- Al Capp
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.8 (MingW32)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iEYEARECAAYFAkf8OPUACgkQbtAgaoJTgL9anQCbBvBvvBkraRKzy5flCuycYOeS
D3sAn16d9VsMJzZx6wm8TkVfC4BiIS3k
=24fF
-----END PGP SIGNATURE-----

 

[Clement Ow]

Are you sure there are no open file handles? FileMon should reveal this:
http://technet.microsoft.com/en-us/sysinternals/bb896642.aspx

If there are no file handles shown, check if you can move the
directories/files via explorer.

Hmmm, i dont quite understand about the open file handles..
Do you mean if the folder is being in use without the knowledge of the
user which in this case is myslelf? If so, I dont see the files being
used by other programs whatsoever. I even closed explorer and run the
script, but to no avail..

-in desperate need for help

 

[Phillip Gawlowski]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Clement Ow wrote:
| Phillip Gawlowski wrote:
|
|> Are you sure there are no open file handles? FileMon should reveal this:
|> http://technet.microsoft.com/en-us/sysinternals/bb896642.aspx
|>
|> If there are no file handles shown, check if you can move the
|> directories/files via explorer.
|
| Hmmm, i dont quite understand about the open file handles..
| Do you mean if the folder is being in use without the knowledge of the
| user which in this case is myslelf? If so, I dont see the files being
| used by other programs whatsoever. I even closed explorer and run the
| script, but to no avail..

There still could be something using the folder or files, without your
knowledge.

In the following (broad) scenarios, Windows blocks access to files:

A) You don't have permission to change these files. Not even the
Administrator account can change all files in Windows, usually deep
within Windows.

However, access control can change that (a normal user, for example,
doesn't have write access to %programfiles%, where applications install
themselves by default, or even using EFS.

B) The file is still in use by a program. Any program. Either locally or
via a network, Windows cannot delete that file.

You can use FileMon to investigate which files are currently open. Just
closing Explorer doesn't help (a shell can still be pointing there, a
Word document could still be open, a backup tool or anti virus
application is using it...)


- --
Phillip Gawlowski
Twitter: twitter.com/cynicalryan

~ "Until you stalk and overrun, you can't devour anyone."
~ --- Hobbes
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.8 (MingW32)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iEYEARECAAYFAkf8RmYACgkQbtAgaoJTgL+/QQCZAXynHuARN160ZnnJFfY8vKUb
Fo0AnjgNFH4wjtzr8kJFFDzV+soYe3fT
=n/qR
-----END PGP SIGNATURE-----

 

[Todd Benson]

Thanks for the prompt reply, 7student and Robert =) But while
transferring files, it does take a significant amount time though, so
the primary reason does not lie in the arrray?

Yes, that's correct.

If so, is there any way I
can quicken the process of transferring files?

I don't think this is a Ruby problem. Possibly network and/or program?

Though it has little, if anything, to do with your original post; is
there a good reason to include a sub-directory in the same recursive
listing (I'm looking at C:/movtest/testing and
C:/movtest/testing/new)? Just a side question.

Todd

 

[Clement Ow]

Yes, that's correct.


I don't think this is a Ruby problem. Possibly network and/or program?

Though it has little, if anything, to do with your original post; is
there a good reason to include a sub-directory in the same recursive
listing (I'm looking at C:/movtest/testing and
C:/movtest/testing/new)? Just a side question.

This is just for testing purposes.
Apparently, the script fails to run only when I try to move files.
Copying files using the cp_r command seems to be working fine though.
Reckon it has got to do with the 'Read Only' option when i click on the
folder's properties. But then again, when i tried to check randomly any
folder on my local disk, it all shows that it's 'Read Only' and that I
have access permission to modify anything. (i cant uncheck the 'Read
Only' option though)hmmm, does this mean i cant delete folders and files
if this still happens? Pretty wierd since im the administrator here..