writing to different offsets of a file in parallel

[Ironhide]

perl-newbie, so please excuse if this is trivial

I need a simple way to write to different offsets of a file in
parallel.
How can I achieve that with perl?

thanks for your help in advance

 

[Uri Guttman]

I> perl-newbie, so please excuse if this is trivial
I> I need a simple way to write to different offsets of a file in
I> parallel.

I> How can I achieve that with perl?

before you code it in any language, please clarify what your goals
are. why do you need or want to do this? what do you mean by parallel?
anything done is parallel is not simple in general. you need to be much
better at explaining your problem (which is usually half the work in
getting it solved).

uri

 

[Ironhide]

  I> perl-newbie, so please excuse if this is trivial
  I> I need a simple way to write to different offsets of a file in
  I> parallel.

  I> How can I achieve that with perl?

before you code it in any language, please clarify what your goals
are. why do you need or want to do this? what do you mean by parallel?
anything done is parallel is not simple in general. you need to be much
better at explaining your problem (which is usually half the work in
getting it solved).

uri

Well so here goes
I am trying to write unique pattern to every file block, block size is
512 bytes in this case.

So if do this in perl

my $hell=generate_pattern($file_block_number);

for($i=0;$i<=(total_blocks-1);$i++) {
system("/iotool seek=(512*$i) bs=512 pattern=$hell filename SIZE");
}
Seems to be taking much more time than if I simply fire the command
from the unix command line,
which is pretty obvious.

So, since I have to coding up the i/o part in a perl script, I was
wondering if I could make
this write thingy happen in parallel since I have the benefit of
writing different offsets.

Hope this explains the problem.

thanks again..

_gourab

 

[Peter J. Holzer]

I am trying to write unique pattern to every file block, block size is
512 bytes in this case.

So if do this in perl

my $hell=generate_pattern($file_block_number);

for($i=0;$i<=(total_blocks-1);$i++) {
system("/iotool seek=(512*$i) bs=512 pattern=$hell filename SIZE");
}

Are you invoking an external tool for every 512-byte block here?

If you are then you can probably speed this up by two or three orders of
magnitude by just writing directory to the file.

hp

 

[Jürgen Exner]

Well so here goes
I am trying to write unique pattern to every file block, block size is
512 bytes in this case.

So if do this in perl

my $hell=generate_pattern($file_block_number);

for($i=0;$i<=(total_blocks-1);$i++) {
system("/iotool seek=(512*$i) bs=512 pattern=$hell filename SIZE");

By far the most expensive part is this call to system(). As long as you
can't find a better way to generate your data you most likely out of
luck.

}
Seems to be taking much more time than if I simply fire the command
from the unix command line,
which is pretty obvious.

So, since I have to coding up the i/o part in a perl script, I was
wondering if I could make
this write thingy happen in parallel since I have the benefit of
writing different offsets.

That wouldn't help you one bit because you would still execute the most
expensive operation individually for each 512 byte block. Plus you would
have the overhead of synchronizing the parallel operations and the
parallel disk access.

My suggestion: generate and write your target data in much, much larger
blocks, if possible the whole file at once.
If you can't generate the target data in large blocks, then at least
collect it in RAM and then write large chunks to the HD.

jue

 

[Xho Jingleheimerschmidt]

Well so here goes
I am trying to write unique pattern to every file block, block size is
512 bytes in this case.

So if do this in perl

my $hell=generate_pattern($file_block_number);

Where does $file_block_number come from?

for($i=0;$i<=(total_blocks-1);$i++) {

$i certainly seems to be your file block number. So why
is there a different variable with that name, and $i is not
it?

system("/iotool seek=(512*$i) bs=512 pattern=$hell filename SIZE");
}

Well, that certainly sucks. If you did it in a way that didn't suck,
you probably wouldn't need to do it in parallel.

Since you seem to be writing the same pattern to each block in the file
consecutively, why not just do that in the obvious way?

foreach (1..$total_blocks) { print $fh $hell}

Seems to be taking much more time than if I simply fire the command
from the unix command line,
which is pretty obvious.

Well, you are firing the command in a loop. Doing something hundreds of
times does tend to take longer than doing it once.

Xho