How to I do this in Python ?

[Ganesh Pal]

Hello Friends ,

Iam a newbie to python , Iam writing a small script that would generate
various kinds of files
in the specified path . Iam using sub process module to achieve this , I
have stuck with few basic problems , any help on this would be great

Case (a) :

The below code creates the only one spare file named sp1 ,

# Creating sparse files in the sparse path
sparse_path = os.path.join(path,'sparsefiles')
os.makedirs(sparse_path)
os.chdir(sparse_path)
sparsefiles = "dd if=/dev/zero of=sp1 count=0 bs=1 seek=10G"
process_0 = subprocess.Popen(sparsefiles, stdout=subprocess.PIPE,
stderr=subprocess.PIPE, shell=True)

Current output :
Sp1,

How do I loop my script to create 100 of files like sp1 , sp2 ,sp3,..
sp100 .. using the same syntax
" sparsefiles = "dd if=/dev/zero of=sp1 count=0 bs=1 seek=10G "


Case (2) :

Is there a better way to create the files in Python other than using sub
process module and running dd command as shown below ..

Example :

# creating sparse File
sparse_path = os.path.join(path,'sparsefiles')
os.makedirs(sparse_path)
os.chdir(sparse_path)
sparsefiles = "dd if=/dev/zero of=sp1 count=0 bs=1 seek=10G"
process_0 = subprocess.Popen(sparsefiles, stdout=subprocess.PIPE,
stderr=subprocess.PIPE, shell=True)

# Creating Regular files
Regular_path = os.path.join(path,'regularfiles')
os.makedirs(Regular_path)
os.chdir(Regular_path)
regularfiles = " dd if=/dev/urandom of=file1 count=0 bs=1 seek=10"
process_1 = subprocess.Popen(regularfiles, stdout=subprocess.PIPE,
stderr=subprocess.PIPE, shell=True)


My goal is to create various kinds of files like sparse, regular
,directories, hard and symlinks etc
what would be the best way to do achieve this ?

Regards,
Ganesh

 

[Steven D'Aprano]

Hi Ganesh, and welcome!

Unfortunately, you ask your questions in reverse order. The most general
(and important) question comes last, and the least important first, so
I'm going to slice-and-dice your post and answer from most general to
least.

My goal is to create various kinds of files like sparse, regular
,directories, hard and symlinks etc
what would be the best way to do achieve this ?

Use your shell, such as bash or csh or equivalent. For simple tasks like
that, it will solve the problem much more simply than Python.

There are three good reasons for doing this in Python:

- "This is only a small part of a larger Python application."

- "I'm doing this to learn how to use Python."

- "I really hate my shell."

But of you just want to get the job done, and don't care what language
you use, use the shell.


Now, having said that, I'm going to assume you have a good reason to use
Python:

Case (2) :

Is there a better way to create the files in Python other than using
sub process module and running dd command as shown below ..

Example :

# creating sparse File
sparse_path = os.path.join(path,'sparsefiles')
os.makedirs(sparse_path)
os.chdir(sparse_path)
sparsefiles = "dd if=/dev/zero of=sp1 count=0 bs=1 seek=10G"
process_0 = subprocess.Popen(sparsefiles, stdout=subprocess.PIPE,
stderr=subprocess.PIPE, shell=True)

# Creating Regular files
Regular_path = os.path.join(path,'regularfiles')
os.makedirs(Regular_path)
os.chdir(Regular_path)
regularfiles = " dd if=/dev/urandom of=file1 count=0 bs=1 seek=10"
process_1 = subprocess.Popen(regularfiles, stdout=subprocess.PIPE,
stderr=subprocess.PIPE, shell=True)

What do you mean by "better"? There's probably nothing that will be
faster at rapidly copying bytes from one file to another than dd. But not
with a blocksize of 1 byte at a time. It's more usual to set bs=512 or
bs=1024.

Oh, I see you're not actually writing anything to the file (count=0). In
that case, instead of using dd, you should use touch.

I'm not sure that shell=True is a good idea.

In Python, to create a new empty file, setting its contents to empty if
it already exists:

open("filename", "w").close()


That will open the file, creating it if it doesn't exist, emptying it if
it does, then close it.

To touch a file without emptying it:

open("filename", "a").close()


To make a sparse file, assuming your file system supports it, I believe
you actually have to write at least one byte to the file:

f = open('foo', 'w')
f.seek(10000)
f.write('\0')
f.close()



[...]

How do I loop my script to create 100 of files like sp1 , sp2 ,sp3,..
sp100 .. using the same syntax

To generate the various file names, you need to loop over a counter from
1 to 100, and stick the count into a string. Use a for-loop and the range
function:

for i in range(1, 101):
filename = "sp" + str(i)
open(filename, "a").close()


If you are a C programmer, you might prefer this style:

filename = "sp%d" % i

Or a more object-oriented style:

filename = "sp{:d}".format(i)

 

[Roy Smith]

# Creating sparse files in the sparse path
sparse_path = os.path.join(path,'sparsefiles')
os.makedirs(sparse_path)
os.chdir(sparse_path)
sparsefiles = "dd if=/dev/zero of=sp1 count=0 bs=1 seek=10G"
process_0 = subprocess.Popen(sparsefiles, stdout=subprocess.PIPE,
stderr=subprocess.PIPE, shell=True)

There is no need to shell out to dd just to do this. All dd is doing
for you is seeking to the offset you specify and closing the file. You
can do that entirely in Python code.

http://docs.python.org/2.7/library/stdtypes.html#file.seek

 

[random832]

There is no need to shell out to dd just to do this. All dd is doing
for you is seeking to the offset you specify and closing the file. You
can do that entirely in Python code.

http://docs.python.org/2.7/library/stdtypes.html#file.seek

*ahem*

http://docs.python.org/2.7/library/stdtypes.html#file.truncate

The dd recipe was written some time over a decade ago as the easiest way
to achieve this at a shell prompt (before the truncate tool was added to
coreutils), and no-one's ever thought twice about it.

 

[Ganesh Pal]

Hi Steven ,

Firstly thanks for responding to the question and also guiding me on how to
post the question in the right order ( general to least important order )

Please find the comments >>> inline

My goal is to create various kinds of files like sparse, regular
,directories, hard and symlinks etc
what would be the best way to do achieve this ?

Use your shell, such as bash or csh or equivalent. For simple tasks like
that, it will solve the problem much more simply than Python.

There are three good reasons for doing this in Python:

- "This is only a small part of a larger Python application."

- "I'm doing this to learn how to use Python."

- "I really hate my shell."

But of you just want to get the job done, and don't care what language
you use, use the shell.


Now, having said that, I'm going to assume you have a good reason to use
Python:
learn python and avoid using shell commands as much as possible , I want
to avoid shell and use Python even if its costly.

Example : I don' want to create sparse files using dd command ( #dd
if=/dev/zero of=sparse-file bs=1 count=1 seek=1024k ) If the same can be
done fseek .

But I guess we might have to d

Case (2) :

Is there a better way to create the files in Python other than using
sub process module and running dd command as shown below ..

Example :

# creating sparse File
sparse_path = os.path.join(path,'sparsefiles')
os.makedirs(sparse_path)
os.chdir(sparse_path)
sparsefiles = "dd if=/dev/zero of=sp1 count=0 bs=1 seek=10G"
process_0 = subprocess.Popen(sparsefiles, stdout=subprocess.PIPE,
stderr=subprocess.PIPE, shell=True)

# Creating Regular files
Regular_path = os.path.join(path,'regularfiles')
os.makedirs(Regular_path)
os.chdir(Regular_path)
regularfiles = " dd if=/dev/urandom of=file1 count=0 bs=1 seek=10"
process_1 = subprocess.Popen(regularfiles, stdout=subprocess.PIPE,
stderr=subprocess.PIPE, shell=True)

What do you mean by "better"? There's probably nothing that will be
faster at rapidly copying bytes from one file to another than dd. But not
with a blocksize of 1 byte at a time. It's more usual to set bs=512 or
bs=1024.

Oh, I see you're not actually writing anything to the file (count=0). In
that case, instead of using dd, you should use touch.

I'm not sure that shell=True is a good idea.

In Python, to create a new empty file, setting its contents to empty if
it already exists:

open("filename", "w").close()


That will open the file, creating it if it doesn't exist, emptying it if
it does, then close it.

To touch a file without emptying it:

open("filename", "a").close()


To make a sparse file, assuming your file system supports it, I believe
you actually have to write at least one byte to the file:

f = open('foo', 'w')
f.seek(10000)
f.write('\0')
f.close()
fine . I was today playing with the temp-file module to create temporary
files and directories ,
Iam yet to explore it completely, but in current context , I had a
quick question using temp-file module is its possible to create empty
temporary files and save them on disk in the user-defined path ?
If " yes " then can this also be an alternative and does this
have any drawback ?


On Fri, Aug 16, 2013 at 4:29 PM, Steven D'Aprano <

Hi Ganesh, and welcome!

Unfortunately, you ask your questions in reverse order. The most general
(and important) question comes last, and the least important first, so
I'm going to slice-and-dice your post and answer from most general to
least.

My goal is to create various kinds of files like sparse, regular
,directories, hard and symlinks etc
what would be the best way to do achieve this ?

Use your shell, such as bash or csh or equivalent. For simple tasks like
that, it will solve the problem much more simply than Python.

There are three good reasons for doing this in Python:

- "This is only a small part of a larger Python application."

- "I'm doing this to learn how to use Python."

- "I really hate my shell."

But of you just want to get the job done, and don't care what language
you use, use the shell.


Now, having said that, I'm going to assume you have a good reason to use
Python:

Case (2) :

Is there a better way to create the files in Python other than using
sub process module and running dd command as shown below ..

Example :

# creating sparse File
sparse_path = os.path.join(path,'sparsefiles')
os.makedirs(sparse_path)
os.chdir(sparse_path)
sparsefiles = "dd if=/dev/zero of=sp1 count=0 bs=1 seek=10G"
process_0 = subprocess.Popen(sparsefiles, stdout=subprocess.PIPE,
stderr=subprocess.PIPE, shell=True)

# Creating Regular files
Regular_path = os.path.join(path,'regularfiles')
os.makedirs(Regular_path)
os.chdir(Regular_path)
regularfiles = " dd if=/dev/urandom of=file1 count=0 bs=1 seek=10"
process_1 = subprocess.Popen(regularfiles, stdout=subprocess.PIPE,
stderr=subprocess.PIPE, shell=True)

What do you mean by "better"? There's probably nothing that will be
faster at rapidly copying bytes from one file to another than dd. But not
with a blocksize of 1 byte at a time. It's more usual to set bs=512 or
bs=1024.

Oh, I see you're not actually writing anything to the file (count=0). In
that case, instead of using dd, you should use touch.

I'm not sure that shell=True is a good idea.

In Python, to create a new empty file, setting its contents to empty if
it already exists:

open("filename", "w").close()


That will open the file, creating it if it doesn't exist, emptying it if
it does, then close it.

To touch a file without emptying it:

open("filename", "a").close()


To make a sparse file, assuming your file system supports it, I believe
you actually have to write at least one byte to the file:

f = open('foo', 'w')
f.seek(10000)
f.write('\0')
f.close()



[...]

How do I loop my script to create 100 of files like sp1 , sp2 ,sp3,..
sp100 .. using the same syntax

To generate the various file names, you need to loop over a counter from
1 to 100, and stick the count into a string. Use a for-loop and the range
function:

for i in range(1, 101):
filename = "sp" + str(i)
open(filename, "a").close()


If you are a C programmer, you might prefer this style:

filename = "sp%d" % i

Or a more object-oriented style:

filename = "sp{:d}".format(i)

 

[Steven D'Aprano]

Please find the comments >>> inline

Please don't do that!

"Arrows" > are used for quoting in emails. If you prefix your *new*
comments using >>> it looks like they were quoted *three messages back*.

You should be able to configure your email or news client to prefix
quoted text with a >, and then you just type your own comments with no
prefix, like I'm doing here. Even Gmail can do that.

You seem to have copied-and-pasted my response into a new email, and then
added your comments. Am I right? The normal way to reply to an email is
to use Reply or Reply All.


[...]

fine . I was today playing with the temp-file module to create
temporary files and directories ,
Iam yet to explore it completely, but in current context , I
had a
quick question using temp-file module is its possible to create empty
temporary files and save them on disk in the user-defined path ?

You shouldn't normally care about where temporary files are stored, since
they're temporary and will disappear as soon as you are done with them.
But yes, tempfile has the ability to control where the files are stored.
Both tempfile.NamedTemporaryFile and tempfile.TemporaryFile take an
optional directory argument.

To read the documentation, run these two commands at the interactive
prompt:


import tempfile
help(tempfile)


or read it on the web:

http://docs.python.org/2/library/tempfile.html
http://docs.python.org/3/library/tempfile.html

If " yes " then can this also be an alternative and does this
have any drawback ?

Alternative to what? If you mean, alternative to *not* storing it in the
user's directory, then yes, it is

Drawbacks -- yes. I hate it when applications dump temporary files in my
home directory.

 

[Ganesh Pal]

Please don't do that!

"Arrows" > are used for quoting in emails. If you prefix your *new*
comments using >>> it looks like they were quoted *three messages back*.

You should be able to configure your email or news client to prefix
quoted text with a >, and then you just type your own comments with no
prefix, like I'm doing here. Even Gmail can do that.

You seem to have copied-and-pasted my response into a new email, and then
added your comments. Am I right? The normal way to reply to an email is
to use Reply or Reply All.

sure , hence forth will take of the same

You shouldn't normally care about where temporary files are stored, since
they're temporary and will disappear as soon as you are done with them.
But yes, tempfile has the ability to control where the files are stored.
Both tempfile.NamedTemporaryFile and tempfile.TemporaryFile take an
optional directory argument.

To read the documentation, run these two commands at the interactive
prompt:


import tempfile
help(tempfile)


or read it on the web:

http://docs.python.org/2/library/tempfile.html
http://docs.python.org/3/library/tempfile.html




Alternative to what? If you mean, alternative to *not* storing it in the
user's directory, then yes, it is

Drawbacks -- yes. I hate it when applications dump temporary files in my
home directory.

Thanks for the links and information on temporary files