Opinion on best practice...

  • Thread starter Anthony Correia
  • Start date
A

Anthony Correia

I need to pick up a language that would cover the Linux platform. I use Powershell for a scripting language on the Windows side of things. Very simple copy files script. Is this the best way to do it?

import os

objdir = ("C:\\temp2")
colDir = os.listdir(objdir)
for f in colDir:
activefile = os.path.join(objdir + "\\" + f)
print ("Removing " + activefile + " from " + objdir)
os.remove(activefile)

In Powershell I would just do:

$colDir = gci -path "c:\temp2"
ForEach($file in $colDir)
 
M

Michael Torrie

I need to pick up a language that would cover the Linux platform. I
use Powershell for a scripting language on the Windows side of things.
Very simple copy files script. Is this the best way to do it?
import os

objdir = ("C:\\temp2")
colDir = os.listdir(objdir)
for f in colDir:
activefile = os.path.join(objdir + "\\" + f)
print ("Removing " + activefile + " from " + objdir)
os.remove(activefile)

In Powershell I would just do:

$colDir = gci -path "c:\temp2"
ForEach($file in $colDir)

Yes you could do something like that in Python. I know on linux there
is a module called shutils that provides more shell file handling
abilities to python. Your script as it is listed has a few bugs, though:
- os.listdir() walks the current working directory, not necessarily c:\temp2
- instead of using "\\" in the os.path.join, you can use os.path.sep to
get the current platform's separator character

While I do use python for shell scripting, by itself, it's not very good
at it. I ended up writing my own wrapper function to run external
commands, feed them std-in, and capture standard out and standard error.
I wrapped one of the subprocess module functions. Not sure which. I
can post the file to the list if anyone wants to see it. Also as I
mentioned, on unix machines there is the shutils module.

Still, though, sometimes a proper shell like bash or zsh is going to
work out better for shell scripting than python. Plus if you're going
to be proficient on the Linux command line you need to know their basics
anyway.
 
U

Ulrich Eckhardt

Am 05.02.2013 05:14, schrieb Anthony Correia:
I need to pick up a language that would cover the Linux platform. I use Powershell for a scripting language on the Windows side of things. Very simple copy files script. Is this the best way to do it?

import os

objdir = ("C:\\temp2")

Drop the parens here.
colDir = os.listdir(objdir)
for f in colDir:
activefile = os.path.join(objdir + "\\" + f)

The point of os.path.join is exactly that you don't have to spell out
the system-specific file separator. In this case, it gets called with a
single string, which it return as-is.
print ("Removing " + activefile + " from " + objdir)

Instead of using + to concat strings, use the format() functionn:

"removing {} from {}".format(activefile, objdir)

Also, if you are using Python 2, print is a statement, not a function,
so you could drop the parens here, too. I would not recommend that
though! Instead, "from __future__ import print_function" and keep using
print() as a function, just like in Python 3.

In general, I would not have stored the result of os.listdir() in a
separate variable but iterated over it directly. For large dirs, it
could also be better to use os.path.walk(), because that doesn't first
build a list and then iterate over the list but iterates over the single
elements directly. This avoids the memory allocation overhead, although
it is unlikely to make a difference in this case and a Premature
Optimization(tm).

Welcome to Python!

Uli
 
T

Terry Reedy

I need to pick up a language that would cover the Linux platform. I use Powershell for a scripting language on the Windows side of things. Very simple copy files script. Is this the best way to do it?

import os

objdir = ("C:\\temp2")
colDir = os.listdir(objdir)
for f in colDir:
activefile = os.path.join(objdir + "\\" + f)
print ("Removing " + activefile + " from " + objdir)
os.remove(activefile)

In Powershell I would just do:

$colDir = gci -path "c:\temp2"
ForEach($file in $colDir)

Python comes with a shutil module with multiple copy functions.
 
P

Peter Otten

Ulrich said:
separate variable but iterated over it directly. For large dirs, it
could also be better to use os.path.walk(), because that doesn't first
build a list and then iterate over the list but iterates over the single
elements directly. This avoids the memory allocation overhead, although
it is unlikely to make a difference in this case and a Premature
Optimization(tm).

Not true. os.walk() uses os.listdir() internally.
 
U

Ulrich Eckhardt

Am 05.02.2013 11:35, schrieb Peter Otten:
Ulrich said:
[...] use os.path.walk(), because that doesn't first build a list and
then iterate over the list but iterates over the single elements directly.
[...]
Not true. os.walk() uses os.listdir() internally.

Oh. 8|

Thanks for proofreading what I wrote, I must have been confusing it with
something else.

Thanks&sorry!

Uli
 
G

Grant Edwards

I need to pick up a language that would cover the Linux platform.

Well, you haven't really described what it is you're trying to do, but
it looks to me like bash and the usual set of shell utilities (e.g.
find) is what you need rather than Python.
I use Powershell for a scripting language on the Windows side of
things. Very simple copy files script. Is this the best way to do
it?

That depends. What is "it"?
import os

objdir = ("C:\\temp2")
colDir = os.listdir(objdir)
for f in colDir:
activefile = os.path.join(objdir + "\\" + f)
print ("Removing " + activefile + " from " + objdir)
os.remove(activefile)

In Powershell I would just do:

$colDir = gci -path "c:\temp2"
ForEach($file in $colDir)

Sorry, I'm a Linux guy. I have no clue what that means.
 
G

Grant Edwards

Python comes with a shutil module with multiple copy functions.

Most of the time when I find myself using the shutil module, it means
the program should have been written in bash. :)
 
N

Neil Cerutti

Hooray for common sense! Python is great, but it's silly to use
Python (unless there is good reason) when a simple shell script
will do the job.

Python is an excellent option for writing shell scripts,
particularly if your shell is cmd.exe.
 
E

Ethan Furman

Python is an excellent option for writing shell scripts,
particularly if your shell is cmd.exe.

I believe having your shell be cmd.exe qualifies as a "good reason"!

:)

~Ethan~
 
G

Grant Edwards

Python is an excellent option for writing shell scripts,
particularly if your shell is cmd.exe.

The OP stated explicitly that the target OS was Linux:

Don't get me wrong -- I think Python is great -- but when the target
OS is Linux, and what you want to do are file find, move, copy,
rename, delete operations, then I still say bash should be what you
try first.
 
G

Grant Edwards

I believe having your shell be cmd.exe qualifies as a "good reason"!

Except the OP said he wanted to pick a language for Linux. If he has
cmd.exe as his shell on Linux, then he's beyond help...
 
C

Chris Angelico

Except the OP said he wanted to pick a language for Linux. If he has
cmd.exe as his shell on Linux, then he's beyond help...

Python is not an "excellent option". It's a bad fit for shell
scripting, it just happens to be way better than a weak shell. Having
grown up on command.com, I found OS/2's cmd.exe to be a massive
improvement, and Windows's cmd.exe to be rather less impressive... but
both of them pale into insignificance compared to bash. OS/2 has a
better shell scripting language (REXX), but Windows doesn't, and
Python ain't it.

(Hmm. Google Chrome doesn't put its red squiggly lines under "ain't",
but it does complain about "virtualization". Seems to me something,
uhh, ain't right there.)

ChrisA
 
I

Ian Kelly

The OP stated explicitly that the target OS was Linux:

The impression I got was that he was looking for something that would
be cross-platform and work in both Windows and Linux. Python is a
reasonable choice for that if you're not willing to deal with Cygwin.
 
S

Steven D'Aprano

Chris said:
Python is not an "excellent option". It's a bad fit for shell
scripting, it just happens to be way better than a weak shell. Having
grown up on command.com, I found OS/2's cmd.exe to be a massive
improvement, and Windows's cmd.exe to be rather less impressive... but
both of them pale into insignificance compared to bash.

I have to disagree with much of this. bash is a poorly designed language
which, in my opinion, is only good enough for short (under 20 lines)
throw-away scripts.

[begin rant]

bash has terse *and* cryptic syntax even less maintainable than Perl. E.g.
this is, apparently, the right way to get the length of an array called R:

rlen=${#R[@]}

(taken from here:

http://www.bashcookbook.com/bashinfo/source/bash-4.2/examples/functions/array-stuff

so I hope that they know what they're talking about.)

Note that sometimes # begins a comment, and sometimes it doesn't.

This is how you test whether something is not the name of a directory:

[[ -d $dir ]] || {
echo "$FUNCNAME: $dir is not a directory" >&2
return 2
}


http://www.bashcookbook.com/bashinfo/source/bash-4.2/examples/functions/emptydir

Arithmetic requires either calling an external program, or special magic
syntax:

z=`expr $z + 3`
i=$(( i + 1 ))

Spaces are magic -- these two lines do radically different things:

i = j
i=j

There's also a "let" command, but it's a kludge:

k=$(( i+2 ))
let k=i+2

are equivalent, but

k=$(( i*2 ))
let k=i*2

are not!

bash is even more weakly typed than Perl. As far as I can tell, bash is
completely untyped -- everything is text, all the time, even arrays.

[steve@ando ~]$ cat testarray.sh
colors=('Blue' 'Red')
echo $((colors + 2))

[steve@ando ~]$ sh testarray.sh
2

Quotation marks have magic powers, and the rules for them are so complicated
that working out how to correctly quote expressions is probably the single
most difficult part of bash scripting.

bash teaches bad programming habits, such as reliance on global and
environment variables, and lacks "modern" (30+ years old) features such as
exception handling, objects, and even proper functions that return a
result. Yes, that's right, bash functions communicate their result by
side-effect.

http://www.linuxjournal.com/content/return-values-bash-functions

Because bash scripts are mostly written by non-programmers, the culture as a
whole has a cargo-cult mentality about it. I'm sure that there are some
people who actually do know bash, it's strengths and weaknesses, well, but
in my experience most people just blindly copy code snippets and recipes
around without really understanding them. As a consequence, we have this:

http://partmaps.org/era/unix/award.html

Almost nothing in bash is portable. Because bash does so little, and relies
on external programs for so much, even if you write the bash parts portably
(which basically means writing for the Bourne shell, which is even more
lacking than bash), you *still* aren't portable because there's no
guarantee that the external programs exist or do what you expect on some
other system.

To do anything meaningful in bash, you need to be an expert on passing work
off to other programs, e.g. sed, awk, grep, tr, cut, ed, ps, dc, wc, ...
any of which may be missing, or have different semantics from what you
expect (ps in particular is notorious for the widely variable command
switches it takes), or output their results in unexpected ways.

Pretty much of all bash scripting is kludge on top of kludge, special
behaviour, small syntax differences making large differences in behaviour,
surprising gotchas, side-effects and magic. If you took the Zen of Python,
and pretty much reversed everything, you might have the Zen of Bash:

Implicit is better than explicit.
Complex is better than simple.
Complicated is better than complex.
Dense is better than sparse.
Readability doesn't count.
Special cases break the rules.
Errors should pass silently.
In the face of ambiguity, guess.
There should be many obscure ways to do it.
Namespaces are terrible, let's make everything global!

I would not hesitate to use Python, or some other high-level language like
Ruby, over bash for anything non-trivial that I cared about. It might not
be as terse and compact as a well-written bash script, but that's a *good*
thing, and a poorly-written bash script is likely to be even more verbose
and difficult to write than a poorly-written Python script.
 
N

Nobody

Hooray for common sense! Python is great, but it's silly to use Python
(unless there is good reason) when a simple shell script will do the job.

A shell script is only the better option if (almost) the *only* thing the
script needs to do is to execute commands.

The moment you start trying to "process" data, it's time to put up with
the verbosity of subprocess.Popen() so that you can use a well-designed
language for the rest of it.

Shells are designed first and foremost for interactive use, and everything
else is compromised by that fact.

Minimising the number of keystrokes is a great idea for something which
will be typed, executed and forgotten. It's an awful idea if you're going
to modify the script in six months' time.

Making the execution of commands a fundamental language feature is a great
idea if that's most of what you do, but not such a great idea if you'll be
doing a lot else besides (because most of the available syntax has been
"consumed" by command execution).
 
M

Michael Torrie

A shell script is only the better option if (almost) the *only* thing the
script needs to do is to execute commands.

The moment you start trying to "process" data, it's time to put up with
the verbosity of subprocess.Popen() so that you can use a well-designed
language for the rest of it.

Agreed. And most of the time a script in bash is going to do something
like:

grep blah /from/file | cut -f 1 -d ' ' | uniq | sort

Python generators happen to be very efficient at recreating this sort of
thing entirely in python. A great presentation that all system
programmers should read is found here:

http://www.dabeaz.com/generators/

Very cool stuff. Grep is just a generator function. cut is just a
generator function, and so on. And it can be pretty darn fast too.
 
D

Dennis Lee Bieber

better shell scripting language (REXX), but Windows doesn't, and
Python ain't it.
Win7 includes PowerShell, and it can be downloaded for WinXP.

Granted, PowerShell also incorporates a security system that might
make running scripts a bit of a pain <G> (I think the default is that it
will only run scripts with known "signatures"; but can be set to allow
local machine sourced scripts to run without checking).

PowerShell is meant to be used for administrative level scripting,
replacing such things as WSH.
 
C

Chris Angelico

Chris said:
Python is not an "excellent option". It's a bad fit for shell
scripting, it just happens to be way better than a weak shell. Having
grown up on command.com, I found OS/2's cmd.exe to be a massive
improvement, and Windows's cmd.exe to be rather less impressive... but
both of them pale into insignificance compared to bash.

I have to disagree with much of this. bash is a poorly designed language
which, in my opinion, is only good enough for short (under 20 lines)
throw-away scripts.

[snip rant]

Your points are valid, and yet I still can't push Python as a command
execution language better than bash scripting. "Nobody" was right: as
soon as you are actually *processing* data, you want something better
than bash.

If I want to make a simple command that pulls the latest project
changes and then shows me those changes, I'm going to write a bash
script that does this:

git pull --rebase
git log --oneline origin/master..master
gitk &

Actually the script's a bit more detailed than that (variable branch,
makes a tag 'lastsync' so that I can go back to it later, passes gitk
some args so it shows me the commit most likely to be of first
interest, etc), but it's still pretty much all command execution. Now
maybe if I were using Mercurial, I could script some of that using
Python; but that would be a special-case that stems from Mercurial
actually having been written in Python, and it doesn't help if you're
working with some other set of commands that don't happen to be
importable.

But what tends to happen at work is that a script like that grows
until it hits a couple dozen lines of code, and then it gets
transparently rewritten into some other language (change the shebang
at the top, nobody needs to know/care that it got a complete rewrite).
My weapon of choice for these rewrites is usually Pike, not Python,
mainly because we don't currently have any significant amount of
Python in our codebase, but either would most likely do. (I haven't
looked into subprocess and how portable my code would be across
2.6/2.7/3.3, though.) The fact remains, though, that an applications
language is definitely *second choice* to bash scripting when it comes
to tasks that involve lots of command execution.

It feels silly enough translating this OS/2 batch script:

@logon SOME_USER /pSOME_PASS /vl
@e:\rexx\load
@db2 start database manager
@exit

into this REXX script:

/* */
"@logon SOME_USER /pSOME_PASS /vl"
"@e:\rexx\load"
"@db2 start database manager"
"@exit"

And that's a change I've made myriad times (to \startup.cmd on many an
OS/2 boot drive) It's far far worse translating it into Python, Pike,
or any other "good language".

ChrisA
 
R

rusi

I would not hesitate to use Python, or some other high-level language like
Ruby, over bash for anything non-trivial that I cared about. It might not
be as terse and compact as a well-written bash script, but that's a *good*
thing, and a poorly-written bash script is likely to be even more verbose
and difficult to write than a poorly-written Python script.

Maybe you should look at scsh or rc http://paganbooks.eu/software/article/rc-duff

Heres the author of scsh (which would not be saying much were it not
for his uber-geek status)
(from http://www.scsh.net/docu/scsh-paper/scsh-paper-Z-H-1.html )
----------------
Shell programming terrifies me. There is something about writing a
simple shell script that is just much, much more unpleasant than
writing a simple C program, or a simple COMMON LISP program, or a
simple Mips assembler program. Is it trying to remember what the rules
are for all the different quotes? Is it having to look up the multi-
phased interaction between filename expansion, shell variables,
quotation, backslashes and alias expansion? Maybe it's having to
subsequently look up which of the twenty or thirty flags I need for my
grep, sed, and awk invocations. Maybe it just gets on my nerves that I
have to run two complete programs simply to count the number of files
in a directory (ls | wc -l), which seems like several orders of
magnitude more cycles than was really needed.
 

Ask a Question

Want to reply to this thread or ask your own question?

You'll need to choose a username for the site, which only take a couple of moments. After that, you can post your question and our members will help you out.

Ask a Question

Members online

No members online now.

Forum statistics

Threads
473,755
Messages
2,569,536
Members
45,007
Latest member
obedient dusk

Latest Threads

Top