approach to writing functions

[Ben Finney]

I understand that most people write functions to reuse code, no?

No. I write functions to make complex code sequences more simple --
i.e. for abstraction. Parcel up the behaviour that can conceptually be
considered a single action, and put it away in a function.

This has the valuable benefit that the function can then be re-used in
other places needing the same action; but that's not the reason I write
them to begin with.

So, I assume it would be better to write functions that are very
specific, as opposed to those that are more generic.

I've no idea how "re-use the code" leads you to think of more specific
functions -- surely the trend would be to more *generic* functions, that
can thus be re-used in more places?

Is this wrong? I know functions were not intended for this, but if I
don't use them in this manner, I might as well do everything globally,
and I don't see any difference in the two approaches.

I recommend you get a copy of Code Complete (Steve McConnell, published
by Microsoft Press). It is an extremely comprehensive and approachable
tome on the actual practice of writing code.

In particular, it explains just about every reason to use abstraction in
data, code and algorithms. Some you may have heard before, but my
feeling from your message is that a lot of it will be new -- and all of
it is valuable.

 

[Bart Nessux]

I understand that most people write functions to reuse code, no? So, I
assume it would be better to write functions that are very specific, as
opposed to those that are more generic.

However, I have difficulty doing this. My code doesn't need to be
super-modular (I don't need functions that can be used in dozens of
different programs). So, my functions don't tend to be portbale and can
sometimes span one, or perhaps two pages.

Is this wrong? I know functions were not intended for this, but if I don't
use them in this manner, I might as well do everything globally, and I
don't see any difference in the two approaches.

Say I have a program that contains no functions. Say that it is all global
and it's written like a shell script. Say that it does what I intend it to
do exactly. More experienced programmers fuss that I have not used
functions to write it. They complain about global variables, etc. But, when
I use functions and enclose everything in a one big function, I am in
essence doing exactly what I was doing globally.

Just asking for a bit of guidance. If my program works, should it be
re-written to use functions or classes? Isn't Python flexible enough to
allow for many approaches to how it is used? I mean, I know nothing of OO
programming, but Python is still a *very* useful language to me *and* to
the programmer who is an OO god. So, can't there be room for all
approaches... the formal, enlightened, abstract method of the gods and the
informal, pragmatic, get-it-done method of the common man?

What do you guys think?

 

[Bart Nessux]

I've no idea how "re-use the code" leads you to think of more specific
functions -- surely the trend would be to more *generic* functions, that
can thus be re-used in more places?

Yes, I got that part backwards. I meant generic. Thank you for pointing it
out. I'm dyslexic. To me, left is right and right is left. I work a lot
with files and file systems in general. I have a function that counts the
number of objects in a FS and returns that. It's very generic and can be
used with most any py script that needs to know how many FS objects are in
a certain path before doing something else.

I recommend you get a copy of Code Complete (Steve McConnell, published
by Microsoft Press). It is an extremely comprehensive and approachable
tome on the actual practice of writing code.

In particular, it explains just about every reason to use abstraction in
data, code and algorithms. Some you may have heard before, but my
feeling from your message is that a lot of it will be new -- and all of
it is valuable.

Thank you, I'll look into this

 

[Terry Reedy]

I understand that most people write functions to reuse code, no?

Another reason is to define, name, and embody a concept, even if the
function is only used once.

Say I have a program that contains no functions. Say that it is all global
and it's written like a shell script. Say that it does what I intend it to
do exactly. More experienced programmers fuss that I have not used
functions to write it. They complain about global variables, etc. But, when
I use functions and enclose everything in a one big function, I am in
essence doing exactly what I was doing globally.

Are you in a position where you *have* to let such people read your code?

Just asking for a bit of guidance. If my program works, should it be
re-written to use functions or classes?

Working correctly is most important. Next is running fast enough. Then
you can consider whether you or another person can read, edit, or reuse six
months from now. As for rewriting, would *you* gain some personal benefit
from doing so?

Isn't Python flexible enough to allow for many approaches to how it is

used?

The language itself is. So are most users, I believe. It is intentionally
not a straightjacket language, even though some people find freedom from
braces to be constricting.

Python was made for people, not people for Python. Ditto for programming
theories.

Terry J. Reedy

 

[Rainer Deyke]

Another reason is to define, name, and embody a concept, even if the
function is only used once.

Another reason is that you need a callable object, whether it embodies some
high level concept or not.

 

[Paul Prescod]

I understand that most people write functions to reuse code, no? So, I
assume it would be better to write functions that are very specific, as
opposed to those that are more generic.

However, I have difficulty doing this. My code doesn't need to be
super-modular (I don't need functions that can be used in dozens of
different programs). So, my functions don't tend to be portbale and can
sometimes span one, or perhaps two pages.

It's sort of a self-fulfilling prophecy. If you don't try to write
reusable functions your code won't be reusable.

But, to be fair to you, it is harder to find code to reuse in Python
than in lower-level languages because so much comes out of the box.

Even so, I very rarely write a program that does not reuse code in one
way or another. Do you find that there are chunks of text that are
almost identical sprinkled throughout your program?

Say I have a program that contains no functions. Say that it is all global
and it's written like a shell script. Say that it does what I intend it to
do exactly. More experienced programmers fuss that I have not used
functions to write it. They complain about global variables, etc. But, when
I use functions and enclose everything in a one big function, I am in
essence doing exactly what I was doing globally.

If the logic is simple, what you are doing is not necessarily bad style.
But if the logic is complicated then eschewing functions can make it
more, not less, complicated. For one thing you get so many indentation
levels that it becomes hard to keep it in your head. For another thing,
when you try to read your code six months from now it will not be in
bite-sized bits you can keep in your head but rather in one long stream.
Imagine a university textbook without chapters. It is somewhat harder
to navigate the one long stream. By convention, programmers use
functions as "chapters."

What do you guys think?

If I were you I would post one of these scripts and see if the function
and OO fans can make it easier to read or maintain by using structured
programming.

Paul Prescod

 

[Ben Finney]

Imagine a university textbook without chapters. It is somewhat harder
to navigate the one long stream. By convention, programmers use
functions as "chapters."

Naah. The modules are the "chapters". Classes are "sections", and
functions are "subsections".

And every sentence should fit on one 80-column line, or be broken into
multiple sentences that do.

 

[Anton Vredegoor]

[bart]

Working correctly is most important. Next is running fast enough. Then
you can consider whether you or another person can read, edit, or reuse six
months from now. As for rewriting, would *you* gain some personal benefit
from doing so?

I disagree with the order in which you list these things, but that
might be caused by whether one sees Python as a language to express
ideas or as a tool to accomplish a more specific task. For example I
am not a native English speaker, but I guess nobody would think it
more important to avoid all spelling errors than to get the idea
across.

Posting scripts and ideas that are in this stadium is a bit dangerous
and one may end up with some egg (or pie!) in ones face, but since
software development is promoted most by removing errors in the
*early* stages it's a good strategy I think. Of course this shouldn't
result in long posts here with completely unqualified code.

However, asking specific questions about smaller functions is better
than posting 100+ lines of code and requesting that someone explains
why the script doesn't do what it is meant to do while it is still
unclear what a poster really wants the script to do.

While developing code I'm imagining a herd of virtual Python gurus
looking over my shoulder assisting me and requiring my code to be
readable ...

What I am trying to say is that if one develops code *as if* posting
it here, errors are detected sooner. And later errors are more costly
than early errors.

Anyway, I want more people to post code, even if it is not perfect!

Only after a lot of "eyeballing" and splitting the code up into
functions one should go to the next phase of testing the script for
possible errors and behavior in worst case scenarios. Of course Python
also accommodates for a grand unified style of programming, and both
styles can add value.

In the end well tested Python scripts often are readable and "obvious"
whichever method is used to produce them, which I think is a nice
effect of readable computer languages in general.

Anton

 

[Bart Nessux]

If I were you I would post one of these scripts and see if the function
and OO fans can make it easier to read or maintain by using structured
programming.

OK, here is a script that works w/o OO or functions. It's very useful...
never fails. It helps sys admins track machines that have mobile users
who are using DHCP.

#!/usr/bin/python
#
# Tested on Linux Machines
# Run as cron as root
# May work on OS X too
#
# Dec 29, 2003 works on Mac OSX 10.3 like this:
# chown 'root' && chgrp 'wheel' && chmod '755'
# place in /usr/bin
#

from email.MIMEText import MIMEText
import smtplib
import os

u = "User" #Change This to user's name.
f = "(e-mail address removed)"
t = "(e-mail address removed)"

fp0 = os.popen("/sbin/ifconfig en0 inet", "r")
fp1 = os.popen("/usr/bin/uptime", "r")
fp2 = os.popen("/usr/bin/uname -a", "r")
fp3 = os.popen("/usr/bin/wc -l /etc/passwd", "r")
msg = MIMEText("-- IFCONFIG --\n\n" + fp0.read() + "\n-- UPTIME --\n\n"
+ fp1.read() + "\n-- UNAME --\n\n" + fp2.read() + "\n-- PASSWD LC
--\n\n" + fp3.read())
fp0.close()
fp1.close()
fp2.close()
fp3.close()

msg["Subject"] = "%s's ifconfig Report" % u
msg["From"] = f
msg["To"] = t

h = "smtp.vt.edu"
s = smtplib.SMTP(h)
s.sendmail(f, t, msg.as_string())
s.quit()

 

[Terry Reedy]

[bart]

Working correctly is most important. Next is running fast enough. Then
you can consider whether you or another person can read, edit, or reuse six
months from now. As for rewriting, would *you* gain some personal benefit
from doing so?

I disagree with the order in which you list these things, but that
might be caused by whether one sees Python as a language to express
ideas or as a tool to accomplish a more specific task.

I gather you are putting my list in the second category. I see Python as
being excellent for both uses, and I think that part of its excellence is
that it works both ways as executable humancode.

The OP was asking about specific-task production code. I believe that he
should first be praised for meeting the prime directive for such, that it
work correctly, before being critiqued for secondary stylistic goals.

For example I
am not a native English speaker, but I guess nobody would think it
more important to avoid all spelling errors than to get the idea
across.

When writing to communicate ideas to other people, 'working correctly'
means successful communication of the intended idea. For this reason, I
sometimes post untested code that may not be exactly right but which
communicates an idea. But when I do so, I label it as 'untested' or
'something like' to communicate that it is idea-passing code rather than
tested execution code.

Terry J. Reedy

 

[Terry Reedy]

If I were you I would post one of these scripts and see if the function
and OO fans can make it easier to read or maintain by using structured
programming.

OK, here is a script that works w/o OO or functions. It's very useful...
never fails. It helps sys admins track machines that have mobile users
who are using DHCP.

#!/usr/bin/python
#
# Tested on Linux Machines
# Run as cron as root
# May work on OS X too
#
# Dec 29, 2003 works on Mac OSX 10.3 like this:
# chown 'root' && chgrp 'wheel' && chmod '755'
# place in /usr/bin
#

from email.MIMEText import MIMEText
import smtplib
import os

u = "User" #Change This to user's name.
f = "(e-mail address removed)"
t = "(e-mail address removed)"

fp0 = os.popen("/sbin/ifconfig en0 inet", "r")
fp1 = os.popen("/usr/bin/uptime", "r")
fp2 = os.popen("/usr/bin/uname -a", "r")
fp3 = os.popen("/usr/bin/wc -l /etc/passwd", "r")
msg = MIMEText("-- IFCONFIG --\n\n" + fp0.read() + "\n-- UPTIME --\n\n"
+ fp1.read() + "\n-- UNAME --\n\n" + fp2.read() + "\n-- PASSWD LC
--\n\n" + fp3.read())
fp0.close()
fp1.close()
fp2.close()
fp3.close()

msg["Subject"] = "%s's ifconfig Report" % u
msg["From"] = f
msg["To"] = t

h = "smtp.vt.edu"
s = smtplib.SMTP(h)
s.sendmail(f, t, msg.as_string())
s.quit()

Your whole script is a coherent no-argument function: create and mail a
status message. C requires you to wrap such within the file as a
syntactically explicit 'main' function. Python, being against busywork for
the sake of busywork, does not.

The *possible* advantage of doing something like this:

def sendreport():
<your current code

if __name__ == '__main__':
sendreport()

is that you could then import the module and use sendreport from another
script. You might then parameterize it to make some of the constants like
u (user) variable. You could then do things like

for u in userlist: sendreport(u)

But the value of adding the wrapper and boilerplate and attendant
flexibility depends on your situation. It also rests on getting the basic
functionality correct.

[OO fans might suggest defining a stat_rep class with a send method, but I
only see that a having much value if you want the learning experience or if
you need to keep unsent messages around.]

As for the code itself. I would probably use more mnemonic names like
f_inet, f_time, f_name, and f_user, but this is a minor stylistic nit. I
might also read and maybe close each pipe as created, but maybe you want
the processes to run as close to simultaneous as possible.

Terry J. Reedy

 

[Peter Hansen]

fp0 = os.popen("/sbin/ifconfig en0 inet", "r")
fp1 = os.popen("/usr/bin/uptime", "r")
fp2 = os.popen("/usr/bin/uname -a", "r")
fp3 = os.popen("/usr/bin/wc -l /etc/passwd", "r")
msg = MIMEText("-- IFCONFIG --\n\n" + fp0.read() + "\n-- UPTIME --\n\n"
+ fp1.read() + "\n-- UNAME --\n\n" + fp2.read() + "\n-- PASSWD LC
--\n\n" + fp3.read())
fp0.close()
fp1.close()
fp2.close()
fp3.close()

This sequence has duplication, so it could stand some refactoring. Write
a routine that executes a command and returns the result, then call it
with various commands, something like this:

def cmd(c):
f = os.popen(c, 'r')
try:
result = f.read()
finally:
f.close()
return result

responses = [cmd(c) for c in
['/sbin/ifconfig en0 inet', '/usr/bin/uptime', '/usr/bin/uname -a',
'/usb/bin/wc -l /etc/passwd']]

msg = MIMEText('blah %s, blah %s, blah %s, blah %s' % responses)

That's only one example of what you could do to simplify/make reusable
code.

The best thing I know of to decide when to make a function is when you have
code that is about to take responsibility for two different things. Your
code has responsibility for setting up user info, retrieving command output,
execute four commands, generating the mail body from the results,
setting up the entire mail message, and sending the mail message. That's
six different areas of responsibility (give or take) and there should
probably be at least three or four functions in there to handle some
of those in a cleaner fashion.

Always ask yourself "what is this chunk of code responsible for?" If
the answer is more than one thing, consider splitting it up. This is,
by the way, effectively the concept of "cohesion", and you would do well
to learn about "cohesion" and "coupling". You always strive for low
coupling (connections between different things) and high cohesion
(responsibility for only closely related things) in code, if you
want clean design.

-Peter

 

[Joe Mason]

Working correctly is most important. Next is running fast enough. Then
you can consider whether you or another person can read, edit, or reuse six
months from now. As for rewriting, would *you* gain some personal benefit
from doing so?

Not true. I would say being able to read, edit and reuse is most
important, then working correctly, then running fast enough. Because
there are always going to be bugs that you don't find for a few months,
so even if you think it works correctly, you're probably wrong. As for
working fast enough, being able to read the code makes it much easier
to optimize it later IF you find out you have to.

Joe

 

[Joe Mason]

fp0 = os.popen("/sbin/ifconfig en0 inet", "r")
fp1 = os.popen("/usr/bin/uptime", "r")
fp2 = os.popen("/usr/bin/uname -a", "r")
fp3 = os.popen("/usr/bin/wc -l /etc/passwd", "r")
msg = MIMEText("-- IFCONFIG --\n\n" + fp0.read() + "\n-- UPTIME --\n\n"
+ fp1.read() + "\n-- UNAME --\n\n" + fp2.read() + "\n-- PASSWD LC
--\n\n" + fp3.read())
fp0.close()
fp1.close()
fp2.close()
fp3.close()

This sequence has duplication, so it could stand some refactoring. Write
a routine that executes a command and returns the result, then call it
with various commands, something like this:

def cmd(c):
f = os.popen(c, 'r')
try:
result = f.read()
finally:
f.close()
return result

responses = [cmd(c) for c in
['/sbin/ifconfig en0 inet', '/usr/bin/uptime', '/usr/bin/uname -a',
'/usb/bin/wc -l /etc/passwd']]

msg = MIMEText('blah %s, blah %s, blah %s, blah %s' % responses)

That's only one example of what you could do to simplify/make reusable
code.

The best thing I know of to decide when to make a function is when you have
code that is about to take responsibility for two different things. Your
code has responsibility for setting up user info, retrieving command output,
execute four commands, generating the mail body from the results,
setting up the entire mail message, and sending the mail message. That's
six different areas of responsibility (give or take) and there should
probably be at least three or four functions in there to handle some
of those in a cleaner fashion.

Hee hee. As the guy who disagreed with Terry before on philosophy, I'm
going to agree with him now on practicals. There's no reason to split
up this code into three or four functions. The only bit I don't like is
having all the read() calls inside the MIMEText string where it's easy
to miss them. I would use Peter's cmd function, and just replace "fp? =
os.open(...)" with "r? = cmd(...)", and then change all the variable
names in MSG. (If I were writing it for the first time, I might use
that loop comprehension depending on how many commands there are. It's
a very useful shortcut if you're doing the same thing a lot, but it's
one more layer of comprehension you have to look past to see what the
code is doing.)

Joe

 

[Peter Hansen]

Hee hee. As the guy who disagreed with Terry before on philosophy, I'm
going to agree with him now on practicals.

Actually all three of us are agreeing on practicals then, because I
wouldn't actually suggest anyone waste time chopping a little utility
like that up into many functions after it has already been written
and is working.

There's no reason to split
up this code into three or four functions.

Agreed. On the other hand, there's *excellent* reasons to write it
as three or four functions in the first place, especially if one is
trying to learn how to write well crafted code. You can't learn how
best to make use of "functions" (or any other software construct) by
just listening to others spout off about it... you gotta try it out
yourself (advice to the OP).

If you can't see the distinction, I'll make myself clearer in a subsequent
message.

(If I were writing it for the first time, I might use
that loop comprehension depending on how many commands there are. It's
a very useful shortcut if you're doing the same thing a lot, but it's
one more layer of comprehension you have to look past to see what the
code is doing.)

I dislike having many data-specific variables (e.g. "user", "password",
"address", "port", etc) kicking around if they are there only to hold
data for a line or two but won't be used more than once.

In other words, if you have four things to retrieve, and will immediately
pass them on to a function (or in this case a string formatting operator)
that will use them in the same order, the simplest thing is just to
assign them as a tuple never handle them individually. That's why I
tended towards the list-comprehension approach above.

Chances are my own code wouldn't have ended up with that particular line
in that particular place, so I doubt in my own version of this I would
actually have had the same list comprehension.

-Peter

 

[Terry Reedy]

Not true.

I an not sure if programming philosophy is in the realm of true and false
;-)
I could be persuaded that points two and three might be reversed.

I would say being able to read, edit and reuse is most
important, then working correctly, then running fast enough. Because
there are always going to be bugs that you don't find for a few months,
so even if you think it works correctly, you're probably wrong.

In a 100,000 line app, sure, there 'inevitably' will be behaviors less that
perfect.
But in the context of writing one page functions, I find this to be a
flabbergasting statement. If the OP's example were to raise a syntax error
or other exception, or gather info about the wrong user, or gather
incorrect info, or email to the wrong address, or print instead of
emailing, then it would be useless and would need to be fixed now and not
months from now.

One can often expand the limits of 'correct' to include 'acceptible but
less than optimal' and maybe even 'occasional mysterious failure' (as with
Windows and Windows app crashes), but there are usually limits beyond which
one moves toward uselessness.

Terry J. Reedy

 

[Joe Mason]

In a 100,000 line app, sure, there 'inevitably' will be behaviors less that
perfect.
But in the context of writing one page functions, I find this to be a
flabbergasting statement. If the OP's example were to raise a syntax error
or other exception, or gather info about the wrong user, or gather
incorrect info, or email to the wrong address, or print instead of
emailing, then it would be useless and would need to be fixed now and not
months from now.

I include misdesigns and lack of functionality here... Apart from that
the ordering doesn't really matter for small functions because it's
trivial to be both clear and correct.

I thought we were talking about how to break down large apps into
functions, though - I didn't read teh original post closely enough.

Joe

 

[Joe Mason]

Agreed. On the other hand, there's *excellent* reasons to write it
as three or four functions in the first place, especially if one is
trying to learn how to write well crafted code. You can't learn how
best to make use of "functions" (or any other software construct) by
just listening to others spout off about it... you gotta try it out
yourself (advice to the OP).

I disagree with that, too, actually. There's a point at which splitting
up into tiny, tiny functions makes it hard to ingest code. If I want to
track exactly what the program is doing here (spawn 4 procs, read lines
from them, and output the results), it's easier to actually have that in
front of me than to have to dig through a large call tree of nicely
decoupled functions to find each step. Poorly designed oode leads to
spaghetti, but overdesigned code can lead to the overall system being
harder to learn.

Oops, gotta go - I was going to expand on this but my test seems to have
started spewing data at me, so it's back to work.

Joe