Dealing with exceptions

[bvdp]

Every time I write a program with exception handling (and I suppose that includes just about every program I write!) I need to scratch my brain when Icreate try blocks.

For example, I'm writing a little program do copy specific files to a USB stick. To do the actual copy I'm using:

try:
shutil.copy(s, os.path.join(usbpath, songname))
except ...

now, I need to figure out just what exceptions to handle. Let's see:

IOError that means that the disk is full or otherwise buggered. Better dump out of the loop.

But, I know there can be other errors as well. Doing some tests, I know that certain filenames are invalid (I think a "?" or unicode char is invalid when writing to a FAT32 filesystem). And, so what exception is that? Withoutactually creating the error, I can't figure it out.

In this case, I can run the program an number of times and parse out the errors and write code to catch various things. But, I think I'm missing something completely. Guess what I'm looking for is a list of possible (probable?) errors for the shutil.copy() command. And, in a much bigger manual, for most other commands.

Maybe I'm just venting about FAT32 filesystems

 

[Kwpolska]

Every time I write a program with exception handling (and I suppose that includes just about every program I write!) I need to scratch my brain whenI create try blocks.

For example, I'm writing a little program do copy specific files to a USBstick. To do the actual copy I'm using:

try:
shutil.copy(s, os.path.join(usbpath, songname))
except ...

now, I need to figure out just what exceptions to handle. Let's see:

IOError that means that the disk is full or otherwise buggered. Betterdump out of the loop.

But, I know there can be other errors as well. Doing some tests, I know that certain filenames are invalid (I think a "?" or unicode char is invalidwhen writing to a FAT32 filesystem). And, so what exception is that? Without actually creating the error, I can't figure it out.

In this case, I can run the program an number of times and parse out the errors and write code to catch various things. But, I think I'm missing something completely. Guess what I'm looking for is a list of possible (probable?) errors for the shutil.copy() command. And, in a much bigger manual, for most other commands.

Maybe I'm just venting about FAT32 filesystems

IOError and OSError should cover all copy problems, I think.

Also, you can do `except:` for a catch-all, but it is discouraged
unless you have REALLY good reasons to do this. And, most of the
time, you don’t.

 

[Ian Kelly]

Every time I write a program with exception handling (and I suppose that includes just about every program I write!) I need to scratch my brain whenI create try blocks.

For example, I'm writing a little program do copy specific files to a USBstick. To do the actual copy I'm using:

try:
shutil.copy(s, os.path.join(usbpath, songname))
except ...

now, I need to figure out just what exceptions to handle. Let's see:

IOError that means that the disk is full or otherwise buggered. Betterdump out of the loop.

But, I know there can be other errors as well. Doing some tests, I know that certain filenames are invalid (I think a "?" or unicode char is invalidwhen writing to a FAT32 filesystem). And, so what exception is that? Without actually creating the error, I can't figure it out.

OSError. In Python 3, I expect it would more specifically be a
FileNotFoundError, which is a subclass of OSError.

In this case, I can run the program an number of times and parse out the errors and write code to catch various things. But, I think I'm missing something completely. Guess what I'm looking for is a list of possible (probable?) errors for the shutil.copy() command. And, in a much bigger manual, for most other commands.

OSError will cover a wide swath of possible exceptions here.

 

[Ian Kelly]

IOError and OSError should cover all copy problems, I think.

And it may be worth pointing out here that as of Python 3.3, IOError
is just a synonym for OSError.

 

[Chris Angelico]

For example, I'm writing a little program do copy specific files to a USB stick. To do the actual copy I'm using:

try:
shutil.copy(s, os.path.join(usbpath, songname))
except ...

now, I need to figure out just what exceptions to handle.

Here's a bit of a left-field thought: Maybe none of them.

What are you actually doing when you get an exception? Can you
plausibly recover? If not - that is, if you're going to abort the
whole operation anyway - then save yourself the trouble of writing the
try/catch, and just let the exception propagate up (to the console, if
nowhere else).

On the other hand, if you want to simply report the error and continue
on (meaning you get as many songs copied as possible), then do what
others have recommended and catch OSError.

ChrisA

 

[Mark Lawrence]

OSError. In Python 3, I expect it would more specifically be a
FileNotFoundError, which is a subclass of OSError.

This wasn't introduced until Python 3.3 see
http://docs.python.org/3/whatsnew/3.3.html#pep-3151-reworking-the-os-and-io-exception-hierarchy

 

[Devin Jeanpierre]

Here's a bit of a left-field thought: Maybe none of them.

What are you actually doing when you get an exception? Can you
plausibly recover? If not - that is, if you're going to abort the
whole operation anyway - then save yourself the trouble of writing the
try/catch, and just let the exception propagate up (to the console, if
nowhere else).

He can't know if he should handle the errors if he doesn't know what
those errors are. Thus the question.

-- Devin

 

[bvdp]

IOError and OSError should cover all copy problems, I think.

How do you know that? I can figure it out as well by running the program, but I'd like to make the determination of what to catch when I'm writing the code.

 

[bvdp]

IOError and OSError should cover all copy problems, I think.

How do you know that? I can figure it out as well by running the program, but I'd like to make the determination of what to catch when I'm writing the code.

 

[bvdp]

Here's a bit of a left-field thought: Maybe none of them.

Not far left at all

What are you actually doing when you get an exception? Can you

plausibly recover? If not - that is, if you're going to abort the

whole operation anyway - then save yourself the trouble of writing the

try/catch, and just let the exception propagate up (to the console, if

nowhere else).

My first cut of the program did exactly that. Just abort the whole thing, figuring that the disk was full or buggered.

What I ran into was that half way through the process I ended up with a filename which the FAT32 didn't like. So, my brain-dead idea was to catch those (and ignore them) and continue on. But then I have to distinguish betweena bad filename and "real" errors.

But, you are probably correct if you say I should check the filename first Is there a module for that?

 

[bvdp]

Here's a bit of a left-field thought: Maybe none of them.

Not far left at all

What are you actually doing when you get an exception? Can you

plausibly recover? If not - that is, if you're going to abort the

whole operation anyway - then save yourself the trouble of writing the

try/catch, and just let the exception propagate up (to the console, if

nowhere else).

My first cut of the program did exactly that. Just abort the whole thing, figuring that the disk was full or buggered.

What I ran into was that half way through the process I ended up with a filename which the FAT32 didn't like. So, my brain-dead idea was to catch those (and ignore them) and continue on. But then I have to distinguish betweena bad filename and "real" errors.

But, you are probably correct if you say I should check the filename first Is there a module for that?

 

[Rick Johnson]

Every time I write a program with exception handling (and
I suppose that includes just about every program I write!)
I need to scratch my brain when I create try blocks.

For example, I'm writing a little program do copy specific
files to a USB stick. To do the actual copy I'm using:

try:
shutil.copy(s, os.path.join(usbpath, songname))
except ...

now, I need to figure out just what exceptions to handle.
Let's see:

IOError that means that the disk is full or otherwise
buggered. Better dump out of the loop.

But, I know there can be other errors as well. Doing some
tests, I know that certain filenames are invalid (I think
a "?" or unicode char is invalid when writing to a FAT32
filesystem). And, so what exception is that? Without
actually creating the error, I can't figure it out.

Well how can you expect an error to be thrown without creating the scenario that will throw one? *wink*

In this case, I can run the program an number of times and
parse out the errors and write code to catch various
things. But, I think I'm missing something completely.
Guess what I'm looking for is a list of possible
(probable?) errors for the shutil.copy() command. And, in
a much bigger manual, for most other commands.

No. What you are doing is *misunderstanding* that well written methods must follow the values of:

"doing one thing and doing it well"

In the case of "shutil.copy", that "one thing" is "coping files"; but that "one thing" does NOT include:

* resolving file paths
* validating file path chars on an OS by OS basis
* ensuring disc space is adequate
* etc...

Methods that take inputs, like in this case: "src" and "dst" file paths, should not be responsible for the validity of the inputs. It is the responsibility of the caller to validate these inputs PRIOR to injecting them into the method.

But even *IF* a convincing argument could be made for methods to validate inputs, then at what point do we draw the line? How many foolish samples of the possible permutation set should we consider? Should we inform the user of such foolish usage as:

shutil.copy(src=1, dst=2, follow_symlinks=isinstance)
shutil.copy(src='cat', dst='hat', follow_symlinks=list)

Now whilst these example are quite absurd, they are in-fact possibilities that must be considered *IF* we intend to follow your "wish" until it's logical conclusion.

 

[Terry Reedy]

But, I know there can be other errors as well. Doing some tests, I
know that certain filenames are invalid (I think a "?" or unicode
char is invalid when writing to a FAT32 filesystem). And, so what
exception is that? Without actually creating the error, I can't
figure it out.

So use the interactive interpreter (or idle, or ipython) and create the
error. You should always have it open when editing. Using less time that
it took you to write the above. 3.3, win7, (idle)
Traceback (most recent call last):
File "<pyshell#2>", line 1, in <module>
open('sdjhfjshdfkjsh')
FileNotFoundError: [Errno 2] No such file or directory: 'sdjhfjshdfkjsh'

Now, does shutil pass on FileNotFoundError? I will let you experiment.

There are error conditions that are hard to generate, but a bad file
name is not one of them.

 

[Chris Angelico]

Traceback (most recent call last):
File "<pyshell#2>", line 1, in <module>
open('sdjhfjshdfkjsh')
FileNotFoundError: [Errno 2] No such file or directory: 'sdjhfjshdfkjsh'

Now, does shutil pass on FileNotFoundError? I will let you experiment.

There are error conditions that are hard to generate, but a bad file name is
not one of them.

That's actually a perfectly valid file name, but one that doesn't
happen to have a corresponding file. However, the same technique will
work with the OP's description of "a filename which the FAT32 didn't
like" too.

ChrisA

 

[Terry Reedy]

open('sdjhfjshdfkjsh')

Traceback (most recent call last):
File "<pyshell#2>", line 1, in <module>
open('sdjhfjshdfkjsh')
FileNotFoundError: [Errno 2] No such file or directory: 'sdjhfjshdfkjsh'

Now, does shutil pass on FileNotFoundError? I will let you experiment.

There are error conditions that are hard to generate, but a bad file name is
not one of them.

That's actually a perfectly valid file name, but one that doesn't
happen to have a corresponding file. However, the same technique will
work with the OP's description of "a filename which the FAT32 didn't
like" too.

Yeah, that gives a different error and message.

open('a~`!@#$%^&*()_-+={[}]|\<,>.?/')

Traceback (most recent call last):
File "<pyshell#3>", line 1, in <module>
open('a~`!@#$%^&*()_-+={[}]|\<,>.?/')
OSError: [Errno 22] Invalid argument: 'a~`!@#$%^&*()_-+={[}]|\\<,>.?/'

 

[Chris Angelico]

open('sdjhfjshdfkjsh')

Traceback (most recent call last):
File "<pyshell#2>", line 1, in <module>
open('sdjhfjshdfkjsh')
FileNotFoundError: [Errno 2] No such file or directory: 'sdjhfjshdfkjsh'

Now, does shutil pass on FileNotFoundError? I will let you experiment.

There are error conditions that are hard to generate, but a bad file name
is
not one of them.

That's actually a perfectly valid file name, but one that doesn't
happen to have a corresponding file. However, the same technique will
work with the OP's description of "a filename which the FAT32 didn't
like" too.

Yeah, that gives a different error and message.

open('a~`!@#$%^&*()_-+={[}]|\<,>.?/')

Traceback (most recent call last):
File "<pyshell#3>", line 1, in <module>
open('a~`!@#$%^&*()_-+={[}]|\<,>.?/')
OSError: [Errno 22] Invalid argument: 'a~`!@#$%^&*()_-+={[}]|\\<,>.?/'

That should be:

OSError: Profanity not permitted on respectable file systems

ChrisA

 

[Steven D'Aprano]

Also, you can do `except:` for a catch-all, but it is discouraged unless
you have REALLY good reasons to do this. And, most of the time, you
don’t.

`except Exception` is to be much preferred over a bare except. It
excludes KeyboardInterruptError (the user hits Ctrl-C) and SystemExit,
which you normally either want to let through, or handle separately.


But yes, in general, only catch the minimum you *know* you need to catch
and can deal with. Anything else is a bug in your code that needs to be
fixed, and you can't fix it if you never see the exception.

 

[Chris Angelico]

But yes, in general, only catch the minimum you *know* you need to catch
and can deal with. Anything else is a bug in your code that needs to be
fixed, and you can't fix it if you never see the exception.

With the exception (if you'll excuse the expression) of "framework"
systems, where there's a distinct separation between "inside" and
"outside". Often then, the "outside" will catch any exception thrown
by the "inside" and deal with it in some generic way (for instance, a
web server might log the details and return HTTP 500 to the client,
then go back and handle the next request). Effectively, this is doing
the job of the top-level exception handler: log the exception (to the
console) and terminate.

ChrisA

 

[Steven D'Aprano]

How do you know that? I can figure it out as well by running the
program, but I'd like to make the determination of what to catch when
I'm writing the code.

In my experience, I would say:

20% by reading the documentation, 20% by experience, and 60% by
experimentation at the interactive interpreter.

Generally if you read the docs, you will get some idea of the exceptions
you can expect to get. E.g. the docs for open claim:

"Raise IOError upon failure."

(call "help(open)" in the interactive interpreter).

Experience and experimentation come into it in the (unfortunately very
common case) where the docs don't describe the exceptions you can expect.

For the specific case of IOError and OSError, another very useful skill
is googling for the specific errno you can test for:

try:
something()
except IOError as e:
if e.errno == whatever:
do_this()
else:
raise


In Python 3.3, this becomes much nicer with individual exceptions for the
most common errnos.

 

[Nobody]

Also, you can do `except:` for a catch-all, but it is discouraged unless
you have REALLY good reasons to do this. And, most of the time, you
don’t.

Most of the time you probably want to catch either Exception (which
excludes GeneratorExit, KeyboardInterrupt and SystemExit) or StandardError
(which excludes the above pluse warnings and StopIteration).

Only in specific circumstances can you reasonably go finer than that,
partly due to the set of exceptions a method may throw seldom being
documented, and partly due to duck typing; a parameter which is intended
to be a file object could realistically be any object which supports the
appropriate methods (e.g. .read()), and there's no guarantee that those
methods will have the same set of possible exceptions as a real file
object.