Why is the argparse module so inflexible?

[Andrew Berg]

I've begun writing a program with an interactive prompt, and it needs to parse input from the user. I thought the argparse module would be
great for this, but unfortunately it insists on calling sys.exit() at any sign of trouble instead of letting its ArgumentError exception
propagate so that I can handle it. Overriding ArgumentParser.error doesn't really help since methods like parse_known_args just send a
message to be relayed to the user as an argument after swallowing ArgumentError (which does have useful information in its attributes). If I
wanted to figure out what actually caused the exception to be raised, I'd have to parse the message, which is ugly at best. I understand
that most people do want argparse to just display a message and terminate if the arguments supplied aren't useful, but there's a lot of
potential in the module that is crippled outside the main use case. I have to wonder why a module that is meant to be imported would ever
call sys.exit(), even if that is what the caller would normally do if presented with an exception.

 

[Roy Smith]

I've begun writing a program with an interactive prompt, and it needs to
parse input from the user. I thought the argparse module would be
great for this, but unfortunately it insists on calling sys.exit() at any
sign of trouble instead of letting its ArgumentError exception
propagate so that I can handle it.

[...] there's a lot of potential in the module that is crippled
outside the main use case.

Having used (and written) a number of different ways of dealing with CLI
parsing (in several languages), I can tell you that argparse is pretty
cool. CLI parsing is amazingly complicated. Argparse turns that into a
job which is only moderately complicated and slightly annoying.
Compared to the alternatives, that's a big win.

Can you give us a concrete example of what you're trying to do?

You might look into "type=". It's normally used for things like
"type=int" or "type=float", but it could give it any user-defined
function as a type and this essentially becomes a hook to insert your
own code into the middle of the processing. Sometimes that can be
warped into doing all sorts of useful things.

 

[Andrew Berg]

Can you give us a concrete example of what you're trying to do?

The actual code I've written so far isn't easily condensed into a short simple snippet.
I'm trying to use argparse to handle all the little details of parsing and verifying arguments in the precmd hook for a cmd.Cmd child class.
argparse's help system is more sophisticated than cmd's help and does all the work of verifying arguments.
The problem I keep running into is that I can't handle any bad input very well. I would have to override every method that catches
ArgumentError in order to get a useful exception that I would then handle.
If I input something that begins with '-' that isn't recognized, parse_args doesn't even raise the exception; it just quits. In this case,
the message gets mangled if error is overridden, and I don't know why.

You might look into "type=". It's normally used for things like
"type=int" or "type=float", but it could give it any user-defined
function as a type and this essentially becomes a hook to insert your
own code into the middle of the processing. Sometimes that can be
warped into doing all sorts of useful things.

I don't think that would solve my problem, but it would probably be quite useful.

 

[Dave Angel]

The actual code I've written so far isn't easily condensed into a short simple snippet.
I'm trying to use argparse to handle all the little details of parsing and verifying arguments in the precmd hook for a cmd.Cmd child class.
argparse's help system is more sophisticated than cmd's help and does all the work of verifying arguments.
The problem I keep running into is that I can't handle any bad input very well. I would have to override every method that catches
ArgumentError in order to get a useful exception that I would then handle.
If I input something that begins with '-' that isn't recognized, parse_args doesn't even raise the exception; it just quits.

No, it raises the SystemExit exception. if you don't catch it, then the
program exits. Perhaps it's not clear to you, but sys.exit() just
raises the SystemExit exception, as Joshua pointed out.

In this case,
the message gets mangled if error is overridden, and I don't know why.

I can't help there.

 

[Robert Kern]

No, it raises the SystemExit exception. if you don't catch it, then the program
exits. Perhaps it's not clear to you, but sys.exit() just raises the SystemExit
exception, as Joshua pointed out.

Internally, the parser raises ArgumentError which has some useful pieces of
information, specifically, the name of the argument that failed to parse.
Unfortunately, it catches that error in parse_known_args(), then formats that
information into a message string to pass to the error() method, which by
default raises SystemExit with just that message string. It is somewhat
difficult to override the parse_known_args() to not lose that information from
the ArgumentError because you will have to copy-paste the rest of the code in there.

So yes, you can override the error() method or catch SystemExit if all you want
is the formatted message string, but I have to agree that there is a missed
opportunity to make argparse more widely usable for other command-line like
parsing tasks[1].

[1] As an existence proof, I offer you one that I wrote for handling IPython's
%magic commands. I just needed the formatted string, so I could just get away
with overriding error().

https://github.com/ipython/ipython/blob/master/IPython/core/magic_arguments.py

--
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless enigma
that is made terrible by our own mad attempt to interpret it as though it had
an underlying truth."
-- Umberto Eco

 

[Steven D'Aprano]

No, it raises the SystemExit exception. if you don't catch it, then the
program exits. Perhaps it's not clear to you, but sys.exit() just
raises the SystemExit exception, as Joshua pointed out.

[rant]
I think it is lousy design for a framework like argparse to raise a
custom ArgumentError in one part of the code, only to catch it elsewhere
and call sys.exit. At the very least, that ought to be a config option,
and off by default.

Libraries should not call sys.exit, or raise SystemExit. Whether to quit
or not is not the library's decision to make, that decision belongs to
the application layer. Yes, the application could always catch
SystemExit, but it shouldn't have to.

 

[Cameron Simpson]

| [rant]
| I think it is lousy design for a framework like argparse to raise a
| custom ArgumentError in one part of the code, only to catch it elsewhere
| and call sys.exit. At the very least, that ought to be a config option,
| and off by default.
|
| Libraries should not call sys.exit, or raise SystemExit. Whether to quit
| or not is not the library's decision to make, that decision belongs to
| the application layer. Yes, the application could always catch
| SystemExit, but it shouldn't have to.

+1

 

[Ethan Furman]

[rant]
I think it is lousy design for a framework like argparse to raise a
custom ArgumentError in one part of the code, only to catch it elsewhere
and call sys.exit. At the very least, that ought to be a config option,
and off by default.

Libraries should not call sys.exit, or raise SystemExit. Whether to quit
or not is not the library's decision to make, that decision belongs to
the application layer. Yes, the application could always catch
SystemExit, but it shouldn't have to.

So a library that is explicitly designed to make command-line scripts easier and friendlier should quit with a traceback?

Really?

 

[Modulok]

Have you looked into docopt?

-Modulok-

[rant]
I think it is lousy design for a framework like argparse to raise a
custom ArgumentError in one part of the code, only to catch it elsewhere
and call sys.exit. At the very least, that ought to be a config option,
and off by default.

Libraries should not call sys.exit, or raise SystemExit. Whether to quit
or not is not the library's decision to make, that decision belongs to
the application layer. Yes, the application could always catch
SystemExit, but it shouldn't have to.

So a library that is explicitly designed to make command-line scripts
easier and friendlier should quit with a traceback?

Really?

 

[rusi]

[rant]
I think it is lousy design for a framework like argparse to raise a
custom ArgumentError in one part of the code, only to catch it elsewhere
and call sys.exit. At the very least, that ought to be a config option,
and off by default.

Libraries should not call sys.exit, or raise SystemExit. Whether to quit
or not is not the library's decision to make, that decision belongs to
the application layer. Yes, the application could always catch
SystemExit, but it shouldn't have to.

So a library that is explicitly designed to make command-line scripts easier
and friendlier should quit with a traceback?

Really?

So a library that behaves like an app is OK?

Really?

I would have thought that with the sophistication of python's exception mechanism in place there would be corresponding peps in place proscribing suchbehavior. In the same way that if I submitted my super-duper library for inclusion into python with 7 spaces indentation, some kind dev would reply with

http://www.python.org/dev/peps/pep-0008/#indentation

 

[Terry Reedy]

[rant]
I think it is lousy design for a framework like argparse to raise a
custom ArgumentError in one part of the code, only to catch it elsewhere
and call sys.exit. At the very least, that ought to be a config option,
and off by default.

Libraries should not call sys.exit, or raise SystemExit. Whether to quit
or not is not the library's decision to make, that decision belongs to
the application layer. Yes, the application could always catch
SystemExit, but it shouldn't have to.

So a library that is explicitly designed to make command-line scripts easier
and friendlier should quit with a traceback?

Really?

So a library that behaves like an app is OK?

No, Steven is right as a general rule (do not raise SystemExit), but
argparse was considered an exception because its purpose is to turn a
module into an app. With the responses I have seen here, I agree that
this is a bit short-sighted, as inflexible behavior. The tracker issue
could use more review and comment.

 

[Isaac To]

So a library that is explicitly designed to make command-line scripts
easier and friendlier should quit with a traceback?

Really?

Perhaps put the functionality "handling of the exception of library to
sys.exit with a message" into a method so that the user can override it
(e.g., so that it just throw the same exception to the caller of the
library)?

 

[Steven D'Aprano]

[rant]
I think it is lousy design for a framework like argparse to raise a
custom ArgumentError in one part of the code, only to catch it
elsewhere and call sys.exit. At the very least, that OUGHT TO BE A
CONFIG OPTION, and OFF BY DEFAULT.

[emphasis added]

So a library that is explicitly designed to make command-line scripts
easier and friendlier should quit with a traceback?

Really?

Yes, really.

Tracebacks are not that unfriendly, generally speaking. In my experience,
the average non-technical person is no more confused and distressed by a
traceback extending over thirty lines than they are by a one line error
message. As the developer, I should see the tracebacks by default[1]. If
I want to suppress or simplify them, then I should take explicit steps to
do so, either by catching the exception and calling sys.exit myself, or
at least by setting a runtime config option to the library.

This also allows me to enable debugging in my app by showing tracebacks,
or disable it by hiding them. That should be my decision, not the
library. If the library catches exceptions then exits, throwing away
potentially useful information, that makes it difficult to debug anything
relying on the library.

I'm willing to concede that, just maybe, something like argparse could
default to "catch exceptions and exit" ON rather than OFF.


[1] There's something in the Zen of Python about that...

 

[Marcin Szamotulski]

[rant]
I think it is lousy design for a framework like argparse to raise a
custom ArgumentError in one part of the code, only to catch it
elsewhere and call sys.exit. At the very least, that OUGHT TO BE A
CONFIG OPTION, and OFF BY DEFAULT.

[emphasis added]

So a library that is explicitly designed to make command-line scripts
easier and friendlier should quit with a traceback?

Really?

Yes, really.

Tracebacks are not that unfriendly, generally speaking. In my experience,
the average non-technical person is no more confused and distressed by a
traceback extending over thirty lines than they are by a one line error
message. As the developer, I should see the tracebacks by default[1]. If
I want to suppress or simplify them, then I should take explicit steps to
do so, either by catching the exception and calling sys.exit myself, or
at least by setting a runtime config option to the library.

This also allows me to enable debugging in my app by showing tracebacks,
or disable it by hiding them. That should be my decision, not the
library. If the library catches exceptions then exits, throwing away
potentially useful information, that makes it difficult to debug anything
relying on the library.

I'm willing to concede that, just maybe, something like argparse could
default to "catch exceptions and exit" ON rather than OFF.


[1] There's something in the Zen of Python about that...

Although I got confused at the first time I was using argparse (or
optparse which is now obsolte and also has this feature), I see the
value when you write scripts. It is mostly annoying when playing with
it in a console, but there is a very easy (but partial) fix for that:
just subclass argparse.ArgumentParser:

import sys
class ArgumentParser(argparse.ArgumentParser):
def exit(self, status=0, message=None):
if message:
self._print_message(message, sys.stderr)

now the parser will not exit, though there is no ease fix to get the
traceback: self.exit() is called in various places sometimes inside
a try block.

Best regards,
Marcin http://mail.python.org/mailman/listinfo/python-list

 

[Roy Smith]

So a library that behaves like an app is OK?

No, Steven is right as a general rule (do not raise SystemExit), but
argparse was considered an exception because its purpose is to turn a
module into an app. With the responses I have seen here, I agree that
this is a bit short-sighted, as inflexible behavior. The tracker issue
could use more review and comment.[/QUOTE]

What is the tracker issue number or url?

 

[Andrew Berg]

What is the tracker issue number or url?

http://bugs.python.org/issue9938

 

[MRAB]

[rant]
I think it is lousy design for a framework like argparse to raise a
custom ArgumentError in one part of the code, only to catch it
elsewhere and call sys.exit. At the very least, that OUGHT TO BE A
CONFIG OPTION, and OFF BY DEFAULT.

[emphasis added]

So a library that is explicitly designed to make command-line scripts
easier and friendlier should quit with a traceback?

Really?

Yes, really.

[snip]
+1

It's the job of argparse to parse the arguments. What should happen if
they're invalid is for its caller to decide.

 

[Ethan Furman]

I'm willing to concede that, just maybe, something like argparse could
default to "catch exceptions and exit" ON rather than OFF.

On this we can agree.