Pattern for foo tool <-> API <-> shell|GUI

[Anastasios Hatzis]

I'm looking for a pattern where different client implementations can use the
same commands of some fictive tool ("foo") by accessing some kind of API.
Actually I have the need for such pattern for my own tool
(http://openswarm.sourceforge.net). I already started restructuring my code
to separate the actual command implementations from the command-line scripts
(which is optparser-based now) and have some ideas how to proceed. But
probably there is already a good pattern used for Python-based tools.

In the case that some of you are interested into this topic and my recent
thoughts, you may want to have a look at the description below. Any comments
are very much appreciated. Hopefully this list is a good place for discussing
a pattern, otherwise I would be okay to move this to another place. Thank
you.

Here we go:
The tool package itself provides several commands, although not important for
the pattern itself, here some examples: modifying user-specific preferences,
creating and changing project settings files, project-related
code-generation, or combinations of such commands ... later also commands for
transformation between several XML formats etc. The classes which implement
these commands are currently in multiple modules, each having a class named
CmdHandler.

I have some Python scripts (each having a ScriptHandler classes), for use via
command-line. Each ScriptHandler class is responsible to add all related
command-line options and process those provided by the user (based on
optparse library from Python standard lib). The script then calls the
corresponding command and provide the verified options as parameters.

Due to the nature of the tool under specific conditions the following results
may come during command execution:
* successful execution, no interaction
* critical error, execution cancelled
* user interaction needed (e.g. prompt user to approve replace existing
directory (yes/no), prompt user to provide an alternative option)

Command-line interactions work simply with raw_input().

So far this works. Nevertheless, there are some other aspects that could be
improved, but this is another topic: The tool uses custom exceptions (e.g.
for critical errors) and logging features (based on logging from Python
standard lib). Currently no automated tests, but I have to add.

For the next step I plan to support not only my own command-line scripts, but
also a GUI to access the commands, as well as 3rd-party products (themselves
command-line scripts or GUIs, such as foo plugins for any 3rd-party-tools). As
far as I see, these clients need to implement a handler that:
(1) Collecting all required parameters and optional parameters from a user
(2) Provide these parameters for a particular call to command API
(3) Provides some kind of hooks that are called back from the API on specific
events, e.g. Question with user-choice; Information with user-input
(4) Provide a logging handler object from the tool logging
class or a sub-class of that in the case that a client-specific logging object
should be triggered on each debug, message, warning etc.

(1) is very client-specific, e.g. in a GUI via dialogs.

(2) Each command provides a signature for all required/optional parameters.
They are all verified from the command itself, although a client could do
some verification at the first place.

(3) Example use-case: a command needs to know if the user wants the command to
proceed with a particular action, e.g. "Do you want to delete bar.txt?"
with "Yes", "No" and "Cancel" choice. So the client's handler object (which
is provided as first parameter to each command) implements client-specific
features to show the user this question (e.g. pop-up dialog with question and
three buttons), receive the user input (clicking one of the buttons) and pass
this choice back to the foo API. Alternatively some kind of text information
could be required, as in raw_input(), so actually this probably would be two
different interaction features to be implemented.

(4) The foo API also provides a logging class. The client needs to initialize
such an object and provide it as member of the handler object provided to the
API. I wonder if some clients may have own logging features and want to
include all log messages from foo tool to the own logs. In this case a client
could its own sub-class of the foo logging class and extending it with
callbacks to its (client-)native logging object.

What do you think about this?

Best regards,
Anastasios

 

[kyosohma]

I'm looking for a pattern where different client implementations can use the
same commands of some fictive tool ("foo") by accessing some kind of API.
Actually I have the need for such pattern for my own tool
(http://openswarm.sourceforge.net). I already started restructuring my code
to separate the actual command implementations from the command-line scripts
(which is optparser-based now) and have some ideas how to proceed. But
probably there is already a good pattern used for Python-based tools.

In the case that some of you are interested into this topic and my recent
thoughts, you may want to have a look at the description below. Any comments
are very much appreciated. Hopefully this list is a good place for discussing
a pattern, otherwise I would be okay to move this to another place. Thank
you.

Here we go:
The tool package itself provides several commands, although not important for
the pattern itself, here some examples: modifying user-specific preferences,
creating and changing project settings files, project-related
code-generation, or combinations of such commands ... later also commands for
transformation between several XML formats etc. The classes which implement
these commands are currently in multiple modules, each having a class named
CmdHandler.

I have some Python scripts (each having a ScriptHandler classes), for use via
command-line. Each ScriptHandler class is responsible to add all related
command-line options and process those provided by the user (based on
optparse library from Python standard lib). The script then calls the
corresponding command and provide the verified options as parameters.

Due to the nature of the tool under specific conditions the following results
may come during command execution:
* successful execution, no interaction
* critical error, execution cancelled
* user interaction needed (e.g. prompt user to approve replace existing
directory (yes/no), prompt user to provide an alternative option)

Command-line interactions work simply with raw_input().

So far this works. Nevertheless, there are some other aspects that could be
improved, but this is another topic: The tool uses custom exceptions (e.g.
for critical errors) and logging features (based on logging from Python
standard lib). Currently no automated tests, but I have to add.

For the next step I plan to support not only my own command-line scripts, but
also a GUI to access the commands, as well as 3rd-party products (themselves
command-line scripts or GUIs, such as foo plugins for any 3rd-party-tools). As
far as I see, these clients need to implement a handler that:
(1) Collecting all required parameters and optional parameters from a user
(2) Provide these parameters for a particular call to command API
(3) Provides some kind of hooks that are called back from the API on specific
events, e.g. Question with user-choice; Information with user-input
(4) Provide a logging handler object from the tool logging
class or a sub-class of that in the case that a client-specific logging object
should be triggered on each debug, message, warning etc.

(1) is very client-specific, e.g. in a GUI via dialogs.

(2) Each command provides a signature for all required/optional parameters.
They are all verified from the command itself, although a client could do
some verification at the first place.

(3) Example use-case: a command needs to know if the user wants the command to
proceed with a particular action, e.g. "Do you want to delete bar.txt?"
with "Yes", "No" and "Cancel" choice. So the client's handler object (which
is provided as first parameter to each command) implements client-specific
features to show the user this question (e.g. pop-up dialog with question and
three buttons), receive the user input (clicking one of the buttons) and pass
this choice back to the foo API. Alternatively some kind of text information
could be required, as in raw_input(), so actually this probably would be two
different interaction features to be implemented.

(4) The foo API also provides a logging class. The client needs to initialize
such an object and provide it as member of the handler object provided to the
API. I wonder if some clients may have own logging features and want to
include all log messages from foo tool to the own logs. In this case a client
could its own sub-class of the foo logging class and extending it with
callbacks to its (client-)native logging object.

What do you think about this?

Best regards,
Anastasios

I think if you want to use a GUI, wxpython or Tkinter would work well
for you. wxPython has more widgets from the start, but is also more
complex. Tkinter is good for quick and dirty GUIs, but gets
increasingly more complicated to deal with the more complex the GUI
has to be, in general. At least, that's what I am told.

I started out using Tkinter. It's great for prototyping, but I had to
port VBA code to a Python GUI and Tkinter just didn't have the widgets
I needed (including the the submodules PMW and Tix). So I went with
wxPython. I personally like it quite a bit. Plus you can use XRC with
wxPython to help separate out the GUI code from the logic.

Both toolkits can accept user input in myriads of ways and both have
standard dialogs that look almost or exactly native to the OS they are
running on. Download wxPython's demo to see all that it can do.
Tkinter is already installed and it has demos in its subdirectory
under Python.

There's my 2 cents.

Mike

 

[Anastasios Hatzis]

I think if you want to use a GUI, wxpython or Tkinter would work well
for you. wxPython has more widgets from the start, but is also more
complex. Tkinter is good for quick and dirty GUIs, but gets
increasingly more complicated to deal with the more complex the GUI
has to be, in general. At least, that's what I am told.

I started out using Tkinter. It's great for prototyping, but I had to
port VBA code to a Python GUI and Tkinter just didn't have the widgets
I needed (including the the submodules PMW and Tix). So I went with
wxPython. I personally like it quite a bit. Plus you can use XRC with
wxPython to help separate out the GUI code from the logic.

Both toolkits can accept user input in myriads of ways and both have
standard dialogs that look almost or exactly native to the OS they are
running on. Download wxPython's demo to see all that it can do.
Tkinter is already installed and it has demos in its subdirectory
under Python.

There's my 2 cents.

Mike

Mike, thank you for your post and taking your time. I understand the reasons
for your recommendation and totally agree with your considerations. Actually
I already have decided to use wxPython for the GUI job in that project. I
guess I didn't make myself very clear in the OP, sorry for this, so I try
again with hopefully more clarity:

I'm working on a tool which is totally command-line based and consisting of
multiple scripts. The user can execute a Python script in the shell, this
script does some basic verification before delegating a call into my tool's
package and depending on some arguments and options provided in the
command-line, e.g.
$python generate.py myproject --force --verbose
the tool processes whatever necessary. There are multiple command handlers
available in this package which are responsible for different tasks and
depending of the script that has been executed one or more of these command
handlers are fired to do their work

Now I need to extend my tool in that way, that it can be accessed not only
from my own scripts, but also by other, additional "client implementations",
such as my own wxPython-based GUI, as well as Python scripts and GUIs (wx,
Tkinter,...) of other tools that integrate some kind of plug-in. For example
there is a Python-based UML tool realized with wxGTK (AFAIR). The developers
of this tool intent to create a small plug-in, so their users can access
features of my tool inside of this UML tool.

So, I need to create an API that can be used by clients to access specific
features of my tool. And I'm looking for patterns how this could be realized
in a smart Pythonic way.

And I don't think that this is very trivial (at least not for my programming
skill level). In the given example "generate.py" (above) the following
scenario is pretty likely:

(1) User works with UML tool and clicks in some dialog a "generate" button
(2) UML tool triggers this event an calls a magic generate() method of my tool
(via the API I provide for this purpose), like my generate.py script would do
same way
(3) Somewhen with-in this generate process my tool may need to get some
information from the user in order to continue (it is in the nature of the
features that I can't avoid this need of interaction in any case).

Currently, since I have only this command-line script "user-interface", I have
it that way implemented, that the command handler class (responsible to
supervise the entire generate process) just prints a question to the console
and prompts the user to input his choice. This is done by raw_input() called
inside the command and the command handler itself receives the user response
directly from the shell.

As far as I understand now, instead of calling raw_input() - which would make
no sense if the client would be some GUI - my command handler should call
some kind of hook or delegation method in the client, e.g. the method
promptUser(QuestionText, UserChoiceArgs). Any client should implement at
least this method.

If a client's promptUser() method is called the client would be responsible to
realize the appropriate user-dialog, such as a shell input or a wxFrame or
whatever reasonable for the nature of the client. Depending on the
interaction of the user with this user-interface the client then passes the
user-choice back to my tool as return value of promptUser().

In addition there would be other aspects, such as as triggers for logging
messages a.s.o., but this is probably not that important yet.

I don't know if I'm totally wrong with this design. I appreciate that it would
encapsulate the logic to layers of separated concerns and with such API it
would be possible to develop new clients rather easy (which would be a great
benefit, I think). There is just still doubt if there isn't a better way. I
would think that I'm not the first one who has a command-line based tool in
Python and wants to provide an interface to additional client programs,
especially GUIs.

I tried to find some appropriate pattern or tutorial on the net, but haven't
been successful.

Thank you, again.

Best regards,
Anastasios

 

[Steven Bethard]

I'm working on a tool which is totally command-line based and consisting of
multiple scripts. The user can execute a Python script in the shell, this
script does some basic verification before delegating a call into my tool's
package and depending on some arguments and options provided in the
command-line, e.g.
$python generate.py myproject --force --verbose
the tool processes whatever necessary. There are multiple command handlers
available in this package which are responsible for different tasks and
depending of the script that has been executed one or more of these command
handlers are fired to do their work

Side note: you might find argparse (http://argparse.python-hosting.com/)
makes this a bit easier if you have positional arguments or sub-commands::

>>> parser = argparse.ArgumentParser()
>>> parser.add_argument('name')
>>> parser.add_argument('--force', action='store_true')
>>> parser.add_argument('--verbose', action='store_true')
>>> parser.parse_args(['my_project', '--force', '--verbose'])

Namespace(force=True, name='my_project', verbose=True)

>>> parser = argparse.ArgumentParser()
>>> subparsers = parser.add_subparsers()
>>> cmd1_parser = subparsers.add_parser('cmd1')
>>> cmd1_parser.add_argument('--foo')
>>> cmd2_parser = subparsers.add_parser('cmd2')
>>> cmd2_parser.add_argument('bar')
>>> parser.parse_args(['cmd1', '--foo', 'X']) Namespace(foo='X')
>>> parser.parse_args(['cmd2', 'Y'])

Namespace(bar='Y')

And I don't think that this is very trivial (at least not for my programming
skill level). In the given example "generate.py" (above) the following
scenario is pretty likely:

(1) User works with UML tool and clicks in some dialog a "generate" button
(2) UML tool triggers this event an calls a magic generate() method of my tool
(via the API I provide for this purpose), like my generate.py script would do
same way
(3) Somewhen with-in this generate process my tool may need to get some
information from the user in order to continue (it is in the nature of the
features that I can't avoid this need of interaction in any case).

So you're imagining an API something like::

def generate(name,
force=False,
verbose=False,
handler=command_line_handler):
...
choice = handler.prompt_user(question_text, user_choices)
...

where the command-line handler might look something like::

class CommandLineHandler(object):
...
def prompt_user(self, question_text, user_choices):
while True:
choice = raw_input(question_text)
if choice in user_choices:
return choice
print 'invalid choice, choose from %s' % choices

and the GUI client would implement the equivalent thing with dialogs?
That seems basically reasonable to me, though you should be clear in the
documentation of generate() -- and any other methods that accept handler
objects -- exactly what methods the handler must provide.

You also may find that "prompt_user" is a bit too generic -- e.g. a file
chooser dialog looks a lot different from a color chooser dialog -- so
you may need to split this up into "prompt_user_file",
"prompt_user_color", etc. so that handler's don't have to introspect the
question text to know what to do...

STeVe

 

[Anastasios Hatzis]

Side note: you might find argparse (http://argparse.python-hosting.com/)

makes this a bit easier if you have positional arguments or sub-commands::

Steve, thank you, for the note. I didn't know argparse before. I have multiple
scripts since optparse puts all arguments and options into one single help
text, and the arguments and options are too specific for most commands (and
thus the help would be absolutely overloaded and useless for new users). It
seems that argparse has multiple help pages separated for each sub-command,
as far as I understand the page.

So you're imagining an API something like::

def generate(name,
force=False,
verbose=False,
handler=command_line_handler):
...
choice = handler.prompt_user(question_text, user_choices)
...

where the command-line handler might look something like::

class CommandLineHandler(object):
...
def prompt_user(self, question_text, user_choices):
while True:
choice = raw_input(question_text)
if choice in user_choices:
return choice
print 'invalid choice, choose from %s' % choices

and the GUI client would implement the equivalent thing with dialogs?

Exactly.

- Now, as I see your example, I wonder if this would work with a GUI which is
event-driven... I have to look into my wx GUI prototype.

That seems basically reasonable to me, though you should be clear in the
documentation of generate() -- and any other methods that accept handler
objects -- exactly what methods the handler must provide.

You also may find that "prompt_user" is a bit too generic -- e.g. a file
chooser dialog looks a lot different from a color chooser dialog -- so
you may need to split this up into "prompt_user_file",
"prompt_user_color", etc. so that handler's don't have to introspect the
question text to know what to do...

STeVe

Hey, right, good idea. I didn't think about the different task-specific
dialogs in most GUIs. But I see that usability will gain benefit from
differentiated "prompt" methods.

Anastasios