Re: Retrieving the full command line

Discussion in 'Python' started by Tim Golden, Jan 22, 2013.

  1. Tim Golden

    Tim Golden Guest

    On 22/01/2013 14:53, Terry Reedy wrote:
    > On 1/22/2013 4:24 AM, Tim Golden wrote:
    >> [Python 2.7/3.3 (and hg tip) running on Windows. Not Windows-specific,
    >> though].
    >>
    >> I use the python -mpackage incantation to run a package which has a
    >> __main__.py module and which uses relative imports internally.
    >>
    >> I'm developing under cherrypy which includes a reloader for development.
    >> The reloader attempts to rebuild the original
    >> command line by combining sys.executable and sys.argv and then does an
    >> execv.
    >>
    >> There does not appear to be any way within Python of determining the
    >> command line I used. The combination of sys.executable and sys.argv in
    >> this case will look like: "c:\python33\python.exe app/__main__.py". But
    >> running this precludes the use of package-relative imports.

    >
    > If I understand right, the reloader should be updated to translate
    > 'x/__main__.py' to '-m x'. Filenames of form'__x__' are reserved, in a
    > sense, like similar identifiers in programs, and '__main__.py' should
    > not be used for a file meant to executed directly.


    To be clear: it's Python itself, not the reloader, which is coming up
    with __main__.py. sys.executable is "c:\python33\python.exe" and
    sys.argv is ['c:\path\to\__main__.py'] for a program which has been
    started by "c:\python33\python.exe -mpath\to".

    Obviously, there is any number of ways around this specific issue,
    including what you suggest: a canonical rewrite of "python
    path\to\__main__.py" into "python -mpath\to". But it's not clear to me
    that this rewrite should be the responsibility of calling code.

    TJG
    Tim Golden, Jan 22, 2013
    #1
    1. Advertising

  2. On Tue, 22 Jan 2013 15:07:18 +0000, Tim Golden wrote:

    > On 22/01/2013 14:53, Terry Reedy wrote:
    >> On 1/22/2013 4:24 AM, Tim Golden wrote:
    >>> [Python 2.7/3.3 (and hg tip) running on Windows. Not Windows-specific,
    >>> though].
    >>>
    >>> I use the python -mpackage incantation to run a package which has a
    >>> __main__.py module and which uses relative imports internally.
    >>>
    >>> I'm developing under cherrypy which includes a reloader for
    >>> development. The reloader attempts to rebuild the original command
    >>> line by combining sys.executable and sys.argv and then does an execv.
    >>>
    >>> There does not appear to be any way within Python of determining the
    >>> command line I used. The combination of sys.executable and sys.argv in
    >>> this case will look like: "c:\python33\python.exe app/__main__.py".
    >>> But running this precludes the use of package-relative imports.

    >>
    >> If I understand right, the reloader should be updated to translate
    >> 'x/__main__.py' to '-m x'. Filenames of form'__x__' are reserved, in a
    >> sense, like similar identifiers in programs, and '__main__.py' should
    >> not be used for a file meant to executed directly.

    >
    > To be clear: it's Python itself, not the reloader, which is coming up
    > with __main__.py. sys.executable is "c:\python33\python.exe" and
    > sys.argv is ['c:\path\to\__main__.py'] for a program which has been
    > started by "c:\python33\python.exe -mpath\to".


    I don't believe you can give direct paths to the -m flag. It uses the
    normal import mechanism to locate a module or package, so you have to
    give it a name which would be importable.

    c:\python33\python.exe -m app

    would work, where "app" is either a package or module:

    C:\something\on\PYTHONPATH\app\__main__.py
    C:\something\on\PYTHONPATH\app.py


    > Obviously, there is any number of ways around this specific issue,
    > including what you suggest: a canonical rewrite of "python
    > path\to\__main__.py" into "python -mpath\to". But it's not clear to me
    > that this rewrite should be the responsibility of calling code.



    I am a bit disturbed that you cannot distinguish between:

    python C:\something\on\pythonpath\app\__main__.py

    python -m app


    by inspecting the command line. I consider it a bug, or at least a
    misfeature, if Python transforms the command line before making it
    available in sys.argv.


    --
    Steven
    Steven D'Aprano, Jan 22, 2013
    #2
    1. Advertising

  3. On 22 January 2013 23:46, Steven D'Aprano
    <> wrote:
    [SNIP]
    >
    > I am a bit disturbed that you cannot distinguish between:
    >
    > python C:\something\on\pythonpath\app\__main__.py
    >
    > python -m app
    >
    >
    > by inspecting the command line. I consider it a bug, or at least a
    > misfeature, if Python transforms the command line before making it
    > available in sys.argv.


    The purpose of the -m option is that you can run a script that is
    located via the Python import path instead of an explicit file path.
    The idea is that if '/path/to/somewhere' is in sys.path then:
    python -m script arg1 arg2
    is equivalent to
    python /path/to/somewhere/script.py arg1 arg2

    If Python didn't modify sys.argv then 'script.py' would need to be
    rewritten to understand that sys.argv would be in a different format
    when it was invoked using the -m option.

    I believe that it has previously been proposed to include something
    like sys.raw_argv, although perhaps for different reasons. Although
    something that might be more useful to the OP (and that I at one point
    wanted) would be a function similar to execfile but that would launch
    a separate process using the same interpreter as the current process.


    Oscar
    Oscar Benjamin, Jan 23, 2013
    #3
  4. On Wed, 23 Jan 2013 00:53:21 +0000, Oscar Benjamin wrote:

    > On 22 January 2013 23:46, Steven D'Aprano
    > <> wrote: [SNIP]
    >>
    >> I am a bit disturbed that you cannot distinguish between:
    >>
    >> python C:\something\on\pythonpath\app\__main__.py
    >>
    >> python -m app
    >>
    >>
    >> by inspecting the command line. I consider it a bug, or at least a
    >> misfeature, if Python transforms the command line before making it
    >> available in sys.argv.

    >
    > The purpose of the -m option is that you can run a script that is
    > located via the Python import path instead of an explicit file path. The
    > idea is that if '/path/to/somewhere' is in sys.path then:
    > python -m script arg1 arg2
    > is equivalent to
    > python /path/to/somewhere/script.py arg1 arg2
    >
    > If Python didn't modify sys.argv then 'script.py' would need to be
    > rewritten to understand that sys.argv would be in a different format
    > when it was invoked using the -m option.


    I don't think that it would be in a different format. Normally people
    only care about sys.argv[1:], the actual arguments. argv[0], the name of
    the script, already comes in multiple formats: absolute or relative paths.

    Currently, if I have a package __main__.py that prints sys.argv, I get
    results like this:

    steve@runes:~$ python3.3 /home/steve/python/testpackage/__main__.py ham
    spam eggs
    ['/home/steve/python/testpackage/__main__.py', 'ham', 'spam', 'eggs']


    which is correct, that's what I gave on the command line. But:

    steve@runes:~$ python3.3 -m testpackage ham spam eggs
    ['/home/steve/python/testpackage/__main__.py', 'ham', 'spam', 'eggs']


    The second example is lying. It should say:

    ['-m testpackage', 'ham', 'spam', 'eggs']


    If you are one of the few people who care about argv[0], then you are
    already dealing with the fact that the name of the executable script is
    not always an absolute path and therefore can vary greatly from one call
    to another. Hell, if you are on a system with soft links, the name of the
    script in the command line is not even necessarily the name of the
    module. So there's not much more effort involved in dealing with one
    extra case:

    # assuming you care, which most people don't
    if sys.argv[0].startswith('-m'):
    do_something()
    elif os.path.isabs(sys.argv[0]):
    do_this()
    else: # relative path
    do_that()



    --
    Steven
    Steven D'Aprano, Jan 23, 2013
    #4
  5. Tim Golden

    Tim Golden Guest

    On 23/01/2013 03:58, Steven D'Aprano wrote:
    > Currently, if I have a package __main__.py that prints sys.argv, I get
    > results like this:
    >
    > steve@runes:~$ python3.3 /home/steve/python/testpackage/__main__.py ham
    > spam eggs
    > ['/home/steve/python/testpackage/__main__.py', 'ham', 'spam', 'eggs']
    >
    >
    > which is correct, that's what I gave on the command line. But:
    >
    > steve@runes:~$ python3.3 -m testpackage ham spam eggs
    > ['/home/steve/python/testpackage/__main__.py', 'ham', 'spam', 'eggs']
    >
    >
    > The second example is lying. It should say:
    >
    > ['-m testpackage', 'ham', 'spam', 'eggs']


    Thanks for the input, Steven & Oscar.

    Apologies for the confusion over my initial example. Of course, -m runs
    something on sys.path, not something in the filesystem as such. I
    confused myself because, running on Windows where the current directory
    is on the path, I sit in c:\path\to and do python -mapp

    Now I look harder, this discussion is basically issue14208:

    http://bugs.python.org/issue14208

    so I'll probably go and contribute over there.

    TJG
    Tim Golden, Jan 23, 2013
    #5
  6. On 23 January 2013 03:58, Steven D'Aprano
    <> wrote:
    > On Wed, 23 Jan 2013 00:53:21 +0000, Oscar Benjamin wrote:
    >
    >> On 22 January 2013 23:46, Steven D'Aprano
    >> <> wrote: [SNIP]
    >>>

    >> The purpose of the -m option is that you can run a script that is
    >> located via the Python import path instead of an explicit file path. The
    >> idea is that if '/path/to/somewhere' is in sys.path then:
    >> python -m script arg1 arg2
    >> is equivalent to
    >> python /path/to/somewhere/script.py arg1 arg2
    >>
    >> If Python didn't modify sys.argv then 'script.py' would need to be
    >> rewritten to understand that sys.argv would be in a different format
    >> when it was invoked using the -m option.

    >
    > I don't think that it would be in a different format. Normally people
    > only care about sys.argv[1:], the actual arguments. argv[0], the name of
    > the script, already comes in multiple formats: absolute or relative paths.
    >
    > Currently, if I have a package __main__.py that prints sys.argv, I get
    > results like this:
    >
    > steve@runes:~$ python3.3 /home/steve/python/testpackage/__main__.py ham
    > spam eggs
    > ['/home/steve/python/testpackage/__main__.py', 'ham', 'spam', 'eggs']
    >
    >
    > which is correct, that's what I gave on the command line. But:
    >
    > steve@runes:~$ python3.3 -m testpackage ham spam eggs
    > ['/home/steve/python/testpackage/__main__.py', 'ham', 'spam', 'eggs']
    >
    >
    > The second example is lying. It should say:
    >
    > ['-m testpackage', 'ham', 'spam', 'eggs']


    I don't know why you would expect this. I imagined that you would want

    ['-m', 'testpackage', 'ham', 'spam', 'eggs']

    If the two were combined into one string I would expect it to at least
    be a valid argument list:

    ['-mtestpackage', 'ham', 'spam', 'eggs']

    >
    >
    > If you are one of the few people who care about argv[0], then you are
    > already dealing with the fact that the name of the executable script is
    > not always an absolute path and therefore can vary greatly from one call
    > to another. Hell, if you are on a system with soft links, the name of the
    > script in the command line is not even necessarily the name of the
    > module. So there's not much more effort involved in dealing with one
    > extra case:


    Unless I've missed something sys.argv[0] is always a valid path to the
    script. Whether it is absolute or not shouldn't matter. For imported
    modules the path is available from __name__. For a script that is
    executed rather than imported __name__ == "__main__" but the path is
    accessible from sys.argv[0]. If you are one of those people who cares
    about sys.argv[0] then this is probably the value that you wanted it
    to contain.

    If it were important for sys.argv to show how exactly the script was
    located and executed, then why not also include the 'python3.3'
    command line argument (the real argv[0])? sys.argv emulates the argv
    that e.g. a C program would get. The real command line used is not
    exactly the same since a Python script is not a directly executable
    binary, so Python processes the argument list before passing it
    through.

    In the OP's case, the script is never invoked without -m so the following works:

    cmdline = [sys.executable, '-m', __package__] + sys.argv[1:]

    In the more general case it would be better to have an API
    specifically for this purpose.


    Oscar
    Oscar Benjamin, Jan 23, 2013
    #6
  7. On Wed, 23 Jan 2013 10:01:24 +0000, Oscar Benjamin wrote:

    > On 23 January 2013 03:58, Steven D'Aprano
    > <> wrote:
    >> On Wed, 23 Jan 2013 00:53:21 +0000, Oscar Benjamin wrote:
    >>
    >>> On 22 January 2013 23:46, Steven D'Aprano
    >>> <> wrote: [SNIP]
    >>>>
    >>> The purpose of the -m option is that you can run a script that is
    >>> located via the Python import path instead of an explicit file path.
    >>> The idea is that if '/path/to/somewhere' is in sys.path then:
    >>> python -m script arg1 arg2
    >>> is equivalent to
    >>> python /path/to/somewhere/script.py arg1 arg2
    >>>
    >>> If Python didn't modify sys.argv then 'script.py' would need to be
    >>> rewritten to understand that sys.argv would be in a different format
    >>> when it was invoked using the -m option.

    >>
    >> I don't think that it would be in a different format. Normally people
    >> only care about sys.argv[1:], the actual arguments. argv[0], the name
    >> of the script, already comes in multiple formats: absolute or relative
    >> paths.
    >>
    >> Currently, if I have a package __main__.py that prints sys.argv, I get
    >> results like this:
    >>
    >> steve@runes:~$ python3.3 /home/steve/python/testpackage/__main__.py ham
    >> spam eggs
    >> ['/home/steve/python/testpackage/__main__.py', 'ham', 'spam', 'eggs']
    >>
    >>
    >> which is correct, that's what I gave on the command line. But:
    >>
    >> steve@runes:~$ python3.3 -m testpackage ham spam eggs
    >> ['/home/steve/python/testpackage/__main__.py', 'ham', 'spam', 'eggs']
    >>
    >>
    >> The second example is lying. It should say:
    >>
    >> ['-m testpackage', 'ham', 'spam', 'eggs']

    >
    > I don't know why you would expect this. I imagined that you would want
    >
    > ['-m', 'testpackage', 'ham', 'spam', 'eggs']



    No. argv[0] is intended to be the script being called, argv[1:] for the
    arguments to the script. Given the two choices:

    1) Break every Python script that expects argv[1:] to be the arguments
    to the script, forcing them to decide whether they should look at
    argv[1:] or argv[2:] according to whether or not argv[0] == '-m';

    or

    2) don't break anything, but make a very small addition to the semantics
    of argv[0] (was: "the path to the script", add "or -m and the name of
    module/package") that won't break anyone's code;


    there's practically no choice in the matter.



    > If the two were combined into one string I would expect it to at least
    > be a valid argument list:
    >
    > ['-mtestpackage', 'ham', 'spam', 'eggs']


    Okay, fair point. I didn't consider that.

    Note however that there is an ambiguity between calling "python -mspam"
    and calling a script literally named "-mspam". But that same ambiguity
    exists in the shell, so I don't consider it a problem. You cannot call a
    script named -mspam unless you use something like this "python ./-mspam".


    >> If you are one of the few people who care about argv[0], then you are
    >> already dealing with the fact that the name of the executable script is
    >> not always an absolute path and therefore can vary greatly from one
    >> call to another. Hell, if you are on a system with soft links, the name
    >> of the script in the command line is not even necessarily the name of
    >> the module. So there's not much more effort involved in dealing with
    >> one extra case:

    >
    > Unless I've missed something sys.argv[0] is always a valid path to the
    > script. Whether it is absolute or not shouldn't matter.


    Sure. But if you care about argv[0] (say, you want to pull out the name
    of the script at runtime, instead of hard-coding it), then you need to be
    aware that you could be given an absolute path, a relative path, a bare
    script name, or the path of a softlink to the file you actually care
    about. Adding one more trivially simple case is not a large burden.

    People hardly ever care about argv[0]. At least, I don't think I ever
    have. But the OP does, and Python mangling argv[0] is causing him grief
    because it lies, claiming to have called the __main__.py of his package
    directly when in fact he called it with -m.


    > For imported
    > modules the path is available from __name__. For a script that is
    > executed rather than imported __name__ == "__main__" but the path is
    > accessible from sys.argv[0]. If you are one of those people who cares
    > about sys.argv[0] then this is probably the value that you wanted it to
    > contain.


    I'm wary about guessing what people "probably" want, and therefore lying
    about what they actually got. That's DWIM coding, and that almost always
    ends in tears.


    > If it were important for sys.argv to show how exactly the script was
    > located and executed, then why not also include the 'python3.3' command
    > line argument (the real argv[0])? sys.argv emulates the argv that e.g. a
    > C program would get. The real command line used is not exactly the same
    > since a Python script is not a directly executable binary, so Python
    > processes the argument list before passing it through.


    Also a good point. To some degree, we're constrained by backwards
    compatibility -- there's only so much change we can do without breaking
    code, and setting argv[0] to the python executable instead of the script
    is too big a change.

    In any case, you can get that information using sys.executable, or at
    least you can get the path of the actual Python binary, or you can use
    sys.version (or equivalent) to determine which version of Python you're
    using.

    This does mean that you can't play dirty hacks like some C binaries do,
    where they change their behaviour depending on whether you call them via
    one path or another path. vi does that. But that's hardly a big loss.

    Contrariwise, I don't believe that there is currently *any* way to
    distinguish between running a script with or without -m. That should be
    fixed.


    --
    Steven
    Steven D'Aprano, Jan 24, 2013
    #7
  8. On Thu, Jan 24, 2013 at 3:49 PM, Steven D'Aprano
    <> wrote:
    > Note however that there is an ambiguity between calling "python -mspam"
    > and calling a script literally named "-mspam". But that same ambiguity
    > exists in the shell, so I don't consider it a problem. You cannot call a
    > script named -mspam unless you use something like this "python ./-mspam".


    Another spanner for your works: "python -- -mspam" succeeds. That sets
    argv[0] to '-mspam'.

    > People hardly ever care about argv[0]. At least, I don't think I ever
    > have. But the OP does, and Python mangling argv[0] is causing him grief
    > because it lies, claiming to have called the __main__.py of his package
    > directly when in fact he called it with -m.


    Usually when I reference argv[0], $0, or any equivalent, it's for a
    usage display - eg:

    USAGE: $0 [opts] infile [outfile]
    --foo Fooify the file
    --bar Burn your computer to the ground

    So I don't particularly care about symlinks or relative paths (if it
    worked once, it'll probably work another time). But ambiguities may be
    an issue.

    ChrisA
    Chris Angelico, Jan 24, 2013
    #8
  9. On 24 January 2013 04:49, Steven D'Aprano
    <> wrote:
    [SNIP]
    >
    > Contrariwise, I don't believe that there is currently *any* way to
    > distinguish between running a script with or without -m. That should be
    > fixed.


    As I said earlier in the thread, the __package__ module global
    distinguishes the two cases:

    ~$ mkdir pkg
    ~$ touch pkg/__init__.py
    ~$ vim pkg/__main__.py
    ~$ cat pkg/__main__.py
    import sys
    if __package__ is None:
    cmdline = [sys.executable] + sys.argv
    else:
    cmdline = [sys.executable, '-m', __package__] + sys.argv[1:]
    print(cmdline)
    ~$ python pkg/__main__.py arg1 arg2
    ['q:\\tools\\Python27\\python.exe', 'pkg/__main__.py', 'arg1', 'arg2']
    ~$ python -m pkg arg1 arg2
    ['q:\\tools\\Python27\\python.exe', '-m', 'pkg', 'arg1', 'arg2']


    Oscar
    Oscar Benjamin, Jan 24, 2013
    #9
  10. Tim Golden

    Tim Golden Guest

    On 24/01/2013 10:06, Oscar Benjamin wrote:
    > On 24 January 2013 04:49, Steven D'Aprano
    > <> wrote:
    > [SNIP]
    >>
    >> Contrariwise, I don't believe that there is currently *any* way to
    >> distinguish between running a script with or without -m. That should be
    >> fixed.

    >
    > As I said earlier in the thread, the __package__ module global
    > distinguishes the two cases:
    >
    > ~$ mkdir pkg
    > ~$ touch pkg/__init__.py
    > ~$ vim pkg/__main__.py
    > ~$ cat pkg/__main__.py
    > import sys
    > if __package__ is None:
    > cmdline = [sys.executable] + sys.argv
    > else:
    > cmdline = [sys.executable, '-m', __package__] + sys.argv[1:]
    > print(cmdline)
    > ~$ python pkg/__main__.py arg1 arg2
    > ['q:\\tools\\Python27\\python.exe', 'pkg/__main__.py', 'arg1', 'arg2']
    > ~$ python -m pkg arg1 arg2
    > ['q:\\tools\\Python27\\python.exe', '-m', 'pkg', 'arg1', 'arg2']


    Reasonable (and thanks for the clear example), but it doesn't work
    if the package which is reconstructing the command line the package
    which was the target of the original command line. In my case,
    I'm making use of the cherrypy reloader, whose __package__ is
    cherrypy.process. But the command which invoked the program was
    python -m myapp.

    ie I'm issuing "python -m myapp". In myapp.__main__ I'm importing
    cherrypy, itself a package, and somewhere in cherrypy.whatever there is
    code which attempts to reconstruct the command line.

    TJG
    Tim Golden, Jan 24, 2013
    #10
  11. Tim Golden

    Tim Golden Guest

    On 24/01/2013 10:56, Tim Golden wrote:
    > if the package which is reconstructing the command line the package
    > which was the target of the original command line.



    Sorry:

    if the package which is reconstructing the command line *is not*
    the package which was the target of the original command line.

    TJG
    Tim Golden, Jan 24, 2013
    #11
  12. On 24 January 2013 10:56, Tim Golden <> wrote:
    > On 24/01/2013 10:06, Oscar Benjamin wrote:
    >> On 24 January 2013 04:49, Steven D'Aprano
    >> <> wrote:
    >> [SNIP]
    >>>
    >>> Contrariwise, I don't believe that there is currently *any* way to
    >>> distinguish between running a script with or without -m. That should be
    >>> fixed.

    >>
    >> As I said earlier in the thread, the __package__ module global
    >> distinguishes the two cases:
    >>
    >> ~$ mkdir pkg
    >> ~$ touch pkg/__init__.py
    >> ~$ vim pkg/__main__.py
    >> ~$ cat pkg/__main__.py
    >> import sys
    >> if __package__ is None:
    >> cmdline = [sys.executable] + sys.argv
    >> else:
    >> cmdline = [sys.executable, '-m', __package__] + sys.argv[1:]
    >> print(cmdline)
    >> ~$ python pkg/__main__.py arg1 arg2
    >> ['q:\\tools\\Python27\\python.exe', 'pkg/__main__.py', 'arg1', 'arg2']
    >> ~$ python -m pkg arg1 arg2
    >> ['q:\\tools\\Python27\\python.exe', '-m', 'pkg', 'arg1', 'arg2']

    >
    > Reasonable (and thanks for the clear example), but it doesn't work
    > if the package which is reconstructing the command line the package
    > which was the target of the original command line. In my case,
    > I'm making use of the cherrypy reloader, whose __package__ is
    > cherrypy.process. But the command which invoked the program was
    > python -m myapp.
    >
    > ie I'm issuing "python -m myapp". In myapp.__main__ I'm importing
    > cherrypy, itself a package, and somewhere in cherrypy.whatever there is
    > code which attempts to reconstruct the command line.


    Easy enough:

    ~$ mkdir pkg
    ~$ touch pkg/__init__.py
    ~$ vim pkg/__main__.py
    ~$ cat pkg/__main__.py
    import pkg.whatever
    ~$ vim pkg/whatever.py
    ~$ cat pkg/whatever.py
    import sys
    import pkg.__main__ as main
    cmdline = [sys.executable, '-m', main.__package__] + sys.argv[1:]
    print(cmdline)
    ~$ python -m pkg
    ['q:\\tools\\Python27\\python.exe', '-m', 'pkg']
    ~$ python -m pkg arg1 arg32
    ['q:\\tools\\Python27\\python.exe', '-m', 'pkg', 'arg1', 'arg32']

    I don't really understand what your spec is. Why do you need to
    inspect this information from sys.argv? Can you not just always use
    'python -m pkg' as your entry point?


    Oscar
    Oscar Benjamin, Jan 24, 2013
    #12
  13. Tim Golden

    Tim Golden Guest

    On 24/01/2013 11:30, Oscar Benjamin wrote:
    > I don't really understand what your spec is. Why do you need to
    > inspect this information from sys.argv? Can you not just always use
    > 'python -m pkg' as your entry point?


    Sorry about the confusion. I think my original point was simply one
    of surprise that sys.argv wouldn't essentially mirror the elements
    of the command line which I used to get there.
    The specifics of my use-case weren't really too important.

    For completeness, I'm talking about the cherrypy Autoreloader which
    attempts to restart (via execv) whatever process was responsible for
    loading it in the first place, via an identical or equivalent command
    line. The current (cherrypy) code simply joins sys.executable and
    sys.argv but this fails in the face of python -m as we have seen.

    The cherrypy package has no especial knowledge of the structure of the
    application which imported it and so must piece together the command
    line somehow. Clearly, I can take various approaches of the sort
    which you've outlined, or subclass the reloader, or fetch the original
    command line from the OS, etc. It's not that this is a showstopper,
    merely slightly surprising. (To me).

    TJG
    Tim Golden, Jan 24, 2013
    #13
  14. On 24 January 2013 13:45, Tim Golden <> wrote:
    > On 24/01/2013 11:30, Oscar Benjamin wrote:
    >> I don't really understand what your spec is. Why do you need to
    >> inspect this information from sys.argv? Can you not just always use
    >> 'python -m pkg' as your entry point?

    >

    [SNIP]
    >
    > For completeness, I'm talking about the cherrypy Autoreloader which
    > attempts to restart (via execv) whatever process was responsible for
    > loading it in the first place, via an identical or equivalent command
    > line. The current (cherrypy) code simply joins sys.executable and
    > sys.argv but this fails in the face of python -m as we have seen.
    >
    > The cherrypy package has no especial knowledge of the structure of the
    > application which imported it and so must piece together the command
    > line somehow. Clearly, I can take various approaches of the sort
    > which you've outlined, or subclass the reloader, or fetch the original
    > command line from the OS, etc. It's not that this is a showstopper,
    > merely slightly surprising. (To me).


    Ok I understand. Then I guess you want:

    import __main__
    pkg = __main__.__package__


    Oscar
    Oscar Benjamin, Jan 24, 2013
    #14
  15. Tim Golden

    Tim Golden Guest

    On 24/01/2013 15:28, Oscar Benjamin wrote:
    > On 24 January 2013 13:45, Tim Golden <> wrote:
    >> On 24/01/2013 11:30, Oscar Benjamin wrote:
    >>> I don't really understand what your spec is. Why do you need to
    >>> inspect this information from sys.argv? Can you not just always use
    >>> 'python -m pkg' as your entry point?

    >>

    > [SNIP]
    >>
    >> For completeness, I'm talking about the cherrypy Autoreloader which
    >> attempts to restart (via execv) whatever process was responsible for
    >> loading it in the first place, via an identical or equivalent command
    >> line. The current (cherrypy) code simply joins sys.executable and
    >> sys.argv but this fails in the face of python -m as we have seen.
    >>
    >> The cherrypy package has no especial knowledge of the structure of the
    >> application which imported it and so must piece together the command
    >> line somehow. Clearly, I can take various approaches of the sort
    >> which you've outlined, or subclass the reloader, or fetch the original
    >> command line from the OS, etc. It's not that this is a showstopper,
    >> merely slightly surprising. (To me).

    >
    > Ok I understand. Then I guess you want:
    >
    > import __main__
    > pkg = __main__.__package__


    Brilliant. Never thought of importing __main__. Thanks.

    For the benefit of anyone still watching, the code (which has to be
    compatible back to 2.3) looks something like this:

    <code>
    import __main__

    # [.. .snip ...]

    try:
    is_package = bool(__main__.__package__)
    except NameError:
    is_package = False
    if is_package:
    args = [sys.executable, '-m', __main__.__package__] + sys.argv[1:]
    else:
    args = [sys.executable] + sys.argv

    os.chdir(_startup_cwd) # avoids relative/absolute issues
    os.execv(args[0], args)

    </code>

    I don't pretend it's foolproot, but it certainly works for my particular
    case. Nor have I considered it against all the cases identified in PEP
    432: http://www.python.org/dev/peps/pep-0432/#configuring-sys-argv

    TJG
    Tim Golden, Jan 24, 2013
    #15
  16. On 24 January 2013 15:51, Tim Golden <> wrote:
    > On 24/01/2013 15:28, Oscar Benjamin wrote:
    >> On 24 January 2013 13:45, Tim Golden <> wrote:
    >>> On 24/01/2013 11:30, Oscar Benjamin wrote:
    >>>> I don't really understand what your spec is. Why do you need to
    >>>> inspect this information from sys.argv? Can you not just always use
    >>>> 'python -m pkg' as your entry point?
    >>>

    >> [SNIP]
    >>>
    >>> For completeness, I'm talking about the cherrypy Autoreloader which
    >>> attempts to restart (via execv) whatever process was responsible for
    >>> loading it in the first place, via an identical or equivalent command
    >>> line. The current (cherrypy) code simply joins sys.executable and
    >>> sys.argv but this fails in the face of python -m as we have seen.
    >>>
    >>> The cherrypy package has no especial knowledge of the structure of the
    >>> application which imported it and so must piece together the command
    >>> line somehow. Clearly, I can take various approaches of the sort
    >>> which you've outlined, or subclass the reloader, or fetch the original
    >>> command line from the OS, etc. It's not that this is a showstopper,
    >>> merely slightly surprising. (To me).

    >>
    >> Ok I understand. Then I guess you want:
    >>
    >> import __main__
    >> pkg = __main__.__package__

    >
    > Brilliant. Never thought of importing __main__. Thanks.
    >
    > For the benefit of anyone still watching, the code (which has to be
    > compatible back to 2.3) looks something like this:
    >
    > <code>
    > import __main__
    >
    > # [.. .snip ...]
    >
    > try:
    > is_package = bool(__main__.__package__)
    > except NameError:
    > is_package = False
    > if is_package:
    > args = [sys.executable, '-m', __main__.__package__] + sys.argv[1:]
    > else:
    > args = [sys.executable] + sys.argv
    >
    > os.chdir(_startup_cwd) # avoids relative/absolute issues
    > os.execv(args[0], args)
    >
    > </code>
    >
    > I don't pretend it's foolproot, but it certainly works for my particular
    > case. Nor have I considered it against all the cases identified in PEP
    > 432: http://www.python.org/dev/peps/pep-0432/#configuring-sys-argv


    Does it work if you use the -m option to run a module rather than a script?


    Oscar
    Oscar Benjamin, Jan 24, 2013
    #16
  17. On 24 January 2013 16:08, Oscar Benjamin <> wrote:
    > On 24 January 2013 15:51, Tim Golden <> wrote:
    >> On 24/01/2013 15:28, Oscar Benjamin wrote:
    >>> On 24 January 2013 13:45, Tim Golden <> wrote:
    >>>> On 24/01/2013 11:30, Oscar Benjamin wrote:
    >>>>> I don't really understand what your spec is. Why do you need to
    >>>>> inspect this information from sys.argv? Can you not just always use
    >>>>> 'python -m pkg' as your entry point?
    >>>>
    >>> [SNIP]
    >>>>
    >>>> For completeness, I'm talking about the cherrypy Autoreloader which
    >>>> attempts to restart (via execv) whatever process was responsible for
    >>>> loading it in the first place, via an identical or equivalent command
    >>>> line. The current (cherrypy) code simply joins sys.executable and
    >>>> sys.argv but this fails in the face of python -m as we have seen.
    >>>>
    >>>> The cherrypy package has no especial knowledge of the structure of the
    >>>> application which imported it and so must piece together the command
    >>>> line somehow. Clearly, I can take various approaches of the sort
    >>>> which you've outlined, or subclass the reloader, or fetch the original
    >>>> command line from the OS, etc. It's not that this is a showstopper,
    >>>> merely slightly surprising. (To me).
    >>>
    >>> Ok I understand. Then I guess you want:
    >>>
    >>> import __main__
    >>> pkg = __main__.__package__

    >>
    >> Brilliant. Never thought of importing __main__. Thanks.
    >>
    >> For the benefit of anyone still watching, the code (which has to be
    >> compatible back to 2.3) looks something like this:
    >>
    >> <code>
    >> import __main__
    >>
    >> # [.. .snip ...]
    >>
    >> try:
    >> is_package = bool(__main__.__package__)
    >> except NameError:
    >> is_package = False
    >> if is_package:
    >> args = [sys.executable, '-m', __main__.__package__] + sys.argv[1:]
    >> else:
    >> args = [sys.executable] + sys.argv
    >>
    >> os.chdir(_startup_cwd) # avoids relative/absolute issues
    >> os.execv(args[0], args)
    >>
    >> </code>
    >>
    >> I don't pretend it's foolproot, but it certainly works for my particular
    >> case. Nor have I considered it against all the cases identified in PEP
    >> 432: http://www.python.org/dev/peps/pep-0432/#configuring-sys-argv

    >
    > Does it work if you use the -m option to run a module rather than a script?


    Sorry that was written incorrectly. I meant to say: does it work when
    a module is directly on sys.path rather than as a submodule of a
    package? In this case __package__ is set to the empty string if run
    with -m or None if run with a direct path. So the check needs to be
    "__package__ is not None" rather than "bool(__package__)".


    Oscar
    Oscar Benjamin, Jan 24, 2013
    #17
  18. Tim Golden

    Tim Golden Guest

    On 24/01/2013 16:53, Oscar Benjamin wrote:
    >> Does it work if you use the -m option to run a module rather than a script?

    >
    > Sorry that was written incorrectly. I meant to say: does it work when
    > a module is directly on sys.path rather than as a submodule of a
    > package? In this case __package__ is set to the empty string if run
    > with -m or None if run with a direct path. So the check needs to be
    > "__package__ is not None" rather than "bool(__package__)".


    The answer is: it depends. Given the code I outlined earlier:

    A package-based module run via -m (python -m package.module) works
    as described (including the implicit __main__ module, my
    primary use-case).

    A module run from the filesystem (python c:\path\to\module.py) works
    by dropping through through to the not is_package logic branch.

    A module run via -m (python -m module) actually works by accident,
    because it too drops through to the not is_package branch and is
    rerun with its full filesystem path. This doesn't have the same
    problems as running a package from the filesystem because relative
    imports aren't an issue. I don't know if there are any other differences
    between python -mmodule and python c:\path\to\module.py.

    As you say, a more refined check could determine a blank __package__
    as opposed to a None __package__. But this code (cherrypy) must also
    cope with version of Python before 2.6 which didn't even have a
    __package__ attribute, muddying the waters that little bit further.

    TJG
    Tim Golden, Jan 24, 2013
    #18
  19. On 24 January 2013 17:13, Tim Golden <> wrote:
    > On 24/01/2013 16:53, Oscar Benjamin wrote:
    >>> Does it work if you use the -m option to run a module rather than a script?

    >>
    >> Sorry that was written incorrectly. I meant to say: does it work when
    >> a module is directly on sys.path rather than as a submodule of a
    >> package? In this case __package__ is set to the empty string if run
    >> with -m or None if run with a direct path. So the check needs to be
    >> "__package__ is not None" rather than "bool(__package__)".

    >
    > The answer is: it depends. Given the code I outlined earlier:
    >
    > A package-based module run via -m (python -m package.module) works
    > as described (including the implicit __main__ module, my
    > primary use-case).


    Does it work in the "python -m package.module" case? It looks to me as
    if the code below will run "python -m package" instead. I think you
    need to split the module name out of sys.argv[0] and put that into the
    command line as well.

    > import __main__
    >
    > # [.. .snip ...]
    >
    > try:
    > is_package = bool(__main__.__package__)
    > except NameError:
    > is_package = False
    > if is_package:
    > args = [sys.executable, '-m', __main__.__package__] + sys.argv[1:]
    > else:
    > args = [sys.executable] + sys.argv
    >
    > os.chdir(_startup_cwd) # avoids relative/absolute issues
    > os.execv(args[0], args)


    I believe "python -m package" and "python -m package.__main__" are
    equivalent so it doesn't matter in that case.

    >
    > A module run from the filesystem (python c:\path\to\module.py) works
    > by dropping through through to the not is_package logic branch.
    >
    > A module run via -m (python -m module) actually works by accident,
    > because it too drops through to the not is_package branch and is
    > rerun with its full filesystem path. This doesn't have the same
    > problems as running a package from the filesystem because relative
    > imports aren't an issue. I don't know if there are any other differences
    > between python -mmodule and python c:\path\to\module.py.


    There is a (probably pathological) case in which running the script
    via a file path and running it via -m are not equivalent. "python
    dir/script.py" places "dir" at the top of sys.path. "python -m script"
    only works if "dir" is in sys.path but it needn't be at the top. This
    means that the order of sys.path is changed and import conflicts may
    be resolved differently in the two cases:

    ~$ cat ~/.local/lib/python2.7/site-packages/script.py
    print("Running script.py")
    import optparse
    ~$ cat ~/.local/lib/python2.7/site-packages/optparse.py
    raise ImportError('Wrong optparse!')
    ~$ python ~/.local/lib/python2.7/site-packages/script.py
    Running script.py
    Traceback (most recent call last):
    File "/home/oscar/.local/lib/python2.7/site-packages/script.py",
    line 2, in <module>
    import optparse
    File "/home/oscar/.local/lib/python2.7/site-packages/optparse.py",
    line 1, in <module>
    raise ImportError('Wrong optparse!')
    ImportError: Wrong optparse!
    ~$ python -m script
    Running script.py
    ~$ python ~/.local/lib/python2.7/site-packages/script.py
    Running script.py
    Traceback (most recent call last):
    File "/home/oscar/.local/lib/python2.7/site-packages/script.py",
    line 2, in <module>
    import optparse
    File "/home/oscar/.local/lib/python2.7/site-packages/optparse.py",
    line 1, in <module>
    raise ImportError('Wrong optparse!')
    ImportError: Wrong optparse!
    ~$ python -m script
    Running script.py

    >
    > As you say, a more refined check could determine a blank __package__
    > as opposed to a None __package__. But this code (cherrypy) must also
    > cope with version of Python before 2.6 which didn't even have a
    > __package__ attribute, muddying the waters that little bit further.


    Then you won't be able to use this method to get the -m switch to work
    on Python < 2.6. If it's ok to just never try the -m switch on those
    versions then it's as simple as doing:
    pkg = getattr(__main__, '__package__', None)


    Oscar
    Oscar Benjamin, Jan 24, 2013
    #19
  20. Tim Golden

    Tim Golden Guest

    On 24/01/2013 20:01, Oscar Benjamin wrote:
    > On 24 January 2013 17:13, Tim Golden <> wrote:
    >> A package-based module run via -m (python -m package.module) works
    >> as described (including the implicit __main__ module, my
    >> primary use-case).

    >
    > Does it work in the "python -m package.module" case? It looks to me as
    > if the code below will run "python -m package" instead. I think you
    > need to split the module name out of sys.argv[0] and put that into the
    > command line as well.


    Good catch. Unless I am, once again, mistaken, I can't see any way
    of recreating a dotted module path which is running as a main module
    without the somewhat fragile expedient of walking back up the
    filepath and hoping that nothing strange has happened to the imports
    on the way down? (pywin32: I'm looking at you!)

    And we haven't even touched on zipped archives or other invented
    loaders (pulling modules from a database or direct from the internet).

    I think, though, that the point which with I started this thread is
    still valid: that to reconstruct the command line with which Python
    was started, you need an increasingly awkward set of conditions and
    checks. As opposed to asking the interpreter: please give me the command
    line which started you.

    How common that particular requirement actually is, I couldn't say.
    But the two cases which clearly could use it are: the reloader
    setup we've been discussing here; and the apply-new-version-and-restart
    which I'm hoping to use in another setup at work.

    Thanks for all the help and input on this, Oscar

    TJG
    Tim Golden, Jan 24, 2013
    #20
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. Samuel R. Neff
    Replies:
    2
    Views:
    583
    bradley
    Jun 10, 2005
  2. Replies:
    1
    Views:
    165
    Ken Bloom
    May 28, 2007
  3. Tim Golden

    Retrieving the full command line

    Tim Golden, Jan 22, 2013, in forum: Python
    Replies:
    0
    Views:
    92
    Tim Golden
    Jan 22, 2013
  4. Terry Reedy

    Re: Retrieving the full command line

    Terry Reedy, Jan 22, 2013, in forum: Python
    Replies:
    0
    Views:
    110
    Terry Reedy
    Jan 22, 2013
  5. Oscar Benjamin

    Re: Retrieving the full command line

    Oscar Benjamin, Jan 23, 2013, in forum: Python
    Replies:
    0
    Views:
    106
    Oscar Benjamin
    Jan 23, 2013
Loading...

Share This Page