# Python 3 minor irritation

Discussion in 'Python' started by David Monaghan, Feb 4, 2010.

1. ### David MonaghanGuest

I have a small program which reads files from the directory in which it
resides. It's written in Python 3 and when run through IDLE or PythonWin
works fine. If I double-click the file, it works fine in Python 2.6, but in
3 it fails because it looks for the files to load in the Python31 folder,
not the one the script is in.

It's not a big deal, but browsing around I haven't found why the behaviour
has been changed or any comment about it (That might be my poor search
technique, I suppose).

The program fails at:

try:
tutdoc = minidom.parse(".//Myfile.xml")
except IOError:
<snip>

DaveM

David Monaghan, Feb 4, 2010

2. ### Alf P. SteinbachGuest

* David Monaghan:
> I have a small program which reads files from the directory in which it
> resides. It's written in Python 3 and when run through IDLE or PythonWin
> works fine. If I double-click the file, it works fine in Python 2.6, but in
> 3 it fails because it looks for the files to load in the Python31 folder,
> not the one the script is in.
>
> It's not a big deal, but browsing around I haven't found why the behaviour
> has been changed or any comment about it (That might be my poor search
> technique, I suppose).
>
> The program fails at:
>
> try:
> tutdoc = minidom.parse(".//Myfile.xml")
> except IOError:
> <snip>

The "//" is wrong, but should not cause the behavior that you describe.

Try to post a complete smallest possible program that exhibits the problem.

Possibly, in creating that example you'll also find what's cause the problem.

Cheers & hth.,

- Alf

Alf P. Steinbach, Feb 4, 2010

3. ### Benjamin KaplanGuest

On Wed, Feb 3, 2010 at 9:56 PM, Alf P. Steinbach <> wrote:
> * David Monaghan:
>>
>> I have a small program which reads files from the directory in which it
>> resides. It's written in Python 3 and when run through IDLE or PythonWin
>> works fine. If I double-click the file, it works fine in Python 2.6, but
>> in
>> 3 it fails because it looks for the files to load in the Python31 folder,
>> not the one the script is in.
>>
>> It's not a big deal, but browsing around I haven't found why the behaviour
>> has been changed or any comment about it (That might be my poor search
>> technique, I suppose).
>>
>> The program fails at:
>>
>>    try:
>>        tutdoc = minidom.parse(".//Myfile.xml")
>>    except IOError:
>>        <snip>

>
> The "//" is wrong, but should not cause the behavior that you describe.
>
> Try to post a complete smallest possible program that exhibits the problem.
>
> Possibly, in creating that example you'll also find what's cause the
> problem.
>
>
> Cheers & hth.,
>
> - Alf

That is the smallest example the exhibits the problem. It's not an
issue with the Python code, it's an issue with how Windows is running
it. I don't know enough about the way Windows Explorer runs files, but
it seems to be doing the equivalent of
cd C:\Python31
python31.exe C:\full\path\to\script\foo.py

cd C:\full\path\path\to\script
C:\Python31\python.exe foo.py
which is David expected. This throws off the relative filepath.

The easiest way to solve this permanently, by the way, is to not use
relative paths. All it takes is one script to call os.chdir and the
script breaks. You can use __file__ and the os.path module to figure
out exactly where you are in an OS-agnostic way.

import os.path
#get the absolute path to the current script
abs_path = os.path.abspath(__file__)

# get the full path to the directory of the script
directory = os.path.dirname(abs_path)

#get the full path to your file
my_file = os.path.join(directory, "MyFile.xml")

Benjamin Kaplan, Feb 4, 2010
4. ### Alf P. SteinbachGuest

* Benjamin Kaplan:
> On Wed, Feb 3, 2010 at 9:56 PM, Alf P. Steinbach <> wrote:
>> * David Monaghan:
>>> I have a small program which reads files from the directory in which it
>>> resides. It's written in Python 3 and when run through IDLE or PythonWin
>>> works fine. If I double-click the file, it works fine in Python 2.6, but
>>> in
>>> 3 it fails because it looks for the files to load in the Python31 folder,
>>> not the one the script is in.
>>>
>>> It's not a big deal, but browsing around I haven't found why the behaviour
>>> has been changed or any comment about it (That might be my poor search
>>> technique, I suppose).
>>>
>>> The program fails at:
>>>
>>> try:
>>> tutdoc = minidom.parse(".//Myfile.xml")
>>> except IOError:
>>> <snip>

>> The "//" is wrong, but should not cause the behavior that you describe.
>>
>> Try to post a complete smallest possible program that exhibits the problem.
>>
>> Possibly, in creating that example you'll also find what's cause the
>> problem.
>>
>>
>> Cheers & hth.,
>>
>> - Alf

>
> That is the smallest example the exhibits the problem.

No, it doesn't seem to exhibit the problem at all.

> It's not an
> issue with the Python code, it's an issue with how Windows is running
> it. I don't know enough about the way Windows Explorer runs files, but
> it seems to be doing the equivalent of
> cd C:\Python31
> python31.exe C:\full\path\to\script\foo.py
>
> cd C:\full\path\path\to\script
> C:\Python31\python.exe foo.py
> which is David expected. This throws off the relative filepath.

No, this is not what happens.

What happens is that when you double-click the script, then __file__ is set to
the absolute path instead of just the filename, at least on my machine (XP);
whether the full path or just the filename is passed on the OS level depends,
however, on the Windows version.

The current directory is always (initially) correct, as the script's directory.

> The easiest way to solve this permanently, by the way, is to not use
> relative paths. All it takes is one script to call os.chdir and the
> script breaks. You can use __file__ and the os.path module to figure
> out exactly where you are in an OS-agnostic way.
>
> import os.path
> #get the absolute path to the current script
> abs_path = os.path.abspath(__file__)

According to the docs: "On most platforms, this is equivalent to
normpath(join(os.getcwd(), path))."

Therefore, if getcwd() is not the script's directory, as you hypothesize above,
then most likely the result of this code is Not The Path You're Looking For.

However, since the current directory is in fact OK, the above is one way to get
a sort of canonical (a where-you-know-the-format) representation of __file__.

> # get the full path to the directory of the script
> directory = os.path.dirname(abs_path)

This is very much likely to yield the same result as calling os.getcwd(); see above.

> #get the full path to your file
> my_file = os.path.join(directory, "MyFile.xml")

Cheers & hth.,

- Alf

Alf P. Steinbach, Feb 4, 2010
5. ### Benjamin KaplanGuest

On Wed, Feb 3, 2010 at 11:18 PM, Alf P. Steinbach <> wrote:
> * Benjamin Kaplan:
>>
>> On Wed, Feb 3, 2010 at 9:56 PM, Alf P. Steinbach <> wrote:
>>>
>>> * David Monaghan:
>>>>
>>>> I have a small program which reads files from the directory in which it
>>>> resides. It's written in Python 3 and when run through IDLE or PythonWin
>>>> works fine. If I double-click the file, it works fine in Python 2.6, but
>>>> in
>>>> 3 it fails because it looks for the files to load in the Python31
>>>> folder,
>>>> not the one the script is in.
>>>>
>>>> It's not a big deal, but browsing around I haven't found why the
>>>> behaviour
>>>> has been changed or any comment about it (That might be my poor search
>>>> technique, I suppose).
>>>>
>>>> The program fails at:
>>>>
>>>>   try:
>>>>       tutdoc = minidom.parse(".//Myfile.xml")
>>>>   except IOError:
>>>>       <snip>
>>>
>>> The "//" is wrong, but should not cause the behavior that you describe.
>>>
>>> Try to post a complete smallest possible program that exhibits the
>>> problem.
>>>
>>> Possibly, in creating that example you'll also find what's cause the
>>> problem.
>>>
>>>
>>> Cheers & hth.,
>>>
>>> - Alf

>>
>> That is the smallest example the exhibits the problem.

>
> No, it doesn't seem to exhibit the problem at all.
>
>
>> It's not an
>> issue with the Python code, it's an issue with how Windows is running
>> it. I don't know enough about the way Windows Explorer runs files, but
>> it seems to be doing the equivalent of
>> cd C:\Python31
>> python31.exe C:\full\path\to\script\foo.py
>>
>> cd C:\full\path\path\to\script
>> C:\Python31\python.exe foo.py
>> which is David expected. This throws off the relative filepath.

>
> No, this is not what happens.
>
> What happens is that when you double-click the script, then __file__ is set
> to the absolute path instead of just the filename, at least on my machine
> (XP); whether the full path or just the filename is passed on the OS level
> depends, however, on the Windows version.
>
> The current directory is always (initially) correct, as the script's
> directory.
>

Read my first paragraph again- it has absolutely nothing to do with
Python. It has to do with how the Windows is running Python 3 on the
OP's computer. The OP's description said that it was looking for the
file .\\myfile.xml in C:\Python31 which means it translated '.', the
current working directory, to be C:\Python31 and not the directory the
script is in.

>
>> The easiest way to solve this permanently, by the way, is to not use
>> relative paths. All it takes is one script to call os.chdir and the
>> script breaks. You can use __file__ and the os.path module to figure
>> out exactly where you are in an OS-agnostic way.
>>
>> import os.path
>> #get the absolute path to the current script
>> abs_path = os.path.abspath(__file__)

>
> According to the docs: "On most platforms, this is equivalent to
> normpath(join(os.getcwd(), path))."
>

os.path.abspath will always work in this case (unless something
changes the current working directory before that statement runs)
because __file__ is given either as an absolute path or as relative to
the current working directory.

----- /Users/bkaplan/test/test.py
print(__file__)
import os.path
print(os.path.abspath(__file__))
------------------

$cd /users/bkaplan$ python3 test/test.py
python3 test/test.py
test/test.py
/Users/bkaplan/test/test.py

$cd /users/bkaplan/test$ python3 test.py
test.py
/Users/bkaplan/test/test.py

If abspath is given an absolute path, it won't touch it

Output from double clicking on the file:
/Users/bkaplan/test/test.py
/Users/bkaplan/test/test.py

> Therefore, if getcwd() is not the script's directory, as you hypothesize
> above, then most likely the result of this code is Not The Path You're
> Looking For.
>

Except that if the cwd is not the directory the script is running in,
__file__ should still point to it either through an absolute path
(highly likely since it's run through Windows Explorer) or some
(however convoluted) relative path.

> However, since the current directory is in fact OK, the above is one way to
> get a sort of canonical (a where-you-know-the-format) representation of
> __file__.
>
>
>>  # get the full path to the directory of the script
>> directory = os.path.dirname(abs_path)

>
> This is very much likely to yield the same result as calling os.getcwd();
> see above.
>

Not if there is more than one folder between __file__ and cwd. For
instance, if the script is in a package several directories down from
the main script.

>
>> #get the full path to your file
>> my_file = os.path.join(directory, "MyFile.xml")

>
>
> Cheers & hth.,
>
> - Alf
> --
> http://mail.python.org/mailman/listinfo/python-list
>

Benjamin Kaplan, Feb 4, 2010
6. ### Alf P. SteinbachGuest

* Benjamin Kaplan:
> On Wed, Feb 3, 2010 at 11:18 PM, Alf P. Steinbach <> wrote:
>> * Benjamin Kaplan:
>>> On Wed, Feb 3, 2010 at 9:56 PM, Alf P. Steinbach <> wrote:
>>>> * David Monaghan:
>>>>> I have a small program which reads files from the directory in which it
>>>>> resides. It's written in Python 3 and when run through IDLE or PythonWin
>>>>> works fine. If I double-click the file, it works fine in Python 2.6, but
>>>>> in
>>>>> 3 it fails because it looks for the files to load in the Python31
>>>>> folder,
>>>>> not the one the script is in.
>>>>>
>>>>> It's not a big deal, but browsing around I haven't found why the
>>>>> behaviour
>>>>> has been changed or any comment about it (That might be my poor search
>>>>> technique, I suppose).
>>>>>
>>>>> The program fails at:
>>>>>
>>>>> try:
>>>>> tutdoc = minidom.parse(".//Myfile.xml")
>>>>> except IOError:
>>>>> <snip>
>>>> The "//" is wrong, but should not cause the behavior that you describe.
>>>>
>>>> Try to post a complete smallest possible program that exhibits the
>>>> problem.
>>>>
>>>> Possibly, in creating that example you'll also find what's cause the
>>>> problem.
>>>>
>>>>
>>>> Cheers & hth.,
>>>>
>>>> - Alf
>>> That is the smallest example the exhibits the problem.

>> No, it doesn't seem to exhibit the problem at all.
>>
>>
>>> It's not an
>>> issue with the Python code, it's an issue with how Windows is running
>>> it. I don't know enough about the way Windows Explorer runs files, but
>>> it seems to be doing the equivalent of
>>> cd C:\Python31
>>> python31.exe C:\full\path\to\script\foo.py
>>>
>>> cd C:\full\path\path\to\script
>>> C:\Python31\python.exe foo.py
>>> which is David expected. This throws off the relative filepath.

>> No, this is not what happens.
>>
>> What happens is that when you double-click the script, then __file__ is set
>> to the absolute path instead of just the filename, at least on my machine
>> (XP); whether the full path or just the filename is passed on the OS level
>> depends, however, on the Windows version.
>>
>> The current directory is always (initially) correct, as the script's
>> directory.
>>

>
> Read my first paragraph again- it has absolutely nothing to do with
> Python. It has to do with how the Windows is running Python 3 on the
> OP's computer.

I'm not sure what you're arguing here.

But anyway, first, I haven't stated that "it" is (only) an issue with Python: on
the contrary, I explained how the program is run by Windows Explorer, where in
Windows XP the full path is passed by Windows Explorer, and how that results in
(can result in) a full path in __file__ when a script is run by double-clicking.

Second, as stated your hypothesis about current directory is Wrong(TM).

So the explanation of an incorrect result involves not only Windows Explorer,
the Windows version, and Python, but also the OP's code is involved.

Hence the request for a minimal, complete example -- which is nearly always a
good idea.

> The OP's description said that it was looking for the
> file .\\myfile.xml in C:\Python31 which means it translated '.', the
> current working directory, to be C:\Python31 and not the directory the
> script is in.

I fail to reproduce that behavior.

By the way, are you the OP (just with another name)?

If not, have you reproduced the described behavior? Then kindly post the code.

>>> The easiest way to solve this permanently, by the way, is to not use
>>> relative paths. All it takes is one script to call os.chdir and the
>>> script breaks. You can use __file__ and the os.path module to figure
>>> out exactly where you are in an OS-agnostic way.
>>>
>>> import os.path
>>> #get the absolute path to the current script
>>> abs_path = os.path.abspath(__file__)

>> According to the docs: "On most platforms, this is equivalent to
>> normpath(join(os.getcwd(), path))."
>>

>
> os.path.abspath will always work in this case (unless something
> changes the current working directory before that statement runs)

Which change is what you surmised as a cause of the original problem.

Hello.

> because __file__ is given either as an absolute path or as relative to
> the current working directory.

No, that's incorrect.

<code py="3.1">
import os

print( "Current directory: [" + os.getcwd() + "]" )
print( "__file__ = " + __file__ )
os.chdir( "blah" )
print( "Current directory: [" + os.getcwd() + "]" )
print( "__file__ = " + __file__ )
</code>

<example>
C:\Documents and Settings\Alf\test> python curdir.py
Current directory: [C:\Documents and Settings\Alf\test]
__file__ = curdir.py
Current directory: [C:\Documents and Settings\Alf\test\blah]
__file__ = curdir.py <-- *Not* relative to cwd.

C:\Documents and Settings\Alf\test> _
</example>

[snip]
> Except that if the cwd is not the directory the script is running in,
> __file__ should still point to it either through an absolute path
> (highly likely since it's run through Windows Explorer) or some
> (however convoluted) relative path.

Don't know about "should", but if you're talking reality, no, that's incorrect;
see above.

[snip]

Cheers & hth.,

- Alf

Alf P. Steinbach, Feb 4, 2010
7. ### Alf P. SteinbachGuest

* Alf P. Steinbach:
> * Benjamin Kaplan:
>> On Wed, Feb 3, 2010 at 11:18 PM, Alf P. Steinbach <> wrote:
>>> * Benjamin Kaplan:
>>>> The easiest way to solve this permanently, by the way, is to not use
>>>> relative paths. All it takes is one script to call os.chdir and the
>>>> script breaks. You can use __file__ and the os.path module to figure
>>>> out exactly where you are in an OS-agnostic way.
>>>>
>>>> import os.path
>>>> #get the absolute path to the current script
>>>> abs_path = os.path.abspath(__file__)
>>> According to the docs: "On most platforms, this is equivalent to
>>> normpath(join(os.getcwd(), path))."
>>>

>>
>> os.path.abspath will always work in this case (unless something
>> changes the current working directory before that statement runs)

>
> Which change is what you surmised as a cause of the original problem.
>
> Hello.
>
>
>> because __file__ is given either as an absolute path or as relative to
>> the current working directory.

>
> No, that's incorrect.

Oh sorry, now I see what you mean. I read it too literally. You mean that at
script startup __file__ is a valid relative or absolute path to the script.

But anyways, Windows Explorer doesn't change the current directory to that of
the associated program, at least not in Windows XP.

Where there *is* a difference with double-clicking the script is in the path
that ends up as __file__.

[snip my silly code counter-example + even more silly comment]

Cheers,

- Alf

Alf P. Steinbach, Feb 4, 2010
8. ### Gabriel GenellinaGuest

En Thu, 04 Feb 2010 02:46:45 -0300, Alf P. Steinbach <>
escribió:

> Oh sorry, now I see what you mean. I read it too literally. You mean
> that at script startup __file__ is a valid relative or absolute path to
> the script.
>
> But anyways, Windows Explorer doesn't change the current directory to
> that of the associated program, at least not in Windows XP.

But the associated program might change the current directory - that's not
the case with the default associations created by the Python installer,
but one should verify this.
To the OP: please create this small test script

import os
print("curdir=", os.getcwd())
print("__file__=", __file__)
input()

and post what you get when you double-click on it.

Also, from the command line, execute:

D:\temp>reg query HKCR\.py

! REG.EXE VERSION 3.0

HKEY_CLASSES_ROOT\.py
<Sin nombre> REG_SZ Python.File
Content Type REG_SZ text/plain

HKEY_CLASSES_ROOT\.py\PersistentHandler

D:\temp>reg query HKCR\Python.File\shell\open\command

! REG.EXE VERSION 3.0

HKEY_CLASSES_ROOT\Python.File\shell\open\command
<Sin nombre> REG_SZ "D:\Apps\Python26\python.exe" "%1" %*

The first command says that files ending with .py are of type
"Python.File", the second one says which command is executed to open a
"Python.File" file.

--
Gabriel Genellina

Gabriel Genellina, Feb 4, 2010
9. ### Gib BogleGuest

Gabriel Genellina wrote:
> But the associated program might change the current directory - that's
> not the case with the default associations created by the Python
> installer, but one should verify this.
> To the OP: please create this small test script
>
> import os
> print("curdir=", os.getcwd())
> print("__file__=", __file__)
> input()
>
> and post what you get when you double-click on it.
>
> Also, from the command line, execute:
>
> D:\temp>reg query HKCR\.py
>
> ! REG.EXE VERSION 3.0
>
> HKEY_CLASSES_ROOT\.py
> <Sin nombre> REG_SZ Python.File
> Content Type REG_SZ text/plain
>
> HKEY_CLASSES_ROOT\.py\PersistentHandler

I'm interested in this, because I'm using Windows XP, and when I execute this
command I see the first part but not the second (with PersistentHandler). Is
this related to the fact that when I double-click on a .py file the command
window disappears after the execution is completed?

Gib Bogle, Feb 4, 2010
10. ### Gabriel GenellinaGuest

En Thu, 04 Feb 2010 17:50:29 -0300, Gib Bogle
<> escribió:
> Gabriel Genellina wrote:

>> Also, from the command line, execute:
>> D:\temp>reg query HKCR\.py
>> ! REG.EXE VERSION 3.0
>> HKEY_CLASSES_ROOT\.py
>> <Sin nombre> REG_SZ Python.File
>> Content Type REG_SZ text/plain
>> HKEY_CLASSES_ROOT\.py\PersistentHandler

>
> I'm interested in this, because I'm using Windows XP, and when I execute
> this command I see the first part but not the second (with
> PersistentHandler).

Sorry, I should have removed that line. This is just my setup; a normal
Python install doesn't create that registry entry. It allows Desktop
Search (or Windows Search, or whatever it is called nowadays; the F3 key)
to search inside .py files (default behavior is to just ignore their
contents).
See http://support.microsoft.com/kb/309173

> Is this related to the fact that when I double-click on a .py file the
> command window disappears after the execution is completed?

(I bet the "Persistent" word confused you.) No, as you can see, it's
completely unrelated. AFAIK, there is no way (on XP and later at least) to
keep a console window open after the program exited. Three choices:

- Open a cmd window and execute the script there. You may drag&drop the
file over the window to avoid typing the full path (I think this last part
does not work on Vista nor Win7)

- Add a raw_input() [2.x] or input() [3.x] line at the end of the script

- Rename it with a '.cmd' extension and add this line at the very top:

@(C:\Python26\Python -x %~f0 %* || pause) && goto:EOF

(see this post by Duncan Booth last month:

--
Gabriel Genellina

Gabriel Genellina, Feb 5, 2010
11. ### Gib BogleGuest

Gabriel Genellina wrote:

> Sorry, I should have removed that line. This is just my setup; a normal
> Python install doesn't create that registry entry. It allows Desktop
> Search (or Windows Search, or whatever it is called nowadays; the F3
> key) to search inside .py files (default behavior is to just ignore
> their contents).
> See http://support.microsoft.com/kb/309173
>
>> Is this related to the fact that when I double-click on a .py file the
>> command window disappears after the execution is completed?

>
> (I bet the "Persistent" word confused you.) No, as you can see, it's
> completely unrelated. AFAIK, there is no way (on XP and later at least)
> to keep a console window open after the program exited. Three choices:
>
> - Open a cmd window and execute the script there. You may drag&drop the
> file over the window to avoid typing the full path (I think this last
> part does not work on Vista nor Win7)
>
> - Add a raw_input() [2.x] or input() [3.x] line at the end of the script
>
> - Rename it with a '.cmd' extension and add this line at the very top:
>
> @(C:\Python26\Python -x %~f0 %* || pause) && goto:EOF
>
> (see this post by Duncan Booth last month:
>

Thanks Gabriel. I didn't know about the drag&drop capability (learn something
every day, forget two things). BTW input() works for me in 2.5.
Cheers
Gib

Gib Bogle, Feb 5, 2010
12. ### David MonaghanGuest

On Thu, 04 Feb 2010 00:39:01 +0000, David Monaghan
<> wrote:

>I have a small program which reads files from the directory in which it
>resides. It's written in Python 3 and when run through IDLE or PythonWin
>works fine. If I double-click the file, it works fine in Python 2.6, but in
>3 it fails because it looks for the files to load in the Python31 folder,
>not the one the script is in.
>
>It's not a big deal, but browsing around I haven't found why the behaviour
>has been changed or any comment about it (That might be my poor search
>technique, I suppose).
>
>The program fails at:
>
> try:
> tutdoc = minidom.parse(".//Myfile.xml")
> except IOError:
> <snip>

I very much appreciate all the help offered on this, but feel a bit of an
idiot now as I can't reproduce the behaviour (it had happened on two
separate machines).

What I am still getting is a similar problem on my work computer with the
program on a network hard drive. Not always - it'll run on repeated
attempts, then fail for a few, then work again. When it failed I ran the
script as suggested:

import os
print("curdir=", os.getcwd())
print("__file__=", __file__)
input()

and got the response:

curdir= H:\
__file__= H:\FRCR\FRCR2010\Course documents\FRCR2009\Script1.py

so it's 'sticking' at H:

For interest, I ran the script from IDLE, too, and PythonWin, on three
separate computers (2 Python3, 1 Python2.6)

With that I get a NameError for __file__

curdir= H:\FRCR\FRCR2010\Course documents\FRCR2009
Traceback (most recent call last):
File "H:\FRCR\FRCR2010\Course documents\FRCR2009\Script1.py", line 3,
in <module>
print("__file__=", __file__)
NameError: name '__file__' is not defined

What have I done wrong?

DaveM

David Monaghan, Feb 5, 2010