Python 3 minor irritation

[David Monaghan]

I have a small program which reads files from the directory in which it
resides. It's written in Python 3 and when run through IDLE or PythonWin
works fine. If I double-click the file, it works fine in Python 2.6, but in
3 it fails because it looks for the files to load in the Python31 folder,
not the one the script is in.

It's not a big deal, but browsing around I haven't found why the behaviour
has been changed or any comment about it (That might be my poor search
technique, I suppose).

The program fails at:

try:
tutdoc = minidom.parse(".//Myfile.xml")
except IOError:
<snip>

DaveM

 

[Alf P. Steinbach]

* David Monaghan:

I have a small program which reads files from the directory in which it
resides. It's written in Python 3 and when run through IDLE or PythonWin
works fine. If I double-click the file, it works fine in Python 2.6, but in
3 it fails because it looks for the files to load in the Python31 folder,
not the one the script is in.

It's not a big deal, but browsing around I haven't found why the behaviour
has been changed or any comment about it (That might be my poor search
technique, I suppose).

The program fails at:

try:
tutdoc = minidom.parse(".//Myfile.xml")
except IOError:
<snip>

The "//" is wrong, but should not cause the behavior that you describe.

Try to post a complete smallest possible program that exhibits the problem.

Possibly, in creating that example you'll also find what's cause the problem.


Cheers & hth.,

- Alf

 

[Benjamin Kaplan]

* David Monaghan:

The "//" is wrong, but should not cause the behavior that you describe.

Try to post a complete smallest possible program that exhibits the problem.

Possibly, in creating that example you'll also find what's cause the
problem.


Cheers & hth.,

- Alf

That is the smallest example the exhibits the problem. It's not an
issue with the Python code, it's an issue with how Windows is running
it. I don't know enough about the way Windows Explorer runs files, but
it seems to be doing the equivalent of
cd C:\Python31
python31.exe C:\full\path\to\script\foo.py

instead of
cd C:\full\path\path\to\script
C:\Python31\python.exe foo.py
which is David expected. This throws off the relative filepath.

The easiest way to solve this permanently, by the way, is to not use
relative paths. All it takes is one script to call os.chdir and the
script breaks. You can use __file__ and the os.path module to figure
out exactly where you are in an OS-agnostic way.

import os.path
#get the absolute path to the current script
abs_path = os.path.abspath(__file__)

# get the full path to the directory of the script
directory = os.path.dirname(abs_path)

#get the full path to your file
my_file = os.path.join(directory, "MyFile.xml")

 

[Alf P. Steinbach]

* Benjamin Kaplan:

That is the smallest example the exhibits the problem.

No, it doesn't seem to exhibit the problem at all.

It's not an
issue with the Python code, it's an issue with how Windows is running
it. I don't know enough about the way Windows Explorer runs files, but
it seems to be doing the equivalent of
cd C:\Python31
python31.exe C:\full\path\to\script\foo.py

instead of
cd C:\full\path\path\to\script
C:\Python31\python.exe foo.py
which is David expected. This throws off the relative filepath.

No, this is not what happens.

What happens is that when you double-click the script, then __file__ is set to
the absolute path instead of just the filename, at least on my machine (XP);
whether the full path or just the filename is passed on the OS level depends,
however, on the Windows version.

The current directory is always (initially) correct, as the script's directory.

The easiest way to solve this permanently, by the way, is to not use
relative paths. All it takes is one script to call os.chdir and the
script breaks. You can use __file__ and the os.path module to figure
out exactly where you are in an OS-agnostic way.

import os.path
#get the absolute path to the current script
abs_path = os.path.abspath(__file__)

According to the docs: "On most platforms, this is equivalent to
normpath(join(os.getcwd(), path))."

Therefore, if getcwd() is not the script's directory, as you hypothesize above,
then most likely the result of this code is Not The Path You're Looking For.

However, since the current directory is in fact OK, the above is one way to get
a sort of canonical (a where-you-know-the-format) representation of __file__.

# get the full path to the directory of the script
directory = os.path.dirname(abs_path)

This is very much likely to yield the same result as calling os.getcwd(); see above.

#get the full path to your file
my_file = os.path.join(directory, "MyFile.xml")

Cheers & hth.,

- Alf

 

[Benjamin Kaplan]

* Benjamin Kaplan:

No, it doesn't seem to exhibit the problem at all.



No, this is not what happens.

What happens is that when you double-click the script, then __file__ is set
to the absolute path instead of just the filename, at least on my machine
(XP); whether the full path or just the filename is passed on the OS level
depends, however, on the Windows version.

The current directory is always (initially) correct, as the script's
directory.

Read my first paragraph again- it has absolutely nothing to do with
Python. It has to do with how the Windows is running Python 3 on the
OP's computer. The OP's description said that it was looking for the
file .\\myfile.xml in C:\Python31 which means it translated '.', the
current working directory, to be C:\Python31 and not the directory the
script is in.

According to the docs: "On most platforms, this is equivalent to
normpath(join(os.getcwd(), path))."

os.path.abspath will always work in this case (unless something
changes the current working directory before that statement runs)
because __file__ is given either as an absolute path or as relative to
the current working directory.

----- /Users/bkaplan/test/test.py
print(__file__)
import os.path
print(os.path.abspath(__file__))
------------------

$ cd /users/bkaplan
$ python3 test/test.py
python3 test/test.py
test/test.py
/Users/bkaplan/test/test.py

$cd /users/bkaplan/test
$ python3 test.py
test.py
/Users/bkaplan/test/test.py


If abspath is given an absolute path, it won't touch it

Output from double clicking on the file:
/Users/bkaplan/test/test.py
/Users/bkaplan/test/test.py

Therefore, if getcwd() is not the script's directory, as you hypothesize
above, then most likely the result of this code is Not The Path You're
Looking For.

Except that if the cwd is not the directory the script is running in,
__file__ should still point to it either through an absolute path
(highly likely since it's run through Windows Explorer) or some
(however convoluted) relative path.

However, since the current directory is in fact OK, the above is one way to
get a sort of canonical (a where-you-know-the-format) representation of
__file__.



This is very much likely to yield the same result as calling os.getcwd();
see above.

Not if there is more than one folder between __file__ and cwd. For
instance, if the script is in a package several directories down from
the main script.

 

[Alf P. Steinbach]

* Benjamin Kaplan:

Read my first paragraph again- it has absolutely nothing to do with
Python. It has to do with how the Windows is running Python 3 on the
OP's computer.

I'm not sure what you're arguing here.

But anyway, first, I haven't stated that "it" is (only) an issue with Python: on
the contrary, I explained how the program is run by Windows Explorer, where in
Windows XP the full path is passed by Windows Explorer, and how that results in
(can result in) a full path in __file__ when a script is run by double-clicking.

Second, as stated your hypothesis about current directory is Wrong(TM).

So the explanation of an incorrect result involves not only Windows Explorer,
the Windows version, and Python, but also the OP's code is involved.

Hence the request for a minimal, complete example -- which is nearly always a
good idea.

The OP's description said that it was looking for the
file .\\myfile.xml in C:\Python31 which means it translated '.', the
current working directory, to be C:\Python31 and not the directory the
script is in.

I fail to reproduce that behavior.

By the way, are you the OP (just with another name)?

If not, have you reproduced the described behavior? Then kindly post the code.

os.path.abspath will always work in this case (unless something
changes the current working directory before that statement runs)

Which change is what you surmised as a cause of the original problem.

Hello.

because __file__ is given either as an absolute path or as relative to
the current working directory.

No, that's incorrect.


<code py="3.1">
import os

print( "Current directory: [" + os.getcwd() + "]" )
print( "__file__ = " + __file__ )
os.chdir( "blah" )
print( "Current directory: [" + os.getcwd() + "]" )
print( "__file__ = " + __file__ )
</code>


<example>
C:\Documents and Settings\Alf\test> python curdir.py
Current directory: [C:\Documents and Settings\Alf\test]
__file__ = curdir.py
Current directory: [C:\Documents and Settings\Alf\test\blah]
__file__ = curdir.py <-- *Not* relative to cwd.

C:\Documents and Settings\Alf\test> _
</example>


[snip]

Except that if the cwd is not the directory the script is running in,
__file__ should still point to it either through an absolute path
(highly likely since it's run through Windows Explorer) or some
(however convoluted) relative path.

Don't know about "should", but if you're talking reality, no, that's incorrect;
see above.

[snip]


Cheers & hth.,

- Alf

 

[Alf P. Steinbach]

* Alf P. Steinbach:

* Benjamin Kaplan:

Which change is what you surmised as a cause of the original problem.

Hello.



No, that's incorrect.

Oh sorry, now I see what you mean. I read it too literally. You mean that at
script startup __file__ is a valid relative or absolute path to the script.

But anyways, Windows Explorer doesn't change the current directory to that of
the associated program, at least not in Windows XP.

Where there *is* a difference with double-clicking the script is in the path
that ends up as __file__.


[snip my silly code counter-example + even more silly comment]


Cheers,

- Alf

 

[Gabriel Genellina]

Oh sorry, now I see what you mean. I read it too literally. You mean
that at script startup __file__ is a valid relative or absolute path to
the script.

But anyways, Windows Explorer doesn't change the current directory to
that of the associated program, at least not in Windows XP.

But the associated program might change the current directory - that's not
the case with the default associations created by the Python installer,
but one should verify this.
To the OP: please create this small test script

import os
print("curdir=", os.getcwd())
print("__file__=", __file__)
input()

and post what you get when you double-click on it.

Also, from the command line, execute:

D:\temp>reg query HKCR\.py

! REG.EXE VERSION 3.0

HKEY_CLASSES_ROOT\.py
<Sin nombre> REG_SZ Python.File
Content Type REG_SZ text/plain

HKEY_CLASSES_ROOT\.py\PersistentHandler

D:\temp>reg query HKCR\Python.File\shell\open\command

! REG.EXE VERSION 3.0

HKEY_CLASSES_ROOT\Python.File\shell\open\command
<Sin nombre> REG_SZ "D:\Apps\Python26\python.exe" "%1" %*


The first command says that files ending with .py are of type
"Python.File", the second one says which command is executed to open a
"Python.File" file.

 

[Gib Bogle]

But the associated program might change the current directory - that's
not the case with the default associations created by the Python
installer, but one should verify this.
To the OP: please create this small test script

import os
print("curdir=", os.getcwd())
print("__file__=", __file__)
input()

and post what you get when you double-click on it.

Also, from the command line, execute:

D:\temp>reg query HKCR\.py

! REG.EXE VERSION 3.0

HKEY_CLASSES_ROOT\.py
<Sin nombre> REG_SZ Python.File
Content Type REG_SZ text/plain

HKEY_CLASSES_ROOT\.py\PersistentHandler

I'm interested in this, because I'm using Windows XP, and when I execute this
command I see the first part but not the second (with PersistentHandler). Is
this related to the fact that when I double-click on a .py file the command
window disappears after the execution is completed?

 

[Gabriel Genellina]

En Thu, 04 Feb 2010 17:50:29 -0300, Gib Bogle

Gabriel Genellina wrote:

I'm interested in this, because I'm using Windows XP, and when I execute
this command I see the first part but not the second (with
PersistentHandler).

Sorry, I should have removed that line. This is just my setup; a normal
Python install doesn't create that registry entry. It allows Desktop
Search (or Windows Search, or whatever it is called nowadays; the F3 key)
to search inside .py files (default behavior is to just ignore their
contents).
See http://support.microsoft.com/kb/309173

Is this related to the fact that when I double-click on a .py file the
command window disappears after the execution is completed?

(I bet the "Persistent" word confused you.) No, as you can see, it's
completely unrelated. AFAIK, there is no way (on XP and later at least) to
keep a console window open after the program exited. Three choices:

- Open a cmd window and execute the script there. You may drag&drop the
file over the window to avoid typing the full path (I think this last part
does not work on Vista nor Win7)

- Add a raw_input() [2.x] or input() [3.x] line at the end of the script

- Rename it with a '.cmd' extension and add this line at the very top:

@(C:\Python26\Python -x %~f0 %* || pause) && goto:EOF

(see this post by Duncan Booth last month:
http://comments.gmane.org/gmane.comp.python.general/650913 )

 

[Gib Bogle]

Sorry, I should have removed that line. This is just my setup; a normal
Python install doesn't create that registry entry. It allows Desktop
Search (or Windows Search, or whatever it is called nowadays; the F3
key) to search inside .py files (default behavior is to just ignore
their contents).
See http://support.microsoft.com/kb/309173

Is this related to the fact that when I double-click on a .py file the
command window disappears after the execution is completed?

(I bet the "Persistent" word confused you.) No, as you can see, it's
completely unrelated. AFAIK, there is no way (on XP and later at least)
to keep a console window open after the program exited. Three choices:

- Open a cmd window and execute the script there. You may drag&drop the
file over the window to avoid typing the full path (I think this last
part does not work on Vista nor Win7)

- Add a raw_input() [2.x] or input() [3.x] line at the end of the script

- Rename it with a '.cmd' extension and add this line at the very top:

@(C:\Python26\Python -x %~f0 %* || pause) && goto:EOF

(see this post by Duncan Booth last month:
http://comments.gmane.org/gmane.comp.python.general/650913 )

Thanks Gabriel. I didn't know about the drag&drop capability (learn something
every day, forget two things). BTW input() works for me in 2.5.
Cheers
Gib

 

[David Monaghan]

I have a small program which reads files from the directory in which it
resides. It's written in Python 3 and when run through IDLE or PythonWin
works fine. If I double-click the file, it works fine in Python 2.6, but in
3 it fails because it looks for the files to load in the Python31 folder,
not the one the script is in.

It's not a big deal, but browsing around I haven't found why the behaviour
has been changed or any comment about it (That might be my poor search
technique, I suppose).

The program fails at:

try:
tutdoc = minidom.parse(".//Myfile.xml")
except IOError:
<snip>

I very much appreciate all the help offered on this, but feel a bit of an
idiot now as I can't reproduce the behaviour (it had happened on two
separate machines).

What I am still getting is a similar problem on my work computer with the
program on a network hard drive. Not always - it'll run on repeated
attempts, then fail for a few, then work again. When it failed I ran the
script as suggested:

import os
print("curdir=", os.getcwd())
print("__file__=", __file__)
input()

and got the response:

curdir= H:\
__file__= H:\FRCR\FRCR2010\Course documents\FRCR2009\Script1.py

so it's 'sticking' at H:

For interest, I ran the script from IDLE, too, and PythonWin, on three
separate computers (2 Python3, 1 Python2.6)

With that I get a NameError for __file__

curdir= H:\FRCR\FRCR2010\Course documents\FRCR2009
Traceback (most recent call last):
File "H:\FRCR\FRCR2010\Course documents\FRCR2009\Script1.py", line 3,
in <module>
print("__file__=", __file__)
NameError: name '__file__' is not defined

What have I done wrong?

DaveM