"env" parameter to "popen" won't accept Unicode on Windows - minorUnicode bug

J

John Nagle

I passed a dict for the "env" variable to Popen with Unicode strings
for the dictionary values.

Got:

File "D:\Python24\lib\subprocess.py", line 706, in _execute_child
TypeError: environment can only contain strings

It turns out that the strings in the "env" parameter have to be ASCII,
not Unicode, even though Windows fully supports Unicode in CreateProcess.

John Nagle
 
B

Bjoern Schliessmann

John said:
It turns out that the strings in the "env" parameter have to be
ASCII, not Unicode, even though Windows fully supports Unicode in
CreateProcess.

Are you sure it supports Unicode, not UTF8 or UTF16? Probably using
something like u"thestring".encode("utf16") will help.

Regards,


Björn
 
B

Benjamin

Are you sure it supports Unicode, not UTF8 or UTF16? Probably using
something like u"thestring".encode("utf16") will help. Otherwise: bugs.python.org

Regards,

Björn
 
B

Benjamin

I passed a dict for the "env" variable to Popen with Unicode strings
for the dictionary values.

Got:

File "D:\Python24\lib\subprocess.py", line 706, in _execute_child
TypeError: environment can only contain strings

It turns out that the strings in the "env" parameter have to be ASCII,
not Unicode, even though Windows fully supports Unicode in CreateProcess.
 
J

John Nagle

Benjamin said:
Otherwise: bugs.python.org

Whatever translation is necessary should be done in "popen", which
has cases for Windows and POSIX. "popen" is supposed to be cross-platform
to the extent possible. I think it's just something that didn't get fixed
when Unicode support went in.

John Nagle
 
D

Diez B. Roggisch

That's of course nonsense, they don't need to be ascii, they need to be
byte-strings in whatever encoding you like.

John's understanding of the differences between unicode and it's encodings
is a bit blurry, to say the least.
Whatever translation is necessary should be done in "popen", which
has cases for Windows and POSIX. "popen" is supposed to be cross-platform
to the extent possible. I think it's just something that didn't get fixed
when Unicode support went in.

Sure thing, python will just magically convert unicode to the encoding the
program YOU invoke will expect. Right after we introduced the

solve_my_problem()

built-in-function. Any other wishes?

If I write this simple program

------ test.py -------
import os
import sys

ENCODDINGS = ['utf-8', 'latin1']

os.env["MY_VARIABLE"].encode(ENCODINGS[int(sys.argv[1])])
------ test.py -------


how's python supposed to know that

suprocess.call("python test.py 0", env=dict(MY_VARIABLE=u'foo'))

needs to be UTF-8?

Diez
 
B

Brian Smith

Diez said:
Sure thing, python will just magically convert unicode to the
encoding the program YOU invoke will expect. Right after we
introduced the

solve_my_problem()

built-in-function. Any other wishes?

There's no reason to be rude.

Anyway, at least on Windows it makes perfect sense for people to expect
Unicode to be handled automatically. popen() knows that it is running on
Windows, and it knows what encoding Windows needs for its environment
(it's either UCS2 or UTF-16 for most Windows APIs). At least when it
receives a unicode string, it has enough information to apply the
conversion automatically, and doing so saves the caller from having to
figure out what exact encoding is to be used.

- Brian
 
D

Diez B. Roggisch

Brian said:
There's no reason to be rude.

If you'd know John, you'd know there is.
Anyway, at least on Windows it makes perfect sense for people to expect
Unicode to be handled automatically. popen() knows that it is running on
Windows, and it knows what encoding Windows needs for its environment
(it's either UCS2 or UTF-16 for most Windows APIs). At least when it
receives a unicode string, it has enough information to apply the
conversion automatically, and doing so saves the caller from having to
figure out what exact encoding is to be used.


For once, the distinction between windows and other platforms is debatable.
I admit that subprocess contains already quite a few platform specific
aspects, but it's purpose is to abstract these away as much as possible.

However, I'm not sure that just because there are wide-char windows apis
available automatically means that using UCS2/UTF-16 would succeed. A look
into the python sources (PC/_subprocess.c) reveals that someone already
thought about this, but it seems that just setting a
CREATE_UNICODE_ENVIRONMENT in the CreateProcess-function should have been
easy enough to do it if there weren't any troubles to expect.

Additionally, passing unicode to env would also imply that os.environ should
yield unicode as well. Not sure how much code _that_ breaks.

Diez
 
B

Bjoern Schliessmann

Brian said:
popen() knows that it is running on Windows, and it knows what
encoding Windows needs for its environment (it's either UCS2 or
UTF-16 for most Windows APIs). At least when it receives a unicode
string, it has enough information to apply the conversion
automatically, and doing so saves the caller from having to figure
out what exact encoding is to be used.

So you propose Python should employ a hidden automatism that
automagically guesses the right encoding? Why not leave it
explicitly/consistently and let the user decide? What will happen
if a future Windows changes its encoding? Will we need another
magic routine to tell it apart?

Regards,


Björn
 
J

John Nagle

Diez said:
That's of course nonsense, they don't need to be ascii, they need to be
byte-strings in whatever encoding you like.


John's understanding of the differences between unicode and it's encodings
is a bit blurry, to say the least.

Who's this guy?
I've been looking at the source code. There's "_PyPopenCreateProcess"
in "posixmodule.c". That one doesn't support passing an environment at
all; see the call to Windows CreateProcess. Is that the one that Popen uses?

Where is "win32process" in the source? It ought to be in Modules, but
it's not.

John Nagle
 
J

John Nagle

Diez said:
If you'd know John, you'd know there is.

?
For once, the distinction between windows and other platforms is debatable.
I admit that subprocess contains already quite a few platform specific
aspects, but it's purpose is to abstract these away as much as possible.

However, I'm not sure that just because there are wide-char windows apis
available automatically means that using UCS2/UTF-16 would succeed. A look
into the python sources (PC/_subprocess.c) reveals that someone already
thought about this, but it seems that just setting a
CREATE_UNICODE_ENVIRONMENT in the CreateProcess-function should have been
easy enough to do it if there weren't any troubles to expect.

The problem is that only the NT-derived Microsoft systems talk Unicode.
The DOS/Win16/Win9x family did not. But they did have CreateProcess.
So the current code will handle Win9x, but not Unicode.

When do we drop support for Win9x? It probably has to happen in
Python 3K, since that's Unicode-everywhere.

John Nagle
 
B

Bjoern Schliessmann

John said:
The problem is that only the NT-derived Microsoft systems
talk Unicode. The DOS/Win16/Win9x family did not. But they did
have CreateProcess. So the current code will handle Win9x, but not
Unicode.

Please explain, I don't understand. If you try using Windows system
functions in older Windows versions, u"mystring" will fail, too.
Those functions need byte strings, not Unicode string instances.
The latter have to be encoded to byte strings to pass them.

Regards,


Björn
 
D

Dennis Lee Bieber

Where is "win32process" in the source? It ought to be in Modules, but
it's not.
Probably because it's part of the PythonWin (or whatever the name is
this month) extension library, not part of base Python -- and is mostly
a lot of compiled PYD files...
--
Wulfraed Dennis Lee Bieber KD6MOG
(e-mail address removed) (e-mail address removed)
HTTP://wlfraed.home.netcom.com/
(Bestiaria Support Staff: (e-mail address removed))
HTTP://www.bestiaria.com/
 

Ask a Question

Want to reply to this thread or ask your own question?

You'll need to choose a username for the site, which only take a couple of moments. After that, you can post your question and our members will help you out.

Ask a Question

Members online

No members online now.

Forum statistics

Threads
473,734
Messages
2,569,441
Members
44,832
Latest member
GlennSmall

Latest Threads

Top