Python configuration question when python scripts are executed usingAppweb as web server.

I

IronyOfLife

Hi All

I have installed python 2.6.2 in windows xp professional machine. I
have set the following environment variables -- PYTHONPATH. It points
to following windows folders: python root folder, the lib folder and
lib-tk folder. I have configured IIS to execute python scripts.

I do not have any issues executing python scripts from the python
interpreter window or from the browser using IIS as the web server.

However, when I use the open source Appweb web browser to execute
python scripts. It is able to execute only simple script without
issues. For example, hello.py executes fine.

hello.py
--------------------------------------------------------
#!/usr/bin/env python

print "Content-type: text/html\n\n"
print "<html>Hello world!</html>"

However I am not able to execute python scripts that contain import
statements. I need to point out here that i do not have issue with
using os or sys packages. When I tried to execute the sample client
script that came with the gnutls wrapper for python, I got error. I
narrowed it down to the "from ... import" line in the example.

from gnutls.crypto import *
from gnutls.connection import *

Next I modified it to the functions used in the client python scripts
(NOTE: I have added some additional functions to connection and crypto
python scripts).
from gnutls.crypto import X509Certificate, X509PrivateKey, X509CRL
from gnutls.connection import AnonClientCred

Next step, in order to debug, I used the module finder to see what is
happening. To being with I used the module finder to find out the
modules loaded for the simple script hello.py.

hello2.py
--------------------------------------------------------
#!/usr/bin/env python

import modulefinder

import pdb

pdb.set_trace()

finder = modulefinder.ModuleFinder()
finder.run_script('hello.py')

print "Content-type: text/html\n\n"
print "<html>"

print sys.exc_info()

print 'Loaded modules:'
for name, mod in finder.modules.iteritems():
print '%s: ' % name,
print ','.join(mod.globalnames.keys()[:3])

print '-'*50
print 'Modules not imported:'
print '\n'.join(finder.badmodules.iterkeys())

print "</html>"

I got the error. Please see below for error as captured from Appweb
logs. See lines 3 to 5.

cgi:5 main 356: CGI Response: read 89 bytes
cgi:6 main 356: Data is:
c:\appweb\web\hello2.py(9)<module>()
Click to expand...
-> finder = modulefinder.ModuleFinder()

(Pdb)

cgi:5 main 356: CGI Response: read 0 bytes
cmd:7 main getExitCode: process 0, code 1
default:1 main Error: 503 "Service Unavailable" for "/hello2.py",
file "c:/appweb/web/hello2.py": CGI process /hello2.py: exited
abnormally with exit code: 1
request:5 main 356: flushOutput: background 0

Once again, let me stress that this issue is happening only when I use
appweb as the web server.

Some details on Appweb that I found out: Appweb opens up a new console
process and executes the python scripts. See sample below:

cmd:5 pool.0 Running: C:\Dev\Ironic\Tools\Python26\python.exe c:/
appweb/web/hello2.py

When python scripts are executed in this fashion, should there be
additional command line arguments be passed? Am I missing any
additional settings. Also can you help with improving the debugging
lines in hello2.py so that I can identify the issue.

Please help.

Thanks.

Ram G.
 
G

Gabriel Genellina

I have installed python 2.6.2 in windows xp professional machine. I
have set the following environment variables -- PYTHONPATH. It points
to following windows folders: python root folder, the lib folder and
lib-tk folder.
Click to expand...

Why? Did you read it somewhere? Usually there is no need to set the
PYTHONPATH variable at all; remove it.
I have configured IIS to execute python scripts.

I do not have any issues executing python scripts from the python
interpreter window or from the browser using IIS as the web server.
Click to expand...

Try again after PYTHONPATH is removed, and make sure you can import all
the required modules from the Python console. Only then try to run the cgi
scripts.
However, when I use the open source Appweb web browser to execute
python scripts. It is able to execute only simple script without
issues. [...]However I am not able to execute python scripts that
contain import
statements. I need to point out here that i do not have issue with
using os or sys packages. When I tried to execute the sample client
script that came with the gnutls wrapper for python, I got error. I
narrowed it down to the "from ... import" line in the example.

from gnutls.crypto import *
from gnutls.connection import *
Click to expand...

Check that the above lines work fine from inside the Python console.
If not, something went wrong when you installed the gnutls Python package.
Next step, in order to debug, I used the module finder to see what is
happening. To being with I used the module finder to find out the
modules loaded for the simple script hello.py.
Click to expand...

Debugging cgi scripts isn't easy sometimes; try adding this line at the
start of your script:

import cgitb;cgitb.enable()

You should get a nicely formatted stack trace whenever an exception is
raised. If you get an HTTP error instead (like 500 Internal Server Error)
it's likely a syntax error, you must fix it before continuing.
Then, you may find that modulefinder is no more needed.
import pdb
pdb.set_trace()
Click to expand...

pdb cannot work inside a CGI script; remove that.
Once again, let me stress that this issue is happening only when I use
appweb as the web server.
Click to expand...

I don't know appweb, but if you first make sure the script works from the
command line, it should be much easier to make it work in the web server.
 
I

IronyOfLife

I have installed python 2.6.2 in windows xp professional machine. I
have set the following environment variables -- PYTHONPATH. It points
to following windows folders: python root folder, the lib folder and
lib-tk folder.
Click to expand...

Why? Did you read it somewhere? Usually there is no need to set the  
PYTHONPATH variable at all; remove it.
I have configured IIS to execute python scripts.
Click to expand...
I do not have any issues executing python scripts from the python
interpreter window or from the browser using IIS as the web server.
Click to expand...

Try again after PYTHONPATH is removed, and make sure you can import all  
the required modules from the Python console. Only then try to run the cgi  
scripts.
However, when I use the open source Appweb web browser to execute
python scripts. It is able to execute only simple script without
issues. [...]However I am not able to execute python scripts that  
contain import
statements. I need to point out here that i do not have issue with
using os or sys packages. When I tried to execute the sample client
script that came with the gnutls wrapper for python, I got error. I
narrowed it down to the "from ... import" line in the example.
Click to expand...
from gnutls.crypto import *
from gnutls.connection import *
Click to expand...

Check that the above lines work fine from inside the Python console.
If not, something went wrong when you installed the gnutls Python package..
Next step, in order to debug, I used the module finder to see what is
happening. To being with I used the module finder to find out the
modules loaded for the simple script hello.py.
Click to expand...

Debugging cgi scripts isn't easy sometimes; try adding this line at the  
start of your script:

import cgitb;cgitb.enable()

You should get a nicely formatted stack trace whenever an exception is  
raised. If you get an HTTP error instead (like 500 Internal Server Error)  
it's likely a syntax error, you must fix it before continuing.
Then, you may find that modulefinder is no more needed.
import pdb
pdb.set_trace()
Click to expand...

pdb cannot work inside a CGI script; remove that.
Once again, let me stress that this issue is happening only when I use
appweb as the web server.
Click to expand...

I don't know appweb, but if you first make sure the script works from the  
command line, it should be much easier to make it work in the web server.
Click to expand...

Hi Gabriel

Thanks for your reply.

As I had mentioned in my original post, I am able to successfully
execute gnutls client python script while using python interpreter and
from the web browser while using IIS as web server. The scripts were
not executing successfully only when I used AppWeb as web server.

Setting PYTHONPATH environment variables is mentioned in Python docs.

I figured out how to get the debug statements to write to Appweb logs
and from there I found out the issue to be ImportError. Python does
provide a very cryptic error message. "(<type
'exceptions.ImportError'>, ImportError('DLL load failed: The specified
module could not be found.',), <traceback object at 0x00B61580>)"

I then downloaded filemon (as suggested in another forum) utility tool
for windows environment. Using the tool I found that python.exe is
trying to load gnutls related dlls from system path and a list of
folders (I dont know why it didnt try to pick up the dlls from all the
path provided in PATH environment variable) none of which contain the
gnutls related dlls.

I solved the issue temporarily by copying the gnutls related dlls to
the path searched by python.exe

Now my gnutls client python script executes successfully when i use
Appweb as the web server,

NOTE: When I use python interpreter or IIS as webserver the path where
python.exe looks for gnutls related dll's are different. In this
scenario python.exe is able to successfully load the required dlls.
 
G

Gabriel Genellina

En Tue said:
Setting PYTHONPATH environment variables is mentioned in Python docs.
Click to expand...

Could you provide a link please? Setting PYTHONPATH should not be
necesary, and in fact it's a very bad idea. Environment variables are
global, but Python modules may depend on the Python version, architecture,
install location... By example, you may install a 64 bits Python 3.1
version *and* a 32 bits Python 2.5 version and they both can coexist
peacefully - but an extension module compiled for the former cannot be
used in the later version. You must build a separate library for each
version, and install them in two separate directories. But since the
PYTHONPATH variable is shared by all installations, which directory should
contain?
It's best not to use PYTHONPATH at all and rely on other alternatives
(like .pth files, that are searched relative to the current Python
executable, so different versions use different configuration files)
I solved the issue temporarily by copying the gnutls related dlls to
the path searched by python.exe
Click to expand...

Glad to see you could finally fix it!
 
I

IronyOfLife

Hi Gabriel

Could you provide a link please? Setting PYTHONPATH should not be
Click to expand...

Here are couple of links that discusses setting PYTHONPATH environment
variable.
http://docs.python.org/using/windows.html
http://www.daimi.au.dk/~chili/PBI/pythonpath.html
necesary, and in fact it's a very bad idea. Environment variables are
global, but Python modules may depend on the Python version, architecture,
install location... By example, you may install a 64 bits Python 3.1
version *and* a 32 bits Python 2.5 version and they both can coexist
peacefully - but an extension module compiled for the former cannot be
used in the later version. You must build a separate library for each
version, and install them in two separate directories. But since the
PYTHONPATH variable is shared by all installations, which directory should
contain?
It's best not to use PYTHONPATH at all and rely on other alternatives
(like .pth files, that are searched relative to the current Python
executable, so different versions use different configuration files)
Click to expand...

I understand your concerns regarding setting of PYTHONPATH while
multiple versions of Python are installed on the same machine. My fix
however does not use PYTHONPATH. The GNUTLS wrapper module for PYTHON
loads the GNUTLS dll's and it was not able to find them. Using FileMon
(win tool) I found out the paths that are scanned and I copied the
dlls to one of such paths. I still do not like this fix. This is a
temporary solution.

Can you explain maybe with some sample how to set .pth files? Maybe
this will resolve my issue.
Glad to see you could finally fix it!
Click to expand...

Thanks very much for your reply.
 
G

Gabriel Genellina

En Thu said:
Here are couple of links that discusses setting PYTHONPATH environment
variable.
http://docs.python.org/using/windows.html
Click to expand...

Ouch, that document should be reworked and updated!
http://www.daimi.au.dk/~chili/PBI/pythonpath.html
Click to expand...

That's mostly obsolete for current versions. Since PEP370 [0] was
implemented, a per-user private directory is already in sys.path now, so
there is no need to play with PYTHONPATH.
I understand your concerns regarding setting of PYTHONPATH while
multiple versions of Python are installed on the same machine. My fix
however does not use PYTHONPATH. The GNUTLS wrapper module for PYTHON
loads the GNUTLS dll's and it was not able to find them. Using FileMon
(win tool) I found out the paths that are scanned and I copied the
dlls to one of such paths. I still do not like this fix. This is a
temporary solution.
Click to expand...

Note that this cannot be fixed from inside Python. When you import a
module, the interpreter scans the directories in ITS search path
(sys.path) looking for a matching module. Once the module is found:
- if it is a Python file, it's executed
- if it is an extension module (a .pyd file, in fact a .dll renamed) it's
loaded using LoadLibraryEx (a Windows function), and finally its
initialization routine is executed.

For LoadLibrary to successfully load the module, all its external
references must be resolved. That is, Windows must locate and load all
other DLLs that this module depends on, using its own search strategy [1],
taking into account the PATH environment variable and many other places.

It is at this stage that you get the error: when the gnutls wrapper
attempts to load the gnutls DLL. That search is done by Windows, not by
Python, and PYTHONPATH has nothing to do with it.

Why different results in IIS and appweb? Note that there exist several
search strategies, they depend on the application home directory, the
location of the .exe, whether SetDllDirectory was called or not, whether
the application has a manifest or not, a .local file or not... Hard to
tell with so many variables.
Can you explain maybe with some sample how to set .pth files? Maybe
this will resolve my issue.
Click to expand...

Yes, but I don't think this will actually help with your issue.

pth files are described here [2]. Suppose you want to add c:\foo\bar to
sys.path, then write a file whatever.pth containing this single line:

c:\foo\bar

and place it on c:\your_python_installation\lib\site-packages
When the interpreter starts, it will find the .pth file, read it, and add
any directory it finds (that actually exists) to sys.path

Note that another Python installation will use a diferent site-packages
directory and won't find this particular .pth file, so different Python
versions don't interfere.
 
I

IronyOfLife

Hi Gabriel

Thanks for the detailed and useful reply.


En Thu, 06 Aug 2009 12:49:30 -0300, IronyOfLife <[email protected]>  
escribió:




Here are couple of links that discusses setting PYTHONPATH environment
variable.
http://docs.python.org/using/windows.html
Click to expand...

Ouch, that document should be reworked and updated!
http://www.daimi.au.dk/~chili/PBI/pythonpath.html
Click to expand...

That's mostly obsolete for current versions. Since PEP370 [0] was  
implemented, a per-user private directory is already in sys.path now, so  
there is no need to play with PYTHONPATH.
Click to expand...

Thanks for the clarification on PYTHONPATH.
I understand your concerns regarding setting of PYTHONPATH while
multiple versions of Python are installed on the same machine. My fix
however does not use PYTHONPATH. The GNUTLS wrapper module for PYTHON
loads the GNUTLS dll's and it was not able to find them. Using FileMon
(win tool) I found out the paths that are scanned and I copied the
dlls to one of such paths. I still do not like this fix. This is a
temporary solution.
Click to expand...

Note that this cannot be fixed from inside Python. When you import a  
module, the interpreter scans the directories in ITS search path  
(sys.path) looking for a matching module. Once the module is found:
- if it is a Python file, it's executed
- if it is an extension module (a .pyd file, in fact a .dll renamed) it's  
loaded using LoadLibraryEx (a Windows function), and finally its  
initialization routine is executed.

For LoadLibrary to successfully load the module, all its external  
references must be resolved. That is, Windows must locate and load all  
other DLLs that this module depends on, using its own search strategy [1],  
taking into account the PATH environment variable and many other places.

It is at this stage that you get the error: when the gnutls wrapper  
attempts to load the gnutls DLL. That search is done by Windows, not by  
Python, and PYTHONPATH has nothing to do with it.

Why different results in IIS and appweb? Note that there exist several  
search strategies, they depend on the application home directory, the  
location of the .exe, whether SetDllDirectory was called or not, whether  
the application has a manifest or not, a .local file or not... Hard to  
tell with so many variables.
Click to expand...

This is fairly easy to explain. When I configured IIS to execute
python scripts, as per the documentation I pass two command line
arguments. Appweb works this way. It opens up a new command process
and in the function parameter for environment it sets only SYSTEMROOT.
It does not set the PATH variable (A BUG IN APPWEB which I have passed
on to them. They have not commented or provided any work around) for
Windows platform.

So while loading the gnutls client script is loaded, the following
path is searched
1. gnutls\library
2. C:\windows
3. C:\WIndows\system
4. C:\WIndows\system32
5. Folder where pyhton.exe is present.

And when I use IIS, the PATH variable is also set. So in this case all
the folders in the PATH are also searched and LoadLibrary is able to
successfully find the GNUTLS related Dlls without any issues.

You were mentioning about .local file or manifest file to specify the
path. I used the python's build command to build the wrapper. Is there
a way to mention in setup.py to generate the .manifest file? I wold
very much appreciate if you can help me with that.

I will also look for answers for how to modify setup.py to generate
manifest file or .local file etc..
Can you explain maybe with some sample how to set .pth files? Maybe
this will resolve my issue.
Click to expand...

Yes, but I don't think this will actually help with your issue.

pth files are described here [2]. Suppose you want to add c:\foo\bar to  
sys.path, then write a file whatever.pth containing this single line:

c:\foo\bar

and place it on c:\your_python_installation\lib\site-packages
When the interpreter starts, it will find the .pth file, read it, and add  
any directory it finds (that actually exists) to sys.path

Note that another Python installation will use a diferent site-packages  
directory and won't find this particular .pth file, so different Python  
versions don't interfere.

---

[0]http://python.org/dev/peps/pep-0370/
[1]http://msdn.microsoft.com/en-us/library/ms682586(VS.85).aspx
[2]http://docs.python.org/library/site.html
Click to expand...
 
G

Gabriel Genellina

Why different results in IIS and appweb? [...]
Click to expand...
This is fairly easy to explain. When I configured IIS to execute
python scripts, as per the documentation I pass two command line
arguments. Appweb works this way. It opens up a new command process
and in the function parameter for environment it sets only SYSTEMROOT.
It does not set the PATH variable (A BUG IN APPWEB which I have passed
on to them. They have not commented or provided any work around) for
Windows platform.
Click to expand...

Perhaps you could configure appweb to call a .bat file like this, instead
of directly invoking python?

set path=%path%;other;directories;added
c:\path\to\python.exe %*

(this is slightly off topic now...)
You were mentioning about .local file or manifest file to specify the
path. I used the python's build command to build the wrapper. Is there
a way to mention in setup.py to generate the .manifest file? I wold
very much appreciate if you can help me with that.

I will also look for answers for how to modify setup.py to generate
manifest file or .local file etc..
Click to expand...

The last part is easy: foo.exe.local is just an empty file in the same
directory as foo.exe - when it exists, DLLs are searched first on the
directory containing the application [1]

With manifest files I can't help. I suggest you create a new thread with
that question, perhaps in the distutils-SIG mailing list [2]

[1] http://msdn.microsoft.com/en-us/library/ms682600(VS.85).aspx
[2] http://mail.python.org/mailman/listinfo/distutils-sig
 

Ask a Question

Want to reply to this thread or ask your own question?

You'll need to choose a username for the site, which only take a couple of moments. After that, you can post your question and our members will help you out.

Ask a Question

Similar Threads

Python client/server that reads HTML body from server 1
Bash scripts for web apps 1
Python battle game help 2
When deployed to Heroku, python setup.py egg info did not run successfully. 1
Help with Visual Lightbox: Scripts 2
Need help with this Python code. 2
noob can't install python modules/scripts 6
executing python scripts that are symlinked 6

Members online

No members online now.
Total: 39 (members: 1, guests: 38)
Robots: 482

Forum statistics

Threads
473,769
Messages
2,569,580
Members
45,054
Latest member
TrimKetoBoost

Latest Threads

Top