Python 2.3.2 & Netware

[Jeff Davey]

I spent the day porting & building a .a that I could link into an NLM
(statically) and run on Netware, for the purpose of using Python in an
application.

So far, everything (from the compiling process) works great. I built a
little test.nlm to see if everything works good:

#include <Python.h>

int main(int argc, char *argv[])
{
Py_Initialize();
PyRun_SimpleString("print 'Hi there'\n");
Py_Finalize();
return 0;
}

I then loaded up this NLM (of like, 3.8MB or so) into protected memory
space on a Netware 5.1 server.

Near instantly, I get an abend of such:
Removed address space because of memory protection violation
Reason: Free detected corrupt preceeding redzone for node

So, I wade through the Python source to discover what's happening:

Py_Initialize() is called
From there we get down to _Py_ReadyTypes();

In _Py_ReadyTypes we get to the first line:
if (PyType_Ready(&PyType_Type) <0)


In PyType_Ready we initialize the base class (ie: recurse once), and
proceed to the following (about to do the type type):

if (add_operators(type) < 0)
goto error;


Following that we get to add_operators:
descr = PyDescr_NewWrapper(type, p, *ptr);

and further down in PyDescr_NewWrapper..
descr = (PyWrapperDescObject *)descr_new(&PyWrapperDescr_Type, type,
base->name);
Just to note, the name here is '__cmp__'

Then in descr_new (getting close now, I promise! :
descr->d_name = PyString_InternFromString(name);

In that particular function, the abend seems to happen in
PyString_InternInPlace(&s)

So Following that, and this is where the abend happens,

In PyString_InternInPlace(PyObject **p) you have this:

if ((t = PyDict_GetItem(interned, (PyObject *)s)) != NULL {
Py_INCREF(t);
Py_DECREF(*p); // THE ABEND IS HERE
*p = t;
return;
}

The abend happens on the Py_DECREF(*p).

Now, I had a hard time trying to figure out exactly what this type of
abend means, but from what I understand, that peice of memory has been
corrupted at some point.

When I ported this, I modified the pyconfig.h by hand, perhaps this
problem is caused by inserting a wrong value (which I sort of doubt
right now...) into the 'type' sizes ?

Let me know if you have any ideas....

Thanks!

Jeff Davey

 

[Brad Clements]

Hmm, I ported Python 1.5.2 to CLIB NetWare 3 years ago (Zope too) and I did
not have this problem. I don't think the code you're looking at has changed
very much since then.

I don't think this statement helps.. Could you try linking your .a with
Python's main to get a command line interface?

i.e. removing your existing 'main' code from the picture might clarify. 3.8
meg seems a bit large, that's with debug info?

 

[Jeff Davey]

I don't think this statement helps.. Could you try linking your .a with
Python's main to get a command line interface?

By python's main, I'm assuming you mean Modules/ccpython.cc ? I converted
this to a C file (it's pretty small) and ran that.... It gets to the same
place before it dies.

i.e. removing your existing 'main' code from the picture might clarify. 3.8
meg seems a bit large, that's with debug info?

Sorry, I didn't realize I had left the -g in there, it's about 800k without.

Jeff

 

[Jeff Davey]

Well, I've gotten a little bit further,

Basically, in the pyconfig.h I changed #define WITH_PYMALLOC to #undef
WITH_PYMALLOC. There was something about PyObject_Malloc that returns
improper memory with Netware.

The new abend I'm getting is a General Protection Fault , tracking that
one down right now. Will let you know when I figure it out.

Jeff

 

[Jeff Davey]

The new abend I'm getting is a General Protection Fault , tracking that
one down right now. Will let you know when I figure it out.

I have a bunch of printfs going to see what's up, basically, it finished the
'Type' type from _Py_ReadyTypes, and starts into bool... Interesting enough,
the printf I have at the end of the PyType_Ready function, gets corrupted
(as in, the static string is overwritten by something else) as it finished
the 'Type' type. It then proceeds to try and PyType_Ready the bool type, it
gets to the bases = Py_BuildValue("(0)", .base); line, inside there it dies:

bases = type->tp_bases;
if (bases == NULL) {
if (bases == NULL)
bases = PyTuple_New(0);
else
bases = Py_BuildValue("0", base); // ABEND is HERE
}
I was going to keep following, but I've sort of come to the conclusion
there's some major memory management problems (overwriting buffers, or
something along that way) with getting Python to run on Netware.

Now, I'm not sure if it'some setting I shouldn't have enabled in the
pyconfig.h, or something else, but I do know that Netware is a LOT pickier
than Linux when it comes to memory.

Any ideas maybe why I'm having so many problems with Python & Memory
Mangament ?

Jeff

 

[Jeff Davey]

Wooot! Got it going

Another braindead mistake in the pyconfig.h.

Now I'm hoping the 300KB memory leak Netware is reporting is because Python
can't find any of the libraries it needs.

but, I have a python.nlm running on my Netware box

Jeff

 

[Brad Clements]

Congrats.. LIBC or CLIB?

Now the real problem is, how will you handle global data in dynamically
loaded extension modules? Since library NLMs do not have per-process private
global data space.

 

[Jeff Davey]

Congrats.. LIBC or CLIB?

LibC, it would have been a super PITA to go CLIB, and really, I didn't
really care about supporting NW 4.11 anymore. Novell as well recommends
to start all software development on LibC.

The nice thing about porting it to LibC is a lot of the libraries that
Python needed were already there (ie: pthreads, etc). LibC has a lot
'unixishness' in it.

Now the real problem is, how will you handle global data in dynamically
loaded extension modules? Since library NLMs do not have per-process private
global data space.

Not quite sure, haven't had time to continue on it. Extremely new to
Python itself, so trying to figure out what Python itself needs. (Ie,
it's Lib, compiled modules, etc).

One thing that is interesting is LibC has support for dlopen & friends.


Jeff