Process crash with no reason

[gil.shinar]

Hi All,

I'm running a program that is acting as a nice interface to sybase'
replication server. The program is using the cherrypy web service for
the GUI. The process is crashing every few days with no reason. In the
log I can see INFO and DEBUG (No ERROR) log lines and I do not get any
TraceBack python's message. This program is running on solaris 9
machine.
Where can I see or what can I do in order to find out what causes the
process to crash?
I have tried simulating a "traceBack" message and I could see this
traceback message in one of the log files I'm using. When the process
crashes without my help, I don't have a clue.
Let me know if you need any other info

Thanks

Gil

 

[Philip Semanchuk]

Hi All,

I'm running a program that is acting as a nice interface to sybase'
replication server. The program is using the cherrypy web service for
the GUI. The process is crashing every few days with no reason. In the
log I can see INFO and DEBUG (No ERROR) log lines and I do not get any
TraceBack python's message. This program is running on solaris 9
machine.
Where can I see or what can I do in order to find out what causes the
process to crash?
I have tried simulating a "traceBack" message and I could see this
traceback message in one of the log files I'm using. When the process
crashes without my help, I don't have a clue.
Let me know if you need any other info

Although Python isn't immune to fatal errors like you describe, I'd
immediately suspect a 3rd-party module instead, esp. one written in C
or C++. Are you using anything like that?

 

[gil.shinar]

Although Python isn't immune to fatal errors like you describe, I'd  
immediately suspect a 3rd-party module instead, esp. one written in C  
or C++. Are you using anything like that?

No I do not.
Is there a way to monitor who had "killed" my process?

Thanks

 

[Tim Golden]

No I do not.

Then how are you interacting with Sybase?

TJG

 

[Philip Semanchuk]

No I do not.

But you're using CherryPy, aren't you? And as Tim Golden pointed out,
you're probably using another 3rd party lib to talk to Sybase. ISTR
that CherryPy is written in pure Python but if you have a database
adapter lib it probably isn't.

Is there a way to monitor who had "killed" my process?

That's more of a Solaris question than a Python one, and I don't know
much about Solaris. Is there a system log where Solaris records when &
why it has killed errant processes?

 

[gil.shinar]

Then how are you interacting with Sybase?

TJG

I'm using python's functions to run sybase sql commands.

Thanks

 

[gil.shinar]

But you're using CherryPy, aren't you? And as Tim Golden pointed out,  
you're probably using another 3rd party lib to talk to Sybase. ISTR  
that CherryPy is written in pure Python but if you have a database  
adapter lib it probably isn't.


That's more of a Solaris question than a Python one, and I don't know  
much about Solaris. Is there a system log where Solaris records when &  
why it has killed errant processes?

Yes I am using CherryPy. One of the developers I'm working with thinks
it is a cherrypy issue. Our python process has nothing to do if
cherrypy is stuck or down. But still, it is strange that there are no
leftovers from the crashed process.
Solaris has system logs that has nothing in them about this process.

Any other suggestion will be great.

Thanks

 

[Philip Semanchuk]

I'm using python's functions to run sybase sql commands.

Can you give a short code sample? I'm unaware of how one would use the
standard Python library to talk to Sybase, unless Sybase has a raw
socket interface or some such.

 

[gil.shinar]

Can you give a short code sample? I'm unaware of how one would use the  
standard Python library to talk to Sybase, unless Sybase has a raw  
socket interface or some such.

First of all I have found the following python's import:

import Sybase


To connect the sybase server we are using the following function:

def _getConnection (self):
db = Sybase.connect(self.dbs, self.login[0], self.login
[1], datetime='auto')
return db


To execute a DBS query, we are using the following function:

def _executeQuery(self, cursor, query, expected_result=None):
try:
self.logger.debug('Executing query:' + query +
', expected:' + str(expected_result))
cursor.execute(query)
except Sybase.DatabaseError, e:
if ((expected_result != None) and ("Msg " +
expected_result in str(e))):
return (DBStatus.SUCCESS,
self._getDBMsg(str(e)))
raise Sybase.DatabaseError, self._getDBMsg(str
(e))

return (DBStatus.SUCCESS, "")

Do you need anything else?

Thanks a lot

 

[Philip Semanchuk]

First of all I have found the following python's import:

import Sybase

This isn't part of the Python standard library. It's a 3rd party
module -- exactly what we were looking for.

Personally, I think this (or another C++ or C-based 3rd party module
that you use heavily) is your prime suspect for the origin of the
crashes you're having. That's not because I think the people who wrote
or maintain it are bad or lazy coders. In fact, it's no reflection on
their skill at all. It's just that a lot more people have used and
exercised the Python standard library modules. A 3rd party module like
this one will be less well-used and therefore less well-tested and
therefore more likely to contain a bug that causes a crash.

That said, I don't know how to advise you to proceed from here. You
could perhaps turn on logging at the database level. I know Postgres,
for instance, can write very detailed logs and so if you get a crash
at 9:33:22 you can look in the log and see what was happening at that
time. If you get several crashes and they all happen when a certain
SQL statement is being executed, that's probably the culprit.

You could also alter the Sybase module to add logging using Python's
logging module. Who knows, it might already be there, waiting to be
turned on with a switch.

But I'm jumping the gun a little. As I said, it could be this module
or another that's causing your problem. It's a lot easier to cause a
hard crash using C or C++ than it is using pure Python, so pure Python
modules would be lower on my list of suspects. Enumerate all of the
modules you're using and find out where they come from. Any of them
that are not in the standard library and are not written in pure
Python should top your list of suspects.

Good luck
Philip

 

[gil.shinar]

This isn't part of the Python standard library. It's a 3rd party  
module -- exactly what we were looking for.

Personally, I think this (or another C++ or C-based 3rd party module  
that you use heavily) is your prime suspect for the origin of the  
crashes you're having. That's not because I think the people who wrote  
or maintain it are bad or lazy coders. In fact, it's no reflection on  
their skill at all. It's just that a lot more people have used and  
exercised the Python standard library modules. A 3rd party module like  
this one will be less well-used and therefore less well-tested and  
therefore more likely to contain a bug that causes a crash.

That said, I don't know how to advise you to proceed from here. You  
could perhaps turn on logging at the database level. I know Postgres,  
for instance, can write very detailed logs and so if you get a crash  
at 9:33:22 you can look in the log and see what was happening at that  
time. If you get several crashes and they all happen when a certain  
SQL statement is being executed, that's probably the culprit.

You could also alter the Sybase module to add logging using Python's  
logging module. Who knows, it might already be there, waiting to be  
turned on with a switch.

But I'm jumping the gun a little. As I said, it could be this module  
or another that's causing your problem. It's a lot easier to cause a  
hard crash using C or C++ than it is using pure Python, so pure Python  
modules would be lower on my list of suspects. Enumerate all of the  
modules you're using and find out where they come from. Any of them  
that are not in the standard library and are not written in pure  
Python should top your list of suspects.

Good luck
Philip

Thanks a lot and sorry for the late response. My main suspect is the
CherryPy.
I'm still investigating it.

Gil

 

[Stephen Hansen]

Thanks a lot and sorry for the late response. My main suspect is the

CherryPy.
I'm still investigating it.

You're asking for advice but not listening to what people are saying
here -- why is CherryPy on the top of your list?

What version of CherryPy is it? 1 or 2?

If its 2, then its pure python and there's a really, really low chance
that its provoking a hard crash w/o a traceback printing before it
dies out. A hard crash which isn't printing an error message is
possible in pure Python I suppose, but I've never seen it. A "Bus
error" for example will kill the interpreter before it can do any
error handling. But I've only ever seen that when interfacing with
third party libraries.

If its 1, there is some C so you can bump it up ... and for that I'd
probably attempt to run the application in a console or with stdout
manually redirected instead of relying on the logging alone, to see if
anything useful gets put out.. like "Bus error", at the time it exits.

But why do you keep discarding the Sybase module?

Its most definitely *not* written in python. What Philip and Tim were
trying to tell you is you are *not* using "python's functions to run
sybase sql commands"

The Sybase module is a python wrapper around code written in C module
which is interfacing with a third party library to talk to Sybase. Are
you at the latest version of Sybase(0.39 if its the same module I'm
aware of?) How about the library underneath it? FreeTDS or Sybase ASE
(whatever that is) or whatever else is being used? A bug in /any/ of
those C layers (notice its *multiple* layers) could propagate up and
provoke a hard crash that Python doesn't catch. So if you look into
those versions perhaps one has a new release that fixes bugs.

--S

 

[gil.shinar]

You're asking for advice but not listening to what people are saying
here -- why is CherryPy on the top of your list?

What version of CherryPy is it? 1 or 2?

If its 2, then its pure python and there's a really, really low chance
that its provoking a hard crash w/o a traceback printing before it
dies out. A hard crash which isn't printing an error message is
possible in pure Python I suppose, but I've never seen it. A "Bus
error" for example will kill the interpreter before it can do any
error handling. But I've only ever seen that when interfacing with
third party libraries.

If its 1, there is some C so you can bump it up ... and for that I'd
probably attempt to run the application in a console or with stdout
manually redirected instead of relying on the logging alone, to see if
anything useful gets put out.. like "Bus error", at the time it exits.

But why do you keep discarding the Sybase module?

Its most definitely *not* written in python. What Philip and Tim were
trying to tell you is you are *not* using "python's functions to run
sybase sql commands"

The Sybase module is a python wrapper around code written in C module
which is interfacing with a third party library to talk to Sybase. Are
you at the latest version of Sybase(0.39 if its the same module I'm
aware of?) How about the library underneath it? FreeTDS or Sybase ASE
(whatever that is) or whatever else is being used? A bug in /any/ of
those C layers (notice its *multiple* layers) could propagate up and
provoke a hard crash that Python doesn't catch. So if you look into
those versions perhaps one has a new release that fixes bugs.

--S

Hi,

After farther investigation and some help from other developers, we
got to the conclusion that the python process does not crash but the
program exits because cherrypy freezes and the python program has
nothing to do when the web service is not responding. My cherrypy
version is 3.0.1
Can any of you help me with cherrypy or should I ask in cherrypy
group?

Thanks

Gil