major performance issue

[cerr]

Hi There,

I have a major performance issue with an application that is inserting
strings into a database.
I'm not quite sure but we see major backlogs and it's only about 6MB
that it would be inserting in 24 hours. That should not be a major
problem i would say.
The way this logserver is working is:
while (1)
{
send message off to syslog on port 1515 which is on the same host.
/*message parsing with strtok and other standard C function calls.*/
sprintf(query,"insert into logs(source,date,time,program,msg)values
('%s','%d-%d-%s','%s','%s','%s[%s:%s');",
_source,
t->tm_year+1900,month,_date,
_time,
_program,
_program,_program_name,newmsg
);
cout << "INSERT Query: " << query << endl;
_update_db(_source,query);
}
This seems to insert only one message per about 500 to 800ms
(estimated). Each message would be around 200 Bytes long. Sometimes
there's multiple messages arriving per second asynchronously.
My problem is, I can not do too much debugging and 'trying things' on
this because this is a system that's been switched live already. My
local system does not seem to be experiencing the same performance
issues.
Any clues where bottle necks may be and how they can be resolved?
Thanks a lot!

roN

 

[Ben Bacarisse]

I have a major performance issue with an application that is inserting
strings into a database.
I'm not quite sure but we see major backlogs and it's only about 6MB
that it would be inserting in 24 hours. That should not be a major
problem i would say.

I don't think you have a C problem...

The way this logserver is working is:
while (1)
{
send message off to syslog on port 1515 which is on the same host.
/*message parsing with strtok and other standard C function calls.*/
sprintf(query,"insert into logs(source,date,time,program,msg)values
('%s','%d-%d-%s','%s','%s','%s[%s:%s');",
_source,
t->tm_year+1900,month,_date,
_time,
_program,
_program,_program_name,newmsg
);
cout << "INSERT Query: " << query << endl;
_update_db(_source,query);
}

.... because this seems to be C++. But even so, I would want to look
at the database first. (Well, after checking that this sprintf does
not overflow it's buffer.)

The first place to ask is in a group that deals with your particular
database (and you should provide a lot more detail than you do here).
If there is no problem with the DB, then there might be one with the
C++ interface to it, and that's harder to offer advice about where
to ask.

A comp.lang.c++ can help with the details of the coding but I doubt
they welcome questions about database performance.

<snip>

 

[cerr]

I have a major performance issue with an application that is inserting
strings into a database.
I'm not quite sure but we see major backlogs and it's only about 6MB
that it would be inserting in 24 hours. That should not be a major
problem i would say.

I don't think you have a C problem...

The way this logserver is working is:
while (1)
{
send message off to syslog on port 1515 which is on the same host.
/*message parsing with strtok and other standard C function calls.*/
sprintf(query,"insert into logs(source,date,time,program,msg)values
('%s','%d-%d-%s','%s','%s','%s[%s:%s');",
                                           _source,
                                           t->tm_year+1900,month,_date,
                                           _time,
                                           _program,
                                           _program,_program_name,newmsg
                                           );
                           cout << "INSERT Query: " << query << endl;
                           _update_db(_source,query);
}

... because this seems to be C++.  But even so, I would want to look
at the database first.  (Well, after checking that this sprintf does
not overflow it's buffer.)

well query is a char array of 1024 elements - which should be enough -
it also does not crash because of a segmentation fault or similar.
The right thing to do is, I understand, to verify the string size and
that it would not overflow the query buffer. But how can I verify the
string size is =< 1024? Before actually putting the string together.

The first place to ask is in a group that deals with your particular
database (and you should provide a lot more detail than you do here).

Yes, right. We're using a mysql db and i thought I would ask here to
get hints of where else the problem may be (other than the actual
execution of the query).
Well I have posted in a mysql NG now as well, they may be able to come
up with some suggestions too.

If there is no problem with the DB, then there might be one with the
C++ interface to it, and that's harder to offer advice about where
to ask.

Right, I understand that but I just looked for a starting point maybe
what to look at and this may bring me further already, thank you!

roN

 

[Jens Thoms Toerring]

I have a major performance issue with an application that is inserting
strings into a database.
I'm not quite sure but we see major backlogs and it's only about 6MB
that it would be inserting in 24 hours. That should not be a major
problem i would say.

I don't think you have a C problem...

The way this logserver is working is:
while (1)
{
send message off to syslog on port 1515 which is on the same host.
/*message parsing with strtok and other standard C function calls.*/
sprintf(query,"insert into logs(source,date,time,program,msg)values
('%s','%d-%d-%s','%s','%s','%s[%s:%s');",
                                           _source,
                                           t->tm_year+1900,month,_date,
                                           _time,
                                           _program,
                                           _program,_program_name,newmsg
                                           );
                           cout << "INSERT Query: " << query << endl;
                           _update_db(_source,query);
}

... because this seems to be C++.  But even so, I would want to look
at the database first.  (Well, after checking that this sprintf does
not overflow it's buffer.)

well query is a char array of 1024 elements - which should be enough -
it also does not crash because of a segmentation fault or similar.
The right thing to do is, I understand, to verify the string size and
that it would not overflow the query buffer. But how can I verify the
string size is =< 1024? Before actually putting the string together.

The first simple test would be to double (or triple or whatever)
the length of the char array. That should be a thing of a few
seconds and, if the problem goes away with that, it's at least
a hint that this might be related to the problem. Then you could
try to use snprintf() with which you can set an upper limit on
the number of chars put into the string and which should return
how many actually where (or would have been) needed. Thus you can
check if the string was successfully output or if you have to try
again with a longer char array. Of course, this advice has to be
taken with a grain of salt since snprintf() is a C99 function and
you seem to be writing C++ and I don't know if it exists in C++...
If not C++ perhaps has some function that automaticallt allocates
a char array large enough (similar to the non-standard asprintf()
function)...

But my (unqualified) gut feeling is also (to be confirmed by the
above test) is that there's some problem with the database and not
something related to C/C++.
Regards, Jens

 

[Richard Bos]

But my (unqualified) gut feeling is also (to be confirmed by the
above test) is that there's some problem with the database and not
something related to C/C++.

My sysadmin instincts tell me to check the network connection as well,
if there is one. It may be that the db is processing packets as fast as
it gets them, which is at 1/1000th the speed at which the application
sends them, because of a bottleneck (or other slowness emitter)
somewhere along the line between app and db.

Richard