major performance issue

Discussion in 'C Programming' started by cerr, Aug 26, 2009.

  1. cerr

    cerr Guest

    Hi There,

    I have a major performance issue with an application that is inserting
    strings into a database.
    I'm not quite sure but we see major backlogs and it's only about 6MB
    that it would be inserting in 24 hours. That should not be a major
    problem i would say.
    The way this logserver is working is:
    while (1)
    {
    send message off to syslog on port 1515 which is on the same host.
    /*message parsing with strtok and other standard C function calls.*/
    sprintf(query,"insert into logs(source,date,time,program,msg)values
    ('%s','%d-%d-%s','%s','%s','%s[%s:%s');",
    _source,
    t->tm_year+1900,month,_date,
    _time,
    _program,
    _program,_program_name,newmsg
    );
    cout << "INSERT Query: " << query << endl;
    _update_db(_source,query);
    }
    This seems to insert only one message per about 500 to 800ms
    (estimated). Each message would be around 200 Bytes long. Sometimes
    there's multiple messages arriving per second asynchronously.
    My problem is, I can not do too much debugging and 'trying things' on
    this because this is a system that's been switched live already. My
    local system does not seem to be experiencing the same performance
    issues.
    Any clues where bottle necks may be and how they can be resolved?
    Thanks a lot!

    roN
     
    cerr, Aug 26, 2009
    #1
    1. Advertising

  2. cerr <> writes:

    > I have a major performance issue with an application that is inserting
    > strings into a database.
    > I'm not quite sure but we see major backlogs and it's only about 6MB
    > that it would be inserting in 24 hours. That should not be a major
    > problem i would say.


    I don't think you have a C problem...

    > The way this logserver is working is:
    > while (1)
    > {
    > send message off to syslog on port 1515 which is on the same host.
    > /*message parsing with strtok and other standard C function calls.*/
    > sprintf(query,"insert into logs(source,date,time,program,msg)values
    > ('%s','%d-%d-%s','%s','%s','%s[%s:%s');",
    > _source,
    > t->tm_year+1900,month,_date,
    > _time,
    > _program,
    > _program,_program_name,newmsg
    > );
    > cout << "INSERT Query: " << query << endl;
    > _update_db(_source,query);
    > }


    .... because this seems to be C++. But even so, I would want to look
    at the database first. (Well, after checking that this sprintf does
    not overflow it's buffer.)

    The first place to ask is in a group that deals with your particular
    database (and you should provide a lot more detail than you do here).
    If there is no problem with the DB, then there might be one with the
    C++ interface to it, and that's harder to offer advice about where
    to ask.

    A comp.lang.c++ can help with the details of the coding but I doubt
    they welcome questions about database performance.

    <snip>
    --
    Ben.
     
    Ben Bacarisse, Aug 26, 2009
    #2
    1. Advertising

  3. cerr

    cerr Guest

    On Aug 26, 1:13 pm, Ben Bacarisse <> wrote:
    > cerr <> writes:
    > > I have a major performance issue with an application that is inserting
    > > strings into a database.
    > > I'm not quite sure but we see major backlogs and it's only about 6MB
    > > that it would be inserting in 24 hours. That should not be a major
    > > problem i would say.

    >
    > I don't think you have a C problem...
    >
    >
    >
    > > The way this logserver is working is:
    > > while (1)
    > > {
    > > send message off to syslog on port 1515 which is on the same host.
    > > /*message parsing with strtok and other standard C function calls.*/
    > > sprintf(query,"insert into logs(source,date,time,program,msg)values
    > > ('%s','%d-%d-%s','%s','%s','%s[%s:%s');",
    > >                                            _source,
    > >                                            t->tm_year+1900,month,_date,
    > >                                            _time,
    > >                                            _program,
    > >                                            _program,_program_name,newmsg
    > >                                            );
    > >                            cout << "INSERT Query: " << query << endl;
    > >                            _update_db(_source,query);
    > > }

    >
    > ... because this seems to be C++.  But even so, I would want to look
    > at the database first.  (Well, after checking that this sprintf does
    > not overflow it's buffer.)


    well query is a char array of 1024 elements - which should be enough -
    it also does not crash because of a segmentation fault or similar.
    The right thing to do is, I understand, to verify the string size and
    that it would not overflow the query buffer. But how can I verify the
    string size is =< 1024? Before actually putting the string together.

    > The first place to ask is in a group that deals with your particular
    > database (and you should provide a lot more detail than you do here).

    Yes, right. We're using a mysql db and i thought I would ask here to
    get hints of where else the problem may be (other than the actual
    execution of the query).
    Well I have posted in a mysql NG now as well, they may be able to come
    up with some suggestions too.

    > If there is no problem with the DB, then there might be one with the
    > C++ interface to it, and that's harder to offer advice about where
    > to ask.

    Right, I understand that but I just looked for a starting point maybe
    what to look at and this may bring me further already, thank you!

    roN
     
    cerr, Aug 26, 2009
    #3
  4. cerr <> wrote:
    > On Aug 26, 1:13 pm, Ben Bacarisse <> wrote:
    > > cerr <> writes:
    > > > I have a major performance issue with an application that is inserting
    > > > strings into a database.
    > > > I'm not quite sure but we see major backlogs and it's only about 6MB
    > > > that it would be inserting in 24 hours. That should not be a major
    > > > problem i would say.

    > >
    > > I don't think you have a C problem...
    > >
    > >
    > >
    > > > The way this logserver is working is:
    > > > while (1)
    > > > {
    > > > send message off to syslog on port 1515 which is on the same host.
    > > > /*message parsing with strtok and other standard C function calls.*/
    > > > sprintf(query,"insert into logs(source,date,time,program,msg)values
    > > > ('%s','%d-%d-%s','%s','%s','%s[%s:%s');",
    > > >                                            _source,
    > > >                                            t->tm_year+1900,month,_date,
    > > >                                            _time,
    > > >                                            _program,
    > > >                                            _program,_program_name,newmsg
    > > >                                            );
    > > >                            cout << "INSERT Query: " << query << endl;
    > > >                            _update_db(_source,query);
    > > > }

    > >
    > > ... because this seems to be C++.  But even so, I would want to look
    > > at the database first.  (Well, after checking that this sprintf does
    > > not overflow it's buffer.)


    > well query is a char array of 1024 elements - which should be enough -
    > it also does not crash because of a segmentation fault or similar.
    > The right thing to do is, I understand, to verify the string size and
    > that it would not overflow the query buffer. But how can I verify the
    > string size is =< 1024? Before actually putting the string together.


    The first simple test would be to double (or triple or whatever)
    the length of the char array. That should be a thing of a few
    seconds and, if the problem goes away with that, it's at least
    a hint that this might be related to the problem. Then you could
    try to use snprintf() with which you can set an upper limit on
    the number of chars put into the string and which should return
    how many actually where (or would have been) needed. Thus you can
    check if the string was successfully output or if you have to try
    again with a longer char array. Of course, this advice has to be
    taken with a grain of salt since snprintf() is a C99 function and
    you seem to be writing C++ and I don't know if it exists in C++...
    If not C++ perhaps has some function that automaticallt allocates
    a char array large enough (similar to the non-standard asprintf()
    function)...

    But my (unqualified) gut feeling is also (to be confirmed by the
    above test) is that there's some problem with the database and not
    something related to C/C++.
    Regards, Jens
    --
    \ Jens Thoms Toerring ___
    \__________________________ http://toerring.de
     
    Jens Thoms Toerring, Aug 26, 2009
    #4
  5. cerr

    Richard Bos Guest

    (Jens Thoms Toerring) wrote:

    > But my (unqualified) gut feeling is also (to be confirmed by the
    > above test) is that there's some problem with the database and not
    > something related to C/C++.


    My sysadmin instincts tell me to check the network connection as well,
    if there is one. It may be that the db is processing packets as fast as
    it gets them, which is at 1/1000th the speed at which the application
    sends them, because of a bottleneck (or other slowness emitter)
    somewhere along the line between app and db.

    Richard
     
    Richard Bos, Aug 31, 2009
    #5
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. kokwooi

    Major Major Problem With ASP.NET

    kokwooi, Sep 18, 2003, in forum: ASP .Net
    Replies:
    6
    Views:
    691
    NuTcAsE
    Sep 19, 2003
  2. Keith

    Major security issue?

    Keith, Feb 1, 2004, in forum: ASP .Net
    Replies:
    5
    Views:
    388
  3. Amir Ghezelbash

    Major Security Issue

    Amir Ghezelbash, Apr 22, 2004, in forum: ASP .Net
    Replies:
    4
    Views:
    333
    Arthur Nesterovsky
    Apr 23, 2004
  4. Jef Driesen
    Replies:
    2
    Views:
    1,768
    Jef Driesen
    Jan 12, 2006
  5. ara howard
    Replies:
    0
    Views:
    255
    ara howard
    Oct 28, 2003
Loading...

Share This Page