plauged by seg faults

[cerr]

Hi There,

My daemon app keeps terminating and i'm trying to figure out why and
where. I have a good idea tho. I'm using a push - pop FIFO system for
log messages. I have a syslog message after the push and after the pop
function and see both appearingh all the time and hence i think that
both, push() and pop() works fine. However, the messages get popped
off in a thread from where i send the data off. I'm using following
code:

pthread_mutex_lock(&log_mtx);
Qstr=pop(&msglist, &locLen);
syslog(LOG_ERR, "DUBUG: popped succesfully!");
strncpy(res,Qstr,LOG_BUF_SIZE);
res[LOG_BUF_SIZE]='\0';
strcat(res,"\n");
pthread_mutex_unlock(&log_mtx);
if (log_sock>-1) {
sndlen=strlen(res);
reclen=send(log_sock, res, sndlen, 0);
if (reclen != sndlen) {
if (reclen==-1){
syslog(LOG_ERR, "nlog: send failed...,errno: %s!", strerror
( errno ));
}
syslog(LOG_ERR, "nlog: \"%s\" not sent, strlen(%d) - sent %d",res,
sndlen, reclen);
}
else {
syslog(LOG_ERR, "DEBUG: sent okay, receive ack");

The last message I see in syslog before my app crashes is "DUBUG:
popped succesfully!"
Now the declaration of those variables are as follows:
char res[LOG_BUF_SIZE+2]={0};
char *Qstr;
int locLen;
int sndlen;
while msglist is from type char** and QLen from type int and declared
globally.
LOG_BUF_SIZE is a define and set to 512.

Does anyone have a clue where my app is terminating and or should i be
looking somewhere else all together? The system worked fine before i
implemented this new FIFO system...
Thanks for hints and suggestions!
Ron

 

[Beej Jorgensen]

pthread_mutex_lock(&log_mtx);
Qstr=pop(&msglist, &locLen);
syslog(LOG_ERR, "DUBUG: popped succesfully!");
strncpy(res,Qstr,LOG_BUF_SIZE);

Based on the information you've given, I'd suspect that pop() was
returning an invalid pointer or NULL...? Maybe you could log the value
of Qstr just to be sure. (The crash could absolutely be elsewhere,
though.)

Can you get a core dump? If you, you might be able to bring it up in a
debugger and have it show you exactly the line it's crashing on:

http://beej.us/guide/bggdb/#coredump

-Beej

 

[David Resnick]

Hi There,

My daemon app keeps terminating and i'm trying to figure out why and
where. I have a good idea tho. I'm using a push - pop FIFO system for
log messages. I have a syslog message after the push and after the pop
function and see both appearingh all the time and hence i think that
both, push() and pop() works fine. However, the messages get popped
off in a thread from where i send the data off. I'm using following
code:

pthread_mutex_lock(&log_mtx);
      Qstr=pop(&msglist, &locLen);
      syslog(LOG_ERR, "DUBUG: popped succesfully!");
      strncpy(res,Qstr,LOG_BUF_SIZE);
      res[LOG_BUF_SIZE]='\0';
      strcat(res,"\n");
      pthread_mutex_unlock(&log_mtx);
      if (log_sock>-1) {
        sndlen=strlen(res);
        reclen=send(log_sock, res, sndlen, 0);
        if (reclen != sndlen) {
          if (reclen==-1){
            syslog(LOG_ERR, "nlog: send failed...,errno: %s!", strerror
( errno ));
          }
          syslog(LOG_ERR, "nlog: \"%s\" not sent, strlen(%d) - sent %d",res,
sndlen, reclen);
        }
        else {
          syslog(LOG_ERR, "DEBUG: sent okay, receive ack");

The last message I see in syslog before my app crashes is "DUBUG:
popped succesfully!"
Now the declaration of those variables are as follows:
  char res[LOG_BUF_SIZE+2]={0};
  char *Qstr;
  int locLen;
  int sndlen;
while msglist is from type char** and QLen from type int and declared
globally.
LOG_BUF_SIZE is a define and set to 512.

Does anyone have a clue where my app is terminating and or should i be
looking somewhere else all together? The system worked fine before i
implemented this new FIFO system...
Thanks for hints and suggestions!
Ron

I'd check the return value of pop. Is it a reasonable pointer? NULL?

If that doesn't solve it, I'd try tools, like valgrind. Or running in
a debugger, or enabling cores.

-David

 

[cerr]

Based on the information you've given, I'd suspect that pop() was
returning an invalid pointer or NULL...?  

Yup, added a
if (Qstr!=NULL)
syslog(LOG_ERR, "DUBUG: popped succesfully!");
else {
syslog(LOG_ERR, "DUBUG: popped not succesful!");
break; /* the break would break back to the thread's while(1) loop*/
}

Maybe you could log the value
of Qstr just to be sure.  (The crash could absolutely be elsewhere,
though.)

Can you get a core dump?  If you, you might be able to bring it up in a
debugger and have it show you exactly the line it's crashing on:

http://beej.us/guide/bggdb/#coredump

Here it says that it would say smthng like "Segmentation fault (core
dumped)" but i don't even get ANY back log on the screen cause the app
is running in daemon mode...
I unfortunately am not able to easily launch it in a debugger because
my app is printing a timestamp to a semaphore which is read out by a
watchdog. In case it doesn't get updated, the watchdog immediately
reboots my system. On top of that I have a hardware heartbeat that
needs to be provided in order to keep the power up and running....

 

[cerr]

Hi There,

My daemon app keeps terminating and i'm trying to figure out why and
where. I have a good idea tho. I'm using a push - pop FIFO system for
log messages. I have a syslog message after the push and after the pop
function and see both appearingh all the time and hence i think that
both, push() and pop() works fine. However, the messages get popped
off in a thread from where i send the data off. I'm using following
code:

pthread_mutex_lock(&log_mtx);
      Qstr=pop(&msglist, &locLen);
      syslog(LOG_ERR, "DUBUG: popped succesfully!");
      strncpy(res,Qstr,LOG_BUF_SIZE);
      res[LOG_BUF_SIZE]='\0';
      strcat(res,"\n");
      pthread_mutex_unlock(&log_mtx);
      if (log_sock>-1) {
        sndlen=strlen(res);
        reclen=send(log_sock, res, sndlen, 0);
        if (reclen != sndlen) {
          if (reclen==-1){
            syslog(LOG_ERR, "nlog: send failed...,errno: %s!", strerror
( errno ));
          }
          syslog(LOG_ERR, "nlog: \"%s\" not sent, strlen(%d) - sent %d",res,
sndlen, reclen);
        }
        else {
          syslog(LOG_ERR, "DEBUG: sent okay, receive ack");

The last message I see in syslog before my app crashes is "DUBUG:
popped succesfully!"
Now the declaration of those variables are as follows:
  char res[LOG_BUF_SIZE+2]={0};
  char *Qstr;
  int locLen;
  int sndlen;
while msglist is from type char** and QLen from type int and declared
globally.
LOG_BUF_SIZE is a define and set to 512.

Does anyone have a clue where my app is terminating and or should i be
looking somewhere else all together? The system worked fine before i
implemented this new FIFO system...
Thanks for hints and suggestions!
Ron

I'd check the return value of pop.  Is it a reasonable pointer?  NULL?

Yep as Beej had suggested, I included a check for NULL:
if (Qstr!=NULL)
syslog(LOG_ERR, "DUBUG: popped succesfully!");
else {
syslog(LOG_ERR, "DUBUG: popped not succesful!");
break;
}

If that doesn't solve it, I'd try tools, like valgrind.  Or running in
a debugger, or enabling cores.

will try to use valgrind - don't know if it will run on my embedded
platform tho...
Debugger unfortunately will not be an option.... enabling cores - what
do you mean?

Thanks!
Ron

 

[cerr]

Hi There,
My daemon app keeps terminating and i'm trying to figure out why and
where. I have a good idea tho. I'm using a push - pop FIFO system for
log messages. I have a syslog message after the push and after the pop
function and see both appearingh all the time and hence i think that
both, push() and pop() works fine. However, the messages get popped
off in a thread from where i send the data off. I'm using following
code:
pthread_mutex_lock(&log_mtx);
      Qstr=pop(&msglist, &locLen);
      syslog(LOG_ERR, "DUBUG: popped succesfully!");
      strncpy(res,Qstr,LOG_BUF_SIZE);
      res[LOG_BUF_SIZE]='\0';
      strcat(res,"\n");
      pthread_mutex_unlock(&log_mtx);
      if (log_sock>-1) {
        sndlen=strlen(res);
        reclen=send(log_sock, res, sndlen, 0);
        if (reclen != sndlen) {
          if (reclen==-1){
            syslog(LOG_ERR, "nlog: send failed...,errno: %s!", strerror
( errno ));
          }
          syslog(LOG_ERR, "nlog: \"%s\" not sent, strlen(%d) - sent %d",res,
sndlen, reclen);
        }
        else {
          syslog(LOG_ERR, "DEBUG: sent okay, receive ack");
The last message I see in syslog before my app crashes is "DUBUG:
popped succesfully!"
Now the declaration of those variables are as follows:
  char res[LOG_BUF_SIZE+2]={0};
  char *Qstr;
  int locLen;
  int sndlen;
while msglist is from type char** and QLen from type int and declared
globally.
LOG_BUF_SIZE is a define and set to 512.
Does anyone have a clue where my app is terminating and or should i be
looking somewhere else all together? The system worked fine before i
implemented this new FIFO system...
Thanks for hints and suggestions!
Ron

I'd check the return value of pop.  Is it a reasonable pointer?  NULL?

Yep as Beej had suggested, I included a check for NULL:
      if (Qstr!=NULL)
        syslog(LOG_ERR, "DUBUG: popped succesfully!");
      else {
        syslog(LOG_ERR, "DUBUG: popped not succesful!");
        break;
      }

If that doesn't solve it, I'd try tools, like valgrind.  Or running in
a debugger, or enabling cores.

will try to use valgrind - don't know if it will run on my embedded
platform tho...

I just compiled valgrind on our dev system but no way i'll be able to
copy it onto our target system as the /opt/valgrind-3.2.3 directory is
81MB big... ... ah, i'm getting desperate.... what can i do?
Other hints, recommodations and suggestions are highly appreciated!

Thanks!
Ron

 

[Seebs]

I just compiled valgrind on our dev system but no way i'll be able to
copy it onto our target system as the /opt/valgrind-3.2.3 directory is
81MB big... ... ah, i'm getting desperate.... what can i do?
Other hints, recommodations and suggestions are highly appreciated!

The threading stuff will be rough, and I can't help much with it.

The first thing I'd do, honestly, would be to run the app in a debugger.
If you can get gdb or the equivalent on your target, then you should be able
to find out where it's crashing and in which thread.

-s

 

[Ian Collins]

I just compiled valgrind on our dev system but no way i'll be able to
copy it onto our target system as the /opt/valgrind-3.2.3 directory is
81MB big... ... ah, i'm getting desperate.... what can i do?
Other hints, recommodations and suggestions are highly appreciated!

Test your code on a hosted platform before running it on the embedded
target. Assuming your target is Linux, this shouldn't be too hard.

 

[cerr]

The threading stuff will be rough, and I can't help much with it.

The first thing I'd do, honestly, would be to run the app in a debugger.
If you can get gdb or the equivalent on your target, then you should be able
to find out where it's crashing and in which thread.

Well yeah, that's the problem, I've tried using the remote GDB feature
which would work nicely but all the watchdogs prevent me from using
it, once the process halt, my system reboots rightaway....
I'm starting to think that my push() or my pop() function may cause
the problem evene tho it still returns from each of them.
I may paste them here for display (someone may be able to help...):

int push(char ***list, char *str, int curlen){
char **temp;
int i=0;

temp = realloc((*list),(curlen+1)*sizeof(*list));
if (temp==NULL){
syslog(LOG_ERR, "push(): Error reallocating memory for msglist");
for (i=curlen;i>=0;i--) {
free((*list));
QLen--;
}
free(list);
return -1;
}
(*list)=temp;
(*list)[curlen]=malloc (strlen(str)+1);
strcpy((*list)[curlen],str);

return ++curlen;
}
//-------------------------------------------------------
char *pop(char ***list, int *size) {
if(!*size) return 0;

char *first = (*list)[0];

memmove(*list, *list + 1, (*size) * sizeof(**list));
(*size) --;
(*list) = realloc(*list, (*size) * sizeof(**list));

return first;
}
//-------------------------------------------------------

Then I'm calling the push() like this:
pthread_mutex_lock(&log_mtx);
strcpy(string,Header);
strcat(string,buf);
syslog(level, buf);
QLen=push(&msglist, string, QLen);
//WriteToFile(string);
syslog(LOG_ERR, "DUBUG: pushed succesfully!");
pthread_mutex_unlock(&log_mtx);
and the pop() call is pasted above.


Thanks x1000 for all the ideas and hints!

 

[Seebs]

Well yeah, that's the problem, I've tried using the remote GDB feature
which would work nicely but all the watchdogs prevent me from using
it, once the process halt, my system reboots rightaway....

Disable watchdogs for debugging.

int push(char ***list, char *str, int curlen){
char **temp;
int i=0;

temp = realloc((*list),(curlen+1)*sizeof(*list));

Although it almost certainly doesn't matter, I'm pretty sure you want
sizeof(**list) here for consistency.

I would also say that this function is crying out to be cleaned
up, in a couple of very significant ways. You should not have the
length be a separate argument; "list" should be of a type such that
you can always do push(list, value); without having to know.

//-------------------------------------------------------
char *pop(char ***list, int *size) {
if(!*size) return 0;

char *first = (*list)[0];

memmove(*list, *list + 1, (*size) * sizeof(**list));
(*size) --;
(*list) = realloc(*list, (*size) * sizeof(**list));

return first;
}

Same issues here -- size tracked elsewhere.

Suggestion (totally untested, etc.):

struct list {
struct list *next;
char *s;
};

void push(struct list **l, char *s) {
struct list *new;
if (!l)
return;
new = malloc(sizeof(*new));
new->next = *l;
new->s = s;
*l = new;
}

char *pop(struct list **l) {
struct list *old;
char *s;
if (!l)
return NULL;
old = *l;
if (old) {
*l = old->next;
s = old->s;
free(old);
return s;
} else {
return NULL;
}
}

Usage:
struct list *l = NULL;
push(&l, "foo");
push(&l, "bar");
pop(&l); /* => "bar" */
pop(&l); /* => "foo" */
pop(&l); /* => NULL */

Your current model is HUGELY vulnerable to possible clashes involving
the current length getting out of sync somehow, but you never need it if
you use an actual list.

If you do want to use an array, think about something like this:
struct list {
char **data;
size_t curlen;
size_t allocated;
};

void push(struct list *l, char *s) {
if (l->curlen >= l->allocated) {
char **new = malloc(allocated * 2 * sizeof(*data));
if (new) {
memcpy(new, data, curlen * sizeof(*data));
free(l->data);
l->data = new;
l->allocated *= 2;
} else {
assert(!"I wrote an error handler");
}
}
l->data[l->curlen++] = s;
}

char *pop(struct list *l) {
if (l->curlen > 0) {
return l->data[--l->curlen];
}
}

(And yeah, this code is probably full of errors.)

-s

 

[Beej Jorgensen]

Yup, added a
if (Qstr!=NULL)
syslog(LOG_ERR, "DUBUG: popped succesfully!");
else {
syslog(LOG_ERR, "DUBUG: popped not succesful!");
break; /* the break would break back to the thread's while(1) loop*/
}

Did that "fix" it?

Here it says that it would say smthng like "Segmentation fault (core
dumped)" but i don't even get ANY back log on the screen cause the app
is running in daemon mode...

This is better asked on comp.unix.programmer, but you can use
setrlimit() to unlimit the writing of core files. If it's a daemon, it
should be cwd /, but it might not be able to write the file there
depending on which user its running as... but I've never tried any of
this with a daemon so YMMV.

-Beej

 

[Moi]

Hi There,

My daemon app keeps terminating and i'm trying to figure out why and
where. I have a good idea tho. I'm using a push - pop FIFO system for
log messages. I have a syslog message after the push and after the pop
function and see both appearingh all the time and hence i think that
both, push() and pop() works fine. However, the messages get popped off
in a thread from where i send the data off. I'm using following code:

pthread_mutex_lock(&log_mtx);
Qstr=pop(&msglist, &locLen);
syslog(LOG_ERR, "DUBUG: popped succesfully!");
strncpy(res,Qstr,LOG_BUF_SIZE);
res[LOG_BUF_SIZE]='\0';

This looks wrong. I think you mean:
res[LOG_BUF_SIZE-1] = 0;

But you would also mean that your strncpy should copy one character less.

strcat(res,"\n");

This looks clumsy to me.
It also implies that your strncpy should copy yet one less character.

pthread_mutex_unlock(&log_mtx);
if (log_sock>-1) {

You could have tested this earlier and bailed out ASAP.

sndlen=strlen(res);

This is ugly. IMHO.

HTH,
AvK

 

[David Resnick]

Hi There,
My daemon app keeps terminating and i'm trying to figure out why and
where. I have a good idea tho. I'm using a push - pop FIFO system for
log messages. I have a syslog message after the push and after the pop
function and see both appearingh all the time and hence i think that
both, push() and pop() works fine. However, the messages get popped
off in a thread from where i send the data off. I'm using following
code:
pthread_mutex_lock(&log_mtx);
      Qstr=pop(&msglist, &locLen);
      syslog(LOG_ERR, "DUBUG: popped succesfully!");
      strncpy(res,Qstr,LOG_BUF_SIZE);
      res[LOG_BUF_SIZE]='\0';
      strcat(res,"\n");
      pthread_mutex_unlock(&log_mtx);
      if (log_sock>-1) {
        sndlen=strlen(res);
        reclen=send(log_sock, res, sndlen, 0);
        if (reclen != sndlen) {
          if (reclen==-1){
            syslog(LOG_ERR, "nlog: send failed...,errno: %s!", strerror
( errno ));
          }
          syslog(LOG_ERR, "nlog: \"%s\" not sent, strlen(%d) - sent %d",res,
sndlen, reclen);
        }
        else {
          syslog(LOG_ERR, "DEBUG: sent okay, receive ack");
The last message I see in syslog before my app crashes is "DUBUG:
popped succesfully!"
Now the declaration of those variables are as follows:
  char res[LOG_BUF_SIZE+2]={0};
  char *Qstr;
  int locLen;
  int sndlen;
while msglist is from type char** and QLen from type int and declared
globally.
LOG_BUF_SIZE is a define and set to 512.
Does anyone have a clue where my app is terminating and or should i be
looking somewhere else all together? The system worked fine before i
implemented this new FIFO system...
Thanks for hints and suggestions!
Ron
I'd check the return value of pop.  Is it a reasonable pointer?  NULL?

Yep as Beej had suggested, I included a check for NULL:
      if (Qstr!=NULL)
        syslog(LOG_ERR, "DUBUG: popped succesfully!");
      else {
        syslog(LOG_ERR, "DUBUG: popped not succesful!");
        break;
      }

will try to use valgrind - don't know if it will run on my embedded
platform tho...

I just compiled valgrind on our dev system but no way i'll be able to
copy it onto our target system as the /opt/valgrind-3.2.3 directory is
81MB big... ... ah, i'm getting desperate.... what can i do?
Other hints, recommodations and suggestions are highly appreciated!

Thanks!
Ron

I assumed you were on a linux flavor. Enabling cores means usually
setting ulimit to allow cores to dump on fatal signals, probably not
relevant to your embedded system. Umm, add more syslogs, repeat as
needed, to narrow down your problem seems like a good bet. You should
be able to figure out exactly what bad pointer causes the segfault,
and trace the problem that way.

You could crash on push if the malloc failed, which I assume is
possible on your limited memory embedded system. You could be messed
up on pop if somehow a shrinking realloc fails.

-David

 

[Ben Bacarisse]

My daemon app keeps terminating and i'm trying to figure out why and
where. I have a good idea tho. I'm using a push - pop FIFO system for
log messages. I have a syslog message after the push and after the pop
function and see both appearingh all the time and hence i think that
both, push() and pop() works fine. However, the messages get popped off
in a thread from where i send the data off. I'm using following code:

pthread_mutex_lock(&log_mtx);
Qstr=pop(&msglist, &locLen);
syslog(LOG_ERR, "DUBUG: popped succesfully!");
strncpy(res,Qstr,LOG_BUF_SIZE);
res[LOG_BUF_SIZE]='\0';

This looks wrong. I think you mean:
res[LOG_BUF_SIZE-1] = 0;

Yes, it *looks* wrong, but...

But you would also mean that your strncpy should copy one character less.


This looks clumsy to me.
It also implies that your strncpy should copy yet one less
character.

res is declared to be BUF_SIZE+2 characters long in code now snipped.
The real problem for anyone reading this is the BUF_SIZE is
miss-named. It is not the size of the buffer.

<snip>

 

[Squeamizh]

Disable watchdogs for debugging.  

This is the real answer for the OP. Even if you solve this problem
without gdb, you'll run into another unsolvable problem sooner or
later. Disabling the watchdog timer for debugging is mandatory.

 

[Moi]

On Tue, 05 Jan 2010 11:40:30 -0800, cerr wrote:

off in a thread from where i send the data off. I'm using following
code:

pthread_mutex_lock(&log_mtx);
Qstr=pop(&msglist, &locLen);
syslog(LOG_ERR, "DUBUG: popped succesfully!");
strncpy(res,Qstr,LOG_BUF_SIZE);
res[LOG_BUF_SIZE]='\0';

This looks wrong. I think you mean:
res[LOG_BUF_SIZE-1] = 0;

Yes, it *looks* wrong, but...



res is declared to be BUF_SIZE+2 characters long in code now snipped.
The real problem for anyone reading this is the BUF_SIZE is miss-named.
It is not the size of the buffer.

Yes, I could not agree more.
What the OP also does wrong is putting the declaration of res[]
_after_ its use.
This is disastrous for lazy people like me, who bail out on the first
spotted error, or the first page with (assumed) errors on it.

Also, I think the usage of the mutex-locks is rather strange.
I would expect the locks to be _inside_ the push() pop() functions.
The locks also do not preserves the order of the messages, if that would
be important to the OP.

AvK

 

[cerr]

This is the real answer for the OP.  Even if you solve this problem
without gdb, you'll run into another unsolvable problem sooner or
later.  Disabling the watchdog timer for debugging is mandatory.

Exactly but I don't know yet how i can do this, need to figure that
out first, it seems as if there's several processes that depend on
each other and if one fails, the system reboots...

 

[cerr]

Hi There,

My daemon app keeps terminating and i'm trying to figure out why and
where. I have a good idea tho. I'm using a push - pop FIFO system for
log messages. I have a syslog message after the push and after the pop
function and see both appearingh all the time and hence i think that
both, push() and pop() works fine. However, the messages get popped off
in a thread from where i send the data off. I'm using following code:

pthread_mutex_lock(&log_mtx);
      Qstr=pop(&msglist, &locLen);
      syslog(LOG_ERR, "DUBUG: popped succesfully!");
      strncpy(res,Qstr,LOG_BUF_SIZE);
      res[LOG_BUF_SIZE]='\0';

This looks wrong. I think you mean:
        res[LOG_BUF_SIZE-1] = 0;

But you would also mean that your strncpy should copy one character less.

why would this be LOG_BUF_SIZE-1? I defined the variable like
char res[LOG_BUF_SIZE+2]={0};
and hence there's space for a null character at the end...

This looks clumsy to me.
It also implies that your strncpy should copy yet one less character.

yes, you're right here, it should be:
strncpy(res,Qstr,LOG_BUF_SIZE-1);
then if it didn't copy the last null character to close the string i
add it one element before the last:
res[LOG_BUF_SIZE-1]='\0';
because i need to have enough room for a new line:
strcat(res,"\n");

would this do it you think?

You could have tested this earlier and bailed out ASAP.

check if the siocket is alright...yeah i can move this further up.

This is ugly. IMHO.

That may be true and i do appreciate the help i'm getting to clean-up
this code.

Thanks a lot!
Ron

 

[Squeamizh]

Exactly but I don't know yet how i can do this, need to figure that
out first, it seems as if there's several processes that depend on
each other and if one fails, the system reboots...

The watchdog probably has to be initialized at some point during boot
up. Just find that line and comment it out. Anyway, I hate to sound
like a nanny about this, but being able to disable the watchdog timer
really is the only sensible way to develop.

 

[cerr]

On Tue, 05 Jan 2010 11:40:30 -0800, cerr wrote:
off in a thread from where i send the data off. I'm using following
code:
pthread_mutex_lock(&log_mtx);
      Qstr=pop(&msglist, &locLen);
      syslog(LOG_ERR, "DUBUG: popped succesfully!");
      strncpy(res,Qstr,LOG_BUF_SIZE);
      res[LOG_BUF_SIZE]='\0';
This looks wrong. I think you mean:
        res[LOG_BUF_SIZE-1] = 0;

Yes, it *looks* wrong, but...

res is declared to be BUF_SIZE+2 characters long in code now snipped.
The real problem for anyone reading this is the BUF_SIZE is miss-named.
It is not the size of the buffer.

Yes, I could not agree more.
What the OP also does wrong is putting the declaration of res[]
_after_ its use.
This is disastrous for lazy people like me, who bail out on the first
spotted error, or the first page with (assumed) errors on it.

Also, I think the usage of the mutex-locks is rather strange.
I would expect the locks to be _inside_ the push() pop() functions.
The locks also do not preserves the order of the messages, if that would
be important to the OP.

Hm, why would you put them inside? I pout them outside so i can hold
the pop() and push() functions stand-alone, so they can be used in
multi threaded environments but if it's single threaded you don't
require mutexes, eh?
Also why would the order of messages not be preserved? i don't
understand...

Ron