Efficient 'logging' mechanism?

[idesilva]

Hi,

I have an application which sends/receives messages at a very high
rate. This app needs to 'log' the contents of these messages. Since the
msg rate is high, 'logging' each and every msg to a disk-file reduces
the app's performace significantly. I want to device a 'logging'
mechanism where the 'log' is actually buffered and it will be written
to the disk periodically. The time of writing to disk should be
controlled by the app rather than by the OS. At times, I also want to
reset the in-memory log without writing to disk.

Currently I use an 'ofstream' object for this purpose. But it does not
allow me to *control* buffering. I tried out another option as follows.

I extended the MFC's CMemFile class.

class CMemLogFile : public CMemFile
{
public:
CMemLogFile(CString sName);
virtual ~CMemLogFile();

// Base class overrides
void Close();

void Truncate();
void WriteToDisk();

private:
CFile m_file;
CString m_sName;
};

CMemLogFile::CMemLogFile(CString sName)
{
m_sName = sName;
m_file.Open(sName, CFile::modeCreate|CFile::modeWrite);
}

void CMemLogFile::Close()
{
m_file.Close();
CMemFile::Close();
}

void CMemLogFile::Truncate()
{
this->SetLength(0);
}

void CMemLogFile::WriteToDisk()
{
char buf[READ_LEN] = "";
this->SeekToBegin();

int iBytesRead = 0;

do
{
iBytesRead = this->Read(buf, READ_LEN);
m_file.Write(buf, iBytesRead);

} while ( iBytesRead >= READ_LEN);

m_file.Flush();
}

This works fine except for the fact that I don't have the advance
formatting options such as setw() that ofstream provides.

My questions are:
1. Can I control the buffering of an ofstream? (Not using 'endl' and
the default state 'nounitbuf' doesn't seem to work)

2. Failing (1), is there a way that we can increase the buffer size of
an ofstream buffer?

3. How can you reset an ofstream buffer? (Just like CFile::SetLength(0)
truncates)

 

[Siemel Naran]

Currently I use an 'ofstream' object for this purpose. But it does not
allow me to *control* buffering. I tried out another option as follows.

If you want to increase the length of the buffer, take a look at function
streambuf::setp(char * begin, char * end). If I remember correctly, the
streambuf does not own the buffer, so you're responsible for deleting the

My questions are:
1. Can I control the buffering of an ofstream? (Not using 'endl' and
the default state 'nounitbuf' doesn't seem to work)

What kind of control are you after? The endl function flushes the stream,
that is writes the buffer to the actual output file. If nounitbuf is on (or
off, I don't remember), then every write will always flush the buffer.

2. Failing (1), is there a way that we can increase the buffer size of
an ofstream buffer?

3. How can you reset an ofstream buffer? (Just like CFile::SetLength(0)
truncates)

One calls functions like ostream::setp.

 

[Saurabh Aggrawal]

Hi,
I think you should put two threads, one would be the main thread for
processing the message and other would create the log file. Make one
queue in each of the thread. As soon as you get the message push it to
the queue of both the threads. Or you can use the command pattern, as
you like.. I think this will substantially reduce the pressure from
the main thread.

Saurabh Aggrawal
Sr. S/w Programmer

 

[=?iso-8859-1?q?Stephan_Br=F6nnimann?=]

Hi,

I have an application which sends/receives messages at a very high
rate. This app needs to 'log' the contents of these messages. Since the
msg rate is high, 'logging' each and every msg to a disk-file reduces
the app's performace significantly. I want to device a 'logging'
mechanism where the 'log' is actually buffered and it will be written
to the disk periodically. The time of writing to disk should be
controlled by the app rather than by the OS. At times, I also want to
reset the in-memory log without writing to disk.

Currently I use an 'ofstream' object for this purpose. But it does not
allow me to *control* buffering. I tried out another option as follows.

I extended the MFC's CMemFile class.

class CMemLogFile : public CMemFile
{
public:
CMemLogFile(CString sName);
virtual ~CMemLogFile();

// Base class overrides
void Close();

void Truncate();
void WriteToDisk();

private:
CFile m_file;
CString m_sName;
};

CMemLogFile::CMemLogFile(CString sName)
{
m_sName = sName;
m_file.Open(sName, CFile::modeCreate|CFile::modeWrite);
}

void CMemLogFile::Close()
{
m_file.Close();
CMemFile::Close();
}

void CMemLogFile::Truncate()
{
this->SetLength(0);
}

void CMemLogFile::WriteToDisk()
{
char buf[READ_LEN] = "";
this->SeekToBegin();

int iBytesRead = 0;

do
{
iBytesRead = this->Read(buf, READ_LEN);
m_file.Write(buf, iBytesRead);

} while ( iBytesRead >= READ_LEN);

m_file.Flush();
}

This works fine except for the fact that I don't have the advance
formatting options such as setw() that ofstream provides.

My questions are:
1. Can I control the buffering of an ofstream? (Not using 'endl' and
the default state 'nounitbuf' doesn't seem to work)

2. Failing (1), is there a way that we can increase the buffer size of
an ofstream buffer?

3. How can you reset an ofstream buffer? (Just like CFile::SetLength(0)
truncates)

Don't try to play around with the internals of ofstream: you're
likely to move into a area of C++ where most of the things are
implementation specific.
An alternative approach is to provide the necessary output
operators in the class CMemLogFile itself and temporary write
to std:stringstream member

<pre>
class CMemLogFile {
public:

template<typename T> Logfile& operator<<(const T& t);
Logfile& operator<<(std:stream& (*func)(std:stream&));
Logfile& operator<<(std::ios& (*func)(std::ios&));
Logfile& operator<<(std::ios_base& (*func)(std::ios_base&));

private:
std:stringstream buffer_;
};

Logfile& Logfile:perator<<(const T& t)
{
buffer_ << t;
return *this;
}

Logfile:perator<<(std:stream& (*func)(std:stream&))
{
buffer_ << func;
return *this;
}
</pre>

and similar for the 2 other output operators.

Hope this helps,
Stephan Brönnimann
(e-mail address removed)
http://www.osb-systems.com
Open source rating and billing engine for communication networks.

 

[Dietmar Kuehl]

follows.

If you want to increase the length of the buffer, take a look at function
streambuf::setp(char * begin, char * end).

'setp()' is a protected member function and not intended for user
access.
In fact, I would expect that a file buffer will fail with some rather
strange problems if you start fiddling with its internal buffering. I
guess you ment 'setbuf()' and 'pubsetbuf()'. Note, however, that these
functions are not required to have any effect except that
'pubsetbuf(0, 0)' is supposed to turn off buffering for file buffers.

If I remember correctly, the
streambuf does not own the buffer, so you're responsible for deleting the
memory (either by calling delete[], using std::vector<char>, etc).

If you are referring to 'pubsetbuf()', this is true. However, I
consider
this function to be mostly useless. In particular, it is useless with
respect to controlling the buffer size (e.g. my implementation
deliberately
ignores any passed buffer).

What kind of control are you after? The endl function flushes the stream,
that is writes the buffer to the actual output file. If nounitbuf is on (or
off, I don't remember), then every write will always flush the

buffer.

If 'unitbuf' is set the stream flushes the the buffer after each
operation
(i.e. when the internal 'sentry' object is destructed).

One calls functions like ostream::setp.

There is no member 'setp()' in 'ostream'. There is, however, a
protected
member 'setp()' in stream buffers.

I would implement a specialized stream buffer for this which just
buffers
the output until it is explicitly flushed. If accidental 'endl's should
also not flush the buffer, flushing could be triggered using a separate
function. Of course, the stream buffer would use a file stream buffer
underneath.

 

[msalters]

Hi,

I have an application which sends/receives messages at a very high
rate. This app needs to 'log' the contents of these messages. Since the
msg rate is high, 'logging' each and every msg to a disk-file reduces
the app's performace significantly. I want to device a 'logging'
mechanism where the 'log' is actually buffered and it will be written
to the disk periodically. The time of writing to disk should be
controlled by the app rather than by the OS. At times, I also want to
reset the in-memory log without writing to disk.

Currently I use an 'ofstream' object for this purpose. But it does not
allow me to *control* buffering. ...

Incorrect assumption.
[...]

My questions are:
1. Can I control the buffering of an ofstream? (Not using 'endl' and
the default state 'nounitbuf' doesn't seem to work)

Yes. Insert your own buffer mechanism between the ofstream and its
std::streambuf. The member you need is .rdbuf, use the old one
as the output destination for your own streambuf.

3. How can you reset an ofstream buffer? (Just like CFile::SetLength(0)
truncates)

It's your own object. Just like you want to. Since you probably will
keep a std::vector<char> in it, it's probably vector<char>::clear().
Regards,
Michiel Salters

 

[Siemel Naran]

Siemel Naran wrote:

If you are referring to 'pubsetbuf()', this is true. However, I
consider
this function to be mostly useless. In particular, it is useless with
respect to controlling the buffer size (e.g. my implementation
deliberately
ignores any passed buffer).

So what's the point of the function?

There is no member 'setp()' in 'ostream'. There is, however, a
protected
member 'setp()' in stream buffers.

Oops, I mean seekp.

I would implement a specialized stream buffer for this which just
buffers
the output until it is explicitly flushed. If accidental 'endl's should
also not flush the buffer, flushing could be triggered using a separate
function. Of course, the stream buffer would use a file stream buffer
underneath.

How would you distinguish o << flush from o.flush()?