snapshot reading a fast changing file

[Ralf]

Hi,
there is a small text file (about 2 KB) completely recreated
every second by some other process.
I need to read a snapshot of this file.
What is the best method to do this?

Currently I read it as fast as possible line by line
and save it in a string vector.
There must be some faster and eleganter way I think
using memory stream etc. but haven't figured out how to do it.
Can someone please help? TIA

 

[Pascal J. Bourguignon]

Hi,
there is a small text file (about 2 KB) completely recreated
every second by some other process.
I need to read a snapshot of this file.
What is the best method to do this?

Currently I read it as fast as possible line by line
and save it in a string vector.
There must be some faster and eleganter way I think
using memory stream etc. but haven't figured out how to do it.
Can someone please help? TIA

It depends on the meaning of "recreated".

If the writer keeps the file open and truncates and rewrites it,
without closing it, it's rather hard. But there's a low probability
that being true.

So lets concentrated on the other cases:

If the writer closes the file and reopens it, overwritting, (therefore
it's still the same i-node everytime), you will have to steal it under
it.

The writer may also delete the file before recreating it. Then we
will have a different i-node everytime.


The basic idea is to keep a reference to the i-node, and remove it
from the directory (unlinking it).

const char* path="/tmp/the-snapshoot-file";
while(1){
int f=open(path,O_RDONLY);
if(f>=0){
int res=unlink(path);
sleep(1); /* we wait to let the other process fill the file.
It would be better to have a positive IPC between
the processes... */
unsigned char buffer[4096];
int size=read(f,buffer,sizeof(buffer));
have_fun(buffer,size);
close(f);
}
}

If the writer opens the file in overwrite mode, and we open and unlink
it while it is writting, then when it closes the file the only
reference to the i-node will be f. We sleep for the period to be sure
writing is complete, but on linux we could use inotify to get the
IN_CLOSE_WRITE event (in anycase we'd need a timeout for the case
where the writer has closed the file before we complete the unlink).

If the writer deletes the file, either we have it open and either us
or the writer will fail in unlink, or we try to open between the
deletion and the creation, so we try again immediately. Here, we could
also use inotify listening for an IN_CREATE event instead of polling
actively like this.

 

[Ralf]

"Eleganter?"

If you're on Unix (or at least POSIX), you may be interested in read(2).

Yes, it's Linux. Using read() is ok, but then I would like
to read from that memory location line-by-line using stream operations.
How can this be done?

If possible, have the writing process ("producer") write to a temporary
file, then move the temporary file atomically to the "real" path; on
POSIX, see rename(2). Once the reading process ("consumer") opens the
file, the producer will not be able to modify it.

Unfortunately there is no possibililty to control that other process,
it is even a system process.

BTW, would memory mapped file operations would be of any help here?

 

[Ralf]

I'm not sure what you need. Posix read would be an alternative to
istreams, and would read fixed-size buffers rather than variable-length
lines.

I mean reading the whole file using read() or fread() into a memory block
is fine, but for processing the data in this memory block
I would like to process it line-by-line.
I vaguely remember there is a possibility to attach a C++ file stream
to a memory block but don't remember the details... :-(

 

[Ralf]

I mean reading the whole file using read() or fread() into a memory block
is fine, but for processing the data in this memory block I would like
to process it line-by-line.
I vaguely remember there is a possibility to attach a C++ file stream
to a memory block but don't remember the details... :-(

Ok, problem solved.
Actually it was the istringstream class and its getline() member
that I was looking for...