snapshot reading a fast changing file

Discussion in 'C++' started by Ralf, Feb 24, 2009.

  1. Ralf

    Ralf Guest

    Hi,
    there is a small text file (about 2 KB) completely recreated
    every second by some other process.
    I need to read a snapshot of this file.
    What is the best method to do this?

    Currently I read it as fast as possible line by line
    and save it in a string vector.
    There must be some faster and eleganter way I think
    using memory stream etc. but haven't figured out how to do it.
    Can someone please help? TIA
     
    Ralf, Feb 24, 2009
    #1
    1. Advertising

  2. Ralf <> writes:

    > Hi,
    > there is a small text file (about 2 KB) completely recreated
    > every second by some other process.
    > I need to read a snapshot of this file.
    > What is the best method to do this?
    >
    > Currently I read it as fast as possible line by line
    > and save it in a string vector.
    > There must be some faster and eleganter way I think
    > using memory stream etc. but haven't figured out how to do it.
    > Can someone please help? TIA


    It depends on the meaning of "recreated".

    If the writer keeps the file open and truncates and rewrites it,
    without closing it, it's rather hard. But there's a low probability
    that being true.

    So lets concentrated on the other cases:

    If the writer closes the file and reopens it, overwritting, (therefore
    it's still the same i-node everytime), you will have to steal it under
    it.

    The writer may also delete the file before recreating it. Then we
    will have a different i-node everytime.


    The basic idea is to keep a reference to the i-node, and remove it
    from the directory (unlinking it).

    const char* path="/tmp/the-snapshoot-file";
    while(1){
    int f=open(path,O_RDONLY);
    if(f>=0){
    int res=unlink(path);
    sleep(1); /* we wait to let the other process fill the file.
    It would be better to have a positive IPC between
    the processes... */
    unsigned char buffer[4096];
    int size=read(f,buffer,sizeof(buffer));
    have_fun(buffer,size);
    close(f);
    }
    }

    If the writer opens the file in overwrite mode, and we open and unlink
    it while it is writting, then when it closes the file the only
    reference to the i-node will be f. We sleep for the period to be sure
    writing is complete, but on linux we could use inotify to get the
    IN_CLOSE_WRITE event (in anycase we'd need a timeout for the case
    where the writer has closed the file before we complete the unlink).

    If the writer deletes the file, either we have it open and either us
    or the writer will fail in unlink, or we try to open between the
    deletion and the creation, so we try again immediately. Here, we could
    also use inotify listening for an IN_CREATE event instead of polling
    actively like this.

    --
    __Pascal Bourguignon__
     
    Pascal J. Bourguignon, Feb 24, 2009
    #2
    1. Advertising

  3. Ralf

    Ralf Guest

    Jeff Schwab wrote:
    > Ralf wrote:
    >>
    >> there is a small text file (about 2 KB) completely recreated
    >> every second by some other process.
    >> I need to read a snapshot of this file.
    >> What is the best method to do this?
    >>
    >> Currently I read it as fast as possible line by line
    >> and save it in a string vector.
    >> There must be some faster and eleganter

    >
    > "Eleganter?"


    :)

    > If you're on Unix (or at least POSIX), you may be interested in read(2).


    Yes, it's Linux. Using read() is ok, but then I would like
    to read from that memory location line-by-line using stream operations.
    How can this be done?

    >> way I think
    >> using memory stream etc. but haven't figured out how to do it.
    >> Can someone please help? TIA

    >
    > If possible, have the writing process ("producer") write to a temporary
    > file, then move the temporary file atomically to the "real" path; on
    > POSIX, see rename(2). Once the reading process ("consumer") opens the
    > file, the producer will not be able to modify it.


    Unfortunately there is no possibililty to control that other process,
    it is even a system process.

    BTW, would memory mapped file operations would be of any help here?
     
    Ralf, Feb 24, 2009
    #3
  4. Ralf

    Ralf Guest

    Jeff Schwab wrote:
    > Ralf wrote:
    >
    >> Using read() is ok, but then I would like to read from that memory
    >> location line-by-line using stream operations.
    >> How can this be done?

    >
    > I'm not sure what you need. Posix read would be an alternative to
    > istreams, and would read fixed-size buffers rather than variable-length
    > lines.


    I mean reading the whole file using read() or fread() into a memory block
    is fine, but for processing the data in this memory block
    I would like to process it line-by-line.
    I vaguely remember there is a possibility to attach a C++ file stream
    to a memory block but don't remember the details... :-(
     
    Ralf, Feb 24, 2009
    #4
  5. Ralf

    Ralf Guest

    Ralf wrote:
    > Jeff Schwab wrote:
    >> Ralf wrote:
    >>
    >>> Using read() is ok, but then I would like to read from that memory
    >>> location line-by-line using stream operations.
    >>> How can this be done?

    >>
    >> I'm not sure what you need. Posix read would be an alternative to
    >> istreams, and would read fixed-size buffers rather than
    >> variable-length lines.

    >
    > I mean reading the whole file using read() or fread() into a memory block
    > is fine, but for processing the data in this memory block I would like
    > to process it line-by-line.
    > I vaguely remember there is a possibility to attach a C++ file stream
    > to a memory block but don't remember the details... :-(


    Ok, problem solved.
    Actually it was the istringstream class and its getline() member
    that I was looking for... :)
     
    Ralf, Feb 24, 2009
    #5
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. Steven Thomas

    How to open a Snapshot file from ASP.NET

    Steven Thomas, Nov 18, 2004, in forum: ASP .Net
    Replies:
    3
    Views:
    847
    Eliahu
    Nov 19, 2004
  2. Replies:
    0
    Views:
    672
  3. Michele Simionato

    Python is darn fast (was: How fast is Python)

    Michele Simionato, Aug 23, 2003, in forum: Python
    Replies:
    13
    Views:
    569
  4. Juha Nieminen
    Replies:
    22
    Views:
    1,036
    Kai-Uwe Bux
    Oct 12, 2007
  5. Stef Mientki
    Replies:
    8
    Views:
    372
    Stef Mientki
    Jan 24, 2008
Loading...

Share This Page