multi-thread and ifstream

[David Breton]

Hi,

I'm a bit confused with ifstream and I was hoping someone here could
illuminate me.

I've got a simple multi-threaded program. The only operation supported
by the program is a DB lookup. The main thread basically listens to an
end user and creates a 'worker' thread to perform the lookup. The
length of the lookups and their frequencies is such that at any given
time there could be many of these 'worker' threads querying the DB at
once. The DB is a set of binary file whose total size is roughly 1TB.
So far no problem.

While trying to improve the response time, it occurred to me to create a
file which caches frequently asked queries. My cache file is 85MB. I
define it like this:

ifstream cache_file;
cache_file.open(cache_filename.c_str(), ios::in|ios::binary);

The 'worker' threads need to be able to do this:

cache_file.seekg(offset);
cache_file.read(buffer, 8);

Here is the bit that confuses me.

Case 1) Suppose I define the cache_file in the main thread and let the
worker threads share it then I think I would have a problem in the
following situation, am I right?
thread1: seekg(offset1)
thread2: seekg(offset2)
thread1: read(buffer, 8) <-- Is thread1 reading from offset1 or offset2?

Case 2) Suppose that instead, I define the cache_file ifstream locally
in each worker thread. Wouldn't it be possible that I end up loading
multiple copies of my 85MB cache file into memory?

This is what I mean about being confused about ifstream. On the one
hand I don't want it possible to have more than one copy of the cache
file in memory, on the other hand I don't really want to revert to using
mutex locks to read. So what's the solution?

Any help would be appreciated
Thanks.

David

 

[James Kanze]

I'm a bit confused with ifstream and I was hoping someone here
could illuminate me.

I've got a simple multi-threaded program. The only operation
supported by the program is a DB lookup. The main thread
basically listens to an end user and creates a 'worker' thread
to perform the lookup. The length of the lookups and their
frequencies is such that at any given time there could be many
of these 'worker' threads querying the DB at once. The DB is
a set of binary file whose total size is roughly 1TB. So far
no problem.

While trying to improve the response time, it occurred to me
to create a file which caches frequently asked queries. My
cache file is 85MB. I define it like this:

ifstream cache_file;
cache_file.open(cache_filename.c_str(), ios::in|ios::binary);

The 'worker' threads need to be able to do this:

cache_file.seekg(offset);
cache_file.read(buffer, 8);

Here is the bit that confuses me.

Case 1) Suppose I define the cache_file in the main thread and
let the worker threads share it then I think I would have a
problem in the following situation, am I right?
thread1: seekg(offset1)
thread2: seekg(offset2)
thread1: read(buffer, 8) <-- Is thread1 reading from offset1 or offset2?

Probably. You'd have to read the documentation for your
implementation (the standard doesn't mention threads---yet), but
I don't see how it could be otherwise.

Case 2) Suppose that instead, I define the cache_file ifstream
locally in each worker thread. Wouldn't it be possible that I
end up loading multiple copies of my 85MB cache file into
memory?

Multiple copies of a given window into the file, perhaps. Each
thread will have its own buffer. (Again, probably. The
standard doesn't say anything about this, but I can't imagine an
implementation where this wasn't the case.)

This is what I mean about being confused about ifstream. On
the one hand I don't want it possible to have more than one
copy of the cache file in memory, on the other hand I don't
really want to revert to using mutex locks to read. So what's
the solution?

As long as you're only reading, there shouldn't be any problem.
To be effective, however, you'll probably end up writing to the
cache as well, and there, you will need locks, and some sort of
shared buffer.