How to open VERY large file using std::ifstream

C

Charlie

Dear all,

I am currently writting a trace analyzer in C++.
It always fails to open a very large input file (3.7Gb).

I tried on a simple program, same thing happens:
==============================
#include <iostream>
#include <fstream>
using namespace std;

int main(int argc, char *argv[])
{
std::ifstream loop_file(argv[1]);

if (!loop_file)
cout<<"cannot open file: "<<argv[1];
else
cout<<"successfully opened"<<endl;
}
==================
# ./tracer input.data
cannot open file input.data
But it works well on smaller input file, (e.g 200Mb)

I guess there is a limit of size on "std::ifstream" object.
I wonder if there is method I could open this large input file
in C++?

Any suggestion or pointer is highly appreciated!

Many thanks!
Charlie
 
I

Ian McCulloch

Charlie said:
Dear all,

I am currently writting a trace analyzer in C++.
It always fails to open a very large input file (3.7Gb).

I tried on a simple program, same thing happens:
==============================
#include <iostream>
#include <fstream>
using namespace std;

int main(int argc, char *argv[])
{
std::ifstream loop_file(argv[1]);

if (!loop_file)
cout<<"cannot open file: "<<argv[1];
else
cout<<"successfully opened"<<endl;
}
==================
# ./tracer input.data
cannot open file input.data
But it works well on smaller input file, (e.g 200Mb)

I guess there is a limit of size on "std::ifstream" object.
I wonder if there is method I could open this large input file
in C++?

There is no limit imposed by C++, but on 32-bit machines you will need what
is called "large file support" in the operating system to open files larger
than 2^31 bytes. Linux definitely supports this and surely Windows does
too, but you will have to consult your compiler documentation as to how to
make this work for you. It is probably a compiler switch of some sort.

HTH,
Ian McCulloch
 
E

eed132

I don't know if this will fix your problem, but you could try to go
back to the older C style input/output routines fopen, fscanf, fprintf,
etc. If that works you could give it a try.

One thing to be aware of is that I've heard that mixing stdio and
iostream can lead to bad results, but I doubt it'd affect anythnig in
this case so you would probably be okay. But it's sorta playing with
fire, at least without knowing more about how much they interact.
Perhaps someone else can say something about that. It probably wouldn't
cause any problems (in fact I wouldn't be surprised if one is usually
implemented in terms of the other), but it might be wise to stay away
from iostream if you are using these functions. Perhaps Google could
say more.
 
M

msalters

Charlie said:
Dear all,

I am currently writting a trace analyzer in C++.
It always fails to open a very large input file (3.7Gb).

I tried on a simple program, same thing happens: (OK)
I guess there is a limit of size on "std::ifstream" object.
I wonder if there is method I could open this large input file
in C++?

The problem isn't C++, but your particular implementation. A 64
bits implementation is unlikely to choke on it. A good 32 bits
implementation might also be able to handle it. Try MSVC 8, IIRC
it can handle these sizes. I do suspect that gcc can too, but I'm
less sure about that.

HTH,
Michiel Salters
 
Joined
Aug 21, 2007
Messages
1
Reaction score
0
I have several question want to ask you about trace analyzer

Dear Charlie,

I am currently writting a trace analyzer in C++,too.
but I have some matter with analyzer.My trace analyzer use xml file to analyze text file,analyze time is to long.So I want to know that your trace analyzer about analyze method.My trace file size about 200M to 1G.

my e-mail:[email protected]
thanks!
very
 

Ask a Question

Want to reply to this thread or ask your own question?

You'll need to choose a username for the site, which only take a couple of moments. After that, you can post your question and our members will help you out.

Ask a Question

Members online

No members online now.

Forum statistics

Threads
473,733
Messages
2,569,440
Members
44,830
Latest member
ZADIva7383

Latest Threads

Top