perl open function for size bigger than 2 Gig

[Peng Yue]

Hi,
I have problem with open function. It can not open file with size
bigger than 2 Gig. If I open a file to write, when it reach 2 Gig,
file size will never increase. It is a unix system and I have no
problem to make file with bigger than 2 Gig. I can use cat to append a
big file. For example:
cat a.file >> b.file (b.file is already more than 2 gig)

I found one old post in this group said there is no file size
limitation on perl
http://groups.google.com/[email protected]&rnum=1

It seems there is for my case.

Any thought about this?

Peng

 

[Brian McCauley]

I have problem with open function. It can not open file with size
bigger than 2 Gig. If I open a file to write, when it reach 2 Gig,
file size will never increase. It is a unix system and I have no
problem to make file with bigger than 2 Gig. I can use cat to append a
big file. For example:
cat a.file >> b.file (b.file is already more than 2 gig)

Rebuild perl with large file support (a compile time option).

Most new binary distributions of Perl are built this way. So you may
want to get a newer Perl binary from your OS vendor.

 

[Joe Smith]

problem to make file with bigger than 2 Gig.
I found one old post in this group said there is no file size
limitation on perl
http://groups.google.com/[email protected]&rnum=1

You misunderstood that post. There was a 32-bit limit; it has been fixed for
some time now. It appears that the version of perl you're using was not
compiled with large file support. You'll need to upgrade to a more recent
version, or recompile the old version from sources with the right options.

Systems that handle files greater than 2 Gig have settings like this:

linux% perl -V | egrep -i 'LARGE|FILE'
useperlio=define d_sfio=undef uselargefiles=define usesocks=undef
cc='cc', ccflags ='-D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64',
cppflags='-D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64',
Compile-time options: USE_LARGE_FILES

-Joe

 

[Scott W Gifford]

Hi,
I have problem with open function. It can not open file with size
bigger than 2 Gig. If I open a file to write, when it reach 2 Gig,
file size will never increase. It is a unix system and I have no
problem to make file with bigger than 2 Gig. I can use cat to append a
big file. For example:
cat a.file >> b.file (b.file is already more than 2 gig)

As others have said, probably you need to recompile Perl, or else find
a new copy from your vendor.

But if you can't, an easy trick is to just pipe the output to/from a
program that can address large files. For example:

open(F,"| cat >/tmp/bigfile")
or die "error starting cat: $!\n";
print F "x"
for (1..2**32+1);
close(F)
or die "error closing cat: $!\n";

(untested, but you get the idea). I used to have a Solaris system
where I had to do this, for various reasons.

----ScottG.

 

[Peng Yue]

open(F,"| cat >/tmp/bigfile")

or die "error starting cat: $!\n";
print F "x"
for (1..2**32+1);
close(F)
or die "error closing cat: $!\n";

Hi Brian and Joe, thank you for your posts. I check the perl which I
am using, there is no support for largefile. So as what you suggest,
it needs to be recompiled. However, I have sent out request and and
now I am waiting. Scott's solution sounds simple and good. I did
succeed in opening big file:

open(FILE, "cat bigfile | ");

however, I still have trouble with writing a big file. Just following
what Scott suggests, I try to use pipe to pass parameter to other
programs.

open(FILE, "| cat > bigfile") is not working for cat thing whatever
perl pass is a filename.

open(FILE," | echo >> bigfile");
print FILE "HELLO";
is also not working. It only append new line to bigfile. Can not
figure out why echo doesn't work or any other program to try.

Peng

 

[Brian McCauley]

open(FILE, "| cat > bigfile") is not working for cat thing whatever
perl pass is a filename.

Since it is clear that English is not your first language it is
even more important that you try to express yourself by giving minimal
but complete Perl scripts to illustrate your point.

open(FILE," | echo >> bigfile");
print FILE "HELLO";
is also not working. It only append new line to bigfile. Can not
figure out why echo doesn't work

To read a description of the Unix echo command

man echo

or any other program to try.

Well you could use dd but there's not reason why cat shouldn't work.

 

[Scott W Gifford]

(e-mail address removed) (Peng Yue) writes:

[...]

Scott's solution sounds simple and good. I did succeed in opening
big file:

open(FILE, "cat bigfile | ");

however, I still have trouble with writing a big file. Just following
what Scott suggests, I try to use pipe to pass parameter to other
programs.

open(FILE, "| cat > bigfile") is not working for cat thing whatever
perl pass is a filename.

I'm not sure I understand exactly what you're saying here, but you'll
want to check whether the open command succeeded, and also the close
command. Is it printing an error now, or just not working? If it's
not working, is the file not created at all, or is it empty, or what?
Does it work when you specify a filename that doesn't exist, but not
when you specify a filename that does exist? What are permissions on
the directory you're writing into, and what's the current directory
when the cat command is started? If you run cat by hand with the same
arguments and type a few lines, what happens?

open(FILE," | echo >> bigfile");
print FILE "HELLO";
is also not working. It only append new line to bigfile. Can not
figure out why echo doesn't work or any other program to try.

echo just prints its arguments, it doesn't take input from a pipe and
print it too, so this is just echo behaving like it's supposed to.

----ScottG.

 

[Peng Yue]

"cat" is working. I have no idea why it did not work when I tried
yesterday. I must did something stupid. Thanks.