perl open function for size bigger than 2 Gig

Discussion in 'Perl Misc' started by Peng Yue, Sep 8, 2004.

  1. Peng Yue

    Peng Yue Guest

    Hi,
    I have problem with open function. It can not open file with size
    bigger than 2 Gig. If I open a file to write, when it reach 2 Gig,
    file size will never increase. It is a unix system and I have no
    problem to make file with bigger than 2 Gig. I can use cat to append a
    big file. For example:
    cat a.file >> b.file (b.file is already more than 2 gig)

    I found one old post in this group said there is no file size
    limitation on perl
    http://groups.google.com/groups?q=&rnum=1

    It seems there is for my case.

    Any thought about this?

    Peng
    Peng Yue, Sep 8, 2004
    #1
    1. Advertising

  2. Peng Yue wrote:

    > I have problem with open function. It can not open file with size
    > bigger than 2 Gig. If I open a file to write, when it reach 2 Gig,
    > file size will never increase. It is a unix system and I have no
    > problem to make file with bigger than 2 Gig. I can use cat to append a
    > big file. For example:
    > cat a.file >> b.file (b.file is already more than 2 gig)


    Rebuild perl with large file support (a compile time option).

    Most new binary distributions of Perl are built this way. So you may
    want to get a newer Perl binary from your OS vendor.
    Brian McCauley, Sep 8, 2004
    #2
    1. Advertising

  3. Peng Yue

    Joe Smith Guest

    Peng Yue wrote:

    > problem to make file with bigger than 2 Gig.
    > I found one old post in this group said there is no file size
    > limitation on perl
    > http://groups.google.com/groups?q=&rnum=1


    You misunderstood that post. There was a 32-bit limit; it has been fixed for
    some time now. It appears that the version of perl you're using was not
    compiled with large file support. You'll need to upgrade to a more recent
    version, or recompile the old version from sources with the right options.

    Systems that handle files greater than 2 Gig have settings like this:

    linux% perl -V | egrep -i 'LARGE|FILE'
    useperlio=define d_sfio=undef uselargefiles=define usesocks=undef
    cc='cc', ccflags ='-D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64',
    cppflags='-D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64',
    Compile-time options: USE_LARGE_FILES

    -Joe
    Joe Smith, Sep 8, 2004
    #3
  4. (Peng Yue) writes:

    > Hi,
    > I have problem with open function. It can not open file with size
    > bigger than 2 Gig. If I open a file to write, when it reach 2 Gig,
    > file size will never increase. It is a unix system and I have no
    > problem to make file with bigger than 2 Gig. I can use cat to append a
    > big file. For example:
    > cat a.file >> b.file (b.file is already more than 2 gig)


    As others have said, probably you need to recompile Perl, or else find
    a new copy from your vendor.

    But if you can't, an easy trick is to just pipe the output to/from a
    program that can address large files. For example:

    open(F,"| cat >/tmp/bigfile")
    or die "error starting cat: $!\n";
    print F "x"
    for (1..2**32+1);
    close(F)
    or die "error closing cat: $!\n";

    (untested, but you get the idea). I used to have a Solaris system
    where I had to do this, for various reasons. :)

    ----ScottG.
    Scott W Gifford, Sep 8, 2004
    #4
  5. Peng Yue

    Peng Yue Guest

    > open(F,"| cat >/tmp/bigfile")
    > or die "error starting cat: $!\n";
    > print F "x"
    > for (1..2**32+1);
    > close(F)
    > or die "error closing cat: $!\n";

    Hi Brian and Joe, thank you for your posts. I check the perl which I
    am using, there is no support for largefile. So as what you suggest,
    it needs to be recompiled. However, I have sent out request and and
    now I am waiting. Scott's solution sounds simple and good. I did
    succeed in opening big file:

    open(FILE, "cat bigfile | ");

    however, I still have trouble with writing a big file. Just following
    what Scott suggests, I try to use pipe to pass parameter to other
    programs.

    open(FILE, "| cat > bigfile") is not working for cat thing whatever
    perl pass is a filename.

    open(FILE," | echo >> bigfile");
    print FILE "HELLO";
    is also not working. It only append new line to bigfile. Can not
    figure out why echo doesn't work or any other program to try.

    Peng
    Peng Yue, Sep 9, 2004
    #5
  6. Peng Yue wrote:

    > open(FILE, "| cat > bigfile") is not working for cat thing whatever
    > perl pass is a filename.


    Since it is clear that English is not your first language it is
    even more important that you try to express yourself by giving minimal
    but complete Perl scripts to illustrate your point.
    >
    > open(FILE," | echo >> bigfile");
    > print FILE "HELLO";
    > is also not working. It only append new line to bigfile. Can not
    > figure out why echo doesn't work


    To read a description of the Unix echo command

    man echo

    > or any other program to try.


    Well you could use dd but there's not reason why cat shouldn't work.
    Brian McCauley, Sep 9, 2004
    #6
  7. (Peng Yue) writes:

    [...]

    > Scott's solution sounds simple and good. I did succeed in opening
    > big file:
    >
    > open(FILE, "cat bigfile | ");
    >
    > however, I still have trouble with writing a big file. Just following
    > what Scott suggests, I try to use pipe to pass parameter to other
    > programs.
    >
    > open(FILE, "| cat > bigfile") is not working for cat thing whatever
    > perl pass is a filename.


    I'm not sure I understand exactly what you're saying here, but you'll
    want to check whether the open command succeeded, and also the close
    command. Is it printing an error now, or just not working? If it's
    not working, is the file not created at all, or is it empty, or what?
    Does it work when you specify a filename that doesn't exist, but not
    when you specify a filename that does exist? What are permissions on
    the directory you're writing into, and what's the current directory
    when the cat command is started? If you run cat by hand with the same
    arguments and type a few lines, what happens?

    > open(FILE," | echo >> bigfile");
    > print FILE "HELLO";
    > is also not working. It only append new line to bigfile. Can not
    > figure out why echo doesn't work or any other program to try.


    echo just prints its arguments, it doesn't take input from a pipe and
    print it too, so this is just echo behaving like it's supposed to.

    ----ScottG.
    Scott W Gifford, Sep 9, 2004
    #7
  8. Peng Yue

    Peng Yue Guest

    "cat" is working. I have no idea why it did not work when I tried
    yesterday. I must did something stupid. Thanks.
    Peng Yue, Sep 10, 2004
    #8
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. z. f.
    Replies:
    5
    Views:
    568
    David Wang [Msft]
    Jan 31, 2005
  2. Rahmi Acar
    Replies:
    0
    Views:
    326
    Rahmi Acar
    Apr 20, 2004
  3. Andi Tse
    Replies:
    4
    Views:
    661
    Vaclav Haisman
    Sep 24, 2004
  4. Replies:
    2
    Views:
    761
    Filip Larsen
    Apr 10, 2007
  5. Yannick Turgeon

    App getting bigger and bigger

    Yannick Turgeon, Oct 13, 2003, in forum: Perl Misc
    Replies:
    1
    Views:
    122
    Yannick Turgeon
    Oct 14, 2003
Loading...

Share This Page