Convert Storable files between platforms?

Discussion in 'Perl Misc' started by Steuard Jensen, Jul 20, 2004.

  1. Some time ago, I wrote a perl script that uses the Storable module to
    save long term data. The script itself runs fine (or at least, its
    bugs aren't important here).

    My problem is that the server where I run the script recently crashed.
    It has been replaced by a new machine with a different architecture
    (the dead server was an Alpha 21164a, the new one is a dual AMD
    Opteron, both running Linux, I believe), and quite possibly a newer
    version of perl (the current version is 5.8.3).

    I still have my old Storable data files, but they were created using
    "store" rather than the platform-independent "nstore". (I did not
    anticipate this sudden need to change platforms!) So now when I try
    to run my script, I get the message

    "Byte order is not compatible at ../../lib/Storable.pm (autosplit
    into ../../lib/auto/Storable/_retrieve.al) line 323"

    (For the curious, line 323 is where Storable.pm calls its internal C
    routine to read a file.)

    All was not yet lost: the man page for Storable explains that earlier
    versions of Storable used a different header format that caused
    problems on some 64 bit platforms. It claims that this problem can
    lead to an error message like that above, and goes on to say that

    "If you have such data then you you should set
    $Storable::interwork_56_64bit to a true value to make
    this Storable read and write files with the old header."

    I tried writing a little script using that setting to convert the old
    files into the platform independent format (I'll include it at the
    end). But even though I set the "interwork_56_64bit" variable to
    true, I still get the same error message as before.

    So my question is, is my data lost for good? Would I need to find
    another Alpha running perl 5.6 in order to convert it into a platform
    independent format? Is there some handy conversion script out there
    to do the job for me? I'm willing to do a bit of work to fix things,
    but this isn't important enough for me to spend time digging through
    Storable's C code or learning the intricacies of endian-ness to fix
    it. Thanks in advance for any advice you can give.

    Steuard Jensen


    My non-working conversion script:
    ============================================================
    use Storable qw(nstore retrieve);
    $Storable::interwork_56_64bit = 1;

    my $storefile = shift(@ARGV);

    my $data = retrieve($storefile);

    $Storable::interwork_56_64bit = 0;

    nstore($data, $storefile . ".new");
    ============================================================
    (Setting the "interwork" variable back to false probably isn't
    necessary since I'm using nstore, but I decided not to take chances.)
     
    Steuard Jensen, Jul 20, 2004
    #1
    1. Advertising

  2. Steuard Jensen

    J. Romano Guest

    (Steuard Jensen) wrote in message news:<ijgLc.29$>...
    >
    > My problem is that the server where I run the script recently crashed.
    > It has been replaced by a new machine with a different architecture
    > (the dead server was an Alpha 21164a, the new one is a dual AMD
    > Opteron, both running Linux, I believe), and quite possibly a newer
    > version of perl (the current version is 5.8.3).
    >
    > I still have my old Storable data files, but they were created using
    > "store" rather than the platform-independent "nstore". (I did not
    > anticipate this sudden need to change platforms!) So now when I try
    > to run my script, I get the message
    >
    > "Byte order is not compatible at ../../lib/Storable.pm (autosplit
    > into ../../lib/auto/Storable/_retrieve.al) line 323"


    Dear Steuard,

    I've been examining some files created by the Storable::store()
    function, and I'm guessing that you're getting this error because the
    byteorder stored in the datafile doesn't match the byteorder of your
    current architecture.

    Try this little experiment out for me: Type:

    perl -0777 -e "print substr(<>,7,8)" DATAFILE

    (where DATAFILE is the name of your data file). Compare the digits in
    the output to:

    perl -MConfig -le "print @Config{byteorder}"

    What are the results? Are the digits the same and in the same order?

    If the digits are the same but not in the same order, it looks like
    you might have an endian-ness issue. Then what you'll probably have
    to do is reverse the byte-order of your numbers. That shouldn't be
    that hard (ask me how if you don't know how and want to know), but in
    order to do so you must know whether an entry is a string (which
    shouldn't be touched), or an integer, or a floating-point number.

    If the digits are not the same, I really wouldn't know what to do
    other than to find another Alpha 21164a and re-record your structures
    using Storable::nstore().

    > So my question is, is my data lost for good? Would I need to find
    > another Alpha running perl 5.6 in order to convert it into a platform
    > independent format? Is there some handy conversion script out there
    > to do the job for me? I'm willing to do a bit of work to fix things,
    > but this isn't important enough for me to spend time digging through
    > Storable's C code or learning the intricacies of endian-ness to fix
    > it. Thanks in advance for any advice you can give.


    I'm not an expert on Storable (I'm making a big guess on what I
    said above), so if someone else contradicts me, you might consider
    their solution over mine. But if endian-ness really is the problem,
    you may be able to fix this with a fairly simple Perl script, provided
    you know what bytes (that is, entries in the structure you stored) to
    reverse. Otherwise, Steuard, I'm afraid that the best solution will
    be read your stored file on an Alpha 21164a and re-store your data.

    Hope you can find something that works...

    -- Jean-Luc
     
    J. Romano, Jul 21, 2004
    #2
    1. Advertising

  3. Steuard Jensen

    thundergnat Guest

    Steuard Jensen wrote:
    > Some time ago, I wrote a perl script that uses the Storable module to
    > save long term data. The script itself runs fine (or at least, its
    > bugs aren't important here).
    >
    > My problem is that the server where I run the script recently crashed.
    > It has been replaced by a new machine with a different architecture
    > (the dead server was an Alpha 21164a, the new one is a dual AMD
    > Opteron, both running Linux, I believe), and quite possibly a newer
    > version of perl (the current version is 5.8.3).
    >
    > I still have my old Storable data files, but they were created using
    > "store" rather than the platform-independent "nstore". (I did not
    > anticipate this sudden need to change platforms!) So now when I try
    > to run my script, I get the message
    >


    ....

    >
    > I tried writing a little script using that setting to convert the old
    > files into the platform independent format (I'll include it at the
    > end). But even though I set the "interwork_56_64bit" variable to
    > true, I still get the same error message as before.
    >
    > So my question is, is my data lost for good? Would I need to find
    > another Alpha running perl 5.6 in order to convert it into a platform
    > independent format? Is there some handy conversion script out there
    > to do the job for me? I'm willing to do a bit of work to fix things,
    > but this isn't important enough for me to spend time digging through
    > Storable's C code or learning the intricacies of endian-ness to fix
    > it. Thanks in advance for any advice you can give.
    >


    Since the x86 processors use Little-Endian byte order, it would seem
    that the Alpha was using Big-Endian. (Which is odd, because Alphas, by
    default used Little-Endian, though apparantly some were switchable in
    software.)

    http://www.unixpapa.com/incnote/byteorder.html

    Probably, your best bet is to find another Big-Endian machine with perl
    installed to do the conversion on. It need not necessarily be an Alpha
    though, Macs use Big-Endian byte order so should be able to read the
    Big-Endian storable file then write it out again in network byte order.
    (Not true everywhere, but in general, Macs are easier to come by than
    Alphas, and OSX come with perl installed.)
     
    thundergnat, Jul 21, 2004
    #3
  4. Quoth thundergnat <> in article
    <40feb358$0$5632$>:
    > Steuard Jensen wrote:
    > > ...the server where I run the script... has been replaced by a
    > > new machine with a different architecture (the dead server was an
    > > Alpha 21164a, the new one is a dual AMD Opteron...)


    > Since the x86 processors use Little-Endian byte order, it would seem
    > that the Alpha was using Big-Endian.

    ....
    > Probably, your best bet is to find another Big-Endian machine with
    > perl installed to do the conversion on. It need not necessarily be
    > an Alpha though, Macs use Big-Endian...


    I'm pretty sure that the Alpha was also using Little-Endian byte
    order, but I appreciate the suggestion. (I've tried the script on a
    Mac; no luck.) But actually, I think I've found the problem; see my
    (forthcoming) response to another post in this thread for details.
    (Unfortunately, _solving_ the problem might be a challenge, given my
    conclusion there... but I greatly appreciate the help I've gotten here
    in identifying it.)
    Steuard Jensen
     
    Steuard Jensen, Jul 21, 2004
    #4
  5. Quoth (J. Romano) in article
    <>:
    > (Steuard Jensen) wrote:
    > > I still have my old Storable data files, but they were created
    > > using "store" rather than the platform-independent "nstore".

    [And thus I get an "Byte order is not compatible" error.]

    > Try this little experiment out for me: Type:
    > perl -0777 -e "print substr(<>,7,8)" DATAFILE
    > Compare the digits in the output to:
    > perl -MConfig -le "print @Config{byteorder}"
    > What are the results? Are the digits the same and in the same order?


    Aha! I think you've just identified my problem; thank you! My
    results for those two tests are, respectively, "12345678" and "1234"!
    It looks like the new Opteron-based machine is currently running in
    x86 compatibility mode, and thus it's only using 32 bit integers. (I
    will politely refrain from grumbling about the error message saying
    "byte order" rather than "byte number". :) )

    > If the digits are not the same, I really wouldn't know what to do
    > other than to find another Alpha 21164a and re-record your
    > structures using Storable::nstore().


    Well, in this case, the digits are different in a very suggestive way.
    I'm pretty sure that I don't have any integer values in the file that
    wouldn't fit in 32 bits (I wouldn't be surprised if they all fit in 16
    bits, really). Is there any reasonable way to truncate 64 bit
    integers to 32 bits in a case like this? (I could presumably figure
    out which values in my program were actually integers, but I have no
    idea at this point how to identify those entries in the Storable
    file.)

    > Hope you can find something that works...


    Thank you again for your help; if I do find something that works, I
    think it will be due in large part to your suggestions here. I really
    appreciate it!
    Steuard Jensen
     
    Steuard Jensen, Jul 21, 2004
    #5
  6. Steuard Jensen

    Ben Morrow Guest

    Quoth (J. Romano):
    > (Steuard Jensen) wrote in message news:<ijgLc.29$>...
    > >
    > > My problem is that the server where I run the script recently crashed.
    > > It has been replaced by a new machine with a different architecture
    > > (the dead server was an Alpha 21164a, the new one is a dual AMD
    > > Opteron, both running Linux, I believe), and quite possibly a newer
    > > version of perl (the current version is 5.8.3).
    > >
    > > I still have my old Storable data files, but they were created using
    > > "store" rather than the platform-independent "nstore". (I did not
    > > anticipate this sudden need to change platforms!) So now when I try
    > > to run my script, I get the message
    > >
    > > "Byte order is not compatible at ../../lib/Storable.pm (autosplit
    > > into ../../lib/auto/Storable/_retrieve.al) line 323"

    >
    > I've been examining some files created by the Storable::store()
    > function, and I'm guessing that you're getting this error because the
    > byteorder stored in the datafile doesn't match the byteorder of your
    > current architecture.


    Yup. You will also get the error if sizeof(IV) or sizeof(NV) is
    different, at least for files created with recent Storable versions.

    > Try this little experiment out for me: Type:
    >
    > perl -0777 -e "print substr(<>,7,8)" DATAFILE
    >
    > (where DATAFILE is the name of your data file). Compare the digits in
    > the output to:
    >
    > perl -MConfig -le "print @Config{byteorder}"
    >
    > What are the results? Are the digits the same and in the same order?


    An Alpha is a 64bit machine; I'm not sure about an Opteron: is it x86_64
    or is it just an ordinary x86?

    > If the digits are the same but not in the same order, it looks like
    > you might have an endian-ness issue. Then what you'll probably have
    > to do is reverse the byte-order of your numbers. That shouldn't be
    > that hard (ask me how if you don't know how and want to know), but in
    > order to do so you must know whether an entry is a string (which
    > shouldn't be touched), or an integer, or a floating-point number.


    Nope, that won't be the least bit easy. The Storable file format is
    intentionally compact rather than readable: the only straightforward way
    to do it would be to fiddle with Storable.xs to add
    byte-order-conversion to all the retrieve ops. There is also the issue
    of floats: a non-network-order Storable file has floats stored in raw
    native format, and I doubt they are the same on Alpha and Opteron.

    > If the digits are not the same, I really wouldn't know what to do
    > other than to find another Alpha 21164a and re-record your structures
    > using Storable::nstore().


    I think that's the only answer, if the data isn't worth the investment
    of grubbing around in Storable's internals.

    > > So my question is, is my data lost for good? Would I need to find
    > > another Alpha running perl 5.6 in order to convert it into a platform
    > > independent format?


    Not necessarily perl5.6. Just any version of Perl with a Storable
    version at least as recent as the one you created the file with (you can
    print a bit of magic(5) which will tell you the version with

    perl -MStorable -eStorable::show_file_magic

    ).

    > > Is there some handy conversion script out there
    > > to do the job for me?


    No, and because of the issue of floats there probably won't ever be:
    binary float formats aren't standardised at all, so a conversion utility
    would need to know every float format on every machine and how to
    convert them, and then a system of tags would need to be devised to add
    to the header of the file... basically, a non-network Storable file
    isn't portable off the arch it was created on.

    This time, I'd advise using nstore; or, if that's too slow (benchmark
    first to make sure it actually is a problem) set up a litle cron job to
    read the files every night and write network-order backups somewhere...

    Ben

    --
    $.=1;*g=sub{print@_};sub r($$\$){my($w,$x,$y)=@_;for(keys%$x){/main/&&next;*p=$
    $x{$_};/(\w)::$/&&(r($w.$1,$x.$_,$y),next);$y eq\$p&&&g("$w$_")}};sub t{for(@_)
    {$f&&($_||&g(" "));$f=1;r"","::",$_;$_&&&g(chr(0012))}};t #
    $J::u::s::t, $a::n::eek:::t::h::e::r, $P::e::r::l, $h::a::c::k::e::r, $.
     
    Ben Morrow, Jul 21, 2004
    #6
  7. Steuard Jensen

    J. Romano Guest

    (Steuard Jensen) wrote in message news:<48ALc.17$>...
    >
    > Well, in this case, the digits are different in a very suggestive way.
    > I'm pretty sure that I don't have any integer values in the file that
    > wouldn't fit in 32 bits (I wouldn't be surprised if they all fit in 16
    > bits, really). Is there any reasonable way to truncate 64 bit
    > integers to 32 bits in a case like this? (I could presumably figure
    > out which values in my program were actually integers, but I have no
    > idea at this point how to identify those entries in the Storable
    > file.)


    To be honest, I don't think there is a good way to truncate 64-bit
    integers to 32 bits, at least in your Storable datafile. Doing so
    might throw off the offsets used by the Storable module to find data
    values. If you're really adventurous, we can still test it out, but
    we'd still have the problem of identifying the correct values to
    truncate. Oh, and truncating a 64-bit value to a 32-bit value will
    only work for little-endian integers (presuming that they fit in 32
    bits, of course), but totally butcher floating-point values. So let
    me tell you now: even if we did modify your datafile, I'm almost
    certain this technique won't work.

    Here's something you might want to try (let me remind you that I'm
    still guessing and I'm not sure if this will work at all): Type:

    perl -0777 -pe 'substr($_,7,8)="87654321"' DATAFILE > munged_data

    (where DATAFILE is the name of your datafile). Then, using a 64-bit
    big-endian machine, read the "munged_data" file and re-store it using
    Storable::nstore(). The numerical data saved will be wrong, but if
    the 64-bit machine successfully reads the new "munged_data" datafile,
    you might be able to convert it later.

    In order for the above to work, you would have to find a 64-bit
    big-endian machine somewhere. Judging by your e-mail address, I'm
    guessing that you attend a university. From experience I know that
    sometimes large universities have a 64-bit Unix machine "lying around"
    that students and faculty can connect to. However, it's not always
    known that a particular machine has a 64-bit architecture, so you may
    have to ask around if anyone knows of one. Not only that, but I heard
    that Apple recently came out with a 64-bit personal computer. You may
    want to ask some of your Mac-addict friends if they have one (and if
    they'll let you use their computer).

    This won't really help you at this point, but just so you know,
    there is another way to store data out to file that doesn't use the
    Storable module. Instead of using the Storable module like so:

    use Storable qw(store nstore);
    store($data, $storefile); # or nstore($data, $storefile)

    you can use the Data::Dumper module (a standard Perl module) like so:

    use Data::Dumper;
    my $fileString = Dumper $data;
    # Now write the $fileString to the data file:
    open(FH, "> $storefile") or die "Cannot write: $!";
    print FH $fileString;
    close(FH);

    To read it back, instead of saying:

    use Storable qw(retrieve);
    my $data = retrieve($storefile);

    you can say:

    # Open the data file and read in the $fileContent:
    open(FH, $storefile) or die "Cannot read: $!";
    my $fileContent = join('', <FH>);
    close(FH);
    # Extract the $data by eval-ing the datafile content:
    my $data = eval $fileContent;

    I'm sure that the Data::Dumper approach is much slower than using the
    Storable module, but it has all the advantages of being able to easily
    view and modify the datafile with your favorite text-editor. If your
    $data is not too complex (or you don't mind the extra time it takes),
    you may want to consider using the Data::Dumper module to at least
    save an extra datafile. That way, you would have a portable and
    readable backup file in case you should ever switch architectures for
    some unforseen reason.

    How complex is your data, anyway? If it's just an array of
    integers, it might be a simple task to just read through the datafile
    and extract the values (provided you can give me a few sample data
    values to look for). I've done this before with other binary data
    files. But if you're using a complex data structure (like an array of
    arrays, or an array of hashes or objects) it won't be easy to do.

    > Thank you again for your help; if I do find something that works, I
    > think it will be due in large part to your suggestions here. I really
    > appreciate it!


    Well, thank you... it's nice to be appreciated (instead of being
    flamed for making all these guesses... :)

    Maybe what I said in this post will help... maybe it won't... but I
    still hope you find something that works.

    -- Jean-Luc
     
    J. Romano, Jul 23, 2004
    #7
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. Daniel Hembree
    Replies:
    0
    Views:
    1,199
    Daniel Hembree
    Jan 12, 2004
  2. Harsha
    Replies:
    1
    Views:
    578
  3. Replies:
    3
    Views:
    2,442
    Jim Gibson
    Mar 29, 2005
  4. Kenjis Kaan

    Storable module for Activestate 5.6.1??

    Kenjis Kaan, Jun 30, 2003, in forum: Perl Misc
    Replies:
    3
    Views:
    149
    Sisyphus
    Jul 1, 2003
  5. arun
    Replies:
    1
    Views:
    1,421
    Rainer Weikusat
    Mar 8, 2012
Loading...

Share This Page