Convert Storable files between platforms?

[Steuard Jensen]

Some time ago, I wrote a perl script that uses the Storable module to
save long term data. The script itself runs fine (or at least, its
bugs aren't important here).

My problem is that the server where I run the script recently crashed.
It has been replaced by a new machine with a different architecture
(the dead server was an Alpha 21164a, the new one is a dual AMD
Opteron, both running Linux, I believe), and quite possibly a newer
version of perl (the current version is 5.8.3).

I still have my old Storable data files, but they were created using
"store" rather than the platform-independent "nstore". (I did not
anticipate this sudden need to change platforms!) So now when I try
to run my script, I get the message

"Byte order is not compatible at ../../lib/Storable.pm (autosplit
into ../../lib/auto/Storable/_retrieve.al) line 323"

(For the curious, line 323 is where Storable.pm calls its internal C
routine to read a file.)

All was not yet lost: the man page for Storable explains that earlier
versions of Storable used a different header format that caused
problems on some 64 bit platforms. It claims that this problem can
lead to an error message like that above, and goes on to say that

"If you have such data then you you should set
$Storable::interwork_56_64bit to a true value to make
this Storable read and write files with the old header."

I tried writing a little script using that setting to convert the old
files into the platform independent format (I'll include it at the
end). But even though I set the "interwork_56_64bit" variable to
true, I still get the same error message as before.

So my question is, is my data lost for good? Would I need to find
another Alpha running perl 5.6 in order to convert it into a platform
independent format? Is there some handy conversion script out there
to do the job for me? I'm willing to do a bit of work to fix things,
but this isn't important enough for me to spend time digging through
Storable's C code or learning the intricacies of endian-ness to fix
it. Thanks in advance for any advice you can give.

Steuard Jensen


My non-working conversion script:
============================================================
use Storable qw(nstore retrieve);
$Storable::interwork_56_64bit = 1;

my $storefile = shift(@ARGV);

my $data = retrieve($storefile);

$Storable::interwork_56_64bit = 0;

nstore($data, $storefile . ".new");
============================================================
(Setting the "interwork" variable back to false probably isn't
necessary since I'm using nstore, but I decided not to take chances.)

 

[J. Romano]

My problem is that the server where I run the script recently crashed.
It has been replaced by a new machine with a different architecture
(the dead server was an Alpha 21164a, the new one is a dual AMD
Opteron, both running Linux, I believe), and quite possibly a newer
version of perl (the current version is 5.8.3).

I still have my old Storable data files, but they were created using
"store" rather than the platform-independent "nstore". (I did not
anticipate this sudden need to change platforms!) So now when I try
to run my script, I get the message

"Byte order is not compatible at ../../lib/Storable.pm (autosplit
into ../../lib/auto/Storable/_retrieve.al) line 323"

Dear Steuard,

I've been examining some files created by the Storable::store()
function, and I'm guessing that you're getting this error because the
byteorder stored in the datafile doesn't match the byteorder of your
current architecture.

Try this little experiment out for me: Type:

perl -0777 -e "print substr(<>,7,8)" DATAFILE

(where DATAFILE is the name of your data file). Compare the digits in
the output to:

perl -MConfig -le "print @Config{byteorder}"

What are the results? Are the digits the same and in the same order?

If the digits are the same but not in the same order, it looks like
you might have an endian-ness issue. Then what you'll probably have
to do is reverse the byte-order of your numbers. That shouldn't be
that hard (ask me how if you don't know how and want to know), but in
order to do so you must know whether an entry is a string (which
shouldn't be touched), or an integer, or a floating-point number.

If the digits are not the same, I really wouldn't know what to do
other than to find another Alpha 21164a and re-record your structures
using Storable::nstore().

So my question is, is my data lost for good? Would I need to find
another Alpha running perl 5.6 in order to convert it into a platform
independent format? Is there some handy conversion script out there
to do the job for me? I'm willing to do a bit of work to fix things,
but this isn't important enough for me to spend time digging through
Storable's C code or learning the intricacies of endian-ness to fix
it. Thanks in advance for any advice you can give.

I'm not an expert on Storable (I'm making a big guess on what I
said above), so if someone else contradicts me, you might consider
their solution over mine. But if endian-ness really is the problem,
you may be able to fix this with a fairly simple Perl script, provided
you know what bytes (that is, entries in the structure you stored) to
reverse. Otherwise, Steuard, I'm afraid that the best solution will
be read your stored file on an Alpha 21164a and re-store your data.

Hope you can find something that works...

-- Jean-Luc

 

[thundergnat]

Some time ago, I wrote a perl script that uses the Storable module to
save long term data. The script itself runs fine (or at least, its
bugs aren't important here).

My problem is that the server where I run the script recently crashed.
It has been replaced by a new machine with a different architecture
(the dead server was an Alpha 21164a, the new one is a dual AMD
Opteron, both running Linux, I believe), and quite possibly a newer
version of perl (the current version is 5.8.3).

I still have my old Storable data files, but they were created using
"store" rather than the platform-independent "nstore". (I did not
anticipate this sudden need to change platforms!) So now when I try
to run my script, I get the message

....


I tried writing a little script using that setting to convert the old
files into the platform independent format (I'll include it at the
end). But even though I set the "interwork_56_64bit" variable to
true, I still get the same error message as before.

So my question is, is my data lost for good? Would I need to find
another Alpha running perl 5.6 in order to convert it into a platform
independent format? Is there some handy conversion script out there
to do the job for me? I'm willing to do a bit of work to fix things,
but this isn't important enough for me to spend time digging through
Storable's C code or learning the intricacies of endian-ness to fix
it. Thanks in advance for any advice you can give.

Since the x86 processors use Little-Endian byte order, it would seem
that the Alpha was using Big-Endian. (Which is odd, because Alphas, by
default used Little-Endian, though apparantly some were switchable in
software.)

http://www.unixpapa.com/incnote/byteorder.html

Probably, your best bet is to find another Big-Endian machine with perl
installed to do the conversion on. It need not necessarily be an Alpha
though, Macs use Big-Endian byte order so should be able to read the
Big-Endian storable file then write it out again in network byte order.
(Not true everywhere, but in general, Macs are easier to come by than
Alphas, and OSX come with perl installed.)

 

[Steuard Jensen]

Since the x86 processors use Little-Endian byte order, it would seem
that the Alpha was using Big-Endian. ....
Probably, your best bet is to find another Big-Endian machine with
perl installed to do the conversion on. It need not necessarily be
an Alpha though, Macs use Big-Endian...

I'm pretty sure that the Alpha was also using Little-Endian byte
order, but I appreciate the suggestion. (I've tried the script on a
Mac; no luck.) But actually, I think I've found the problem; see my
(forthcoming) response to another post in this thread for details.
(Unfortunately, _solving_ the problem might be a challenge, given my
conclusion there... but I greatly appreciate the help I've gotten here
in identifying it.)
Steuard Jensen

 

[Steuard Jensen]

Quoth (e-mail address removed) (J. Romano) in article
[And thus I get an "Byte order is not compatible" error.]

Try this little experiment out for me: Type:
perl -0777 -e "print substr(<>,7,8)" DATAFILE
Compare the digits in the output to:
perl -MConfig -le "print @Config{byteorder}"
What are the results? Are the digits the same and in the same order?

Aha! I think you've just identified my problem; thank you! My
results for those two tests are, respectively, "12345678" and "1234"!
It looks like the new Opteron-based machine is currently running in
x86 compatibility mode, and thus it's only using 32 bit integers. (I
will politely refrain from grumbling about the error message saying
"byte order" rather than "byte number". )

If the digits are not the same, I really wouldn't know what to do
other than to find another Alpha 21164a and re-record your
structures using Storable::nstore().

Well, in this case, the digits are different in a very suggestive way.
I'm pretty sure that I don't have any integer values in the file that
wouldn't fit in 32 bits (I wouldn't be surprised if they all fit in 16
bits, really). Is there any reasonable way to truncate 64 bit
integers to 32 bits in a case like this? (I could presumably figure
out which values in my program were actually integers, but I have no
idea at this point how to identify those entries in the Storable
file.)

Hope you can find something that works...

Thank you again for your help; if I do find something that works, I
think it will be due in large part to your suggestions here. I really
appreciate it!
Steuard Jensen

 

[Ben Morrow]

Quoth (e-mail address removed) (J. Romano):

I've been examining some files created by the Storable::store()
function, and I'm guessing that you're getting this error because the
byteorder stored in the datafile doesn't match the byteorder of your
current architecture.

Yup. You will also get the error if sizeof(IV) or sizeof(NV) is
different, at least for files created with recent Storable versions.

Try this little experiment out for me: Type:

perl -0777 -e "print substr(<>,7,8)" DATAFILE

(where DATAFILE is the name of your data file). Compare the digits in
the output to:

perl -MConfig -le "print @Config{byteorder}"

What are the results? Are the digits the same and in the same order?

An Alpha is a 64bit machine; I'm not sure about an Opteron: is it x86_64
or is it just an ordinary x86?

If the digits are the same but not in the same order, it looks like
you might have an endian-ness issue. Then what you'll probably have
to do is reverse the byte-order of your numbers. That shouldn't be
that hard (ask me how if you don't know how and want to know), but in
order to do so you must know whether an entry is a string (which
shouldn't be touched), or an integer, or a floating-point number.

Nope, that won't be the least bit easy. The Storable file format is
intentionally compact rather than readable: the only straightforward way
to do it would be to fiddle with Storable.xs to add
byte-order-conversion to all the retrieve ops. There is also the issue
of floats: a non-network-order Storable file has floats stored in raw
native format, and I doubt they are the same on Alpha and Opteron.

If the digits are not the same, I really wouldn't know what to do
other than to find another Alpha 21164a and re-record your structures
using Storable::nstore().

I think that's the only answer, if the data isn't worth the investment
of grubbing around in Storable's internals.

Not necessarily perl5.6. Just any version of Perl with a Storable
version at least as recent as the one you created the file with (you can
print a bit of magic(5) which will tell you the version with

perl -MStorable -eStorable::show_file_magic

).

No, and because of the issue of floats there probably won't ever be:
binary float formats aren't standardised at all, so a conversion utility
would need to know every float format on every machine and how to
convert them, and then a system of tags would need to be devised to add
to the header of the file... basically, a non-network Storable file
isn't portable off the arch it was created on.

This time, I'd advise using nstore; or, if that's too slow (benchmark
first to make sure it actually is a problem) set up a litle cron job to
read the files every night and write network-order backups somewhere...

Ben

 

[J. Romano]

Well, in this case, the digits are different in a very suggestive way.
I'm pretty sure that I don't have any integer values in the file that
wouldn't fit in 32 bits (I wouldn't be surprised if they all fit in 16
bits, really). Is there any reasonable way to truncate 64 bit
integers to 32 bits in a case like this? (I could presumably figure
out which values in my program were actually integers, but I have no
idea at this point how to identify those entries in the Storable
file.)

To be honest, I don't think there is a good way to truncate 64-bit
integers to 32 bits, at least in your Storable datafile. Doing so
might throw off the offsets used by the Storable module to find data
values. If you're really adventurous, we can still test it out, but
we'd still have the problem of identifying the correct values to
truncate. Oh, and truncating a 64-bit value to a 32-bit value will
only work for little-endian integers (presuming that they fit in 32
bits, of course), but totally butcher floating-point values. So let
me tell you now: even if we did modify your datafile, I'm almost
certain this technique won't work.

Here's something you might want to try (let me remind you that I'm
still guessing and I'm not sure if this will work at all): Type:

perl -0777 -pe 'substr($_,7,8)="87654321"' DATAFILE > munged_data

(where DATAFILE is the name of your datafile). Then, using a 64-bit
big-endian machine, read the "munged_data" file and re-store it using
Storable::nstore(). The numerical data saved will be wrong, but if
the 64-bit machine successfully reads the new "munged_data" datafile,
you might be able to convert it later.

In order for the above to work, you would have to find a 64-bit
big-endian machine somewhere. Judging by your e-mail address, I'm
guessing that you attend a university. From experience I know that
sometimes large universities have a 64-bit Unix machine "lying around"
that students and faculty can connect to. However, it's not always
known that a particular machine has a 64-bit architecture, so you may
have to ask around if anyone knows of one. Not only that, but I heard
that Apple recently came out with a 64-bit personal computer. You may
want to ask some of your Mac-addict friends if they have one (and if
they'll let you use their computer).

This won't really help you at this point, but just so you know,
there is another way to store data out to file that doesn't use the
Storable module. Instead of using the Storable module like so:

use Storable qw(store nstore);
store($data, $storefile); # or nstore($data, $storefile)

you can use the Data:umper module (a standard Perl module) like so:

use Data:umper;
my $fileString = Dumper $data;
# Now write the $fileString to the data file:
open(FH, "> $storefile") or die "Cannot write: $!";
print FH $fileString;
close(FH);

To read it back, instead of saying:

use Storable qw(retrieve);
my $data = retrieve($storefile);

you can say:

# Open the data file and read in the $fileContent:
open(FH, $storefile) or die "Cannot read: $!";
my $fileContent = join('', <FH>);
close(FH);
# Extract the $data by eval-ing the datafile content:
my $data = eval $fileContent;

I'm sure that the Data:umper approach is much slower than using the
Storable module, but it has all the advantages of being able to easily
view and modify the datafile with your favorite text-editor. If your
$data is not too complex (or you don't mind the extra time it takes),
you may want to consider using the Data:umper module to at least
save an extra datafile. That way, you would have a portable and
readable backup file in case you should ever switch architectures for
some unforseen reason.

How complex is your data, anyway? If it's just an array of
integers, it might be a simple task to just read through the datafile
and extract the values (provided you can give me a few sample data
values to look for). I've done this before with other binary data
files. But if you're using a complex data structure (like an array of
arrays, or an array of hashes or objects) it won't be easy to do.

Thank you again for your help; if I do find something that works, I
think it will be due in large part to your suggestions here. I really
appreciate it!

Well, thank you... it's nice to be appreciated (instead of being
flamed for making all these guesses...

Maybe what I said in this post will help... maybe it won't... but I
still hope you find something that works.

-- Jean-Luc