Marshal data too short

[Mike Flint]

Hello,

On Ruby 1.8.7.

Having an odd occasional occurrence of the above error - only some times
so I feel it's down to timing caused by file buffer issues (maybe!)

I have a Web Server running under Sinatra/Rack. Part of the process
involves a Cache class which is a singleton. This class checks for the
existence of an index file. If the file does exist, then its contents
are loaded, and then Marshal loaded as a hash. If the file does not
exist, then a Hash is created, and Marshal dumped (ro a string), then
written to the index file.

About 1 in 20 times the Marshal.load gives the 'too short' message.

Once I get this message it remains. That is, an hour later, long after
the index file has been created the error still happens. I can hexdump
the file but the Web Server still gets the error. Looking at the index
file which is 'broken' it seems no different to a good one. They are
both 82 bytes long.

There is only a single user accessing the system when the error occurs.
The website is not busy.

I am using blocks on the read/write of the file so my file IO channel is
being closed correctly between the writing of it, and subsequent reads.
And even if it's not closed correctly, I figure that an hour and several
messages later it will have been closed.

My gut says it's a buffer issue, or a problem with how I'm writing the
dump to the file ... using puts in a subsequent write of the string
representation of the marshalled object rather than directly on the
dump ... but why does it work most of the time?

I've added some debugging which should shed light, but anyone have
thoughts on the matter?

Thanks,
Mike.

 

[Robert Klemme]

Hello,

On Ruby 1.8.7.

Having an odd occasional occurrence of the above error - only some times
so I feel it's down to timing caused by file buffer issues (maybe!)

I have a Web Server running under Sinatra/Rack. Part of the process
involves a Cache class which is a singleton. This class checks for the
existence of an index file. If the file does exist, then its contents
are loaded, and then Marshal loaded as a hash. If the file does not
exist, then a Hash is created, and Marshal dumped (ro a string), then
written to the index file.

Are you using binary IO? I.e.

cache = File.open(cache_file, "rb") {|io| Marshal.load(io)}

File.open(cache_file, "wb") {|io| Marshal.dump(cache, io)}

About 1 in 20 times the Marshal.load gives the 'too short' message.

Once I get this message it remains. That is, an hour later, long after
the index file has been created the error still happens. I can hexdump
the file but the Web Server still gets the error. Looking at the index
file which is 'broken' it seems no different to a good one. They are
both 82 bytes long.

You should probably also remove the file if it cannot be loaded. This
will give you a more robust code base.

There is only a single user accessing the system when the error occurs.
The website is not busy.

Well, since the object is a singleton and most web pages consist of
multiple HTTP requests it may not matter how many different users
actually access it. You get concurrency anyway. For that you may use
file system level locking to avoid writing the file concurrently.

I am using blocks on the read/write of the file so my file IO channel is
being closed correctly between the writing of it, and subsequent reads.
And even if it's not closed correctly, I figure that an hour and several
messages later it will have been closed.

My gut says it's a buffer issue, or a problem with how I'm writing the
dump to the file ... using puts in a subsequent write of the string
representation of the marshalled object rather than directly on the
.dump ... but why does it work most of the time?

Chance. We do not know what you put in the Hash so... But using puts
to write marshaled content is plain wrong. You should be marshalling
directly from and to the IO instance (see above).

I've added some debugging which should shed light, but anyone have
thoughts on the matter?

See above. If it does not work, show the code.

Cheers

robert

 

[Mike Flint]

Hi,

Thanks for the response. I've been distracted from this issue by more
pressing matters so missed the reply.

I think it must be how I'm writing the dump data, and what is in there.

I will convert the code as you suggest and I'm sure that will solve the
problem.

Thanks!

Mike.