custom serialization problem 2 (marshal_dump, marshal_load)

Discussion in 'Ruby' started by sayoyo Sayoyo, May 21, 2008.

  1. Hi,

    I have a problem with the marshal_dump and marshal_load...
    I built two classes XXX, YYY and implemented both methods like this:

    class YYY
    @data = "string"
    @version = 1

    def marshal_dump()
    return [@version,@data]
    end

    def marshal_load(var)
    @version = var[0]
    case @version
    when 1
    @data = var[1]
    else
    #do something else
    end
    end
    end

    class XXX
    @data = Array of YYY
    @version = 1

    def marshal_dump()
    return [@version,@data]
    end

    def marshal_load(var)
    @version = var[0]
    case @version
    when 1
    @data = var[1]
    else
    #do something else
    end
    end
    end

    The serialization works correctly when the number of objects YYY is
    small, but when it got more than 5000, the serialization doesn't work.
    (the serializaed file has more and less 1 meg)


    When I try to not to use marshal_dump and marshal_load. the
    serialization works again????

    Can someone tell me, where is the difference between customized
    serialization and no-customized one???

    Thanks you very much

    Sayoyo
    --
    Posted via http://www.ruby-forum.com/.
    sayoyo Sayoyo, May 21, 2008
    #1
    1. Advertising

  2. sayoyo Sayoyo wrote:
    > Hi,
    >
    > I have a problem with the marshal_dump and marshal_load...
    > I built two classes XXX, YYY and implemented both methods like this:
    >
    > class YYY
    > @data = "string"
    > @version = 1
    >
    > def marshal_dump()
    > return [@version,@data]
    > end
    >
    > def marshal_load(var)
    > @version = var[0]
    > case @version
    > when 1
    > @data = var[1]
    > else
    > #do something else
    > end
    > end
    > end
    >
    > class XXX
    > @data = Array of YYY
    > @version = 1
    >
    > def marshal_dump()
    > return [@version,@data]
    > end
    >
    > def marshal_load(var)
    > @version = var[0]
    > case @version
    > when 1
    > @data = var[1]
    > else
    > #do something else
    > end
    > end
    > end
    >
    > The serialization works correctly when the number of objects YYY is
    > small, but when it got more than 5000, the serialization doesn't work.
    > (the serializaed file has more and less 1 meg)
    >
    >
    > When I try to not to use marshal_dump and marshal_load. the
    > serialization works again????
    >
    > Can someone tell me, where is the difference between customized
    > serialization and no-customized one???
    >
    > Thanks you very much
    >
    > Sayoyo


    What version of ruby? What platform?

    Same problem when you serialize to memory instead of file?

    What exactly is the observed problem? Crash, exception, data error?

    The following version of your code seems to work:

    class YYY
    attr_reader :data, :version

    def initialize
    @data = "string"
    @version = 1
    end

    def marshal_dump()
    return [@version,@data]
    end

    def marshal_load(var)
    @version = var[0]
    case @version
    when 1
    @data = var[1]
    else
    #do something else
    end
    end
    end

    class XXX
    attr_reader :data, :version

    def initialize
    @data = (0..1000).map { YYY.new }
    @version = 1
    end

    def marshal_dump()
    return [@version,@data]
    end

    def marshal_load(var)
    @version = var[0]
    case @version
    when 1
    @data = var[1]
    else
    #do something else
    end
    end
    end

    xxx = XXX.new

    x2 = Marshal.load(Marshal.dump(xxx))

    p x2.data[777]

    File.open('/tmp/foo', "wb") do |f|
    Marshal.dump(xxx, f)
    end

    x3 = File.open('/tmp/foo', "rb") do |f|
    Marshal.load(f)
    end

    p x3.data[888]




    --
    vjoel : Joel VanderWerf : path berkeley edu : 510 665 3407
    Joel VanderWerf, May 21, 2008
    #2
    1. Advertising

  3. Joel VanderWerf wrote:
    > @data = (0..1000).map { YYY.new }


    I tried it for 1_000_000 with no problem, and that generated a 1.4Mb file.

    Ruby 1.8.6-p114, linux.

    --
    vjoel : Joel VanderWerf : path berkeley edu : 510 665 3407
    Joel VanderWerf, May 21, 2008
    #3
  4. Re: custom serialization problem 2 (marshal_dump, marshal_lo

    Hi,

    I'm using ruby 1.8.6 (2007-09-24 patchlevel 111) [x86_64-linux] and
    Rails 1.2.5

    The problem I have, is the class XXX should have an array of YYY as
    data, but it turn out I got only NIL when I deserialized, then I got an
    exception for unable to use [] for a NIL. It happens only when I have a
    lot of data.

    The example I posted here is an simplification of my data, actually I
    have 5 classes, the structure:
    A--
    |-Array of B
    |- C

    B--
    |-Array of D

    C--
    |-Array of E

    and each of them have more and less 10 attributes... When I have 5000 of
    B, the serialized file has almost 1 meg. And I also realized each time I
    serialized the same data, the file got a different size each time...

    How can I verify the serailization?





    --
    Posted via http://www.ruby-forum.com/.
    sayoyo Sayoyo, May 21, 2008
    #4
  5. Re: custom serialization problem 2 (marshal_dump, marshal_lo

    sayoyo Sayoyo wrote:
    > Hi,
    >
    > I'm using ruby 1.8.6 (2007-09-24 patchlevel 111) [x86_64-linux] and
    > Rails 1.2.5
    >
    > The problem I have, is the class XXX should have an array of YYY as
    > data, but it turn out I got only NIL when I deserialized, then I got an
    > exception for unable to use [] for a NIL. It happens only when I have a
    > lot of data.
    >
    > The example I posted here is an simplification of my data, actually I
    > have 5 classes, the structure:
    > A--
    > |-Array of B
    > |- C
    >
    > B--
    > |-Array of D
    >
    > C--
    > |-Array of E
    >
    > and each of them have more and less 10 attributes... When I have 5000 of
    > B, the serialized file has almost 1 meg. And I also realized each time I
    > serialized the same data, the file got a different size each time...
    >
    > How can I verify the serailization?


    I would try to come up with a minimal characterization of the
    problem--minimize the classes, attrs, objects so that if you take away
    anything else the problem goes away. That should help point to the problem.

    Also, you could try running this on other systems and ruby versions....

    --
    vjoel : Joel VanderWerf : path berkeley edu : 510 665 3407
    Joel VanderWerf, May 22, 2008
    #5
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. Replies:
    8
    Views:
    2,215
    deadsea
    Jan 2, 2005
  2. Replies:
    3
    Views:
    1,002
  3. Dimitri Ognibene
    Replies:
    4
    Views:
    757
    Dimitri Ognibene
    Sep 2, 2006
  4. Michael Davis

    Ruby 1.8 and marshal_dump issue

    Michael Davis, Oct 10, 2003, in forum: Ruby
    Replies:
    2
    Views:
    136
    Michael Davis
    Oct 18, 2003
  5. Tim Hunter

    marshal_load, marshal_dump

    Tim Hunter, Dec 13, 2008, in forum: Ruby
    Replies:
    5
    Views:
    94
    Joel VanderWerf
    Dec 16, 2008
Loading...

Share This Page