Marshaling objects partially

Discussion in 'Ruby' started by Ronald Pijnacker, Jul 7, 2003.

  1. Hi!

    I've got a question about streaming objects with Marshal.

    The structure I'd like to stream is cyclicly referenced and the objects
    that I want to dump have some data members that I would not like to
    stream.

    I had a look at _dump and _load, but it seems to me that I'd have to
    solve the cyclic references myself again. Is this true?

    What is the recomended way of streaming such a structure?
    Can I simply tag some members as being 'volatile' so they are no longer
    streamed?

    Regards,

    Ronald
    Ronald Pijnacker, Jul 7, 2003
    #1
    1. Advertising

  2. Hi,

    In message "Marshaling objects partially"
    on 03/07/07, Ronald Pijnacker <> writes:

    |I had a look at _dump and _load, but it seems to me that I'd have to
    |solve the cyclic references myself again. Is this true?

    Yes.

    But if you use Marshal.dump for _dump, Marshal resolves cycles
    (potential duplication might happen).

    class Foo
    def _dump(limit)
    Marshal.dump([...data in array...], limit-1)
    end
    def Foo._load(str)
    data = Marshal.load(str)
    ...build instance using data...
    end
    end

    matz.
    Yukihiro Matsumoto, Jul 7, 2003
    #2
    1. Advertising

  3. > Hi,
    >
    > In message "Marshaling objects partially"
    > on 03/07/07, Ronald Pijnacker <> writes:
    >
    > |I had a look at _dump and _load, but it seems to me that I'd have to
    > |solve the cyclic references myself again. Is this true?
    >
    > Yes.
    >
    > But if you use Marshal.dump for _dump, Marshal resolves cycles
    > (potential duplication might happen).


    Thanks, this works fine.

    Do you have any suggestions on how to orgainize _load and _dump such
    that in a class hierarchy every class level is responsible for streaming
    its own members?

    Ronald.
    Ronald Pijnacker, Jul 8, 2003
    #3
  4. > > Hi,
    > >
    > > In message "Marshaling objects partially"
    > > on 03/07/07, Ronald Pijnacker <> writes:
    > >
    > > |I had a look at _dump and _load, but it seems to me that I'd have to
    > > |solve the cyclic references myself again. Is this true?
    > >
    > > Yes.
    > >
    > > But if you use Marshal.dump for _dump, Marshal resolves cycles
    > > (potential duplication might happen).

    >
    > Thanks, this works fine.
    >
    > Do you have any suggestions on how to orgainize _load and _dump such
    > that in a class hierarchy every class level is responsible for streaming
    > its own members?


    On second tought... it does not seem to work fine for cycles. The
    following example breaks down with a SystemStackError:

    class Foo
    attr_accessor :bar
    def _dump(limit)
    Marshal.dump([@bar], limit-1)
    end
    def Foo._load(str)
    data = Marshal.load(str)
    obj = self.new
    obj.bar = data.shift
    end
    end

    foo1 = Foo.new
    foo2 = Foo.new
    foo1.bar = foo2
    foo2.bar = foo1
    str = Marshal.dump(foo1)

    It works ok without the _dump/_load part.

    Ronald.
    Ronald Pijnacker, Jul 8, 2003
    #4
  5. Ronald Pijnacker

    ts Guest

    >>>>> "R" == Ronald Pijnacker <> writes:

    R> On second tought... it does not seem to work fine for cycles. The
    R> following example breaks down with a SystemStackError:

    Well, you must do like marshal. Try something like this (*not tested*)

    svg% cat b.rb
    #!/usr/bin/ruby
    class Foo
    attr_accessor :bar
    @@dumped = {}
    @@loaded = []
    @@num = 0

    def _dump(limit)
    if @@dumped[self]
    "\000#{@@dumped[self]}\000"
    else
    @@dumped[self] = @@num
    @@num += 1
    Marshal.dump([@bar], limit-1)
    end
    end

    def self._load(str)
    if /\A\000(\d+)\000\z/ =~ str
    obj = @@loaded[$1.to_i]
    else
    obj = self.new
    @@loaded.push(obj)
    obj.bar = Marshal.load(str).shift
    end
    obj
    end
    end

    foo1 = Foo.new
    foo2 = Foo.new
    foo1.bar = foo2
    foo2.bar = foo1
    p foo1
    p Marshal.load(Marshal.dump(foo1))
    svg%

    svg% b.rb
    #<Foo:0x40099100 @bar=#<Foo:0x40099010 @bar=#<Foo:0x40099100 ...>>>
    #<Foo:0x40098ebc @bar=#<Foo:0x40098e30 @bar=#<Foo:0x40098ebc ...>>>
    svg%


    The string "\000#{@@dumped[self]}\000" must be something never used by
    marshal


    Guy Decoux
    ts, Jul 8, 2003
    #5
  6. Hi,

    In message "Re: Marshaling objects partially"
    on 03/07/08, ts <> writes:

    |R> On second tought... it does not seem to work fine for cycles. The
    |R> following example breaks down with a SystemStackError:
    |
    | Well, you must do like marshal. Try something like this (*not tested*)

    Ah, let me think....

    To whom it may concern, can I increase format minor version on 1.8.0
    to solve this issue? You will have compatibility problem by this
    change, but minor version number will change in 1.8.0 anyway, so that
    the damage would be minimal.

    By this change, _dump() and _load() can return/receive arbitrary
    object, not only string.

    matz.
    Yukihiro Matsumoto, Jul 9, 2003
    #6
  7. Ronald Pijnacker

    ts Guest

    >>>>> "T" == Tanaka Akira <> writes:

    T> When object A and B refer each other, which _load is called first?

    I must say that I've not understood

    svg% cat b.rb
    #!/usr/bin/ruby
    class C
    attr_accessor :ref
    @@dumped = {}
    @@loaded = []
    @@num = 0

    def _dump(limit)
    if @@dumped[self]
    "\000#{@@dumped[self]}\000"
    else
    @@dumped[self] = @@num
    @@num += 1
    Marshal.dump([@ref], limit-1)
    end
    end

    def self._load(str)
    if /\A\000(\d+)\000\z/ =~ str
    obj = @@loaded[$1.to_i]
    else
    obj = self.new
    @@loaded.push(obj)
    obj.ref = Marshal.load(str).shift
    end
    obj
    end
    end

    a = C.new
    b = C.new

    a.instance_eval { @ref = b; @xxx = STDIN }
    b.instance_eval { @ref = a; @xxx = STDIN }

    p a
    m = Marshal.dump(a)
    c = Marshal.load(m)
    p c
    p c.ref

    svg%

    svg% b.rb
    #<C:0x40098e30 @ref=#<C:0x40098d40 @ref=#<C:0x40098e30 ...>, @xxx=#<IO:0x4009f348>>, @xxx=#<IO:0x4009f348>>
    #<C:0x40098b60 @ref=#<C:0x40098ad4 @ref=#<C:0x40098b60 ...>>>
    #<C:0x40098ad4 @ref=#<C:0x40098b60 @ref=#<C:0x40098ad4 ...>>>
    svg%


    Guy Decoux
    ts, Jul 9, 2003
    #7
  8. Ronald Pijnacker

    Christoph Guest

    "Tanaka Akira" <> wrote in message
    news:02.aist.go.jp...
    > In article <>,
    > ts <> writes:
    >
    > > I must say that I've not understood

    >
    > > svg% cat b.rb

    >
    > This works well because an object is allocated before child object is
    > loaded as:
    >
    > > obj = self.new
    > > @@loaded.push(obj)
    > > obj.ref = Marshal.load(str).shift

    >
    > But marshal.c cannot know an object allocated by C._load before
    > calling C._load.


    if I run Guys example through a fairly well tested (- the thread part;-)
    "Compare by Value" module

    ---
    module CompareByValue

    def ==(other)
    if not instance_of?(other.class)
    false
    elsif equal?(other)
    true
    elsif id < other.id
    CompareByValue.comp?(self,other)
    else
    CompareByValue.comp?(other,self)
    end
    end

    end


    class << CompareByValue

    # Seen is a book keeping class for currently
    # active thread local CompareByValue comparisons.
    class Seen < Hash

    # Helper class to store Id Pairs.
    const_set :pair, Struct.new:)l,:r)

    def initialize(l,r)
    store(Pair.new(l,r),true)
    end

    def default(key)
    store(key,true)
    nil
    end


    def seen?(l,r)
    self[Pair.new(l,r)]
    end

    def unseen(l,r)
    delete(Pair.new(l,r))
    end

    end

    # symbol mangling
    const_set :SEEN_COMPS, "CBV#{id}_comps".intern
    const_set :NUM_CALLS, "CBV#{id}_calls".intern

    def comp?(l,r)
    return true if thread_local_seen?(l,r)
    begin
    Thread.current[NUM_CALLS] +=1
    return Comp.new.comp?(l,r)
    rescue
    Thread.current[SEEN_COMPS].unseen(l,r)
    raise
    ensure
    if (Thread.current[NUM_CALLS] -= 1).zero?
    Thread.current[SEEN_COMPS].clear
    end
    end
    end

    def thread_local_seen?(l,r)
    if comps = Thread.current[SEEN_COMPS]
    comps.seen?(l,r)
    else
    Thread.current[NUM_CALLS] = 0
    Thread.current[SEEN_COMPS] = Seen.new(l,r)
    false
    end
    end


    class Comp < Hash

    def comp?(l,r)
    return true if l.equal?(r)
    vars = l.instance_variables.sort!
    return false unless vars == r.instance_variables.sort!
    #
    store(l.id,r.id)
    vars.each { |name|
    ll = l.instance_eval(name)
    rr = r.instance_eval(name)
    if CompareByValue === ll
    return false unless ll.instance_of?(rr.class)
    if rr_seen_id = self[ll.id]
    return false unless rr_seen_id == rr.id
    else
    return false unless comp?(ll,rr)
    end
    else
    return false unless ll == rr
    end
    }
    return true
    end

    end

    end



    class C
    include CompareByValue
    attr_accessor :ref
    @@dumped = {}
    @@loaded = []
    @@num = 0

    def _dump(limit)
    if @@dumped[self]
    "\000#{@@dumped[self]}\000"
    else
    @@dumped[self] = @@num
    @@num += 1
    Marshal.dump([@ref], limit-1)
    end
    end

    def self._load(str)
    if /\A\000(\d+)\000\z/ =~ str
    obj = @@loaded[$1.to_i]
    else
    obj = self.new
    @@loaded.push(obj)
    obj.ref = Marshal.load(str).shift
    end
    obj
    end
    end

    Kernel.const_set :A, C.new
    Kernel.const_set :B, C.new

    A.instance_eval { @ref = B; @xxx = 1 }
    B.instance_eval { @ref = A; @xxx = 2 }

    p A
    p B
    p a = Marshal.load(Marshal.dump(A))
    p b = Marshal.load(Marshal.dump(B))
    p B == A # false
    p b == a # true
    p B == a # false
    p b == a.ref # true
    p B == A.ref # true
    p B == a.ref # false
    p b == A.ref # false
    ---

    I am getting an output that does not look quite right to me

    ---
    #<C:0x2788020 @ref=#<C:0x2787918 @ref=#<C:0x2788020 ...>, @xxx=2>, @xxx=1>
    #<C:0x2787918 @ref=#<C:0x2788020 @ref=#<C:0x2787918 ...>, @xxx=1>, @xxx=2>
    #<C:0x2786d30 @ref=#<C:0x2786928 @ref=#<C:0x2786d30 ...>>>
    #<C:0x2786928 @ref=#<C:0x2786d30 @ref=#<C:0x2786928 ...>>>
    false
    true
    false
    true
    true
    false
    false
    ---

    but I am not sure if I would blame this on the current "_dump, _load"
    scheme.


    /Christoph
    Christoph, Jul 10, 2003
    #8
  9. Ronald Pijnacker

    Christoph Guest

    "Christoph" <swap(news_chr)@gmx.net> wrote in message
    news:bei96e$2r6q$...
    ....
    >
    > I am getting an output that does not look quite right to me


    Oh well never mind, I just read the beginning of the thread ...
    i.e. the output looks right to me now;-)


    /Christoph
    Christoph, Jul 10, 2003
    #9
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. Kannan
    Replies:
    0
    Views:
    549
    Kannan
    Mar 11, 2005
  2. isthar

    Marshaling unicode WDDX

    isthar, Jan 5, 2006, in forum: Python
    Replies:
    4
    Views:
    326
    =?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=
    Jan 10, 2006
  3. Replies:
    2
    Views:
    413
    Srini
    Aug 18, 2005
  4. Sree

    Namevaluecollection Vs Webservices(Marshaling)

    Sree, Sep 27, 2004, in forum: ASP .Net Web Services
    Replies:
    0
    Views:
    131
  5. Marshaling IIdentity via web services.

    , Dec 17, 2004, in forum: ASP .Net Web Services
    Replies:
    3
    Views:
    119
    Dilip Krishnan
    Dec 20, 2004
Loading...

Share This Page