Re: Fastest way to serialize arbitrary objects ???

Discussion in 'C++' started by Rui Maciel, Apr 30, 2010.

  1. Rui Maciel

    Rui Maciel Guest

    Peter Olcott wrote:

    > I want to know some ideas to provide very fast object
    > serialization and deserialization for arbitrary objects.


    What do you consider "very fast"?


    Rui Maciel
    Rui Maciel, Apr 30, 2010
    #1
    1. Advertising

  2. Rui Maciel

    Ian Collins Guest

    On 05/ 1/10 01:44 AM, Peter Olcott wrote:
    > "Rui Maciel"<> wrote in message
    > news:4bdaa0e6$0$28484$...
    >> Peter Olcott wrote:
    >>
    >>> I want to know some ideas to provide very fast object
    >>> serialization and deserialization for arbitrary objects.

    >>
    >> What do you consider "very fast"?

    >
    > As close to as fast as possible as can be reasonably
    > implemented. The fastest possible method would be a method
    > that fills all of the data structures with a single block
    > read and nothing else.


    You haven't said what you want to serialise for. If it's for
    transmission or storage, the I/O will be orders of magnitude slower then
    even the most convoluted serialisation process (XML!).

    >> Rui Maciel

    >
    >

    Please remember to trim.

    --
    Ian Collins
    Ian Collins, May 1, 2010
    #2
    1. Advertising

  3. Rui Maciel

    Ian Collins Guest

    On 05/ 1/10 03:37 PM, Peter Olcott wrote:
    > "Ian Collins"<> wrote in message
    > news:...
    >> On 05/ 1/10 01:44 AM, Peter Olcott wrote:
    >>> "Rui Maciel"<> wrote in message
    >>> news:4bdaa0e6$0$28484$...
    >>>> Peter Olcott wrote:
    >>>>
    >>>>> I want to know some ideas to provide very fast object
    >>>>> serialization and deserialization for arbitrary
    >>>>> objects.
    >>>>
    >>>> What do you consider "very fast"?
    >>>
    >>> As close to as fast as possible as can be reasonably
    >>> implemented. The fastest possible method would be a
    >>> method
    >>> that fills all of the data structures with a single block
    >>> read and nothing else.

    >>
    >> You haven't said what you want to serialise for. If it's
    >> for transmission or storage, the I/O will be orders of
    >> magnitude slower then even the most convoluted
    >> serialisation process (XML!).

    >
    > Storage.


    Then you are overly concerned with performance. I nearly always use
    JSON these days and I've never been anything other that I/O bound.

    > Actually I measured the I/O to be between two and
    > ten times (depending on platform) slower than fast
    > serialization. It seems that most of the I/O time is for
    > disk seek + rotational latency, thus a single block read or
    > written to disk is much faster than any smaller portions. My
    > very cheap SATA drive reads 1.0 GB at 300 MB / Sec.


    To the cache maybe, but not to the platters.

    --
    Ian Collins
    Ian Collins, May 1, 2010
    #3
  4. Rui Maciel

    Ian Collins Guest

    On 05/ 2/10 01:55 AM, Peter Olcott wrote:
    > "Ian Collins"<> wrote:
    >> On 05/ 1/10 03:37 PM, Peter Olcott wrote:
    >>> "Ian Collins"<> wrote:
    >>>>
    >>>> You haven't said what you want to serialise for. If
    >>>> it's
    >>>> for transmission or storage, the I/O will be orders of
    >>>> magnitude slower then even the most convoluted
    >>>> serialisation process (XML!).
    >>>
    >>> Storage.

    >>
    >> Then you are overly concerned with performance. I nearly
    >> always use JSON these days and I've never been anything
    >> other that I/O bound.

    >
    > How many MB / Sec can JSON do? (I can do 600).


    In the words of old Rolls Royce literature: the performance is adequate.
    It has never been a bottleneck, so I haven't had cause to measure or
    tweak.

    >>> Actually I measured the I/O to be between two and
    >>> ten times (depending on platform) slower than fast
    >>> serialization. It seems that most of the I/O time is for
    >>> disk seek + rotational latency, thus a single block read
    >>> or
    >>> written to disk is much faster than any smaller portions.
    >>> My
    >>> very cheap SATA drive reads 1.0 GB at 300 MB / Sec.

    >>
    >> To the cache maybe, but not to the platters.

    >
    > My cache is only 16 MB, so 1.0 GB read at 300 MB/Sec must be
    > actual sustained throughput.


    Then you either have a new speed record, or you have measured wrong.

    --
    Ian Collins
    Ian Collins, May 1, 2010
    #4
  5. Rui Maciel

    Ian Collins Guest

    On 05/ 2/10 10:57 AM, Peter Olcott wrote:
    > "Ian Collins"<> wrote in message
    > news:...
    >> On 05/ 2/10 01:55 AM, Peter Olcott wrote:

    >
    >>>> Then you are overly concerned with performance. I
    >>>> nearly
    >>>> always use JSON these days and I've never been anything
    >>>> other that I/O bound.
    >>>
    >>> How many MB / Sec can JSON do? (I can do 600).

    >>
    >> In the words of old Rolls Royce literature: the
    >> performance is adequate. It has never been a bottleneck,
    >> so I haven't had cause to measure or tweak.

    >
    > Thus probably not a small and fast as possible, thus not
    > good enough.


    Obviously it isn't as fast as possible, but it's good enough for
    anything I've used it for. What is your application and how did you
    measure the bottlenecks?

    >> --
    >> Ian Collins

    >


    Please trim!

    --
    Ian Collins
    Ian Collins, May 2, 2010
    #5
  6. Rui Maciel

    Liviu Guest

    "Peter Olcott" <> wrote...
    > "Ian Collins" <> wrote...
    >> On 05/ 2/10 01:55 AM, Peter Olcott wrote:

    >
    >>> My cache is only 16 MB, so 1.0 GB read at 300 MB/Sec
    >>> must be actual sustained throughput.

    >>
    >> Then you either have a new speed record, or you have measured wrong.

    >
    > I think the truth is a third answer, Windows 7 cheats.
    >
    > The red light of the drive is on long after the program returns
    > indicating that the read is still 300 MB / Sec. Windows measures this
    > as 3 seconds, but apparently actually takes much longer.


    Then your program does not measure what you think/claim it does.
    Off-topic win32 hint: lookup FlushFileBuffers in the docs.

    Liviu
    Liviu, May 2, 2010
    #6
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. Mike Larkin
    Replies:
    1
    Views:
    6,061
    Alvin Bruney [MVP - ASP.NET]
    May 23, 2005
  2. Gordz
    Replies:
    3
    Views:
    3,924
    John Oakes
    Jun 7, 2004
  3. Siegfried Ertl
    Replies:
    1
    Views:
    291
    Roedy Green
    Aug 5, 2003
  4. Honestmath
    Replies:
    5
    Views:
    549
    Honestmath
    Dec 13, 2004
  5. Brian
    Replies:
    4
    Views:
    345
    Liviu
    May 2, 2010
Loading...

Share This Page