Pack optimization

Discussion in 'Python' started by BDL, Oct 15, 2009.

  1. BDL

    BDL Guest

    I have a large amount of binary data that needs to be pushed across
    the network. It appears from profiling that the dominant time is
    being taken up by packing the data. (50%) Here is a CME that shows
    the problem.

    from numpy import random
    from struct import pack
    import time

    lenstim = 10084200
    sigma = 0.1
    stim = random.normal(0., sigma, lenstim) # 10084200 gaussian random
    numbers (doubles)

    fmt = '!h'+str(stim.size)+'d' # makes fmt = '!h10084200d'
    cmd = 4

    startTime = time.time()
    packdat = pack( fmt, cmd, *stim )
    elapsed = time.time() - startTime
    print "Time to pack the command and data %.6f seconds " % elapsed

    Is there a faster method to do this? Is it possible to use array?
    Any suggestions would be appreciated.
     
    BDL, Oct 15, 2009
    #1
    1. Advertising

  2. BDL

    Paul Rubin Guest

    BDL <> writes:
    > Is there a faster method to do this? Is it possible to use array?


    array.tostring is the first thing I'd think of.
     
    Paul Rubin, Oct 15, 2009
    #2
    1. Advertising

  3. BDL

    Robert Kern Guest

    On 2009-10-15 13:57 PM, BDL wrote:
    > I have a large amount of binary data that needs to be pushed across
    > the network. It appears from profiling that the dominant time is
    > being taken up by packing the data. (50%) Here is a CME that shows
    > the problem.
    >
    > from numpy import random
    > from struct import pack
    > import time
    >
    > lenstim = 10084200
    > sigma = 0.1
    > stim = random.normal(0., sigma, lenstim) # 10084200 gaussian random
    > numbers (doubles)
    >
    > fmt = '!h'+str(stim.size)+'d' # makes fmt = '!h10084200d'
    > cmd = 4
    >
    > startTime = time.time()
    > packdat = pack( fmt, cmd, *stim )
    > elapsed = time.time() - startTime
    > print "Time to pack the command and data %.6f seconds " % elapsed
    >
    > Is there a faster method to do this? Is it possible to use array?
    > Any suggestions would be appreciated.


    If you're already using numpy, use its routines to convert to string
    representations (like stim.tostring(), but there are better alternatives like
    the NPY file format). Don't use struct.pack() for large homogeneous data.

    If you have further numpy questions, you should ask them on the numpy mailing list:

    http://www.scipy.org/Mailing_Lists

    --
    Robert Kern

    "I have come to believe that the whole world is an enigma, a harmless enigma
    that is made terrible by our own mad attempt to interpret it as though it had
    an underlying truth."
    -- Umberto Eco
     
    Robert Kern, Oct 15, 2009
    #3
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. Matthew Louden

    failed to install .NET Framework Service Pack 2

    Matthew Louden, Dec 9, 2003, in forum: ASP .Net
    Replies:
    3
    Views:
    6,230
    Thomas Scheidegger [MVP]
    Dec 11, 2003
  2. Mike

    no dotnet 1.1 service pack?!

    Mike, Apr 30, 2004, in forum: ASP .Net
    Replies:
    1
    Views:
    468
    news.microsoft.com
    May 3, 2004
  3. Tim Jones
    Replies:
    0
    Views:
    395
    Tim Jones
    Jan 31, 2004
  4. Ravikiran

    Zero Optimization and Sign Optimization???

    Ravikiran, Nov 17, 2008, in forum: C Programming
    Replies:
    22
    Views:
    890
    Thad Smith
    Nov 24, 2008
  5. Alexander Farber

    pack 'C3U*' not same as pack 'C3(xC)*'

    Alexander Farber, Jun 23, 2005, in forum: Perl Misc
    Replies:
    2
    Views:
    152
    Ilmari Karonen
    Jun 23, 2005
Loading...

Share This Page