A question about thrift performance.

Discussion in 'Python' started by Vincent, Jan 6, 2013.

  1. Vincent

    Vincent Guest

    Hi, all

    I have issue of thrift-performance in python, does anyone has an experience on thrift-in-python?

    My question in stackoverflow:

    http://stackoverflow.com/questions/14171227/why-is-thrift-binary-protocol-serialization-so-much-slow


    Copy the question to here(open stackoverflow to check pretty-print question content):

    I'm newbie on thrift. I wrote a thrift server in python, also client in python too.

    Here is my thrift defination:
    struct RatingByReport {
    1: required string ticker,
    2: required i32 cnt_institution,
    3: optional list<string> strong_buy,
    4: optional list<string> buy,
    5: optional list<string> neutral,
    6: optional list<string> sell,
    7: optional list<string> strong_sell,
    8: optional i32 cnt_maintain,
    9: optional i32 cnt_upgrade,
    10: optional i32 cnt_downgrade,
    11: optional i32 avg_score,
    12: optional string adjustment
    }

    struct TableRatingByReport {
    1: required list<string> head,
    2: required list<RatingByReport> body,
    3: optional struct.CadaTranslation translation
    }



    service china{
    void ping(),
    TableRatingByReport rating_byreport(1:string ticker) throws (1:struct.CadaInternalError error)
    }
    Here is my server side:
    handler = StockChinaHandler()
    processor = china.Processor(handler)
    #startup()

    transport = TSocket.TServerSocket(port=30303)
    tfactory = TTransport.TBufferedTransportFactory()
    pfactory = TBinaryProtocol.TBinaryProtocolFactory()

    server = TServer.TSimpleServer(processor, transport, tfactory, pfactory)
    #server = TProcessPoolServer.TProcessPoolServer(processor, transport,
    # tfactory, pfactory)

    print "Start server..."
    import cProfile
    print >>open('/tmp/test.log', 'w'), cProfile.run('server.serve()',
    sort='cumulative')
    #server.serve()
    print "done!"
    Client side:
    # Make socket
    transport = TSocket.TSocket('localhost', 30303)

    # Buffering is critical. Raw sockets are very slow
    transport = TTransport.TBufferedTransport(transport)

    # Wrap in a protocol
    protocol = TBinaryProtocol.TBinaryProtocol(transport)

    # Create a client to use the protocol encoder
    client = china.Client(protocol)

    # Connect!
    transport.open()

    client.ping()
    print "ping()"

    print msg
    msg = client.rating_byreport('2012-01-04')
    print msg
    transport.close()
    cProfile result:
    ncalls tottime percall cumtime percall filename:lineno(function)
    1 0.000 0.000 230.968 230.968 <string>:1(<module>)
    1 0.000 0.000 230.968 230.968 TServer.py:74(serve)
    3 0.000 0.000 225.967 75.322 TSocket.py:172(accept)
    3 0.000 0.000 225.967 75.322 socket.py:194(accept)
    3 225.967 75.322 225.967 75.322 {method 'accept' of '_socket.socket' objects}
    5 0.003 0.001 4.993 0.999 china.py:140(process)
    1 0.000 0.000 3.200 3.200 china.py:177(process_rating_byreport)
    1 0.000 0.000 2.366 2.366 china.py:500(write)
    1 0.003 0.003 2.366 2.366 ttypes.py:515(write)
    1455 0.261 0.000 2.363 0.002 ttypes.py:364(write)
    155556 0.246 0.000 1.995 0.000 TCompactProtocol.py:38(nested)
    145880 0.298 0.000 1.640 0.000 TCompactProtocol.py:255(__writeString)
    18 1.370 0.076 1.370 0.076 {method 'recv' of '_socket.socket' objects}
    5 0.000 0.000 1.292 0.258 TCompactProtocol.py:306(readMessageBegin)
    13 0.000 0.000 1.292 0.099 TCompactProtocol.py:286(__readUByte)
    26 0.000 0.000 1.291 0.050 TTransport.py:54(readAll)
    26 0.000 0.000 1.291 0.050 TTransport.py:154(read)
    5 0.000 0.000 1.291 0.258 TSocket.py:101(read)
    In my case, TableRatingByReport instance got a body with 1400 rows(list\), and It cost over 3 seconds(function *process_rating_byreport*, which is auto generate by thift) to generate binnary content. I don't know why it is so slow.

    Using json to serialize the same data, It's only spend less than 200 ms.

    I'm wondering did I use the uncorrect way to manipulate thrift?

    Thanks.
    Vincent, Jan 6, 2013
    #1
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. Don Beal
    Replies:
    13
    Views:
    839
    Richard Grimes [MVP]
    Sep 29, 2003
  2. jm
    Replies:
    1
    Views:
    507
    alien2_51
    Dec 12, 2003
  3. cjl
    Replies:
    3
    Views:
    986
    John Nagle
    May 21, 2007
  4. Software Engineer
    Replies:
    0
    Views:
    324
    Software Engineer
    Jun 10, 2011
  5. ankit patel
    Replies:
    1
    Views:
    459
Loading...

Share This Page