SimpleJson is slow .... is there any C Compiled version ?

[sanket]

Hello All,

I have created an API which fetches some data from the database.
I am using simplejson to encode it and return it back.

Now the problem is that, this API is being called for millions of
times in a sequence.
I ran a profiler and saw that most of the time is consumed in encoding
my database results in to json.
So I was just wondering is there any C compiled version of simplejson
is available?
or any help regarding this issue would be appreciated.

Thank you,
Sanket

 

[Matt Nordhoff]

Hello All,

I have created an API which fetches some data from the database.
I am using simplejson to encode it and return it back.

Now the problem is that, this API is being called for millions of
times in a sequence.
I ran a profiler and saw that most of the time is consumed in encoding
my database results in to json.
So I was just wondering is there any C compiled version of simplejson
is available?
or any help regarding this issue would be appreciated.

Thank you,
Sanket

simplejson is not the only JSON library out there. For example, there's
python-cjson, which is written entirely in C:

<http://pypi.python.org/pypi/python-cjson>

There's also an enhanced version of it:

<http://python.cx.hu/python-cjson/>

I think simplejson has some small, optional C bits that will improve
performance if you compile them.

Also, be aware that I think simplejson is being integrated into the
stdlib in Python 2.6.

Also, simplejson and python-cjson might not be entirely compatible:
there's one character that one escapes and the other doesn't, or something.
--

 

[sanket]

simplejson is not the only JSON library out there. For example, there's
python-cjson, which is written entirely in C:

<http://pypi.python.org/pypi/python-cjson>

There's also an enhanced version of it:

<http://python.cx.hu/python-cjson/>

I think simplejson has some small, optional C bits that will improve
performance if you compile them.

Also, be aware that I think simplejson is being integrated into the
stdlib in Python 2.6.

Also, simplejson and python-cjson might not be entirely compatible:
there's one character that one escapes and the other doesn't, or something.
--

Thanks Matt.
I used cjson and it appears to be 5-6 times faster.

Thank you,
Sanket

 

[Richard Levasseur]

Also, simplejson and python-cjson might not be entirely compatible:
there's one character that one escapes and the other doesn't, or something.
--

They also have different interface. simplejson uses load/loads/dump/
dumps, whereas cjson using encode/decode (its pretty simple to
monkeypatch cjson with some lambda's so it has the same interface,
though).

Yeah, its frustrating that there are so many json encoders for
python. Its a bit disappointing simplejson is going into the stdlib;
i hope they rewrite it in C?

http://blog.hill-street.net/?p=7
Has a good comparison of encoders.

 

[Dan Stromberg]

They also have different interface. simplejson uses load/loads/dump/
dumps, whereas cjson using encode/decode (its pretty simple to
monkeypatch cjson with some lambda's so it has the same interface,
though).

Yeah, its frustrating that there are so many json encoders for python.
Its a bit disappointing simplejson is going into the stdlib; i hope they
rewrite it in C?

C's great for performance, but if you want something featureful and
reliable, you're probably better off in something else - including but
not limited to python, optionally with psyco.

I wish the pypy people would package up their stuff into a form that you
can just ./configure && make && make install and then #! to - I believe
it'd grow/mature faster than it is if they did (and they may have, but
last time I looked it didn't appear so). But once they do, that might be
a convenient way of getting better performance.

 

[Joshua Kugler]

Hello All,

I have created an API which fetches some data from the database.
I am using simplejson to encode it and return it back.

Now the problem is that, this API is being called for millions of
times in a sequence.
I ran a profiler and saw that most of the time is consumed in encoding
my database results in to json.
So I was just wondering is there any C compiled version of simplejson
is available?
or any help regarding this issue would be appreciated.

simplejson does have a C module. It is compiled automatically when
installed. If you're installing on Windows, then it probably isn't getting
compiled.

j