strptime() in _strptime.py vs lib-dynload/time.so

[igor.tatarinov]

It looks like there are two implementation of strptime() (why?) and
the one that's used by default is the Python version in _strptime.py

Unfortunately, it's pretty slow and takes up a big chunk of my code's
execution time. Is there a way to use the C version instead (is there
a C version in time.so?)

6199597 function calls (6199433 primitive calls) in 45.820
CPU seconds

Ordered by: internal time
List reduced from 183 to 10 due to restriction <10>
ncalls tottime percall cumtime percall
filename:lineno(function)
57765 6.020 0.000 12.970 0.000 /usr/lib64/python2.4/
_strptime.py:273(strptime)
....

 

[igor.tatarinov]

ncalls tottime percall cumtime percall filename:lineno(function)
57765 6.020 0.000 12.970 0.000 /usr/lib64/python2.4/_strptime.py:273(strptime)
...

actually, the C-version of strptime() is also getting called:
57765 0.960 0.000 13.940 0.000 :0(strptime)

that's pretty confusing. Does the Python version call the C-version
(or the other way around)?

Thanks,
igor

 

[Gabriel Genellina]

actually, the C-version of strptime() is also getting called:
57765 0.960 0.000 13.940 0.000 :0(strptime)

that's pretty confusing. Does the Python version call the C-version
(or the other way around)?

The other way. The strptime function inside the time module (timemodule.c)
just calls the strptime function written in Python (_strptime.py).
I have no idea why it's so slow in your case.

 

[igor.tatarinov]

The other way. The strptime function inside the time module (timemodule.c)
just calls the strptime function written in Python (_strptime.py).
I have no idea why it's so slow in your case.

Thanks, that helps. Still I don't understand why the native (C)
version isn't available.
Perhaps, it's broken under Windows or something.

In my test code, strptime() is called approx 60K times (5M times in
the real code)
and that takes 6 CPU secs. Thus, we get 100 usecs per call. That's
much slower
than running strptime from command line. Perhaps profiling slows
things down.

$ python -mtimeit -s 'from time import mktime, strptime'
'mktime(strptime("070501", "%y%m%d"))'
10000 loops, best of 3: 40.4 usec per loop

An easy fix in my case is to avoid strptime() altogether:
python -mtimeit -s 'from time import mktime; s="070501"'
'mktime((int(s[0:2]), int(s[2:4]), int(s[4:6]),0,0,0,0,0,-1))'
100000 loops, best of 3: 8.6 usec per loop

Quite a bit faster.
igor
It would be nice to have access to the native version of strptime()
for performance reasons.

 

[Gabriel Genellina]

The other way. The strptime function inside the time module
(timemodule.c)
just calls the strptime function written in Python (_strptime.py).
I have no idea why it's so slow in your case.

Thanks, that helps. Still I don't understand why the native (C)
version isn't available.
Perhaps, it's broken under Windows or something.

In my test code, strptime() is called approx 60K times (5M times in
the real code)
and that takes 6 CPU secs. Thus, we get 100 usecs per call. That's
much slower
than running strptime from command line. Perhaps profiling slows
things down.

$ python -mtimeit -s 'from time import mktime, strptime'
'mktime(strptime("070501", "%y%m%d"))'
10000 loops, best of 3: 40.4 usec per loop

An easy fix in my case is to avoid strptime() altogether:
python -mtimeit -s 'from time import mktime; s="070501"'
'mktime((int(s[0:2]), int(s[2:4]), int(s[4:6]),0,0,0,0,0,-1))'
100000 loops, best of 3: 8.6 usec per loop

I think that if your date format is simple enough, parsing it as above may
be even faster than using the native strptime.

It would be nice to have access to the native version of strptime()
for performance reasons.

You can see the story of strptime at http://bugs.python.org/issue474274
Looks like a native strptime may not be always present, or unreliable.
Perhaps if certain platform has strptime available and it's reliable and
produces the right results, at configure time it could be detected and
used instead of the pure Python implementation...