String concatenation - which is the fastest way ?

P

przemolicc

Hello,

I'd like to write a python (2.6/2.7) script which connects to database, fetches
hundreds of thousands of rows, concat them (basically: create XML)
and then put the result into another table. Do I have any choice
regarding string concatenation in Python from the performance point of view?
Since the number of rows is big I'd like to use the fastest possible library
(if there is any choice). Can you recommend me something ?

Regards
Przemyslaw Bak (przemol)
 
S

Steven D'Aprano

Hello,

I'd like to write a python (2.6/2.7) script which connects to database,
fetches hundreds of thousands of rows, concat them (basically: create XML)
and then put the result into another table. Do I have any choice
regarding string concatenation in Python from the performance point of
view ? Since the number of rows is big I'd like to use the fastest
possible library (if there is any choice). Can you recommend me something
?

For fast string concatenation, you should use the string.join method:

substrings = ['a', 'bb', 'ccc', 'dddd']
body = ''.join(substrings)

Using string addition in a loop, like this:

# Don't do this!
body = ''
for sub in substrings:
body += sub

risks being *extremely* slow for large numbers of substrings. (To be
technical, string addition can O(N**2), while ''.join is O(N).) This
depends on many factors, including the operating system's memory
management, and the Python version and implementation, so repeated addition
may be fast on one machine and slow on another. Better to always use join,
which is consistently fast.

You should limit string addition to small numbers of substrings:

result = head + body + tail # This is okay.
 

Ask a Question

Want to reply to this thread or ask your own question?

You'll need to choose a username for the site, which only take a couple of moments. After that, you can post your question and our members will help you out.

Ask a Question

Members online

Forum statistics

Threads
473,836
Messages
2,569,748
Members
45,545
Latest member
rapter____0

Latest Threads

Top