better way for ' '.join(args) + '\n'?

[Ulrich Eckhardt]

Hi!

General advise when assembling strings is to not concatenate them
repeatedly but instead use string's join() function, because it avoids
repeated reallocations and is at least as expressive as any alternative.

What I have now is a case where I'm assembling lines of text for driving
a program with a commandline interface. In this scenario, I'm currently
doing this:

args = ['foo', 'bar', 'baz']
line = ' '.join(args) + '\n'

So, in other words, I'm avoiding all the unnecessary copying, just to
make another copy to append the final newline.

The only way around this that I found involves creating an intermediate
sequence like ['foo', ' ', 'bar', ' ', 'baz', '\n']. This can be done
rather cleanly with a generator:

def helper(s):
for i in s[:-1]:
yield i
yield ' '
yield s[-1]
yield '\n'
line = ''.join(tmp(args))

Efficiency-wise, this is satisfactory. However, readability counts and
that is where this version fails and that is the reason why I'm writing
this message. So, dear fellow Pythonistas, any ideas to improve the
original versions efficiency while preserving its expressiveness?

Oh, for all those that are tempted to tell me that this is not my
bottleneck unless it's called in a very tight loop, you're right.
Indeed, the overhead of the communication channel TCP between the two
programs is by far dwarving the few microseconds I could save here. I'm
still interested in learning new and better solutions though.


Cheers!

Uli

 

[Peter Otten]

Hi!

General advise when assembling strings is to not concatenate them
repeatedly but instead use string's join() function, because it avoids
repeated reallocations and is at least as expressive as any alternative.

What I have now is a case where I'm assembling lines of text for driving
a program with a commandline interface. In this scenario, I'm currently
doing this:

args = ['foo', 'bar', 'baz']
line = ' '.join(args) + '\n'

So, in other words, I'm avoiding all the unnecessary copying, just to
make another copy to append the final newline.

The only way around this that I found involves creating an intermediate
sequence like ['foo', ' ', 'bar', ' ', 'baz', '\n']. This can be done
rather cleanly with a generator:

def helper(s):
for i in s[:-1]:
yield i
yield ' '
yield s[-1]
yield '\n'
line = ''.join(tmp(args))

Efficiency-wise, this is satisfactory.

No, it is not. In a quick timeit test it takes 5 to 10 times as long as the
original. Remember that function calls are costly, and that with s[:-1] you
are trading the extra string for an extra list. Also, you are doubling the
loop implicit in str.join() with the explicit one in your oh-so-efficient
generator.

However, readability counts and
that is where this version fails and that is the reason why I'm writing
this message. So, dear fellow Pythonistas, any ideas to improve the
original versions efficiency while preserving its expressiveness?

Oh, for all those that are tempted to tell me that this is not my
bottleneck unless it's called in a very tight loop, you're right.
Indeed, the overhead of the communication channel TCP between the two
programs is by far dwarving the few microseconds I could save here. I'm
still interested in learning new and better solutions though.

Even if it were the bottleneck the helper generator approach would still be
unhelpful.

 

[Steven D'Aprano]

Hi!

General advise when assembling strings is to not concatenate them
repeatedly but instead use string's join() function, because it avoids
repeated reallocations and is at least as expressive as any alternative.

What I have now is a case where I'm assembling lines of text for driving
a program with a commandline interface. In this scenario, I'm currently
doing this:

args = ['foo', 'bar', 'baz']
line = ' '.join(args) + '\n'

So, in other words, I'm avoiding all the unnecessary copying, just to
make another copy to append the final newline.

*shrug*

The difference between ' '.join(sequence) and (' '.join(sequence) + '\n')
is, in Big Oh analysis, insignificant. The first case does O(N)
operations, the second does O(N) + O(N) = 2*O(N) operations, which is
still O(N). In effect, the two differ only by an approximately constant
factor.

If you really care, and you don't mind ending your last line with a
space, just append '\n' to the sequence before calling join.

The only way around this that I found involves creating an intermediate
sequence like ['foo', ' ', 'bar', ' ', 'baz', '\n']. This can be done
rather cleanly with a generator:

def helper(s):
for i in s[:-1]:
yield i
yield ' '
yield s[-1]
yield '\n'
line = ''.join(tmp(args))

Efficiency-wise, this is satisfactory.

Have you actually tested this? I would not be the least surprised if
that's actually less efficient than the (' '.join(seq) + '\n') version.

 

[Hubert Grünheidt]

Hi Ulrich,

is this acceptable?

args = ['foo', 'bar', 'baz']
args.append('\n')
line = ' '.join(args)

Cheers,
Hubert

 

[Tycho Andersen]

Hi!

General advise when assembling strings is to not concatenate them
repeatedly but instead use string's join() function, because it
avoids repeated reallocations and is at least as expressive as any
alternative.

What I have now is a case where I'm assembling lines of text for
driving a program with a commandline interface. In this scenario,
I'm currently doing this:

args = ['foo', 'bar', 'baz']
line = ' '.join(args) + '\n'

Assuming it's the length of the list that's the problem, not the
length of the strings in the list...

args = ['foo', 'bar', 'baz']
args[-1] = args[-1] + '\n'
line = ' '.join(args)

\t

 

[Dave Angel]

Hi!

General advise when assembling strings is to not concatenate them
repeatedly but instead use string's join() function, because it
avoids repeated reallocations and is at least as expressive as any
alternative.

What I have now is a case where I'm assembling lines of text for
driving a program with a commandline interface. In this scenario,
I'm currently doing this:

args = ['foo', 'bar', 'baz']
line = ' '.join(args) + '\n'

Assuming it's the length of the list that's the problem, not the
length of the strings in the list...

args = ['foo', 'bar', 'baz']
args[-1] = args[-1] + '\n'
line = ' '.join(args)

\t

Main problem with that is the trailing space before the newline. If
that's not a problem, then fine.

Not sure why we try so hard to optimize something that's going to take
negligible time.

 

[Tycho Andersen]

Assuming it's the length of the list that's the problem, not the
length of the strings in the list...

args = ['foo', 'bar', 'baz']
args[-1] = args[-1] + '\n'
line = ' '.join(args)

\t

Main problem with that is the trailing space before the newline. If
that's not a problem, then fine.

What trailing space before the newline? The other solutions have it,
the above does not. However, the above does mutate args, which isn't
all that great. Alas, if you want the performance of mutable
structures, you're probably going to have to mutate something. (In any
case, it's easy enough to change it back, though ugly.)

Not sure why we try so hard to optimize something that's going to take
negligible time.

The same reason some people enjoy sporting events: it's fun

\t

 

[Thomas Rachel]

Am 26.10.2012 09:49 schrieb Ulrich Eckhardt:

Hi!

General advise when assembling strings is to not concatenate them
repeatedly but instead use string's join() function, because it avoids
repeated reallocations and is at least as expressive as any alternative.

What I have now is a case where I'm assembling lines of text for driving
a program with a commandline interface.

Stop.

In this case, you think too complicated.

Just do

subprocess.Popen(['prog', 'foo', 'bar', 'baz'])

- is the most safest thing for this use case.

If it should not be possible for any reason, you should be aware of any
traps you could catch - e.g., if you want to feed your string to a
Bourne shell, you should escape the strings properly.

In such cases, I use


def shellquote(*strs):
r"""Input: file names, output: ''-enclosed strings where every ' is
replaced with '\''. Intended for usage with the shell."""
# just take over everything except ';
# replace ' with '\''
# The shell sees ''' as ''\'''\'''\'''. Ugly, but works.
return " ".join([
"'"+st.replace("'","'\\''")+"'"
for st in strs
])


so I can use

shellquote('program name', 'argu"ment 1', '$arg 2',
"even args containing a ' are ok")

For Windows, you'll have to modify this somehow.


HTH,

Thomas

 

[Prasad, Ramit]

Am 26.10.2012 09:49 schrieb Ulrich Eckhardt:

Hi!

General advise when assembling strings is to not concatenate them
repeatedly but instead use string's join()function, because it avoids
repeated reallocations and is at least as expressive as any alternative.

What I have now is a case where I'm assembling lines of text for driving
a program with a commandline interface.

Stop.

In thiscase, you think too complicated.

Just do

subprocess.Popen(['prog', 'foo', 'bar', 'baz'])

- is the most safest thing for this use case.

If it should not be possible for any reason, you should be aware of any
traps you could catch - e.g., if you want to feed your string to a
Bourne shell, you should escape the strings properly.

In such cases,I use


def shellquote(*strs):
r"""Input: file names, output: ''-enclosed strings where every ' is
replaced with '\''. Intended for usage with the shell."""
# just take over everything except ';
# replace ' with '\''
# The shell sees ''' as ''\'''\'''\'''. Ugly, but works.
return " ".join([
"'"+st.replace("'","'\\''")+"'"
for st in strs
])


so I can use

shellquote('program name', 'argu"ment 1', '$arg 2',
"even args containing a ' are ok")

For Windows, you'll have to modify this somehow.

The subprocess module suggests using pipes.quote for escaping.

a
('program name', 'argu"ment 1', '$arg 2', "even args containing a ' are ok")
import pipes
map(pipes.quote, a)
["'program name'", '\'argu"ment 1\'', "'$arg 2'", '\'even args containing a \'"\'"\' are ok\'']
' '.join(a)

'\'program name\' \'argu"ment 1\' \'$arg 2\' \'even args containing a \'\\\'\' are ok\''


Ramit Prasad


This email is confidential and subject to important disclaimers and
conditions including on offers for the purchase or sale of
securities, accuracy and completeness ofinformation, viruses,
confidentiality, legal privilege, and legal entity disclaimers,
available at http://www.jpmorgan.com/pages/disclosures/email.

 

[Ramchandra Apte]

On 10/26/2012 05:26 PM, Tycho Andersen wrote:

Assuming it's the length of the list that's the problem, not the
length of the strings in the list...

args = ['foo', 'bar', 'baz']
args[-1] = args[-1] + '\n'
line = ' '.join(args)

\t

Main problem with that is the trailing space before the newline. If

that's not a problem, then fine.

What trailing space before the newline? The other solutions have it,

the above does not. However, the above does mutate args, which isn't

all that great. Alas, if you want the performance of mutable

structures, you're probably going to have to mutate something. (In any

case, it's easy enough to change it back, though ugly.)

Not sure why we try so hard to optimize something that's going to take

negligible time.

The same reason some people enjoy sporting events: it's fun Me too



\t

 

[Ramchandra Apte]

On 10/26/2012 05:26 PM, Tycho Andersen wrote:

Assuming it's the length of the list that's the problem, not the
length of the strings in the list...

args = ['foo', 'bar', 'baz']
args[-1] = args[-1] + '\n'
line = ' '.join(args)

\t

Main problem with that is the trailing space before the newline. If

that's not a problem, then fine.

What trailing space before the newline? The other solutions have it,

the above does not. However, the above does mutate args, which isn't

all that great. Alas, if you want the performance of mutable

structures, you're probably going to have to mutate something. (In any

case, it's easy enough to change it back, though ugly.)

Not sure why we try so hard to optimize something that's going to take

negligible time.

The same reason some people enjoy sporting events: it's fun Me too



\t