Problem writing some strings (UnicodeEncodeError)

[Paulo da Silva]

Hi!

I am using a python3 script to produce a bash script from lots of
filenames got using os.walk.

I have a template string for each bash command in which I replace a
special string with the filename and then write the command to the bash
script file.

Something like this:

shf=open(bashfilename,'w')
filenames=getfilenames() # uses os.walk
for fn in filenames:
...
cmd=templ.replace("<fn>",fn)
shf.write(cmd)

For certain filenames I got a UnicodeEncodeError exception at
shf.write(cmd)!
I use utf-8 and have # -*- coding: utf-8 -*- in the source .py.

How can I fix this?

Thanks for any help/comments.

 

[Peter Otten]

I am using a python3 script to produce a bash script from lots of
filenames got using os.walk.

I have a template string for each bash command in which I replace a
special string with the filename and then write the command to the bash
script file.

Something like this:

shf=open(bashfilename,'w')
filenames=getfilenames() # uses os.walk
for fn in filenames:
...
cmd=templ.replace("<fn>",fn)
shf.write(cmd)

For certain filenames I got a UnicodeEncodeError exception at
shf.write(cmd)!
I use utf-8 and have # -*- coding: utf-8 -*- in the source .py.

How can I fix this?

Thanks for any help/comments.

You make it harder to debug your problem by not giving the complete
traceback. If the error message contains 'surrogates not allowed' like in
the demo below
.... f.write("\udcef")
....
Traceback (most recent call last):
File "<stdin>", line 2, in <module>
UnicodeEncodeError: 'utf-8' codec can't encode character '\udcef' in
position 0: surrogates not allowed

you have filenames that are not valid UTF-8 on your harddisk.

A possible fix would be to use bytes instead of str. For that you need to
open `bashfilename` in binary mode ("wb") and pass bytes to the os.walk()
call.

Or you just go and fix the offending names.

 

[Emile van Sebille]

Hi!

I am using a python3 script to produce a bash script from lots of
filenames got using os.walk.

I have a template string for each bash command in which I replace a
special string with the filename and then write the command to the bash
script file.

Something like this:

shf=open(bashfilename,'w')
filenames=getfilenames() # uses os.walk
for fn in filenames:
...
cmd=templ.replace("<fn>",fn)
shf.write(cmd)

For certain filenames I got a UnicodeEncodeError exception at
shf.write(cmd)!
I use utf-8 and have # -*- coding: utf-8 -*- in the source .py.

How can I fix this?

Not sure exactly, but I'd try


shf=open(bashfilename,'wb')

as a start.

HTH,

Emile

 

[Paulo da Silva]

Em 12-01-2014 16:23, Peter Otten escreveu:

You make it harder to debug your problem by not giving the complete
traceback. If the error message contains 'surrogates not allowed' like in
the demo below

... f.write("\udcef")
...
Traceback (most recent call last):
File "<stdin>", line 2, in <module>
UnicodeEncodeError: 'utf-8' codec can't encode character '\udcef' in
position 0: surrogates not allowed

That is the situation. I just lost it and it would take a few houres to
repeat the situation. Sorry.

you have filenames that are not valid UTF-8 on your harddisk.

A possible fix would be to use bytes instead of str. For that you need to
open `bashfilename` in binary mode ("wb") and pass bytes to the os.walk()
call.

This is my 1st time with python3, so I am confused!

As much I could understand it seems that os.walk is returning the
filenames exactly as they are on disk. Just bytes like in C.

My template is a string. What is the result of the replace command? Is
there any change in the filename from os.walk contents?

Now, if the result of the replace has the replaced filename unchanged
how do I "convert" it to bytes type, without changing its contents, so
that I can write to the bashfile opened with "wb"?

Or you just go and fix the offending names.

This is impossible in my case.
I need a bash script with the names as they are on disk.

 

[Peter Otten]

Em 12-01-2014 16:23, Peter Otten escreveu:

That is the situation. I just lost it and it would take a few houres to
repeat the situation. Sorry.


This is my 1st time with python3, so I am confused!

As much I could understand it seems that os.walk is returning the
filenames exactly as they are on disk. Just bytes like in C.

No, they are decoded with the preferred encoding. With UTF-8 that can fail,
and if it does the surrogateescape error handler replaces the offending
bytes with special codepoints:

....
8

['\udce4\udcf6\udcfc']

You can bypass the decoding process by providing a bytes argument to
os.listdir() (or os.walk() which uses os.listdir() internally):
[b'\xe4\xf6\xfc']

To write these raw bytes into a file the file has of course to be binary,
too.

My template is a string. What is the result of the replace command? Is
there any change in the filename from os.walk contents?

Now, if the result of the replace has the replaced filename unchanged
how do I "convert" it to bytes type, without changing its contents, so
that I can write to the bashfile opened with "wb"?


This is impossible in my case.
I need a bash script with the names as they are on disk.

I think instead of the hard way sketched out above it will be sufficient to
specify the error handler when opening the destination file

shf = open(bashfilename, 'w', errors="surrogateescape")

but I have not tried it myself. Also, some bytes may need to be escaped,
either to be understood by the shell, or to address security concerns:
.... print(template.replace("<fn>", filename))
....
ls foo; rm bar

 

[Paulo da Silva]

I think instead of the hard way sketched out above it will be sufficient to
specify the error handler when opening the destination file

shf = open(bashfilename, 'w', errors="surrogateescape")

This seems to fix everything!
I tried with a small test set and it worked.

but I have not tried it myself. Also, some bytes may need to be escaped,
either to be understood by the shell, or to address security concerns:

Since I am puting the file names between "", the only char that needs to
be escaped is the " itself.

I'm gonna try with the real thing.

Thank you very much for the fixing and for everything I have learned here.

 

[Peter Otten]

Since I am puting the file names between "", the only char that needs to
be escaped is the " itself.

What about the escape char?

 

[Paulo da Silva]

Em 12-01-2014 20:29, Peter Otten escreveu:

What about the escape char?

Just this fn=fn.replace('"','\\"')

So far I didn't find any problem, but the script is still running.

 

[Peter Otten]

Em 12-01-2014 20:29, Peter Otten escreveu:
Just this fn=fn.replace('"','\\"')

So far I didn't find any problem, but the script is still running.

To be a bit more explicit:
.... print(template.replace("<fn>", filename.replace('"', '\\"')))
....
ls "\\"; rm whatever; ls \"

 

[Peter Otten]

To be a bit more explicit:

... print(template.replace("<fn>", filename.replace('"', '\\"')))
...
ls "\\"; rm whatever; ls \"

The complete session:
.... print(template.replace("<fn>", filename.replace('"', '\\"')))
....
ls "\\"; rm whatever; ls \"


Shell variable substitution is another problem. c.l.py is probably not the
best place to get the complete list of possibilities.

 

[Paulo da Silva]

Em 13-01-2014 08:58, Peter Otten escreveu:

The complete session:

... print(template.replace("<fn>", filename.replace('"', '\\"')))
...
ls "\\"; rm whatever; ls \"


Shell variable substitution is another problem. c.l.py is probably not the
best place to get the complete list of possibilities.

I see what you mean.
This is a tedious problem. Don't know if there is a simple solution in
python for this. I have to think about it ...
On a more general and serious application I would not produce a bash
script. I would do all the work in python.

That's not the case, however. This is a few times execution script for a
very special purpose. The only problem was the occurrence of some
Portuguese characters in old filenames encoded in another code than
utf-8. Very few also include the ".

The worst thing that could happen was the bash script to abort. Then it
would be easy to fix it using a simple editor.

 

[Peter Otten]

Em 13-01-2014 08:58, Peter Otten escreveu:
I see what you mean.
This is a tedious problem. Don't know if there is a simple solution in
python for this. I have to think about it ...
On a more general and serious application I would not produce a bash
script. I would do all the work in python.

That's not the case, however. This is a few times execution script for a
very special purpose. The only problem was the occurrence of some
Portuguese characters in old filenames encoded in another code than
utf-8. Very few also include the ".

The worst thing that could happen was the bash script to abort. Then it
would be easy to fix it using a simple editor.

I looked around in the stdlib and found shlex.quote(). It uses ' instead of
" which simplifies things, and special-cases only ':
'alpha'"'"'beta'

So the answer is simpler than I had expected.

 

[Paulo da Silva]

Em 13-01-2014 17:29, Peter Otten escreveu:

I looked around in the stdlib and found shlex.quote(). It uses ' instead of
" which simplifies things, and special-cases only ':

'alpha'"'"'beta'

So the answer is simpler than I had expected.

Yes, it should work, at least in this case.
Although python oriented, it seems to work to bash also.
I need to remove the "" from the templates and use shlex.quote for
filenames. I'll give it a try.

Thanks