Ldap module and base64 oncoding

[Joseph L. Casale]

I have some data I am working with that is not being interpreted as a string requiring
base64 encoding when sent to the ldif module for output.

The base64 string parsed is ZGV0XDMzMTB3YmJccGc= and the raw string is det\3310wbb\pg.
I'll admit my understanding of the handling requirements of non ascii data in 2.7 is weak
and as such I am failing at adjusting the regex that deduces is the string contains characters
requiring base64 encoding when being output.

Any insight, or nudges in the right direction would be appreciated!
Thanks,
jlc

 

[Michael Ströder]

I have some data I am working with that is not being interpreted as a string requiring
base64 encoding when sent to the ldif module for output.

The base64 string parsed is ZGV0XDMzMTB3YmJccGc= and the raw string is det\3310wbb\pg.
I'll admit my understanding of the handling requirements of non ascii data in 2.7 is weak
and as such I am failing at adjusting the regex that deduces is the string contains characters
requiring base64 encoding when being output.

I'm not sure what exactly you're asking for.
Especially "is not being interpreted as a string requiring base64 encoding" is
written without giving the right context.

So I'm just guessing that this might be the usual misunderstandings with use
of base64 in LDIF. Read more about when LDIF requires base64-encoding here:

http://tools.ietf.org/html/rfc2849

To me everything looks right:

Python 2.7.3 (default, Apr 14 2012, 08:58:41) [GCC] on linux2
Type "help", "copyright", "credits" or "license" for more information.
What do you think is a problem?

Ciao, Michael.

 

[Joseph L. Casale]

I'm not sure what exactly you're asking for.

Especially "is not being interpreted as a string requiring base64 encoding" is
written without giving the right context.

So I'm just guessing that this might be the usual misunderstandings with use
of base64 in LDIF. Read more about when LDIF requires base64-encoding here:

http://tools.ietf.org/html/rfc2849

To me everything looks right:

Python 2.7.3 (default, Apr 14 2012, 08:58:41) [GCC] on linux2
Type "help", "copyright", "credits" or "license" for more information.
What do you think is a problem?

Michael,
Thanks for the reply. The issues I am sure are in my code, I read the ldif source file and up
with a values such as 'det\3310wbb\pg' after the base64 encoded entries aredecoded.

The problem I am having is when I add this to an add/mod entry list and write it back out.
As it does not get re-encoded to base64 the ldif file ends up seeing a textentry with a ^]
character which if I re-read it with the parser it causes the handle methodto break midway
through the entry dict and so the last half re-appears disjoint without a dn.

Like I said, I am pretty sure its my poor misunderstanding of decoding and encoding.
I am using the build from http://www.lfd.uci.edu/~gohlke/pythonlibs/ on a windows
2008 r2 server.

I have re-implemented handle to create a cidict holding all the dn/entry's that are parsed as
I then perform some processing such as manipulating attribute values in theentry dict. I
am pretty sure I am breaking things here. The data I am reading is coming from utf-16-le
encoded files and has Unicode characters as the source directory is globally available, being
written to in just about every country.

Is there a process for manipulating/adding data to the entry dict before I write it out that I
should adhere to? For example, if I am adding a new attribute to be composed of part of
another parsed attr for use in a modlist:

{'customAttr': ['foo.{}.bar'.format(entry['uid'])]}

By looking at the value from above, 'det\3310wbb\pg', I gather the entry dict was parsed
into byte strings. I should have decoded this, where as some of the data isUnicode and
as such I should have encoded it?

I really appreciate the time.

Grazie per tutto,
jlc

 

[Michael Ströder]

I'm not sure what exactly you're asking for.
Especially "is not being interpreted as a string requiring base64 encoding" is
written without giving the right context.

So I'm just guessing that this might be the usual misunderstandings with use
of base64 in LDIF. Read more about when LDIF requires base64-encoding here:

http://tools.ietf.org/html/rfc2849

To me everything looks right:

Python 2.7.3 (default, Apr 14 2012, 08:58:41) [GCC] on linux2
Type "help", "copyright", "credits" or "license" for more information.

'ZGV0XDMzMTB3YmJccGc='.decode('base64').decode('utf-8') u'det\\3310wbb\\pg'

What do you think is a problem?

Thanks for the reply. The issues I am sure are in my code, I read the ldif source file and up
with a values such as 'det\3310wbb\pg' after the base64 encoded entries are decoded.

The problem I am having is when I add this to an add/mod entry list and write it back out.
As it does not get re-encoded to base64 the ldif file ends up seeing a text entry with a ^]
character which if I re-read it with the parser it causes the handle method to break midway
through the entry dict and so the last half re-appears disjoint without a dn.

Like I said, I am pretty sure its my poor misunderstanding of decoding and encoding.
I am using the build from http://www.lfd.uci.edu/~gohlke/pythonlibs/ on a windows
2008 r2 server.

I have re-implemented handle to create a cidict holding all the dn/entry's that are parsed as
I then perform some processing such as manipulating attribute values in the entry dict. I
am pretty sure I am breaking things here. The data I am reading is coming from utf-16-le
encoded files and has Unicode characters as the source directory is globally available, being
written to in just about every country.

Processing LDIF is one thing, doing LDAP operations another.

LDIF itself is meant to be ASCII-clean. But each attribute value can carry any
byte sequence (e.g. attribute 'jpegPhoto'). There's no further processing by
module LDIF - it simply returns byte sequences.

The access protocol LDAPv3 mandates UTF-8 encoding for Unicode strings on the
wire if attribute syntax is DirectoryString, IA5String (mainly ASCII) or similar.

So if you're LDIF input returns UTF-16 encoded attribute values for e.g.
attribute 'cn' or 'o' or another attribute not being of OctetString or Binary
syntax something's wrong with the producer of the LDIF data.

Is there a process for manipulating/adding data to the entry dict before I write it out that I
should adhere to? For example, if I am adding a new attribute to be composed of part of
another parsed attr for use in a modlist:

{'customAttr': ['foo.{}.bar'.format(entry['uid'])]}

By looking at the value from above, 'det\3310wbb\pg', I gather the entry dict was parsed
into byte strings. I should have decoded this, where as some of the data is Unicode and
as such I should have encoded it?

I wonder what the string really is. At least the base64-encoding you provided
before decodes as UTF-8 but I'm not sure whether it's the right sequence of
Unicode code points you're expecting.
u'det\\3310wbb\\pg'

I still can't figure out what you're really doing though. I'd recommend to
strip down your operations to a very simple test code snippet illustrating the
issue and post that here.

Ciao, Michael.

 

[Joseph L. Casale]

Hi Michael,

Processing LDIF is one thing, doing LDAP operations another.

LDIF itself is meant to be ASCII-clean. But each attribute value can carry any
byte sequence (e.g. attribute 'jpegPhoto'). There's no further processingby
module LDIF - it simply returns byte sequences.

The access protocol LDAPv3 mandates UTF-8 encoding for Unicode strings onthe
wire if attribute syntax is DirectoryString, IA5String (mainly ASCII) or similar.

So if you're LDIF input returns UTF-16 encoded attribute values for e.g.
attribute 'cn' or 'o' or another attribute not being of OctetString or Binary
syntax something's wrong with the producer of the LDIF data.

That could be, I am using ms's ldifde.exe to dump a domino and AD directoryfor
comparative processing. The problem is I don't have much control on the data in
the directory and I do know that DN's have non ascii characters unique to the

I wonder what the string really is. At least the base64-encoding you provided
before decodes as UTF-8 but I'm not sure whether it's the right sequence of
Unicode code points you're expecting.

u'det\\3310wbb\\pg'

I still can't figure out what you're really doing though. I'd recommend to
strip down your operations to a very simple test code snippet illustrating the
issue and post that here.

So I have removed all my likely broken attempts at working with this data and will
soon have some simple code but at this point I may have an indication of what is
awry with my data.

After parsing the data for a user I am simply taking a value from the ldif file and writing
it back out to another which fails, the value parsed is:

officestreetaddress:: T3R0by1NZcOfbWVyLVN0cmHDn2UgMQ==


File "C:\Python27\lib\site-packages\ldif.py", line 202, in unparse
self._unparseChangeRecord(record)
File "C:\Python27\lib\site-packages\ldif.py", line 181, in _unparseChangeRecord
self._unparseAttrTypeandValue(mod_type,mod_val)
File "C:\Python27\lib\site-packages\ldif.py", line 142, in _unparseAttrTypeandValue
self._unfoldLDIFLine(':: '.join([attr_type,base64.encodestring(attr_value).replace('\n','')]))
File "C:\Python27\lib\base64.py", line 315, in encodestring
pieces.append(binascii.b2a_base64(chunk))
UnicodeEncodeError: 'ascii' codec can't encode character u'\xdf' in position 7: ordinal not in range(128)

c:\python27\lib\base64.py(315)encodestring()

-> pieces.append(binascii.b2a_base64(chunk))
(Pdb) l
310 def encodestring(s):
311 """Encode a string into multiple lines of base-64 data."""
312 pieces = []
313 for i in range(0, len(s), MAXBINSIZE):
314 chunk = s[i : i + MAXBINSIZE]
315 -> pieces.append(binascii.b2a_base64(chunk))
316 return "".join(pieces)
317
318
319 def decodestring(s):
320 """Decode a string."""
(Pdb) args
s = Otto-Meßmer-Straße 1

So moving up a frame or two and looking at the entry dict, I see a modlist entry of:
('streetAddress', [u'Otto-Me\xdfmer-Stra\xdfe 1']) which is correct:

In [2]: 'T3R0by1NZcOfbWVyLVN0cmHDn2UgMQ=='.decode('base64').decode('utf-8')
Out[2]: u'Otto-Me\xdfmer-Stra\xdfe 1'

Looking at the stack trace, I think I see the issue:
(Pdb) import base64
(Pdb) base64.encodestring(u'Otto-Me\xdfmer-Stra\xdfe 1'.encode('utf-8')).replace('\n','')
'T3R0by1NZcOfbWVyLVN0cmHDn2UgMQ=='

I now have the exact the value I started with. Ensuring where I ever handlethe original
values that I return utf-8 decoded objects for use in a modlist to later write and Sub
classing LDIFWriter and overriding _unparseAttrTypeandValue to do the encoding has
eliminated all the errors.

What remains finally is ldifde.exe's output of what looks like U+00BF, or an inverted question
mark for some values, otherwise this issue looks solved.

Thanks for everything,
jlc

 

[dieter]

...
After parsing the data for a user I am simply taking a value from the ldif file and writing
it back out to another which fails, the value parsed is:

officestreetaddress:: T3R0by1NZcOfbWVyLVN0cmHDn2UgMQ==


File "C:\Python27\lib\site-packages\ldif.py", line 202, in unparse
self._unparseChangeRecord(record)
File "C:\Python27\lib\site-packages\ldif.py", line 181, in _unparseChangeRecord
self._unparseAttrTypeandValue(mod_type,mod_val)
File "C:\Python27\lib\site-packages\ldif.py", line 142, in _unparseAttrTypeandValue
self._unfoldLDIFLine(':: '.join([attr_type,base64.encodestring(attr_value).replace('\n','')]))
File "C:\Python27\lib\base64.py", line 315, in encodestring
pieces.append(binascii.b2a_base64(chunk))
UnicodeEncodeError: 'ascii' codec can't encode character u'\xdf' in position 7: ordinal not in range(128)

c:\python27\lib\base64.py(315)encodestring()

-> pieces.append(binascii.b2a_base64(chunk))

This looks like a coding bug: "chunk" seems to be a unicode string;
"b2a_base64" expects an encoded string ("str/bytes"); as
a consequence, Python tries to convert the unicode to "str" by
encoding with its "default enconding" ("ascii" by default) - and
fails.

You could try to find out, why "chunk" is unicode (rather than "str").
That will probably bring you to the real problem.

 

[Michael Ströder]

After parsing the data for a user I am simply taking a value from the ldif file and writing
it back out to another which fails, the value parsed is:

officestreetaddress:: T3R0by1NZcOfbWVyLVN0cmHDn2UgMQ==


File "C:\Python27\lib\site-packages\ldif.py", line 202, in unparse
self._unparseChangeRecord(record)
File "C:\Python27\lib\site-packages\ldif.py", line 181, in _unparseChangeRecord
self._unparseAttrTypeandValue(mod_type,mod_val)
File "C:\Python27\lib\site-packages\ldif.py", line 142, in _unparseAttrTypeandValue
self._unfoldLDIFLine(':: '.join([attr_type,base64.encodestring(attr_value).replace('\n','')]))
File "C:\Python27\lib\base64.py", line 315, in encodestring
pieces.append(binascii.b2a_base64(chunk))
UnicodeEncodeError: 'ascii' codec can't encode character u'\xdf' in position 7: ordinal not in range(128)

Note that all modules in python-ldap up to 2.4.10 including module 'ldif'
expect raw byte strings to be passed as arguments. It seems to me you're
passing a Unicode object in the entry dictionary which will fail in case an
attribute value contains NON-ASCII chars.

python-ldap expects raw strings since it's not schema-aware and therefore does
not have any knowledge about the LDAP syntax used for a particular attribute
type. So automagically convert Unicode strings will likely fail in many cases.
=> The calling application has to deal with it.

I now have the exact the value I started with. Ensuring where I ever handle the original
values that I return utf-8 decoded objects for use in a modlist to later write and Sub
classing LDIFWriter and overriding _unparseAttrTypeandValue to do the encoding has
eliminated all the errors.

Don't muck with overriding _unparseAttrTypeandValue(). Simply pass the
properly encoded data into ldif module.

Ciao, Michael.

 

[Joseph L. Casale]

Note that all modules in python-ldap up to 2.4.10 including module 'ldif'

expect raw byte strings to be passed as arguments. It seems to me you're
passing a Unicode object in the entry dictionary which will fail in case an
attribute value contains NON-ASCII chars.

Yup, I was.

python-ldap expects raw strings since it's not schema-aware and thereforedoes
not have any knowledge about the LDAP syntax used for a particular attribute
type. So automagically convert Unicode strings will likely fail in many cases.
=> The calling application has to deal with it.

I see, that recco went a long a way in cleaning up my code actually and making the
handling of decoding and encoding more consistent.

Don't muck with overriding _unparseAttrTypeandValue(). Simply pass the
properly encoded data into ldif module.

I had some time today, so I attempted to open the ldif files in binary modeto simply
work with the raw byte strings but the moment the first entry was parsed, parse()
stumbled on a character in the first entries dict and passed a dn of None for the last half?

If the option to avoid worrying about decoding and encoding could work, I would be
happy to process the whole lot in byte strings. Any idea what may cause this?

Thanks a lot Michael,
jlc

 

[Michael Ströder]

I had some time today, so I attempted to open the ldif files in binary mode to simply
work with the raw byte strings but the moment the first entry was parsed, parse()
stumbled on a character in the first entries dict and passed a dn of None for the last half?

Without seeing the LDIF data and your code I can't tell what's going on.

If the option to avoid worrying about decoding and encoding could work, I would be
happy to process the whole lot in byte strings. Any idea what may cause this?

I would not claim that module 'ldif' has really awesome docs.
But did you follow the example with LDIFParser in the docs?

http://www.python-ldap.org/doc/html/ldif.html#example

It illustrates that for LDIF stream processing one basically derives a class
from ldif.LDIFParser overriding method handle(). The most basic test would be
something like this:

[..]
def handle(self,dn,entry):
print '***dn',repr(dn)
pprint.pprint(entry)

And then carefully look at the output.

Ciao, Michael.