Ldap module and base64 oncoding

Discussion in 'Python' started by Joseph L. Casale, May 24, 2013.

  1. I have some data I am working with that is not being interpreted as a string requiring
    base64 encoding when sent to the ldif module for output.

    The base64 string parsed is ZGV0XDMzMTB3YmJccGc= and the raw string is det\3310wbb\pg.
    I'll admit my understanding of the handling requirements of non ascii data in 2.7 is weak
    and as such I am failing at adjusting the regex that deduces is the string contains characters
    requiring base64 encoding when being output.

    Any insight, or nudges in the right direction would be appreciated!
    Thanks,
    jlc
     
    Joseph L. Casale, May 24, 2013
    #1
    1. Advertising

  2. Joseph L. Casale wrote:
    > I have some data I am working with that is not being interpreted as a string requiring
    > base64 encoding when sent to the ldif module for output.
    >
    > The base64 string parsed is ZGV0XDMzMTB3YmJccGc= and the raw string is det\3310wbb\pg.
    > I'll admit my understanding of the handling requirements of non ascii data in 2.7 is weak
    > and as such I am failing at adjusting the regex that deduces is the string contains characters
    > requiring base64 encoding when being output.


    I'm not sure what exactly you're asking for.
    Especially "is not being interpreted as a string requiring base64 encoding" is
    written without giving the right context.

    So I'm just guessing that this might be the usual misunderstandings with use
    of base64 in LDIF. Read more about when LDIF requires base64-encoding here:

    http://tools.ietf.org/html/rfc2849

    To me everything looks right:

    Python 2.7.3 (default, Apr 14 2012, 08:58:41) [GCC] on linux2
    Type "help", "copyright", "credits" or "license" for more information.
    >>> 'ZGV0XDMzMTB3YmJccGc='.decode('base64').decode('utf-8')

    u'det\\3310wbb\\pg'
    >>>


    What do you think is a problem?

    Ciao, Michael.
     
    Michael Ströder, May 26, 2013
    #2
    1. Advertising

  3. > I'm not sure what exactly you're asking for.
    > Especially "is not being interpreted as a string requiring base64 encoding" is
    > written without giving the right context.
    >
    > So I'm just guessing that this might be the usual misunderstandings with use
    > of base64 in LDIF. Read more about when LDIF requires base64-encoding here:
    >
    > http://tools.ietf.org/html/rfc2849
    >
    > To me everything looks right:
    >
    > Python 2.7.3 (default, Apr 14 2012, 08:58:41) [GCC] on linux2
    > Type "help", "copyright", "credits" or "license" for more information.
    > >>> 'ZGV0XDMzMTB3YmJccGc='.decode('base64').decode('utf-8')

    > u'det\\3310wbb\\pg'
    > >>>

    >
    > What do you think is a problem?


    Michael,
    Thanks for the reply. The issues I am sure are in my code, I read the ldif source file and up
    with a values such as 'det\3310wbb\pg' after the base64 encoded entries aredecoded.

    The problem I am having is when I add this to an add/mod entry list and write it back out.
    As it does not get re-encoded to base64 the ldif file ends up seeing a textentry with a ^]
    character which if I re-read it with the parser it causes the handle methodto break midway
    through the entry dict and so the last half re-appears disjoint without a dn.

    Like I said, I am pretty sure its my poor misunderstanding of decoding and encoding.
    I am using the build from http://www.lfd.uci.edu/~gohlke/pythonlibs/ on a windows
    2008 r2 server.

    I have re-implemented handle to create a cidict holding all the dn/entry's that are parsed as
    I then perform some processing such as manipulating attribute values in theentry dict. I
    am pretty sure I am breaking things here. The data I am reading is coming from utf-16-le
    encoded files and has Unicode characters as the source directory is globally available, being
    written to in just about every country.

    Is there a process for manipulating/adding data to the entry dict before I write it out that I
    should adhere to? For example, if I am adding a new attribute to be composed of part of
    another parsed attr for use in a modlist:

    {'customAttr': ['foo.{}.bar'.format(entry['uid'])]}

    By looking at the value from above, 'det\3310wbb\pg', I gather the entry dict was parsed
    into byte strings. I should have decoded this, where as some of the data isUnicode and
    as such I should have encoded it?

    I really appreciate the time.

    Grazie per tutto,
    jlc
     
    Joseph L. Casale, May 26, 2013
    #3
  4. Joseph L. Casale wrote:
    >> I'm not sure what exactly you're asking for.
    >> Especially "is not being interpreted as a string requiring base64 encoding" is
    >> written without giving the right context.
    >>
    >> So I'm just guessing that this might be the usual misunderstandings with use
    >> of base64 in LDIF. Read more about when LDIF requires base64-encoding here:
    >>
    >> http://tools.ietf.org/html/rfc2849
    >>
    >> To me everything looks right:
    >>
    >> Python 2.7.3 (default, Apr 14 2012, 08:58:41) [GCC] on linux2
    >> Type "help", "copyright", "credits" or "license" for more information.
    >>>>> 'ZGV0XDMzMTB3YmJccGc='.decode('base64').decode('utf-8')

    >> u'det\\3310wbb\\pg'
    >>>>>

    >>
    >> What do you think is a problem?

    >
    > Thanks for the reply. The issues I am sure are in my code, I read the ldif source file and up
    > with a values such as 'det\3310wbb\pg' after the base64 encoded entries are decoded.
    >
    > The problem I am having is when I add this to an add/mod entry list and write it back out.
    > As it does not get re-encoded to base64 the ldif file ends up seeing a text entry with a ^]
    > character which if I re-read it with the parser it causes the handle method to break midway
    > through the entry dict and so the last half re-appears disjoint without a dn.
    >
    > Like I said, I am pretty sure its my poor misunderstanding of decoding and encoding.
    > I am using the build from http://www.lfd.uci.edu/~gohlke/pythonlibs/ on a windows
    > 2008 r2 server.
    >
    > I have re-implemented handle to create a cidict holding all the dn/entry's that are parsed as
    > I then perform some processing such as manipulating attribute values in the entry dict. I
    > am pretty sure I am breaking things here. The data I am reading is coming from utf-16-le
    > encoded files and has Unicode characters as the source directory is globally available, being
    > written to in just about every country.


    Processing LDIF is one thing, doing LDAP operations another.

    LDIF itself is meant to be ASCII-clean. But each attribute value can carry any
    byte sequence (e.g. attribute 'jpegPhoto'). There's no further processing by
    module LDIF - it simply returns byte sequences.

    The access protocol LDAPv3 mandates UTF-8 encoding for Unicode strings on the
    wire if attribute syntax is DirectoryString, IA5String (mainly ASCII) or similar.

    So if you're LDIF input returns UTF-16 encoded attribute values for e.g.
    attribute 'cn' or 'o' or another attribute not being of OctetString or Binary
    syntax something's wrong with the producer of the LDIF data.

    > Is there a process for manipulating/adding data to the entry dict before I write it out that I
    > should adhere to? For example, if I am adding a new attribute to be composed of part of
    > another parsed attr for use in a modlist:
    >
    > {'customAttr': ['foo.{}.bar'.format(entry['uid'])]}
    >
    > By looking at the value from above, 'det\3310wbb\pg', I gather the entry dict was parsed
    > into byte strings. I should have decoded this, where as some of the data is Unicode and
    > as such I should have encoded it?


    I wonder what the string really is. At least the base64-encoding you provided
    before decodes as UTF-8 but I'm not sure whether it's the right sequence of
    Unicode code points you're expecting.

    >>> 'ZGV0XDMzMTB3YmJccGc='.decode('base64').decode('utf-8')

    u'det\\3310wbb\\pg'

    I still can't figure out what you're really doing though. I'd recommend to
    strip down your operations to a very simple test code snippet illustrating the
    issue and post that here.

    Ciao, Michael.
     
    Michael Ströder, May 26, 2013
    #4
  5. Hi Michael,

    > Processing LDIF is one thing, doing LDAP operations another.
    >
    > LDIF itself is meant to be ASCII-clean. But each attribute value can carry any
    > byte sequence (e.g. attribute 'jpegPhoto'). There's no further processingby
    > module LDIF - it simply returns byte sequences.
    >
    > The access protocol LDAPv3 mandates UTF-8 encoding for Unicode strings onthe
    > wire if attribute syntax is DirectoryString, IA5String (mainly ASCII) or similar.
    >
    > So if you're LDIF input returns UTF-16 encoded attribute values for e.g.
    > attribute 'cn' or 'o' or another attribute not being of OctetString or Binary
    > syntax something's wrong with the producer of the LDIF data.


    That could be, I am using ms's ldifde.exe to dump a domino and AD directoryfor
    comparative processing. The problem is I don't have much control on the data in
    the directory and I do know that DN's have non ascii characters unique to the

    > I wonder what the string really is. At least the base64-encoding you provided
    > before decodes as UTF-8 but I'm not sure whether it's the right sequence of
    > Unicode code points you're expecting.
    >
    > >>> 'ZGV0XDMzMTB3YmJccGc='.decode('base64').decode('utf-8')

    > u'det\\3310wbb\\pg'
    >
    > I still can't figure out what you're really doing though. I'd recommend to
    > strip down your operations to a very simple test code snippet illustrating the
    > issue and post that here.


    So I have removed all my likely broken attempts at working with this data and will
    soon have some simple code but at this point I may have an indication of what is
    awry with my data.

    After parsing the data for a user I am simply taking a value from the ldif file and writing
    it back out to another which fails, the value parsed is:

    officestreetaddress:: T3R0by1NZcOfbWVyLVN0cmHDn2UgMQ==


    File "C:\Python27\lib\site-packages\ldif.py", line 202, in unparse
    self._unparseChangeRecord(record)
    File "C:\Python27\lib\site-packages\ldif.py", line 181, in _unparseChangeRecord
    self._unparseAttrTypeandValue(mod_type,mod_val)
    File "C:\Python27\lib\site-packages\ldif.py", line 142, in _unparseAttrTypeandValue
    self._unfoldLDIFLine(':: '.join([attr_type,base64.encodestring(attr_value).replace('\n','')]))
    File "C:\Python27\lib\base64.py", line 315, in encodestring
    pieces.append(binascii.b2a_base64(chunk))
    UnicodeEncodeError: 'ascii' codec can't encode character u'\xdf' in position 7: ordinal not in range(128)

    > c:\python27\lib\base64.py(315)encodestring()

    -> pieces.append(binascii.b2a_base64(chunk))
    (Pdb) l
    310 def encodestring(s):
    311 """Encode a string into multiple lines of base-64 data."""
    312 pieces = []
    313 for i in range(0, len(s), MAXBINSIZE):
    314 chunk = s[i : i + MAXBINSIZE]
    315 -> pieces.append(binascii.b2a_base64(chunk))
    316 return "".join(pieces)
    317
    318
    319 def decodestring(s):
    320 """Decode a string."""
    (Pdb) args
    s = Otto-Meßmer-Straße 1

    So moving up a frame or two and looking at the entry dict, I see a modlist entry of:
    ('streetAddress', [u'Otto-Me\xdfmer-Stra\xdfe 1']) which is correct:

    In [2]: 'T3R0by1NZcOfbWVyLVN0cmHDn2UgMQ=='.decode('base64').decode('utf-8')
    Out[2]: u'Otto-Me\xdfmer-Stra\xdfe 1'

    Looking at the stack trace, I think I see the issue:
    (Pdb) import base64
    (Pdb) base64.encodestring(u'Otto-Me\xdfmer-Stra\xdfe 1'.encode('utf-8')).replace('\n','')
    'T3R0by1NZcOfbWVyLVN0cmHDn2UgMQ=='

    I now have the exact the value I started with. Ensuring where I ever handlethe original
    values that I return utf-8 decoded objects for use in a modlist to later write and Sub
    classing LDIFWriter and overriding _unparseAttrTypeandValue to do the encoding has
    eliminated all the errors.

    What remains finally is ldifde.exe's output of what looks like U+00BF, or an inverted question
    mark for some values, otherwise this issue looks solved.

    Thanks for everything,
    jlc
     
    Joseph L. Casale, May 27, 2013
    #5
  6. Joseph L. Casale

    dieter Guest

    "Joseph L. Casale" <> writes:
    > ...
    > After parsing the data for a user I am simply taking a value from the ldif file and writing
    > it back out to another which fails, the value parsed is:
    >
    > officestreetaddress:: T3R0by1NZcOfbWVyLVN0cmHDn2UgMQ==
    >
    >
    > File "C:\Python27\lib\site-packages\ldif.py", line 202, in unparse
    > self._unparseChangeRecord(record)
    > File "C:\Python27\lib\site-packages\ldif.py", line 181, in _unparseChangeRecord
    > self._unparseAttrTypeandValue(mod_type,mod_val)
    > File "C:\Python27\lib\site-packages\ldif.py", line 142, in _unparseAttrTypeandValue
    > self._unfoldLDIFLine(':: '.join([attr_type,base64.encodestring(attr_value).replace('\n','')]))
    > File "C:\Python27\lib\base64.py", line 315, in encodestring
    > pieces.append(binascii.b2a_base64(chunk))
    > UnicodeEncodeError: 'ascii' codec can't encode character u'\xdf' in position 7: ordinal not in range(128)
    >
    >> c:\python27\lib\base64.py(315)encodestring()

    > -> pieces.append(binascii.b2a_base64(chunk))


    This looks like a coding bug: "chunk" seems to be a unicode string;
    "b2a_base64" expects an encoded string ("str/bytes"); as
    a consequence, Python tries to convert the unicode to "str" by
    encoding with its "default enconding" ("ascii" by default) - and
    fails.

    You could try to find out, why "chunk" is unicode (rather than "str").
    That will probably bring you to the real problem.
     
    dieter, May 27, 2013
    #6
  7. Joseph L. Casale wrote:
    > After parsing the data for a user I am simply taking a value from the ldif file and writing
    > it back out to another which fails, the value parsed is:
    >
    > officestreetaddress:: T3R0by1NZcOfbWVyLVN0cmHDn2UgMQ==
    >
    >
    > File "C:\Python27\lib\site-packages\ldif.py", line 202, in unparse
    > self._unparseChangeRecord(record)
    > File "C:\Python27\lib\site-packages\ldif.py", line 181, in _unparseChangeRecord
    > self._unparseAttrTypeandValue(mod_type,mod_val)
    > File "C:\Python27\lib\site-packages\ldif.py", line 142, in _unparseAttrTypeandValue
    > self._unfoldLDIFLine(':: '.join([attr_type,base64.encodestring(attr_value).replace('\n','')]))
    > File "C:\Python27\lib\base64.py", line 315, in encodestring
    > pieces.append(binascii.b2a_base64(chunk))
    > UnicodeEncodeError: 'ascii' codec can't encode character u'\xdf' in position 7: ordinal not in range(128)


    Note that all modules in python-ldap up to 2.4.10 including module 'ldif'
    expect raw byte strings to be passed as arguments. It seems to me you're
    passing a Unicode object in the entry dictionary which will fail in case an
    attribute value contains NON-ASCII chars.

    python-ldap expects raw strings since it's not schema-aware and therefore does
    not have any knowledge about the LDAP syntax used for a particular attribute
    type. So automagically convert Unicode strings will likely fail in many cases.
    => The calling application has to deal with it.

    > I now have the exact the value I started with. Ensuring where I ever handle the original
    > values that I return utf-8 decoded objects for use in a modlist to later write and Sub
    > classing LDIFWriter and overriding _unparseAttrTypeandValue to do the encoding has
    > eliminated all the errors.


    Don't muck with overriding _unparseAttrTypeandValue(). Simply pass the
    properly encoded data into ldif module.

    Ciao, Michael.
     
    Michael Ströder, May 27, 2013
    #7
  8. > Note that all modules in python-ldap up to 2.4.10 including module 'ldif'
    > expect raw byte strings to be passed as arguments. It seems to me you're
    > passing a Unicode object in the entry dictionary which will fail in case an
    > attribute value contains NON-ASCII chars.


    Yup, I was.

    > python-ldap expects raw strings since it's not schema-aware and thereforedoes
    > not have any knowledge about the LDAP syntax used for a particular attribute
    > type. So automagically convert Unicode strings will likely fail in many cases.
    > => The calling application has to deal with it.


    I see, that recco went a long a way in cleaning up my code actually and making the
    handling of decoding and encoding more consistent.

    > Don't muck with overriding _unparseAttrTypeandValue(). Simply pass the
    > properly encoded data into ldif module.


    I had some time today, so I attempted to open the ldif files in binary modeto simply
    work with the raw byte strings but the moment the first entry was parsed, parse()
    stumbled on a character in the first entries dict and passed a dn of None for the last half?

    If the option to avoid worrying about decoding and encoding could work, I would be
    happy to process the whole lot in byte strings. Any idea what may cause this?

    Thanks a lot Michael,
    jlc
     
    Joseph L. Casale, May 28, 2013
    #8
  9. Joseph L. Casale wrote:
    > I had some time today, so I attempted to open the ldif files in binary mode to simply
    > work with the raw byte strings but the moment the first entry was parsed, parse()
    > stumbled on a character in the first entries dict and passed a dn of None for the last half?


    Without seeing the LDIF data and your code I can't tell what's going on.

    > If the option to avoid worrying about decoding and encoding could work, I would be
    > happy to process the whole lot in byte strings. Any idea what may cause this?


    I would not claim that module 'ldif' has really awesome docs.
    But did you follow the example with LDIFParser in the docs?

    http://www.python-ldap.org/doc/html/ldif.html#example

    It illustrates that for LDIF stream processing one basically derives a class
    from ldif.LDIFParser overriding method handle(). The most basic test would be
    something like this:

    [..]
    def handle(self,dn,entry):
    print '***dn',repr(dn)
    pprint.pprint(entry)

    And then carefully look at the output.

    Ciao, Michael.
     
    Michael Ströder, May 28, 2013
    #9
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. Replies:
    1
    Views:
    537
    Raymond DeCampo
    Feb 21, 2006
  2. rcmn
    Replies:
    1
    Views:
    343
    =?ISO-8859-1?Q?Michael_Str=F6der?=
    Nov 6, 2006
  3. Jason Wold

    using LDAP Controls in ruby-ldap

    Jason Wold, Nov 4, 2004, in forum: Ruby
    Replies:
    5
    Views:
    260
  4. Carlos Nepomuceno

    RE: Ldap module and base64 oncoding

    Carlos Nepomuceno, May 25, 2013, in forum: Python
    Replies:
    0
    Views:
    78
    Carlos Nepomuceno
    May 25, 2013
  5. Joseph L. Casale

    RE: Ldap module and base64 oncoding

    Joseph L. Casale, May 25, 2013, in forum: Python
    Replies:
    1
    Views:
    100
    Michael Ströder
    May 26, 2013
Loading...

Share This Page