ascii character - removing chars from string

Discussion in 'Python' started by bruce, Jul 4, 2006.

  1. bruce

    bruce Guest

    hi...

    i'm running into a problem where i'm seeing non-ascii chars in the parsing
    i'm doing. in looking through various docs, i can't find functions to
    remove/restrict strings to valid ascii chars.

    i'm assuming python has something like

    valid_str = strip(invalid_str)

    where 'strip' removes/strips out the invalid chars...

    any ideas/thoughts/pointers...

    thanks

    -bruce
     
    bruce, Jul 4, 2006
    #1
    1. Advertising

  2. bruce

    Guest

    bruce:
    > valid_str = strip(invalid_str)
    > where 'strip' removes/strips out the invalid chars...


    This isn't short but it is fast:
    import string
    valid_chars = string.lowercase + string.uppercase + \
    string.digits +
    """|!'\\"£$%&/()=?^*é§_:;>+,.-<\n \t"""
    all_chars = "".join(map( chr, range(256)) )
    comp_valid_chars = "".join( set(all_chars).difference(valid_chars) )
    print "test string".translate(all_chars, comp_valid_chars)


    Shorter and a bit slower alternative:
    import string
    valid_chars_set = set(string.lowercase + string.uppercase
    + string.digits +
    """|!'\\"£$%&/()=?^*é§_:;>+,.-<\n \t""")
    print filter(lambda c: c in valid_chars_set, "test string")

    You can add the chars you want to the string of accepted ones.

    Bye,
    bearophile
     
    , Jul 4, 2006
    #2
    1. Advertising

  3. bruce

    John Machin Guest

    On 4/07/2006 9:27 AM, bruce wrote:
    > hi...
    >
    > i'm running into a problem where i'm seeing non-ascii chars in the parsing
    > i'm doing. in looking through various docs, i can't find functions to
    > remove/restrict strings to valid ascii chars.
    >


    It's possible that you would be better off handling those characters in
    some fashion other than blowing them away. What are the characters that
    you are seeing, and what is the problem that they are causing you?
     
    John Machin, Jul 4, 2006
    #3
  4. bruce

    Rune Strand Guest

    bruce wrote:
    > hi...
    >
    > i'm running into a problem where i'm seeing non-ascii chars in the parsing
    > i'm doing. in looking through various docs, i can't find functions to
    > remove/restrict strings to valid ascii chars.
    >
    > i'm assuming python has something like
    >
    > valid_str = strip(invalid_str)
    >
    > where 'strip' removes/strips out the invalid chars...
    >
    > any ideas/thoughts/pointers...


    If you're able to define the invalid_chars, the most convenient is
    probably to use the strip() method:
    >>> a_string = "abcdef"
    >>> invalid_chars = 'abc'
    >>> a_string.strip(invalid_chars)

    'def'
     
    Rune Strand, Jul 4, 2006
    #4
  5. bruce

    Simon Forman Guest

    bruce wrote:
    > hi...
    >
    > i'm running into a problem where i'm seeing non-ascii chars in the parsing
    > i'm doing. in looking through various docs, i can't find functions to
    > remove/restrict strings to valid ascii chars.
    >
    > i'm assuming python has something like
    >
    > valid_str = strip(invalid_str)
    >
    > where 'strip' removes/strips out the invalid chars...
    >
    > any ideas/thoughts/pointers...
    >
    > thanks
    >
    > -bruce


    You might be able to use the translate() and maketrans() string
    methods. See
    http://groups.google.ca/group/comp...._frm/thread/261516dd06ee32e6/9a02a21c95bd0ec9
    second-to-last post for an example.
     
    Simon Forman, Jul 4, 2006
    #5
  6. bruce

    Simon Forman Guest

    bruce wrote:
    > hi...
    >
    > update. i'm getting back html, and i'm getting strings like " foo &nbsp;"
    > which is valid HTML as the '&nbsp;' is a space.


    &, n, b, s, p, ; Those are all ascii characters.

    > i need a way of stripping/removing the '&nbsp;' from the string
    >
    > the &nbsp; needs to be treated as a single char...
    >
    > text = "foo cat &nbsp;"
    >
    > ie ok_text = strip(text)
    >
    > ok_text = "foo cat"


    Do you really want to remove those html entities? Or would you rather
    convert them back into the actual text they represent? Do you just
    want to deal with &nbsp;'s? Or maybe the other possible entities that
    might appear also?

    Check out htmlentitydefs.entitydefs (see
    http://docs.python.org/lib/module-htmlentitydefs.html) it's kind of
    ugly looking so maybe use pprint to print it:

    >>> import htmlentitydefs, pprint
    >>> pprint.pprint(htmlentitydefs.entitydefs)

    {'AElig': 'Æ',
    'Aacute': 'Á',
    'Acirc': 'Â',
    ..
    ..
    ..
    'nbsp': '\xa0',
    ..
    ..
    ..
    etc...


    HTH,
    ~Simon

    "You keep using that word. I do not think it means what you think it
    means."
    -Inigo Montoya, "The Princess Bride"
     
    Simon Forman, Jul 4, 2006
    #6
  7. bruce

    Simon Forman Guest

    bruce wrote:
    > simon...
    >
    > the '&nbsp;' is not to be seen/viewed as text/ascii.. it's a representation
    > of a hex 'u\xa0' if i recall...


    Did you not see this part of the post that you're replying to?

    > 'nbsp': '\xa0',


    My point was not that '\xa0' is an ascii character... It was that your
    initial request was very misleading:

    "i'm running into a problem where i'm seeing non-ascii chars in the
    parsing i'm doing. in looking through various docs, i can't find
    functions to remove/restrict strings to valid ascii chars."

    That's why you got three different answers to the wrong question.

    You weren't "seeing non-ascii chars" at all. You were seeing ascii
    representations of html entities that, in the case of '&nbsp;', happen
    to represent non-ascii values.

    >
    > i'm looking to remove or replace the insances with a ' ' (space)


    Simplicity:

    s.replace('&nbsp;', ' ')

    ~Simon

    "You keep using that word. I do not think it means what you think it
    means."
    -Inigo Montoya, "The Princess Bride"

    >
    > -bruce
    >
    >
    > -----Original Message-----
    > From: python-list-bounces+bedouglas=
    > [mailto:python-list-bounces+bedouglas=]On Behalf
    > Of Simon Forman
    > Sent: Monday, July 03, 2006 7:17 PM
    > To:
    > Subject: Re: ascii character - removing chars from string
    >
    >
    > bruce wrote:
    > > hi...
    > >
    > > update. i'm getting back html, and i'm getting strings like " foo &nbsp;"
    > > which is valid HTML as the '&nbsp;' is a space.

    >
    > &, n, b, s, p, ; Those are all ascii characters.
    >
    > > i need a way of stripping/removing the '&nbsp;' from the string
    > >
    > > the &nbsp; needs to be treated as a single char...
    > >
    > > text = "foo cat &nbsp;"
    > >
    > > ie ok_text = strip(text)
    > >
    > > ok_text = "foo cat"

    >
    > Do you really want to remove those html entities? Or would you rather
    > convert them back into the actual text they represent? Do you just
    > want to deal with &nbsp;'s? Or maybe the other possible entities that
    > might appear also?
    >
    > Check out htmlentitydefs.entitydefs (see
    > http://docs.python.org/lib/module-htmlentitydefs.html) it's kind of
    > ugly looking so maybe use pprint to print it:
    >
    > >>> import htmlentitydefs, pprint
    > >>> pprint.pprint(htmlentitydefs.entitydefs)

    > {'AElig': 'Æ',
    > 'Aacute': 'Á',
    > 'Acirc': 'Â',
    > .
    > .
    > .
    > 'nbsp': '\xa0',
    > .
    > .
    > .
    > etc...
    >
    >
    > HTH,
    > ~Simon
    >
    > "You keep using that word. I do not think it means what you think it
    > means."
    > -Inigo Montoya, "The Princess Bride"
    >
    > --
    > http://mail.python.org/mailman/listinfo/python-list
     
    Simon Forman, Jul 4, 2006
    #7
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. bruce
    Replies:
    1
    Views:
    618
    Marc 'BlackJack' Rintsch
    Jul 4, 2006
  2. bruce
    Replies:
    2
    Views:
    569
    Fredrik Lundh
    Jul 4, 2006
  3. bruce
    Replies:
    1
    Views:
    2,474
    Steven D'Aprano
    Jul 4, 2006
  4. TOXiC
    Replies:
    5
    Views:
    1,259
    TOXiC
    Jan 31, 2007
  5. Replies:
    3
    Views:
    116
    Benoit Lefebvre
    Dec 24, 2007
Loading...

Share This Page