Replacing utf-8 characters

Discussion in 'Python' started by Mike, Oct 5, 2005.

  1. Mike

    Mike Guest

    Hi, I am using Python to scrape web pages and I do not have problem
    unless I run into a site that is utf-8. It seems & is changed to &
    when the site is utf-8.

    If I try to replace it with .replace('&','&') it for some reason
    does not replace it.

    For example:

    The url in the page looks like this

    However when I pull it into python the URL ends up looking like this
    (notice the & instead of just & in the URL)

    Any ideas?
    Mike, Oct 5, 2005
    1. Advertisements

  2. Some code would be helpful: the "&" is in the page source to start
    with (which is as it ought to be). What are you using to parse the HTML?
    Richard Brodie, Oct 5, 2005
    1. Advertisements

  3. Mike

    Mike Guest

    For example this is what I am trying to do that is not working.

    The contents of link is the reuters web page, containing


    link = link.replace('&','&')

    But if I now view the the contents link it shows it the same as when it
    was assigned.
    Mike, Oct 5, 2005
  4. Mike

    Steve Holden Guest

    You must be doing *something* wrong:

    Steve Holden, Oct 5, 2005
  5. Mike

    Mike Guest

    What you and I typed was ascii. The value of link came from importing
    that utf-8 web page into that variable. That is why I think it is not
    working. But not sure what the solution is.
    Mike, Oct 5, 2005
  6. How about using the universal feedparser from to fetch
    and parse the RSS from Reuters? That's what I do and it works like a

    .... print rss.entries[0][what]
    .... print

    Top court seems closely divided on suicide law

    During arguments, the justices sharply questioned both sides on whether then-Attorney General John Ashcroft had the power under federal law in 2001 to bar distribution of controlled drugs to assist suicides, regardless of state law.

    Klaus Alexander Seistrup, Oct 5, 2005
  7. Mike

    Mike Guest

    In playing with this I found link.replace does work but when I use


    it replaces it with & instead of just &. link.replace is working
    for me since if I changed the second option from & to something else I
    see the change.

    So it seems link.replace() function reads whether the first option is
    utf-8 and converts the second option automatically to utf-8? How do I
    prevent that?

    Thanks again.
    Mike, Oct 5, 2005
  8. Mike

    David Bolen Guest

    Are you sure you're asking what you think you are asking? Both the
    ampersand character (&) and the characters within the ampersand entity
    character reference (&) are ASCII. As it turns out they are also
    legal UTF-8, but I would not call a web page UTF-8 just because I saw
    the sequence of characters "&" within the stream. (That's not to
    say it isn't UTF-8 encoded, just that I don't think that's the issue)

    I'm just guessing, but you do realize that legal HTML should quote all
    uses of the ampersand character with an entity reference, since the
    ampersand itself is reserved for use in such references. This
    includes URL references whether inside attributes or in the body of
    the text.

    So when you see something in a browser in a web page that shows a URL
    that includes "&" such as for separating parameters, internally that
    page is (or should be) stored with "&" for that character. Thus
    if you retrieve the page in code, that's what you'll find. It's the
    browser processing that entity reference that turns it back into the
    "&" for presentation.

    Note that whether or not the page in question is encoded as UTF-8 is a
    completely distinct question - whatever encoding the page is in would
    be used to encode the characters in the entity reference (namely

    I'm assuming that in scraping the page you want to reverse the process
    (e.g., perform the interpretation of the entity references much as a
    browser would) before using that URL for other purposes. If so, the
    string replacement you tried should handle the replacement just fine,
    at least within the value of the URL as managed by your code.

    You then mention it being the same when you view the contents of the
    link, which isn't quite clear to me, but if that means retrieving
    another copy of the link as embedded in an HTML page then yes, it'll
    get quoted again since as initially, you have to quote an ampersand
    as an entity reference within HTML.

    What did you mean by "view the contents link"?

    -- David
    David Bolen, Oct 5, 2005
  9. Not sure what an option is... if you are talking about parameters,
    rest assured that <string>.replace does not know or care whether any
    of its parameters is encoded in UTF-8. Also not sure where you got
    the impression UTF-8 could have to do anything with this.

    =?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=, Oct 5, 2005
    1. Advertisements

Ask a Question

Want to reply to this thread or ask your own question?

You'll need to choose a username for the site, which only take a couple of moments (here). After that, you can post your question and our members will help you out.