Characters aren't displayed correctly

[Hussein B]

I guessed that! I was referring to the fact that you didn't tell us
what values you did eventually supply that made it generate seemingly
reasonable Arabic letters in unicode!!  Was it charset="utf8" that did
the trick?

Yes, it is utf8

Let's suppose that you used charset="XXXX" ... as far as I can tell,
not being a mysqldb user myself, this means that your data tables and/
or your default connection don't use XXXX as an encoding. If so, this
might be an issue you might like to take up with whoever created the
database that you are using.
and I got the following:
u'\u062f\u062e\u0648\u0644 \u0633\u0631\u064a\u0639
\u0634\u0647\u0631'
So characters are getting converted successfully.
I guess so -- U+06nn sure are Arabic characters
However as suggested above, "converted from what?" might be worth
pursuing if you like to understand what is going on instead of just
applying magic recipes ;-)
Well, using the previous recipe for sending the mail:http://code.activestate.com/recipes/473810/
I got the following error:
Traceback (most recent call last):
  File "HtmlMail.py", line 52, in <module>
    s.sendmail(sender, receiver , msg.as_string())
[big snip]
_handle_text
    self._fp.write(payload)
UnicodeEncodeError: 'ascii' codec can't encode characters in position
115-118: ordinal not in range(128)
Again, any ideas guys?
That recipe appears to have been written by an ascii bigot for ascii
bigots :-(
Try reading the docs for email.charset (that's the charset module in
the email package).

Every thing is working now, I did the following:
t = MIMEText(markup.encode('utf-8'), 'html', 'utf-8')
Thank you all guys and especially you John, I owe you a HUGE bottle of
beer

Thanks for the kind thought, but beer decreases grey-cell count and
increases girth ... I don't need any assistance with those matters

Cheers,
John

No problem John.

 

[Steve Holden]

On Mar 2, 2009, at 5:26 PM, John Machin wrote: [...]
I (mostly) agree with your rule. But as I said, there's more than one
way to solve this problem. Or perhaps I should say that there's more
than one way to lead the OP to a solution to this problem. We teach
differently, you and I. I believe there's room in the world for *both*
styles -- perhaps even a third or fourth! =)

Clue stick!

regards
Steve

 

[Steve Holden]

On Mar 2, 2009, at 5:26 PM, John Machin wrote: [...]
I (mostly) agree with your rule. But as I said, there's more than one
way to solve this problem. Or perhaps I should say that there's more
than one way to lead the OP to a solution to this problem. We teach
differently, you and I. I believe there's room in the world for *both*
styles -- perhaps even a third or fourth! =)

Clue stick!

regards
Steve

 

[J. Clifford Dyer]

31 AM, Hussein B wrote:

Hey,
I'm retrieving records from MySQL database that contains non english
characters.
Then I create a String that contains HTML markup and column values
from the previous result set.
+++++
markup = u'''<table>.....'''
for row in rows:
markup = markup + '<tr><td>' + row['id']
markup = markup + '</table>
+++++
Then I'm sending the email according to this tip:
http://code.activestate.com/recipes/473810/
Well, the email contains ????? characters for each non english ones.
Any ideas?

There's so many places where this could go wrong and you haven't
narrowed down the problem.

Are the characters stored in the database correctly?

Are they stored consistently (i.e. all using the same encoding, not
some using utf-8 and others using iso-8859-1)?

What are you getting out of the database? Is it being converted to
Unicode correctly, or at all?

Are you sure that the program you're using to view the email
understands the encoding?

Isolate those questions one at a time. Add some debugging breakpoints.
Ensure that you have what you think you have. You might not fix your
problem, but you will make it much smaller and more specific.

Good luck
Philip

Let me add to that checklist:

Are you sure the email you are creating has the encoding declared
properly in the headers?

My HTML markup contains only table tags (you know, table, tr and td)

Ah. The issue is not with the HTML markup, but the email headers. For
example, the email you sent me has a header that says:

Content-type: text/plain; charset="iso-8859-1"

Guessing from the recipe you linked to, you probably need something
like:

msgRoot['Content-type'] = 'text/plain; charset="utf-16"'

replacing utf-16 with whatever encoding you have encoded your email
with.

Or it may be that the header has to be attached to the individual mime
parts. I'm not as familiar with MIME.

Cheers,
Cliff

Hey Cliff,
I tried your tip and I still get the same thing (?????)
I added print statement to print each value of the result set into the
console, which also prints ???? characters instead of the real
characters values.
Maybe a conversion is happened upon getting the data from the
database?
(the values are stored correctly in the database)

Sorry for responding so late.

Getting ? characters when you print to the console does not
*necessarily* mean python is doing anything wrong. It may be outputting
the correct characters, but your console lacks the proper font for
displaying those characters. Try doing for c in email_body: print
hex(ord(c)), and see if the results correspond to what you're trying to
get.

Cheers,
Cliff