UnicodeDecodeError: 'utf-8' codec can't decode byte 0xb6 in position

[Dave Angel]

Στις 5/7/2013 10:50 πμ, ο/η Dave Angel έγÏαψε:

The line started as:

host = socket.gethostbyaddr( os.environ['REMOTE_ADDR'] )[0]

refactor that to:
remadd = os.environ('REMOVE_ADDR')
tuple3 = socket.gethostbyaddr(remadd)
host = tuple3[0]

and see which one throws the exception. Then once you have that,
examine the exact parameters that might be triggering the problem. In
particular, figure out the exact types and values for remadd and tuple3.

print(type(remadd) + " : " + repr(remadd))

I'am not sure how iam supposed to write this: i just tried this:


try:
remadd = os.environ('REMOVE_ADDR')
tuple3 = socket.gethostbyaddr(remadd)
host = tuple3[0]
except:
host = type(remadd) + " : " + repr(remadd)


but iam getting an internal server error.

I didnt print it as you said but its the same thing host var gets
printed later on.

Now, why would this give an internal server error?

I have no idea what causes an internal server error. It's up to you to
get the output of the expression to some location you can examine.
Easiest way is to run those 3 lines directly on the server, not in the
cgi environment.

But if you don't have any debugging tools, then STOP right now and build
some. Use logging, or redirect print, or do something that the server
folks provide as debugging aids. But just blindly guessing is
ludicrous. So also is throwing out clues by using a bare except.

Assigning that string to host makes no sense at all. And neither does
putting it in the except clause. You want to get that string to YOUR
eyes, not to the server who might get an internal server error.

 

[feedthetroll]

Am Freitag, 5. Juli 2013 10:00:21 UTC+2 schrieb Îίκος Gr33k:

...
I'am not sure how iam supposed to write this: i just tried this:
try:
remadd = os.environ('REMOVE_ADDR')
tuple3 = socket.gethostbyaddr(remadd)
host = tuple3[0]
except:
host = type(remadd) + " : " + repr(remadd)

Hey, if no one told you before:
You are allowed to read what other people suggest you to do, think about itand so correct obvious typos: 'REMO*V*E_ADDR'

By the way, my i cite:
Am Donnerstag, 4. Juli 2013 14:52:59 UTC+2 schrieb Îίκος Gr33k:

...
'108.162.229.97' is the result of:
print( ascii(os.environ['REMOTE_ADDR']) )

Am Donnerstag, 4. Juli 2013 16:48 UTC+2 schrieb Îίκος Gr33k:

 

[Îίκος Gr33k]

Στις 5/7/2013 11:35 πμ, ο/η Dave Angel έγÏαψε:

Στις 5/7/2013 10:50 πμ, ο/η Dave Angel έγÏαψε:

The line started as:

host = socket.gethostbyaddr( os.environ['REMOTE_ADDR'] )[0]

refactor that to:
remadd = os.environ('REMOVE_ADDR')
tuple3 = socket.gethostbyaddr(remadd)
host = tuple3[0]

and see which one throws the exception. Then once you have that,
examine the exact parameters that might be triggering the problem. In
particular, figure out the exact types and values for remadd and tuple3.

print(type(remadd) + " : " + repr(remadd))

I'am not sure how iam supposed to write this: i just tried this:


try:
remadd = os.environ('REMOVE_ADDR')
tuple3 = socket.gethostbyaddr(remadd)
host = tuple3[0]
except:
host = type(remadd) + " : " + repr(remadd)


but iam getting an internal server error.

I didnt print it as you said but its the same thing host var gets
printed later on.

Now, why would this give an internal server error?

I have no idea what causes an internal server error. It's up to you to
get the output of the expression to some location you can examine.
Easiest way is to run those 3 lines directly on the server, not in the
cgi environment.

But if you don't have any debugging tools, then STOP right now and build
some. Use logging, or redirect print, or do something that the server
folks provide as debugging aids. But just blindly guessing is
ludicrous. So also is throwing out clues by using a bare except.

Assigning that string to host makes no sense at all. And neither does
putting it in the except clause. You want to get that string to YOUR
eyes, not to the server who might get an internal server error.

I don't think running it via 'cli' would help much, since its a
cgi-script and ip addr function have no meaning calling them in plain
our of a cgi environment but here it is:


Python 3.3.2 (default, Jun 3 2013, 16:18:05)
[GCC 4.4.7 20120313 (Red Hat 4.4.7-3)] on linux
Type "help", "copyright", "credits" or "license" for more information.Traceback (most recent call last):
File "<stdin>", line 1, in <module>
TypeError: '_Environ' object is not callable

 

[Îίκος Gr33k]

Στις 5/7/2013 11:27 πμ, ο/η Dave Angel έγÏαψε:

Or if the problems cannot be recreated outside the cgi environment, use
the log files you've been logging other problems into. Or simply open a
text file for writing, and add a file= keyword parameter to the print
function call.

print(repr(e), file=myfile)

Yes you are correct, problem need to be recreated within the cgi env.

try:
remadd = os.environ('REMOTE_ADDR')
tuple3 = socket.gethostbyaddr(remadd)
host = tuple3[0]
except Exception as e:
host = repr(e)


http://superhost.gr/?show=log&page=index.html shows explicitly the same
kind of error that python interpreter via cli gave me the same error!

 

[Dave Angel]

I don't think running it via 'cli' would help much, since its a
cgi-script and ip addr function have no meaning calling them in plain
our of a cgi environment but here it is:

No idea how to parse "have no meaning calling them in plain our of a cgi
environment"

Python 3.3.2 (default, Jun 3 2013, 16:18:05)
[GCC 4.4.7 20120313 (Red Hat 4.4.7-3)] on linux
Type "help", "copyright", "credits" or "license" for more information.Traceback (most recent call last):
File "<stdin>", line 1, in <module>
TypeError: '_Environ' object is not callable

But there were two problems with the code you faithfully copied from my
earlier post. One was already pointed out by feedthetroll, that I
accidentally changed REMOTE_ADDR to REMOVE_ADDR.

The other one is perhaps more subtle; I replaced square brackets with
parentheses.

So try again with:

import os
remadd = os.environ['REMOTE_ADDR']

I get an error:

Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/usr/local/lib/python3.3/os.py", line 669, in __getitem__
value = self._data[self.encodekey(key)]
KeyError: b'REMOTE_ADDR'


but presumably your machine actually has such an environment variable.

Isn't that mistake something you could easily have caught? Or were you
just blindly pasting my bugs without understanding what I was trying to
do with refactoring?

Anyway, I can't see any reason why the rest of the sequence shouldn't
behave identically from a terminal as it does in CGI.

 

[Îίκος Gr33k]

Στις 5/7/2013 12:21 μμ, ο/η Dave Angel έγÏαψε:

I don't think running it via 'cli' would help much, since its a
cgi-script and ip addr function have no meaning calling them in plain
our of a cgi environment but here it is:

No idea how to parse "have no meaning calling them in plain our of a cgi
environment"

Python 3.3.2 (default, Jun 3 2013, 16:18:05)
[GCC 4.4.7 20120313 (Red Hat 4.4.7-3)] on linux
Type "help", "copyright", "credits" or "license" for more information.

import os
remadd = os.environ('REMOVE_ADDR')

Traceback (most recent call last):
File "<stdin>", line 1, in <module>
TypeError: '_Environ' object is not callable

But there were two problems with the code you faithfully copied from my
earlier post. One was already pointed out by feedthetroll, that I
accidentally changed REMOTE_ADDR to REMOVE_ADDR.

The other one is perhaps more subtle; I replaced square brackets with
parentheses.

So try again with:

import os
remadd = os.environ['REMOTE_ADDR']

I get an error:

Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/usr/local/lib/python3.3/os.py", line 669, in __getitem__
value = self._data[self.encodekey(key)]
KeyError: b'REMOTE_ADDR'


but presumably your machine actually has such an environment variable.

Isn't that mistake something you could easily have caught? Or were you
just blindly pasting my bugs without understanding what I was trying to
do with refactoring?

Anyway, I can't see any reason why the rest of the sequence shouldn't
behave identically from a terminal as it does in CGI.

Yes i didnt see your typo and i have corrected it:


try:
remadd = os.environ('REMOTE_ADDR')
tuple3 = socket.gethostbyaddr(remadd)
host = tuple3[0]
except Exception as e:
host = repr(e)


Ima still receiving the same kind of erro as i did with cli as well.

You can view the error in the very first line here:

http://superhost.gr/?show=log&page=index.html


which yields: TypeError("'_Environ' object is not callable",)

 

[Lele Gaifax]

They aren't equivalent. "except Exception" won't catch KeyboardInterrupt or
SystemExit or a few others that you really don't want to catch in a generic
error handler. You should almost never have a bare except.

I know, that's why I added "(almost)", I was just trying to explain why
he wasn't able to see the problem.

Thanks for pointing out the difference,
ciao, lele.

 

[Lele Gaifax]

You can view the error in the very first line here:

http://superhost.gr/?show=log&page=index.html

No, visiting that page simply emit the standard Apache error page,
without details.

which yields: TypeError("'_Environ' object is not callable",)

Dave already told you the reason[1]. Îίκος, *read* **and** *understand*
our *whole* answers to your questions, otherwise we are wasting time,
you, and us!

ciao, lele.

[1] “… The other one is perhaps more subtle; I replaced square brackets
with parentheses.â€

 

[Îίκος Gr33k]

Στις 5/7/2013 12:25 μμ, ο/η Îίκος Gr33k έγÏαψε:

try:
remadd = os.environ('REMOTE_ADDR')
tuple3 = socket.gethostbyaddr(remadd)
host = tuple3[0]
except Exception as e:
host = repr(e)

which yields: TypeError("'_Environ' object is not callable",)

Any thoufgs as to why os.environ('REMOTE_ADDR') gives the above error?

I noticed that if i remove my domain from cloudflare the gethostbyaddr
as it uses too months now.

 

[Îίκος Gr33k]

Στις 5/7/2013 1:24 μμ, ο/η Lele Gaifax έγÏαψε:

You can view the error in the very first line here:

http://superhost.gr/?show=log&page=index.html

No, visiting that page simply emit the standard Apache error page,
without details.

which yields: TypeError("'_Environ' object is not callable",)

Dave already told you the reason[1]. Îίκος, *read* **and** *understand*
our *whole* answers to your questions, otherwise we are wasting time,
you, and us!

ciao, lele.

[1] “… The other one is perhaps more subtle; I replaced square brackets
with parentheses.â€

I read carefully all of tour answer please try to load again:

http://superhost.gr/?show=log&page=index.html

Di i miss an explanation on this?

TypeError("'_Environ' object is not callable",)

 

[Îίκος Gr33k]

Στις 5/7/2013 12:21 μμ, ο/η Dave Angel έγÏαψε:

Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/usr/local/lib/python3.3/os.py", line 669, in __getitem__
value = self._data[self.encodekey(key)]
KeyError: b'REMOTE_ADDR

Wait!
Are you saying that the ip address is being returned as a byte string
which then i have to decode with something like:

host = socket.gethostbyaddr( os.environ['REMOTE_HOST'].decode('utf-8') )[0]

?

 

[Lele Gaifax]

Στις 5/7/2013 12:25 μμ, ο/η Îίκος Gr33k έγÏαψε:

try:
remadd = os.environ('REMOTE_ADDR')
tuple3 = socket.gethostbyaddr(remadd)
host = tuple3[0]
except Exception as e:
host = repr(e)

which yields: TypeError("'_Environ' object is not callable",)

Any thoufgs as to why os.environ('REMOTE_ADDR') gives the above error?

Yes, I'd try to understand the error message, and eventually lookup the
documentation on os.environ to see what type of object it is bound
to. The solution should be easy.

ciao, lele.

 

[Îίκος Gr33k]

Στις 5/7/2013 1:36 μμ, ο/η Lele Gaifax έγÏαψε:

Στις 5/7/2013 12:25 μμ, ο/η Îίκος Gr33k έγÏαψε:

try:
remadd = os.environ('REMOTE_ADDR')
tuple3 = socket.gethostbyaddr(remadd)
host = tuple3[0]
except Exception as e:
host = repr(e)

which yields: TypeError("'_Environ' object is not callable",)

Any thoufgs as to why os.environ('REMOTE_ADDR') gives the above error?

Yes, I'd try to understand the error message, and eventually lookup the
documentation on os.environ to see what type of object it is bound
to. The solution should be easy.

Looks now when i print( repr(e)) i get

UnicodeDecodeError('utf-8', b'\xb6\xe3\xed\xf9\xf3

but what string does it try to decode and jeeps failing?

 

[Lele Gaifax]

Looks now when i print( repr(e)) i get

UnicodeDecodeError('utf-8', b'\xb6\xe3\xed\xf9\xf3

but what string does it try to decode and jeeps failing?

Reasonably it's the second one, as the first clearly seems the tag of
the decoder that tried to translate it to Unicode.

As already explained, your immediate goal should be trying to understand
from *where* that byte string is coming. I can't help on that, sorry.

ciao, lele.

 

[Îίκος Gr33k]

Στις 5/7/2013 1:59 μμ, ο/η Lele Gaifax έγÏαψε:

Reasonably it's the second one, as the first clearly seems the tag of
the decoder that tried to translate it to Unicode.

2nd one of what? 2nd byte in order?
Can ou show me form which characters does this string consist of so we
migth have an idea of where its coming from if we know what it looks like?

As already explained, your immediate goal should be trying to understand
from *where* that byte string is coming. I can't help on that, sorry.

Thats what i'm trying to do.
If i completely remove the gethostbyaddr fucntion adds function then
there is nor problem.

The problem is recreating when the script tries to decode a hostname.

For some bizarre reason if i exclude my domain from CloudFlare then the
reverse DNS resolution of the visitors hostname would be returned properly.

I will do it right now for you to see.

 

[Îίκος Gr33k]

Στις 5/7/2013 2:05 μμ, ο/η Îίκος Gr33k έγÏαψε:

Thats what i'm trying to do.
If i completely remove the gethostbyaddr fucntion adds function then
there is nor problem.

The problem is recreating when the script tries to decode a hostname.

For some bizarre reason if i exclude my domain from CloudFlare then the
reverse DNS resolution of the visitors hostname would be returned properly.

I will do it right now for you to see.

Precisely as i have said it would happen:

I removed the domain form CloudFlare and your domains visting my website
appearing as usual.

Its only when i cloudflare it and the UnicodeError happens.

But why?!?! I see no reason as to why when my domain becomes Cloudflared
the gethostbyaddr() fails.

What kind of weird strings does it return back?

 

[Lele Gaifax]

Στις 5/7/2013 1:59 μμ, ο/η Lele Gaifax έγÏαψε:

2nd one of what? 2nd byte in order?

You asked “what string†(although you probably meant “which stringâ€):

UnicodeDecodeError('utf-8', b'\xb6\xe3\xed\xf9\xf3

first string-------^^^^^^^
second string---------------^^^^^^^^^^^^^^^^^^^^^^

ciao, lele.

 

[Îίκος Gr33k]

Στις 5/7/2013 2:16 μμ, ο/η Lele Gaifax έγÏαψε:

UnicodeDecodeError('utf-8', b'\xb6\xe3\xed\xf9\xf3

first string-------^^^^^^^
second string---------------^^^^^^^^^^^^^^^^^^^^^^

Hold on please!

From where do these dashes and carets characters come from?
Also from where do you see 2 strings?

Looking at that: UnicodeDecodeError('utf-8', b'\xb6\xe3\xed\xf9\xf3

ic an understandn onlt that utf-8 has failsed decoding some byte stream
satrting with \xb6

I totally not follow...

 

[feedthetroll]

Am Freitag, 5. Juli 2013 12:33:05 UTC+2 schrieb Îίκος Gr33k:

...
Wait!
Are you saying that the ip address is being returned as a byte string
which then i have to decode with something like:

host = socket.gethostbyaddr( os.environ['REMOTE_HOST'].decode('utf-8') )[0]

Wait!
I get a decode error when python tries to automatically decode a bytestringassuming it to be utf-8 encoded.
I am sure the error will disappear, when I try to decode it explicit using utf-8. Heureka! I got it!

Or in other words:
If a big stone falls on my foot accidently it hurts.
But I am sure it will not hurt, if take that same stone and throw it on my foot.
Heureka! I got it!

P.S.:

Am 14.06.2013 10:35, schrieb Fábio Santos:

 

[feedthetroll]

Am Freitag, 5. Juli 2013 13:27:25 UTC+2 schrieb Îίκος Gr33k:

Στις 5/7/2013 2:16 μμ, ο/η Lele Gaifax έγÏαψε:

Hold on please!
From where do these dashes and carets characters come from?

ROTFL!!!!
Nikos, you made my day, again! Fun is back in these threads!

Also from where do you see 2 strings?

Look, my little dumb baby: The dashes and carets point to the strings.
The first one being 'utf-8', the second one starting with '\xb6\xe3\xed' (being a bytestring, therefore the b before the ')
Sorry, I forgot. You are not using python for your business, therefore you can't know, that strings in python can for example be identified by surrounding '.

ic an understandn onlt that utf-8 has failsed decoding some byte stream
satrting with \xb6

So ... where did you call utf-8() so that it could try to decode something?