Referrer key missing form os.environ dictionary?

[Îίκος]

Hello, i decided am ong other os.environ variables to also grab the
'HTTP_REFERER' fiel but when i try to run my script i was seeing a
KeyError complaining that 'HTTP_REFERER' didnt exist.

So, to see what existed in the os.environ dictionary i issues a print(
os.environ ) to see all available keys and their values:

environ({'SERVER_PORT': '80', 'REQUEST_URI': '/',
'HTTP_ACCEPT_ENCODING': 'gzip,deflate,sdch', 'SERVER_ADDR':
'84.200.17.58', 'DOCUMENT_ROOT': '/home/nikos/public_html',
'HTTP_CONNECTION': 'keep-alive', 'SCRIPT_FILENAME':
'/home/nikos/public_html/cgi-bin/metrites.py', 'SERVER_NAME':
'superhost.gr', 'REMOTE_PORT': '58896', 'HTTP_ACCEPT':
'text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8',
'HTTP_USER_AGENT': 'Mozilla/5.0 (Windows NT 6.2; WOW64)
AppleWebKit/537.36 (KHTML, like Gecko) Chrome/31.0.1650.2
Safari/537.36', 'HTTP_ACCEPT_LANGUAGE': 'en-US,en;q=0.8,el;q=0.6',
'GATEWAY_INTERFACE': 'CGI/1.1', 'HTTP_HOST': 'superhost.gr',
'REDIRECT_URL': '/index.html', 'SERVER_PROTOCOL': 'HTTP/1.1',
'SERVER_SIGNATURE': '
Apache/2.2.25 (Unix) mod_ssl/2.2.25 OpenSSL/1.0.0-fips
mod_auth_passthrough/2.1 mod_bwlimited/1.4 FrontPage/5.0.2.2635 Server
at superhost.gr Port 80
\n', 'HTTP_DNT': '1', 'REQUEST_METHOD': 'GET', 'QUERY_STRING':
'file=/home/nikos/public_html/index.html', 'PATH':
'/sbin:/usr/sbin:/bin:/usr/bin', 'HTTP_COOKIE': 'cf_use_ob=0;
__cfduid=da37079bb377f13e9c50224189ab46ac71379783691866;
__utma=210786583.1207352568.1379497319.1380099225.1380108255.32;
__utmc=210786583;
__utmz=210786583.1379497319.1.1.utmcsr=(direct)|utmccn=(direct)|utmcmd=(none)',
'REMOTE_ADDR': '176.92.73.41', 'REDIRECT_QUERY_STRING':
'file=/home/nikos/public_html/index.html', 'SERVER_SOFTWARE':
'Apache/2.2.25 (Unix) mod_ssl/2.2.25 OpenSSL/1.0.0-fips
mod_auth_passthrough/2.1 mod_bwlimited/1.4 FrontPage/5.0.2.2635',
'SERVER_ADMIN': '(e-mail address removed)', 'SCRIPT_NAME':
'/cgi-bin/metrites.py', 'REDIRECT_STATUS': '200'})


i dont see anywhere a refferer key so to catch it in a variable sting
like this:

referrer = os.environ['HTTP_REFERER']

Do i miss something? its a suprise to me that the environ dictioanry has
almost anythign but a referrer key.

I need your help please.

 

[Îίκος]

Στις 25/9/2013 3:45 μμ, ο/η Îίκος έγÏαψε:

Hello, i decided am ong other os.environ variables to also grab the
'HTTP_REFERER' fiel but when i try to run my script i was seeing a
KeyError complaining that 'HTTP_REFERER' didnt exist.

So, to see what existed in the os.environ dictionary i issues a print(
os.environ ) to see all available keys and their values:

environ({'SERVER_PORT': '80', 'REQUEST_URI': '/',
'HTTP_ACCEPT_ENCODING': 'gzip,deflate,sdch', 'SERVER_ADDR':
'84.200.17.58', 'DOCUMENT_ROOT': '/home/nikos/public_html',
'HTTP_CONNECTION': 'keep-alive', 'SCRIPT_FILENAME':
'/home/nikos/public_html/cgi-bin/metrites.py', 'SERVER_NAME':
'superhost.gr', 'REMOTE_PORT': '58896', 'HTTP_ACCEPT':
'text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8',
'HTTP_USER_AGENT': 'Mozilla/5.0 (Windows NT 6.2; WOW64)
AppleWebKit/537.36 (KHTML, like Gecko) Chrome/31.0.1650.2
Safari/537.36', 'HTTP_ACCEPT_LANGUAGE': 'en-US,en;q=0.8,el;q=0.6',
'GATEWAY_INTERFACE': 'CGI/1.1', 'HTTP_HOST': 'superhost.gr',
'REDIRECT_URL': '/index.html', 'SERVER_PROTOCOL': 'HTTP/1.1',
'SERVER_SIGNATURE': '
Apache/2.2.25 (Unix) mod_ssl/2.2.25 OpenSSL/1.0.0-fips
mod_auth_passthrough/2.1 mod_bwlimited/1.4 FrontPage/5.0.2.2635 Server
at superhost.gr Port 80
\n', 'HTTP_DNT': '1', 'REQUEST_METHOD': 'GET', 'QUERY_STRING':
'file=/home/nikos/public_html/index.html', 'PATH':
'/sbin:/usr/sbin:/bin:/usr/bin', 'HTTP_COOKIE': 'cf_use_ob=0;
__cfduid=da37079bb377f13e9c50224189ab46ac71379783691866;
__utma=210786583.1207352568.1379497319.1380099225.1380108255.32;
__utmc=210786583;
__utmz=210786583.1379497319.1.1.utmcsr=(direct)|utmccn=(direct)|utmcmd=(none)',
'REMOTE_ADDR': '176.92.73.41', 'REDIRECT_QUERY_STRING':
'file=/home/nikos/public_html/index.html', 'SERVER_SOFTWARE':
'Apache/2.2.25 (Unix) mod_ssl/2.2.25 OpenSSL/1.0.0-fips
mod_auth_passthrough/2.1 mod_bwlimited/1.4 FrontPage/5.0.2.2635',
'SERVER_ADMIN': '(e-mail address removed)', 'SCRIPT_NAME':
'/cgi-bin/metrites.py', 'REDIRECT_STATUS': '200'})


i dont see anywhere a refferer key so to catch it in a variable sting
like this:

referrer = os.environ['HTTP_REFERER']

Do i miss something? its a suprise to me that the environ dictioanry has
almost anythign but a referrer key.

I need your help please.

I also tried this:

referer = os.environ.get('HTTP_REFERER', 'UnknownRef')

but that doesn't return anythign either.

 

[Chris “Kwpolska†Warrick]

Hello, i decided am ong other os.environ variables to also grab the
'HTTP_REFERER' fiel but when i try to run my script i was seeing a KeyError
complaining that 'HTTP_REFERER' didnt exist.

So, to see what existed in the os.environ dictionary i issues a print(
os.environ ) to see all available keys and their values:

The Referer header is not mandatory by any means. Your client
probably does not send it.

 

[Îίκος]

Στις 25/9/2013 5:01 μμ, ο/η Chris “Kwpolska†Warrick έγÏαψε:

The Referer header is not mandatory by any means. Your client
probably does not send it.

I would like to check for its existence and retrieve it if possible, if
its not there then default to the string "UnKnown Ref".

I try to do this with:

referer = os.environ.get('HTTP_REFERER', 'UnknownRef')

but that doesn't return anything either.

Can you verify that its the correct way to grab the referral string?

 

[Îίκος]

Στις 25/9/2013 5:01 μμ, ο/η Chris “Kwpolska†Warrick έγÏαψε:

The Referer header is not mandatory by any means. Your client
probably does not send it.

But one other problem appeared too:

(e-mail address removed) [~/www/cgi-bin]# python metrites.py
File "metrites.py", line 27
host = socket.gethostbyaddr( os.environ['REMOTE_ADDR'] )[0] or
'UnKnown Host'
^
SyntaxError: invalid syntax


i dont see anything wrong with that line, and the carret is actually
pointing to the "host".

 

[John Gordon]

referrer = os.environ['HTTP_REFERER']

Do i miss something? its a suprise to me that the environ dictioanry has
almost anythign but a referrer key.

HTTP_REFERER is used to indicate the URL containing the link that led
the user to the current URL. So for example if the user is viewing
foo.html and then clicks a link on that page which leads to bar.html, the
HTTP_REFERER for that request would be foo.html.

However, if the user did not arrive from another page, then HTTP_REFERER
will be missing. This happens when the user types the web address directly
into their browser, or clicks on a bookmark, or many other ways.

Also, obviously, it's up to the browser to truthfully report HTTP_REFERER;
the server itself has no idea what page you were on previously. What
browser are you using?

 

[John Gordon]

referer = os.environ.get('HTTP_REFERER', 'UnknownRef')

but that doesn't return anything either.

When you say it "doesn't return anything", what exactly do you mean? Does
it return None? Does it raise KeyError? Something else?

In any case, I'm surprised that doesn't work. That should be the correct
way to do it.

You could try this:

try:
referer = os.environ.get('HTTP_REFERER', 'UnknownRef')
except KeyError:
referer = None

if not referer:
referer = 'UnknownRef'

 

[Robert Kern]

Στις 25/9/2013 5:01 μμ, ο/η Chris “Kwpolska†Warrick έγÏαψε:

The Referer header is not mandatory by any means. Your client
probably does not send it.

But one other problem appeared too:

(e-mail address removed) [~/www/cgi-bin]# python metrites.py
File "metrites.py", line 27
host = socket.gethostbyaddr( os.environ['REMOTE_ADDR'] )[0] or 'UnKnown Host'
^
SyntaxError: invalid syntax


i dont see anything wrong with that line, and the carret is actually pointing to
the "host".

As has been explained to you before, SyntaxErrors just point to the place where
the parser saw something unexpected, not the exact point where you made the
error. It is not omniscient. If you don't see the problem on the reported line,
you have probably left off a closing bracket or something similar in a previous
line. Look back a few lines.

--
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless enigma
that is made terrible by our own mad attempt to interpret it as though it had
an underlying truth."
-- Umberto Eco

 

[John Gordon]

But one other problem appeared too:

(e-mail address removed) [~/www/cgi-bin]# python metrites.py
File "metrites.py", line 27
host = socket.gethostbyaddr( os.environ['REMOTE_ADDR'] )[0] or
'UnKnown Host'
^
SyntaxError: invalid syntax

i dont see anything wrong with that line, and the carret is actually
pointing to the "host".

Your post has the code broken up into two separate lines. Is that how
the code actually appears? If so, then that's the problem. You can't
arbitrarily break a statement into separate lines.

If not, then you'll have to post more of the code; the problem isn't on
this line.

 

[Steven D'Aprano]

(e-mail address removed) [~/www/cgi-bin]# python metrites.py
File "metrites.py", line 27
host = socket.gethostbyaddr( os.environ['REMOTE_ADDR'] )[0] or
'UnKnown Host'
^
SyntaxError: invalid syntax


i dont see anything wrong with that line, and the carret is actually
pointing to the "host".

If the caret is pointing to "host", then the syntax error isn't
discovered until that point. That means the actual syntax error occurs
before that point, probably on the previous line.

How long have you been doing web development with Python? Six months? A
year? You're not a beginner any more. If you can't solve syntax errors by
yourself by now, it's probably time to give up and find a job more suited
to your skills.

 

[Steven D'Aprano]

I would like to check for its existence and retrieve it if possible, if
its not there then default to the string "UnKnown Ref".

I try to do this with:

referer = os.environ.get('HTTP_REFERER', 'UnknownRef')

but that doesn't return anything either.

Can you verify that its the correct way to grab the referral string?

The Referer is not an environment variable. How would your shell know
what URL you were just browsing?

Have you googled for HTTP Referer? Do you understand what it is?

 

[Tim Chase]

However, if the user did not arrive from another page, then
HTTP_REFERER will be missing. This happens when the user types the
web address directly into their browser, or clicks on a bookmark,
or many other ways.

Also, obviously, it's up to the browser to truthfully report
HTTP_REFERER;

There are browser plugins that allow blocking or manually-overriding
the outbound refer[r]er header which help mitigate data leakage such
as search-engine query strings or work around website limitations. So
server-side code should always assume that the HTTP_REFERER header can
be absent or easily be spoofed, treating it as a hint, not absolute
truth.

-tkc

 

[Îίκος]

Στις 25/9/2013 5:52 μμ, ο/η Steven D'Aprano έγÏαψε:

The Referer is not an environment variable. How would your shell know
what URL you were just browsing?

Have you googled for HTTP Referer? Do you understand what it is?

Yes Steven googleign for 2 hours now.

You were rigth about the carret it was a synatx error above which i
missed and i was breakign my head to the wall to understand what was wot
with the 'host' variable. why python doestn detect the exact synxtax
error and ispoitnign to me to another line making me think the error is
elsewhere?

This indeed works now:

ref = os.environ.get('HTTP_REFERER', 'Άγνωστο Ref')

but iam wondering why this doesnt work also:

ref = os.environ('HTTP_REFERER')

Shouldnt both work?

 

[Steven D'Aprano]

You could try this:

try:
referer = os.environ.get('HTTP_REFERER', 'UnknownRef')
except KeyError:
referer = None
if not referer:
referer = 'UnknownRef'

The get method will not raise KeyError. If the key is missing,
'UnknownRef' will be returned instead.

So when Nikos reports that os.environ.get('HTTP_REFERER', 'UnknownRef')
"doesn't return anything", he is mistaken. It returns 'UnknownRef'.

 

[Ned Batchelder]

You could try this:

try:
referer = os.environ.get('HTTP_REFERER', 'UnknownRef')
except KeyError:
referer = None

if not referer:
referer = 'UnknownRef'

There's no need for the "except KeyError" clause. dict.get never raises
KeyError, this code will assign 'UnknownRef' in the case the environment
variable is missing.

--Ned.

 

[Xaxa Urtiz]

Le mercredi 25 septembre 2013 17:02:45 UTC+2, Ferrous Cranus a écrit :

Στις 25/9/2013 5:52 μμ, ο/η Steven D'Aprano έγÏαψε:



Yes Steven googleign for 2 hours now.



You were rigth about the carret it was a synatx error above which i

missed and i was breakign my head to the wall to understand what was wot

with the 'host' variable. why python doestn detect the exact synxtax

error and ispoitnign to me to another line making me think the error is

elsewhere?



This indeed works now:



ref = os.environ.get('HTTP_REFERER', 'Άγνωστο Ref')



but iam wondering why this doesnt work also:



ref = os.environ('HTTP_REFERER')



Shouldnt both work?

http://www.tutorialspoint.com/python/dictionary_get.htm

 

[Tim Chase]

This indeed works now:

ref = os.environ.get('HTTP_REFERER', 'Άγνωστο Ref')

but iam wondering why this doesnt work also:

ref = os.environ('HTTP_REFERER')

Shouldnt both work?

No...that calls os.environ. You likely *mean*

ref = os.environ['HTTP_REFERER']

using square brackets. However, as previously noted by multiple
respondents, this header is not guaranteed to be sent by the browser
for a variety of reasons, so it may not be in the environment
dictionary at all. Thus you are likely to get KeyError exceptions if
you don't use .get() unless you wrap it in a check:

if "HTTP_REFERER" in os.environ:
ref = os.environ["HTTP_REFERER"]
else:
deal_with_this_situation()

-tkc

 

[Îίκος]

Στις 25/9/2013 6:04 μμ, ο/η Steven D'Aprano έγÏαψε:

The get method will not raise KeyError. If the key is missing,
'UnknownRef' will be returned instead.

So when Nikos reports that os.environ.get('HTTP_REFERER', 'UnknownRef')
"doesn't return anything", he is mistaken. It returns 'UnknownRef'.

Not actually Steven,

the referer key works well but it was the simlutaneous "host" issues
that was giving me false understanding. now it works ok?

So you say if the key is missign the get method wotn return an error but
the default string instead. That is nice to hear.

But answer me plz this:

how caom the http_referer thing works ok now but when i just print all
the key listing of .os.environ ket the http_referer key isnt inside?

 

[Grant Edwards]

And of course that's bolloks:

Python 2.7.5 (default, Aug 29 2013, 15:13:35)
[GCC 4.6.3] on linux2
Type "help", "copyright", "credits" or "license" for more information.
Python 3.2.5 (default, Aug 29 2013, 15:19:46)
[GCC 4.6.3] on linux2
Type "help", "copyright", "credits" or "license" for more information.
It is.

The Referer is not an environment variable.

It is when you're writing a CGI app.

How would your shell know what URL you were just browsing?

Because the HTTP server sets those environment variables before
invoking the CGI app.

 

[Îίκος]

Στις 25/9/2013 6:14 μμ, ο/η Tim Chase έγÏαψε:

This indeed works now:

ref = os.environ.get('HTTP_REFERER', 'Άγνωστο Ref')

but iam wondering why this doesnt work also:

ref = os.environ('HTTP_REFERER')

Shouldnt both work?

No...that calls os.environ. You likely *mean*

ref = os.environ['HTTP_REFERER']

using square brackets. However, as previously noted by multiple
respondents, this header is not guaranteed to be sent by the browser
for a variety of reasons, so it may not be in the environment
dictionary at all. Thus you are likely to get KeyError exceptions if
you don't use .get() unless you wrap it in a check:

if "HTTP_REFERER" in os.environ:
ref = os.environ["HTTP_REFERER"]
else:
deal_with_this_situation()

Thank you very much for the clarification.
I just wanted to know the differecne between the 2 ways of using the
HTTP_REFERER.