cookie lib policy how-tp?

[Riko Wichmann]

dear all,

i try to retrieve information from a secure web site. I use cookielib
and urllib2 for this exercise which works to a certain level. I can
authenticate myself and read the top-level page.

However, from here I need to load a page which is dynamically build from
information available from the page source on the top level. The desired
info is inside a <form> tag which seems to be used by a javascript
routine. But when I look at the page source retrieved with urllib2, the
information I need to build the next URL is mssing.

When I use opera to access this page by hand and look at the sources, I
see the full sources when letting opera identify itself as MSIE 6.0.
When using Mozilla 5.0 I get the same in-complete source file as with
python.
The desired info is included in a <form> tag.

Also, when accessing the web page using opera with those two identities,
I'm asked to accept 2 cookies in case of MSIE6.0 and only one for
mozilla. So, I suspect, the problem is somehow related to the cookies
(maybe!?).

I saw, that I can chance the cookie policy in cookielib. Maybe that will
do the job. However, I don't quite understand what I have to do for that.

Any hints or other thoughts are greatly appreciated.

Thanks and cheers,

Riko

 

[Jonathan Ellis]

When I use opera to access this page by hand and look at the sources, I
see the full sources when letting opera identify itself as MSIE 6.0.
When using Mozilla 5.0 I get the same in-complete source file as with

python.

Sounds like your first step should be to identify yourself as IE.

opener = urllib2.build_opener(...)
opener.addheaders = [("User-Agent", "whatever IE calls itself these
days")]

-Jonathan

 

[Riko Wichmann]

When I use opera to access this page by hand and look at the sources,
I

see the full sources when letting opera identify itself as MSIE 6.0.
When using Mozilla 5.0 I get the same in-complete source file as with

python.

Sounds like your first step should be to identify yourself as IE.

opener = urllib2.build_opener(...)
opener.addheaders = [("User-Agent", "whatever IE calls itself these
days")]

-Jonathan

Tried that already. At least, I hope I guessed at least one of the
possible identifiers correct: MSIE6.0, MSIE 6.0, MSIE/6.0
Unfortunately no different. I also tried to define my cookiejar as a
MSIE cookiejar. However, since I'm working under Linux, I'm not sure
that even works, even though there was no complain:

#----------------------------
import ClientCookie, urllib

cj = ClientCookie.MSIECookieJar()

self.opener =
ClientCookie.build_opener(ClientCookie.HTTPCookieProcessor(cj))
self.opener.addheaders = [('User-Agent', 'MSIE6.0')]
# set login parametes ...
parameters = urllib.urlencode({"~login" : "XX", "~password" :
"XX"})

f = self.opener.open('https:myhost', parameters)

#----------------------------

 

[Marc Christiansen]

Sounds like your first step should be to identify yourself as IE.

opener = urllib2.build_opener(...)
opener.addheaders = [("User-Agent", "whatever IE calls itself these
days")]

-Jonathan

Tried that already. At least, I hope I guessed at least one of the
possible identifiers correct: MSIE6.0, MSIE 6.0, MSIE/6.0

When my opera is set to identify as MSIE, it sends
"Mozilla/4.0 (compatible; MSIE 6.0; X11; Linux i686) Opera 7.54 [en]".

Saluton
Marc

 

[Riko Wichmann]

Tried that already. At least, I hope I guessed at least one of the

possible identifiers correct: MSIE6.0, MSIE 6.0, MSIE/6.0

When my opera is set to identify as MSIE, it sends
"Mozilla/4.0 (compatible; MSIE 6.0; X11; Linux i686) Opera 7.54 [en]".

Hi Marc,

thanks for the hint! that brought me a big step forward!

Cheers,

Riko

 

[Marc Christiansen]

Tried that already. At least, I hope I guessed at least one of the
possible identifiers correct: MSIE6.0, MSIE 6.0, MSIE/6.0

When my opera is set to identify as MSIE, it sends
"Mozilla/4.0 (compatible; MSIE 6.0; X11; Linux i686) Opera 7.54 [en]".

Hi Marc,

thanks for the hint! that brought me a big step forward!

You're welcome
Marc

 

[Fuzzyman]

I think your question has already been answered - but just to clarify a
couple of issues.

Setting a cookie policy will only *restrict* the situations in which
cookies are returned. The default is to *always* return them.

Also using an MSIE instance is useful for creating a CookieJar instance
with your current set of IE cookies in it, but it doesn't make
cookielib behave any more like IE.

(take all that with the usual pinch of salt, but I'm pretty sure it's
correct).

Regards,

Fuzzy
http://www.voidspace.org.uk/atlantibots/pythonutils.html