Looking for a urllib(2) cookie handler

[Mark Carter]

I'm in python 2.6.5, and have Firefox 3.6.13. I would like to download
some html from a site and scrape it programatically. The site requires
a cookie, which I have in Firefox.

Is there a simple python recipe I can use to read the contents of a
url and say "just use the cookie that I have in Firefox"?

 

[Chris Rebert]

I'm in python 2.6.5, and have Firefox 3.6.13. I would like to download
some html from a site and scrape it programatically. The site requires
a cookie, which I have in Firefox.

Is there a simple python recipe I can use to read the contents of a
url and say "just use the cookie that I have in Firefox"?

Untested (3rd Google hit for "import firefox cookies python"):
http://blog.mithis.net/archives/python/90-firefox3-cookies-in-python

At a minimum, I think you'll need to replace:
from pysqlite2 import dbapi2 as sqlite
With:
import sqlite3 as sqlite

You'll also have to figure out where Firefox's `cookies.sqlite` file
is located on your system.

Cheers,
Chris

 

[Mark Carter]

url and say "just use the cookie that I have in Firefox"?

"mechanize" looks kinda like what I want, but i still can't get it to
work properly. So far I have:

import cookielib
import mechanize

cookiefile = "C:\\Users\\$ME\\AppData\\Roaming\\Mozilla\\Firefox\
\Profiles\\zl648qvt.default\\cookies.sqlite"
cookies = mechanize.MozillaCookieJar(filename = cookiefile,
delayload=True)
#cookies = cookielib.MozillaCookieJar()
#cookies = cookielib.MSIECookieJar()
#cookies.load_from_registry() # finds cookie index file from registry
br = mechanize.Browser()
br.set_cookiejar(cookies)
br.addheaders = [('User-agent', 'Mozilla/5.0 (X11; U; Linux i686; en-
US; rv:1.9.0.1) Gecko/2008071615 Fedora/3.0.1-1.fc9 Firefox/3.0.1')]
url = "$URL"
r = br.open(url)

#print cj
#opener =
mechanize.build_opener(mechanize.HTTPCookieProcessor(cookies))


html = r.read()
print html

where $ME and $URL are replaced with suitable values. It doesn't
appear to acutally be "using" the cookies.