urlgrabber cookie handling?

Discussion in 'Python' started by Devraj, Oct 17, 2007.

  1. Devraj

    Devraj Guest

    Hi everyone,

    I have been battling to make my code work with a HTTPS proxy, current
    my code uses urllib2 to to most things and works well, except that
    urllib2 doesn't handle HTTPS proxies.

    Urlgrabber (http://linux.duke.edu/projects/urlgrabber/help/
    urlgrabber.grabber.html) looks very promising except that I can find a
    way to handle cookies in urlgrabber. Is there a way urlgrabber can use
    a HTTPCookieProcess or cookielib.CookieJar object to handle cookies?

    Thanks for sparing your time. Thanks.

    --
    Code referenced in this email is available at http://code.google.com/p/gdatacopier
    Devraj, Oct 17, 2007
    #1
    1. Advertising

  2. Devraj

    John J. Lee Guest

    Devraj <> writes:

    > Hi everyone,
    >
    > I have been battling to make my code work with a HTTPS proxy, current
    > my code uses urllib2 to to most things and works well, except that
    > urllib2 doesn't handle HTTPS proxies.
    >
    > Urlgrabber (http://linux.duke.edu/projects/urlgrabber/help/
    > urlgrabber.grabber.html) looks very promising except that I can find a
    > way to handle cookies in urlgrabber. Is there a way urlgrabber can use
    > a HTTPCookieProcess or cookielib.CookieJar object to handle cookies?


    I don't see a nice way. But then I don't see any HTTPS proxy support
    in urlgrabber... (I looked at version 3.1.0).

    There is a recipe or two on ASPN showing how to support HTTPS proxies
    with urllib2, which gives an idea how to do it, though the code is a
    bit rough (I'd post some code myself, but I did it for work).


    John
    John J. Lee, Oct 17, 2007
    #2
    1. Advertising

  3. Devraj

    Devraj Guest

    Hi John,

    Thanks for getting back to me. I did find the ASPN article. If I
    figure this out then I will make sure I post the code somewhere for
    public consumption.

    On Oct 18, 6:13 am, (John J. Lee) wrote:
    > Devraj <> writes:
    > > Hi everyone,

    >
    > > I have been battling to make my code work with a HTTPS proxy, current
    > > my code uses urllib2 to to most things and works well, except that
    > > urllib2 doesn't handle HTTPS proxies.

    >
    > > Urlgrabber (http://linux.duke.edu/projects/urlgrabber/help/
    > > urlgrabber.grabber.html) looks very promising except that I can find a
    > > way to handle cookies in urlgrabber. Is there a way urlgrabber can use
    > > a HTTPCookieProcess or cookielib.CookieJar object to handle cookies?

    >
    > I don't see a nice way. But then I don't see any HTTPS proxy support
    > in urlgrabber... (I looked at version 3.1.0).
    >
    > There is a recipe or two on ASPN showing how to support HTTPS proxies
    > with urllib2, which gives an idea how to do it, though the code is a
    > bit rough (I'd post some code myself, but I did it for work).
    >
    > John
    Devraj, Oct 18, 2007
    #3
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. Ben
    Replies:
    3
    Views:
    5,796
    Steven Cheng[MSFT]
    Jun 3, 2004
  2. Shapper

    Cookie and Session Cookie Questions.

    Shapper, Apr 27, 2005, in forum: ASP .Net
    Replies:
    1
    Views:
    553
  3. =?Utf-8?B?UGF1bA==?=

    Cookie Question (IP as domain and cookie file location)

    =?Utf-8?B?UGF1bA==?=, Jan 10, 2006, in forum: ASP .Net
    Replies:
    1
    Views:
    2,462
    Bruce Barker
    Jan 10, 2006
  4. ad
    Replies:
    2
    Views:
    7,465
    Kevin Spencer
    Jan 27, 2006
  5. Robert Dailey

    urlgrabber for Python 3.0

    Robert Dailey, Apr 30, 2009, in forum: Python
    Replies:
    1
    Views:
    281
    Terry Reedy
    Apr 30, 2009
Loading...

Share This Page