urllib and sites that require passwds

Discussion in 'Python' started by bob_smith_17280@hotmail.com, Dec 23, 2004.

  1. Guest

    Hello,

    I'm doing a small website survey as a consultant for a company that has
    a large private lan. Basically, I'm trying to determine how many web
    sites there are on their network and what content the sites contain
    (scary how they don't know this, but I suspect many companies are this
    way).

    Everything is going fine so far except for sites that require passwds
    to be accessed. I don't want to view content on these sites, I only
    want to note that they are passwd protected, make a list of them and
    move on. The problem is that urllib hangs waiting for a username/passwd
    to be entered. Is there a graceful way to deal with this?
    Many thanks,
    Bob
     
    , Dec 23, 2004
    #1
    1. Advertising

  2. Fuzzyman Guest

    USe urllib2 which will fail with an exception. You can trap this
    exception and using the code attribute of the exception object,
    determine why it failed. The error code for 'authentication required'
    is 401.

    Off the top of my head :

    import urllib2
    req = urllib2.Request(theurl)
    try:
    handle = urllib2.urlopen(req)
    except IOError, e:
    if not e.hasattr('code'):
    print 'The url appears to be invalid.'
    print e.reason
    else:
    if e.code == 401:
    print theurl, 'is protected with a password.'
    else:
    print 'We failed with error code', e.code
    HTH

    Regards,

    Fuzzy
    http://www.voidspace.org.uk/python/index.shtml
     
    Fuzzyman, Dec 23, 2004
    #2
    1. Advertising

  3. Fuzzyman Guest

    damn... I'm losing my leading spaces.... indentation should be obvious
    anyway... (everything below except is indented at least one step).
    Fuzzy
     
    Fuzzyman, Dec 23, 2004
    #3
  4. Ishwor Guest

    On 23 Dec 2004 06:46:50 -0800, Fuzzyman <> wrote:
    > damn... I'm losing my leading spaces.... indentation should be obvious

    We'll forgive you for that. It was from "top-of-your-head" ~;-)

    > anyway... (everything below except is indented at least one step).
    > Fuzzy

    Its nice that urllib2 returns errcode to process further. doesn't
    urllib do that?
    Anyway i wanted to know if any website which is similar to CPAN
    library website? I mean i want to be able find modules n stuff for
    Python.. It would be really great to know.

    Thanks.

    --
    cheers,
    Ishwor Gurung
     
    Ishwor, Dec 23, 2004
    #4
  5. Fuzzyman Guest

    Ishwor wrote:
    > On 23 Dec 2004 06:46:50 -0800, Fuzzyman <> wrote:
    > > damn... I'm losing my leading spaces.... indentation should be

    obvious
    > We'll forgive you for that. It was from "top-of-your-head" ~;-)
    >


    Hey - I put the indentation in there... it just got stripped out when
    it was posted ! :)

    > > anyway... (everything below except is indented at least one step).
    > > Fuzzy

    > Its nice that urllib2 returns errcode to process further. doesn't
    > urllib do that?


    The OP is saying that it hangs rather than returning an error. I
    haven't tested it. In general urllib2.urlopen is much better than
    urllib.urlopen. urllib has some useful other functions though.

    > Anyway i wanted to know if any website which is similar to CPAN
    > library website? I mean i want to be able find modules n stuff for
    > Python.. It would be really great to know.
    >


    There is PyPi and the Vaults of Parnassus. Neither are really like
    CPAN. There has been lots of talk about it recently - everyone agrees
    we need one... but no one is offering the bandwidth or the code.

    There are lots of modules available though - and usually not too hard
    to track down.

    Regards,

    Fuzzy
    http://www.voidspace.org.uk/python/index.shtml
    > Thanks.
    >
    > --
    > cheers,
    > Ishwor Gurung
     
    Fuzzyman, Dec 23, 2004
    #5
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. Kevin Buchan
    Replies:
    1
    Views:
    501
    Eric Lawrence [MSFT]
    Feb 20, 2004
  2. Stefan Caliandro
    Replies:
    2
    Views:
    667
    Beauregard T. Shagnasty
    Feb 14, 2005
  3. imx
    Replies:
    10
    Views:
    841
  4. Jonathan Gardner

    Asynchronous urllib (urllib+asyncore)?

    Jonathan Gardner, Feb 26, 2008, in forum: Python
    Replies:
    1
    Views:
    500
    Terry Jones
    Feb 27, 2008
  5. Chris McDonald
    Replies:
    0
    Views:
    325
    Chris McDonald
    Nov 1, 2010
Loading...

Share This Page