Trying to make a spider using mechanize

Discussion in 'Python' started by tedpottel@gmail.com, Sep 8, 2008.

  1. Guest

    Hi,

    I can read the home page using the mechanize lib. Is there a way to
    load in web pages using filename.html instad of servername/
    filename.html. Lots of time the links just have the file name. I'm
    trying to read in the links name and then vsit those pages.

    here is the sample code I am ussing.


    import ClientForm
    import mechanize


    #get home page
    request = mechanize.Request("http://www.activetechconsulting.com")
    response = mechanize.urlopen(request)
    print response.read()

    #sub page (this does note work)
    request = mechanize.Request("service.html")
    response = mechanize.urlopen(request)
    print response.read-Ted
    , Sep 8, 2008
    #1
    1. Advertising

  2. James Mills Guest

    Hi,

    Perhaps you might want to
    try out using a sample spider
    I wrote and base your code of
    this ?

    See: http://hg.shortcircuit.net.au/index.wsgi/pymills/file/b9936ae2525c/examples/spider.py

    cheers
    James

    On Tue, Sep 9, 2008 at 2:24 AM, <> wrote:
    > Hi,
    >
    > I can read the home page using the mechanize lib. Is there a way to
    > load in web pages using filename.html instad of servername/
    > filename.html. Lots of time the links just have the file name. I'm
    > trying to read in the links name and then vsit those pages.
    >
    > here is the sample code I am ussing.
    >
    >
    > import ClientForm
    > import mechanize
    >
    >
    > #get home page
    > request = mechanize.Request("http://www.activetechconsulting.com")
    > response = mechanize.urlopen(request)
    > print response.read()
    >
    > #sub page (this does note work)
    > request = mechanize.Request("service.html")
    > response = mechanize.urlopen(request)
    > print response.read-Ted
    > --
    > http://mail.python.org/mailman/listinfo/python-list
    >




    --
    --
    -- "Problems are solved by method"
    James Mills, Sep 8, 2008
    #2
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. baroque Chou

    how google spider access my web site?

    baroque Chou, Jan 26, 2006, in forum: ASP .Net
    Replies:
    7
    Views:
    3,891
    Alan Silver
    Feb 2, 2006
  2. bruce
    Replies:
    0
    Views:
    489
    bruce
    Jul 21, 2008
  3. bruce
    Replies:
    0
    Views:
    622
    bruce
    Jul 21, 2008
  4. Replies:
    1
    Views:
    268
    Gabriel Genellina
    Aug 24, 2008
  5. Replies:
    0
    Views:
    385
Loading...

Share This Page