Python simulate browser activity

Discussion in 'Python' started by choi2k, Mar 16, 2012.

  1. choi2k

    choi2k Guest

    Hi, everyone

    I am trying to write a small application using python but I am not
    sure whether it is possible to do so..
    The application aims to simulate user activity including visit a
    website and perform some interactive actions (click on the menu,
    submit a form, redirect to another pages...etc)
    I have found some libraries / plugins which aims to simulate browser
    activity but ... none of them support AJAX request and/or running
    javascript.

    Here is one of the simple tasks which I would like to do using the
    application:
    1. simulate the user activity, visit the website ("https://
    www.abc.com")
    2. find out the target element by id ("submitBtn") and simulate mouse
    click on the item.
    3. perform some javascript functions which invoked by click event from
    step 2. ( the function is bind to the item by jquery)
    4. post ajax request using javascript and if the page has been
    redirected, load the new page content

    Is it possible to do so?

    Thank you in advance.
     
    choi2k, Mar 16, 2012
    #1
    1. Advertising

  2. choi2k

    Chris Rebert Guest

    On Thu, Mar 15, 2012 at 7:23 PM, choi2k <> wrote:
    > Hi, everyone
    >
    > I am trying to write a small application using python but I am not
    > sure whether it is possible to do so..
    > The application aims to simulate user activity including visit a
    > website and perform some interactive actions (click on the menu,
    > submit a form, redirect to another pages...etc)
    > I have found some libraries / plugins which aims to simulate browser
    > activity but ... none of them support AJAX request and/or running
    > javascript.


    Did you look at Selenium?
    http://seleniumhq.org/

    Cheers,
    Chris
     
    Chris Rebert, Mar 16, 2012
    #2
    1. Advertising

  3. choi2k

    Jerry Hill Guest

    On Thu, Mar 15, 2012 at 10:54 PM, Chris Rebert <> wrote:
    > On Thu, Mar 15, 2012 at 7:23 PM, choi2k <> wrote:
    >> The application aims to simulate user activity including visit a
    >> website and perform some interactive actions (click on the menu,
    >> submit a form, redirect to another pages...etc)

    >
    > Did you look at Selenium?
    > http://seleniumhq.org/


    You might also be interested in Sikuli (http://sikuli.org/), which
    uses Jython to write scripts to automate other programs using
    screenshots. It's pretty neat if you need to automate a GUI that is
    otherwise difficult to deal with.

    --
    Jerry
     
    Jerry Hill, Mar 16, 2012
    #3
  4. choi2k

    Roy Smith Guest

    In article
    <>,
    choi2k <> wrote:

    > Hi, everyone
    >
    > I am trying to write a small application using python but I am not
    > sure whether it is possible to do so..
    > The application aims to simulate user activity including visit a
    > website and perform some interactive actions (click on the menu,
    > submit a form, redirect to another pages...etc)
    > I have found some libraries / plugins which aims to simulate browser
    > activity but ... none of them support AJAX request and/or running
    > javascript.
    >
    > Here is one of the simple tasks which I would like to do using the
    > application:
    > 1. simulate the user activity, visit the website ("https://
    > www.abc.com")
    > 2. find out the target element by id ("submitBtn") and simulate mouse
    > click on the item.
    > 3. perform some javascript functions which invoked by click event from
    > step 2. ( the function is bind to the item by jquery)
    > 4. post ajax request using javascript and if the page has been
    > redirected, load the new page content
    >
    > Is it possible to do so?
    >
    > Thank you in advance.


    Depending on exactly what you're trying to test, this may or may not be
    possible.

    #1 is easy. Just get the URL with urllib (or, even better, Kenneth
    Reitz's very cool requests library (http://docs.python-requests.org/).

    #2 gets a little more interesting, but still not that big a deal. Parse
    the HTML you get back with something like lxml (http://lxml.de/).
    Navigate the DOM with a CSSSelector to find the element you're looking
    for. Use urllib/requests again to get the href.

    #3 is where things get fun. What, exactly, are you trying to test? Are
    you trying to test that the js works, or are you really trying to test
    your server and you're just using the embedded js as a means to generate
    the next HTTP request?

    If the former, then you really want to be exploring something like
    selenium (http://seleniumhq.org/). If the latter, then skip all the js
    crap and just go a GET or POST (back to urllib/requests) on the
    appropriate url.

    #4 falls into the same bucket as #3.

    Once you have figured all this out, you will get a greater appreciation
    for the need to cleanly separate your presentation (HTML, CSS,
    Javascript) from your business logic and data layers. If there is a
    clean interface between them, it becomes easy to test one in isolation
    from the other. If not, then you end up playing games with screen
    scraping via lxml and selenium just to test your database queries.
    Which will quickly lead to you running screaming into the night.
     
    Roy Smith, Mar 16, 2012
    #4
  5. On Fri, Mar 16, 2012 at 1:23 PM, choi2k <> wrote:
    > The application aims to simulate user activity including visit a
    > website and perform some interactive actions (click on the menu,
    > submit a form, redirect to another pages...etc)
    > I have found some libraries / plugins which aims to simulate browser
    > activity but ... none of them support AJAX request and/or running
    > javascript.
    >
    > Here is one of the simple tasks which I would like to do using the
    > application:
    > 4. post ajax request using javascript and if the page has been
    > redirected, load the new page content


    To the web server, everything is a request. Don't think in terms of
    Javascript; there is no Javascript, there is no AJAX, there is not
    even a web browser. There's just a TCP socket connection and an HTTP
    request. That's all you have.

    On that basis, your Python script most definitely can simulate the
    request. It needs simply to make the same HTTP query that the
    Javascript would have made. The only tricky part is figuring out what
    that request would be. If you can use a non-SSL connection, that'll
    make your job easier - just get a proxy or packet sniffer to monitor
    an actual web browser, then save the request headers and body. You can
    then replay that using Python; it's not difficult to handle cookies
    and such, too.

    Hope that helps!

    ChrisA
     
    Chris Angelico, Mar 16, 2012
    #5
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. Stefan Behrens

    Detecting user activity in Python

    Stefan Behrens, Jul 9, 2003, in forum: Python
    Replies:
    0
    Views:
    414
    Stefan Behrens
    Jul 9, 2003
  2. Mark Hammond
    Replies:
    0
    Views:
    657
    Mark Hammond
    Oct 27, 2005
  3. James Hu
    Replies:
    0
    Views:
    659
    James Hu
    Oct 27, 2005
  4. philthym

    Newbie LWP question - simulate browser?

    philthym, May 10, 2004, in forum: Perl Misc
    Replies:
    3
    Views:
    144
    philthym
    May 11, 2004
  5. Lars Madsen

    simulate user activity

    Lars Madsen, Jul 16, 2006, in forum: Perl Misc
    Replies:
    2
    Views:
    260
    Lars Madsen
    Jul 18, 2006
Loading...

Share This Page